Compare commits

...
91 Commits
Author SHA1 Message Date
Laan Tungir 20902e1c5d v2.1.21 - Fixed caching_service_binary_path to ./caching_relay (relative to relay CWD), added caching_relay build step to make_and_restart_relay.sh, added default caching config values (admin npub, 4 bootstrap relays, kinds 0 and 10002) 2026-07-26 14:56:16 -04:00
Laan Tungir 7233818da9 v2.1.20 - Added default caching config values: admin npub as root, 4 bootstrap relays, kinds 0 and 10002 added to default kinds list 2026-07-25 18:21:28 -04:00
Laan Tungir 86cbf7ff16 v2.1.19 - Consolidated caching_relay code into caching/ directory, switched launcher to PostgreSQL mode (-p pg_conn) so all config comes from the config table instead of a JSON config file 2026-07-25 16:27:56 -04:00
Laan Tungir 3035e71ca5 v2.1.18 - Fixed caching service integration: aligned caching_status response schema, fixed launcher to use -c config_path instead of --pg-conn, added caching service config fields to UI, replaced misleading not-implemented placeholder with accurate auth-failure messaging 2026-07-25 11:19:37 -04:00
Laan Tungir 179b160cf3 v2.1.17 - Verified api-worker LISTEN/NOTIFY monitoring on v2.1.16 relay (port 7777): subscriber-cadence, NOTIFY-reactive, and zero-overhead idle tests all pass 2026-07-25 10:17:30 -04:00
Laan Tungir aedd22436e v2.1.16 - Implemented api-worker timer + LISTEN/NOTIFY monitoring (Phases 1,2,5) and fixed thread-connection binding bug that stopped dashboard updates 2026-07-25 10:07:08 -04:00
Laan Tungir 8fdc362a8b v2.1.15 - Fix PG addressable d-tag derivation and replaceable/addressable transactional replace semantics 2026-05-01 19:01:06 -04:00
Laan Tungir 67cf3973c5 v2.1.14 - Fix PostgreSQL replaceable/addressable upsert semantics with NIP-01 conflict ordering 2026-05-01 14:38:57 -04:00
Laan Tungir c02ae2be5b v2.1.13 - Fix PostgreSQL startup detection for existing relays and make admin_pubkey parsing whitespace-tolerant 2026-04-03 16:36:56 -04:00
Laan Tungir fd3efdd01f v2.1.12 - Implement multi-admin auth cache and checks, add admin management commands, and add multi-admin integration test 2026-04-03 15:46:33 -04:00
Laan Tungir 232f93c16f v2.1.11 - Fix api-worker DB lifecycle and NIP test reliability: add dedicated worker DB connection, guard PG expiration cast in REQ path, correct NIP-45 baseline-aware expectation, and harden run_all_tests skip handling 2026-04-03 07:32:14 -04:00
Laan Tungir 9a9365dcfa v2.1.10 - Fix NIP-50 PG search semantics and NIP-17 test compatibility 2026-04-02 13:00:34 -04:00
Laan Tungir 5577b57149 v2.1.9 - Rebrand repository, docs, scripts, and tooling from c-relay to c-relay-pg 2026-04-02 07:06:41 -04:00
Laan Tungir 22c943d495 v2.1.8 - Implement explicit WAL checkpoint scheduling and disable SQLite auto-checkpoint 2026-04-01 21:03:24 -04:00
Laan Tungir 7e3d9b6825 v2.1.7 - Refactor SQLite connection ownership into db_ops, remove direct sqlite usage from runtime modules, and document agent-browser admin testing flow 2026-04-01 19:48:29 -04:00
Laan Tungir 547d22d09f v2.1.6 - Fix strict-mode admin API key access by prewarming config and relay private key caches 2026-04-01 19:08:42 -04:00
Laan Tungir ce9ae5a488 v2.1.5 - Offload duplicate and COUNT paths, add config/NIP-11 caching, and reduce DB reader threads 2026-04-01 16:15:46 -04:00
Laan Tungir 9b248b740f v2.1.4 - Harden async EVENT fallback handling and verify relay/nginx 503 behavior 2026-04-01 15:19:28 -04:00
Laan Tungir 0f77aeb72b v2.1.3 - Fix async EVENT thread safety by splitting store core/post-actions and naming lws main thread 2026-04-01 11:29:14 -04:00
Laan Tungir bb64651a0c v2.1.2 - Fix async REQ callback leak and validate Phase 2+3 subscription offload path 2026-04-01 11:01:38 -04:00
Laan Tungir 17be9c2b03 v2.1.1 - Wire REQ/COUNT/EVENT paths through thread pool sync helpers and stabilize filter limit tests 2026-04-01 09:38:25 -04:00
Laan Tungir f396c622b6 v2.1.0 - Bump to v2.1.0 as SQLite baseline for PostgreSQL fork handoff 2026-04-01 09:14:23 -04:00
Laan Tungir cbd260c1ae v2.0.4 - Complete Phase 1 DB abstraction: opaque db_stmt API, ip_ban migration, and SQLite leakage cleanup 2026-04-01 09:10:42 -04:00
Laan Tungir bc9ac290ab v2.0.3 - Complete Phase 1 SQLite abstraction cleanup across config/api/websockets and add db path accessor 2026-04-01 07:47:21 -04:00
Laan Tungir 192eeb248d v2.0.2 - Stabilize websocket queue draining, add db config count helper, and align NIP-45/50 test expectations 2026-04-01 07:04:58 -04:00
Laan Tungir da6c505420 v2.0.1 - Fix test-mode build/restart pipeline and NIP-11 root path handling; validate core NIP tests 2026-04-01 05:43:57 -04:00
Laan Tungir c077c209e5 v2.0.0 - Major architecture refactor: db_ops abstraction, low-risk SQLite refactor, and thread-pool scaffolding 2026-04-01 05:25:03 -04:00
Laan Tungir 9bf02702c2 v1.2.56 - Update codebase and documentation 2026-04-01 04:51:44 -04:00
Your Name 0920cc092d v1.2.55 - Preserve relay version metadata for API header/title and accept JSON NIP-11 content type 2026-03-23 09:09:50 -04:00
Your Name a89460be5d v1.2.54 - Sync relay_version to compiled CRELAY_VERSION on startup and show version in API header/title 2026-03-23 09:04:49 -04:00
Your Name ff2a3aa335 v1.2.53 - Fix P0 config-value memory leaks in hot paths and add investigation plan 2026-03-19 07:59:40 -04:00
Your Name 94b61e8a7c v1.2.52 - Update nostr_core_lib to v0.4.13 and route nostr logs through relay callback logger 2026-03-13 14:33:08 -04:00
Your Name b0c0754e83 v1.2.51 - Fix NIP-01 non-compliance: limit:0 now correctly returns zero stored events instead of falling through to default_limit 2026-03-04 11:23:58 -04:00
Your Name 3265e3d114 v1.2.50 - Fix NIP-42 relay URL verification and add onauth to all publish flows 2026-03-01 12:20:44 -04:00
Your Name 927659ece1 v1.2.49 - Fix: MEM% bar format (bar then MB), move CPU Core row after CPU Usage 2026-02-25 07:37:50 -04:00
Your Name b6ff4150b4 v1.2.48 - Add CPU% and MEM% ASCII bar graphs to API page stats 2026-02-25 07:33:06 -04:00
Your Name 63bc526163 v1.2.47 - Fix build: use getter function for g_connection_count (was static, can't extern) 2026-02-25 07:24:17 -04:00
Your Name a1f712236a v1.2.46 - Fix: move extern g_connection_count to file scope in api.c 2026-02-25 07:22:14 -04:00
Your Name 06e6c17b7b v1.2.45 - Add WebSocket Connections to system status: api.c sends active_connections, HTML+JS display it 2026-02-25 07:18:42 -04:00
Your Name 0751a7c55c v1.2.44 - IP Bans: show idle+auth bans in stats/status, fix filter buttons, add idle ban columns to query 2026-02-25 07:12:07 -04:00
Your Name f1728932a9 v1.2.43 - IP Bans page: fix filter buttons, add whitelist management UI, compact table rows 2026-02-25 07:08:20 -04:00
Your Name ef8bdef2a8 v1.2.42 - IP Bans page: fix filter buttons, add whitelist management UI, compact table rows 2026-02-25 07:08:12 -04:00
Your Name c11a8ba292 v1.2.41 - Fix IP Bans page: route ip_bans SQL responses to handleIpBansResponse 2026-02-25 06:59:22 -04:00
Your Name 0f124fe575 v1.2.40 - IP Bans page: show whitelisted IPs with star icon, hide Unban button for whitelisted IPs 2026-02-25 06:54:35 -04:00
Your Name 6b20452fab v1.2.39 - Fix IP Bans page: convert rows+columns SQL response format to objects for display 2026-02-25 06:53:50 -04:00
Your Name 10c19bc243 v1.2.38 - Add idle_ban_whitelist config: comma-separated IPs that are never idle-banned 2026-02-25 06:48:41 -04:00
Your Name 3d7aa2196f v1.2.37 - Fix: executeSqlQueryRaw uses relayPool/sendAdminCommand instead of undefined pool variable 2026-02-25 06:43:29 -04:00
Your Name a416c3f275 v1.2.36 - Fix: don't reset relay connection state when external relays time out in subscription onclose 2026-02-25 06:33:25 -04:00
Your Name bba9baabc3 v1.2.35 - Fix: only record idle ban failures for WebSocket connections, not HTTP requests (NIP-11/embedded files) 2026-02-24 17:41:59 -04:00
Your Name 31187c4c4f v1.2.34 - Fix: mark HTTP requests (NIP-11, embedded files) as session_active to prevent idle ban on legitimate HTTP clients 2026-02-24 17:07:33 -04:00
Your Name 55f862b879 v1.2.33 - Set default debug_level to 3 (INFO) 2026-02-24 16:06:19 -04:00
Your Name 76c9b3fcf0 v1.2.32 - Set idle_ban_threshold default to 1 and idle_ban_window_sec default to 30 2026-02-24 15:45:40 -04:00
Your Name 929bd09164 v1.2.31 - Fix idle connection timeout: use global connection list + periodic timer instead of lws_set_timeout which was not firing 2026-02-24 15:44:11 -04:00
Your Name d17f1dd8d5 v1.2.30 - Temporarily bypass admin verification on API page (idle timeout fix pending) 2026-02-24 14:55:39 -04:00
Your Name 207a949835 v1.2.29 - Added idle connection ban system - bans IPs that connect but never send REQ/EVENT (idle timeout or early disconnect) with separate rate limiting from auth failures 2026-02-24 14:18:27 -04:00
Your Name d08f4e4221 v1.2.28 - Add debug_level config setting: live update every 60s without restart, default 0 2026-02-24 08:56:32 -04:00
Your Name c96736fa6a v1.2.27 - Permanent ban escalation: ban_count never resets, IPs with history always get 24h ban on next failure 2026-02-23 18:18:55 -04:00
Your Name f8ec4ae924 v1.2.26 - Persistent IP ban table: save/load to ip_bans DB, auth success tracking, lifetime stats per IP 2026-02-23 18:16:14 -04:00
Your Name fe7304ac7f v1.2.25 - Add NIP-42 AUTH support to web UI: onauth callback in subscribeMany and publish calls 2026-02-23 18:03:29 -04:00
Your Name e5d39c984b v1.2.24 - Fix: run ip_ban maintenance unconditionally every 60s regardless of max_connection_seconds setting 2026-02-23 17:57:20 -04:00
Your Name 5e45f21e35 v1.2.23 - ip_ban: retain ban_count for 24 hours after last ban expiry, then fully clean entry 2026-02-23 17:16:02 -04:00
Your Name 5321a238b8 v1.2.22 - Fix ip_ban cleanup: preserve ban_count on cleanup so exponential backoff persists across ban expiry 2026-02-23 17:12:15 -04:00
Your Name 083bc14972 v1.2.21 - Fix IP ban check: use pss->client_ip (proxy-aware) instead of raw socket IP to match failure recording 2026-02-23 16:26:23 -04:00
Your Name 11aaccba9b v1.2.20 - Add IP auth failure ban system: track failures per IP, ban after threshold with exponential backoff, periodic log stats 2026-02-23 16:02:07 -04:00
Your Name bd1bbd763d v1.2.19 - Add IP auth failure ban system: track failures per IP, ban after threshold with exponential backoff, periodic log stats 2026-02-23 16:00:53 -04:00
Your Name 2bd7aa5a10 v1.2.18 - Proactive auth timeout via lws_set_timeout: close idle unauthenticated connections after nip42_auth_timeout_sec even without REQ 2026-02-23 15:37:45 -04:00
Your Name 361912ec85 v1.2.17 - Add nip42_auth_timeout_sec (default 10s): close unauthenticated connections after timeout to prevent connection accumulation 2026-02-23 15:22:18 -04:00
Your Name 0de491382e v1.2.16 - Fix auth rules UI: change label/placeholder/errors from nsec to npub (public key, not private key) 2026-02-23 14:36:19 -04:00
Your Name 3148bbbee7 v1.2.15 - Admin verification: show 'Waiting for response' with status updates, 60s timeout, no premature access denied 2026-02-23 14:26:02 -04:00
Your Name 3965ba04d8 v1.2.14 - Handle late admin verification response: grant access even if timeout already fired and access-denied overlay was shown 2026-02-23 14:24:55 -04:00
Your Name b96af938bd v1.2.13 - Add WoT status to NIP-11 response: restricted_writes, auth_required, and web_of_trust object when WoT enabled 2026-02-23 14:22:08 -04:00
Your Name 89c8248013 v1.2.12 - Increase admin verification timeout from 5s to 30s; continue waiting after publish timeout under heavy load 2026-02-23 14:16:56 -04:00
Your Name d8f477c6cf v1.2.11 - Early duplicate check before secp256k1 signature verification — skip crypto for events already in DB 2026-02-23 14:01:23 -04:00
Your Name 040eeadb13 v1.2.10 - Zero-copy message queue: eliminate memcpy in broadcast and REQ paths via queue_message_take_ownership() 2026-02-23 13:45:56 -04:00
Your Name 83f8b0ab88 v1.2.9 - Replace cJSON_GetObjectItem with CaseSensitive variant (178 calls) — eliminates 14% CPU from tolower+linear scan in hot path 2026-02-23 13:25:50 -04:00
Your Name b82f7eaaf3 v1.2.8 - Add debug build support: _debug suffix in build_static.sh, deploy_lt_debug.sh with perf profiling workflow 2026-02-23 13:07:53 -04:00
Your Name 0dc5b75d7c v1.2.7 - Add configurable SQLite mmap_size (256MB) and cache_size (64MB) PRAGMAs for ~20% CPU reduction from DB read overhead 2026-02-23 09:48:01 -04:00
Your Name e94b1a81e3 v1.2.6 - Add MAX_MESSAGE_QUEUE_SIZE=500 limit to prevent OOM from unbounded write queue; fix wasted buf allocation in broadcast 2026-02-23 09:12:09 -04:00
Your Name 1adabdbc4e v1.2.5 - Add nip17_admin_enabled config toggle (default off) to prevent OOM from NIP-17 gift wrap decryption flood 2026-02-23 08:58:10 -04:00
Your Name 81d44c3d8c v1.2.4 - Add more characters to valid subscription characters 2026-02-11 05:56:09 -04:00
Your Name 7acc0bdd90 v1.2.3 - Handle rate limiting properly on kind 99999 request 2026-02-09 07:49:43 -04:00
Your Name c4ef71d673 v1.2.2 - Add + to allowed subscription characters 2026-02-07 13:32:00 -04:00
Your Name 086d2af56c v1.2.1 - Handle NDKs pings of kind 99999 2026-02-03 13:19:33 -04:00
Your Name 18a7deec54 v1.2.0 - Schema v11: Added event_json storage for 2500x performance improvement 2026-02-02 20:27:50 -04:00
Your Name c794370a3f v1.1.9 - Add composite index for active_subscriptions_log view optimization (schema v10) - reduces monitoring queries from 540ms to <10ms 2026-02-02 11:38:57 -04:00
Your Name 2ed4b96058 v1.1.8 - Add comprehensive database query logging with timing at debug level 3, fix schema version compatibility (v6-v9), add version logging at startup, allow monitoring throttle=0 to disable monitoring 2026-02-02 11:20:11 -04:00
Your Name c0051b22be v1.1.7 - Add per-connection database query tracking for abuse detection
Implemented comprehensive database query tracking to identify clients causing
high CPU usage through excessive database queries. The relay now tracks and
displays query statistics per WebSocket connection in the admin UI.

Features Added:
- Track db_queries_executed and db_rows_returned per connection
- Calculate query rate (queries/minute) and row rate (rows/minute)
- Display stats in admin UI grouped by IP address and WebSocket
- Show: IP, Subscriptions, Queries, Rows, Query Rate, Duration

Implementation:
- Added tracking fields to per_session_data structure
- Increment counters in handle_req_message() and handle_count_message()
- Extract stats from pss in query_subscription_details()
- Updated admin UI to display IP address and query metrics

Use Case:
Admins can now identify abusive clients by monitoring:
- High query rates (>50 queries/min indicates polling abuse)
- High row counts (>10K rows/min indicates broad filter abuse)
- Query patterns (high queries + low rows = targeted, high both = crawler)

This enables informed decisions about which IPs to blacklist based on
actual resource consumption rather than just connection count.
2026-02-01 16:26:37 -04:00
Your Name 4cc2d2376e v1.1.6 - Optimize: Deduplicate kinds in subscription index to prevent redundant operations
The kind index was adding subscriptions multiple times when filters contained
duplicate kinds (e.g., 'kinds': [1, 1, 1] or multiple filters with same kind).
This caused:
- Redundant malloc/free operations during add/remove
- Multiple index entries for same subscription+kind pair
- Excessive TRACE logging (7+ removals for single subscription)
- Wasted CPU cycles on duplicate operations

Fix:
- Added bitmap-based deduplication in add_subscription_to_kind_index()
- Uses 8KB bitmap (65536 bits) to track which kinds already added
- Prevents adding same subscription to same kind index multiple times
- Reduces index operations by 3-10x for subscriptions with duplicate kinds

Performance Impact:
- Eliminates redundant malloc/free cycles
- Reduces lock contention on kind index operations
- Decreases log volume significantly
- Should reduce CPU usage by 20-40% under production load
2026-02-01 15:59:54 -04:00
Your Name 30dc4bf67d v1.1.5 - Fix CRITICAL segfault: Use wrapper nodes for no-kind-filter subscriptions
The kind index optimization in v1.1.4 introduced a critical bug that caused
segmentation faults in production. The bug was in add_subscription_to_kind_index()
which directly assigned sub->next for no-kind-filter subscriptions, corrupting
the main active_subscriptions linked list.

Root Cause:
- subscription_t has only ONE 'next' pointer used by active_subscriptions list
- Code tried to reuse 'next' for no_kind_filter_subs list
- This overwrote the active_subscriptions linkage, breaking list traversal
- Result: segfaults when iterating subscriptions

Fix:
- Added no_kind_filter_node_t wrapper structure (like kind_subscription_node_t)
- Changed no_kind_filter_subs from subscription_t* to no_kind_filter_node_t*
- Updated add/remove functions to use wrapper nodes
- Updated broadcast function to iterate through wrapper nodes

This follows the same pattern already used for kind_index entries and
prevents any corruption of the subscription structure's next pointer.
2026-02-01 12:37:07 -04:00
Your Name a1928cc5d7 v1.1.4 - Add kind-based index for 10x subscription matching performance improvement 2026-02-01 11:15:26 -04:00
Your Name 7bf0757b1f v1.1.3 - Rename VERSION to CRELAY_VERSION to avoid conflict with nostr_core_lib 2026-02-01 10:02:19 -04:00
193 changed files with 41625 additions and 3067 deletions
+1
View File
@@ -11,3 +11,4 @@ copy_executable_local.sh
nostr_login_lite/
style_guide/
nostr-tools
.test_keys
+2
View File
@@ -0,0 +1,2 @@
ADMIN_PUBKEY='6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3'
SERVER_PRIVKEY='1111111111111111111111111111111111111111111111111111111111111111'
+11 -11
View File
@@ -1,12 +1,12 @@
# AGENTS.md - AI Agent Integration Guide for Architect Mode
**Project-Specific Information for AI Agents Working with C-Relay in Architect Mode**
**Project-Specific Information for AI Agents Working with C-Relay-PG in Architect Mode**
## Critical Architecture Understanding
### System Architecture Overview
C-Relay implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
C-Relay-PG implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
@@ -51,9 +51,9 @@ C-Relay implements a **unique event-based configuration architecture** that fund
## Architectural Decision Analysis
### Configuration System Design
**Traditional Approach vs C-Relay:**
**Traditional Approach vs C-Relay-PG:**
```
Traditional: C-Relay:
Traditional: C-Relay-PG:
config.json → kind 33334 events
ENV variables → cryptographically signed tags
File watching → database polling/restart
@@ -119,18 +119,18 @@ File watching → database polling/restart
```
Developer Machine:
├── ./make_and_restart_relay.sh
├── build/c_relay_x86
├── build/c_relay_pg_x86
├── build/<relay_pubkey>.db
└── relay.log
```
### Production SystemD Deployment
```
/opt/c-relay/:
├── c_relay_x86
/opt/c-relay-pg/:
├── c_relay_pg_x86
├── <relay_pubkey>.db
├── systemd service (c-relay.service)
└── c-relay user isolation
├── systemd service (c-relay-pg.service)
└── c-relay-pg user isolation
```
### Container Deployment Architecture
@@ -144,7 +144,7 @@ Container:
### Reverse Proxy Architecture
```
Internet → Nginx/HAProxy → C-Relay
Internet → Nginx/HAProxy → C-Relay-PG
├── WebSocket upgrade handling
├── SSL termination
└── Rate limiting
@@ -292,7 +292,7 @@ Admin Signs Event → WebSocket Submit → Validate → Store → Restart Requir
**Decision**: Same port serves both protocols
**Consequences**: Simplified deployment, protocol detection overhead, libwebsockets dependency
These architectural decisions form the foundation of C-Relay's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
These architectural decisions form the foundation of C-Relay-PG's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
**
[Response interrupted by a tool use result. Only one tool may be used at a time and should be placed at the end of the message.]
Executable
BIN
View File
Binary file not shown.
+8 -8
View File
@@ -1,6 +1,6 @@
# AGENTS.md - AI Agent Integration Guide
**Project-Specific Information for AI Agents Working with C-Relay**
**Project-Specific Information for AI Agents Working with C-Relay-PG**
## Critical Build Commands
@@ -15,9 +15,9 @@
- Starts relay in background with proper logging
### Architecture-Specific Binary Outputs
- **x86_64**: `./build/c_relay_x86`
- **ARM64**: `./build/c_relay_arm64`
- **Other**: `./build/c_relay_$(ARCH)`
- **x86_64**: `./build/c_relay_pg_x86`
- **ARM64**: `./build/c_relay_pg_arm64`
- **Other**: `./build/c_relay_pg_$(ARCH)`
### Database File Naming Convention
- **Format**: `<relay_pubkey>.db` (NOT `.nrdb` as shown in docs)
@@ -75,10 +75,10 @@
### Process Management
```bash
# Kill existing relay processes
pkill -f "c_relay_"
pkill -f "c_relay_pg_"
# Check running processes
ps aux | grep c_relay_
ps aux | grep c_relay_pg_
# Force kill port binding
fuser -k 8888/tcp
@@ -95,7 +95,7 @@ fuser -k 8888/tcp
- Event configuration tests: `tests/event_config_tests.sh`
### SystemD Integration Considerations
- Service runs as `c-relay` user in `/opt/c-relay`
- Service runs as `c-relay-pg` user in `/opt/c-relay-pg`
- Database files created in WorkingDirectory automatically
- No environment variables needed (event-based config)
- Resource limits: 65536 file descriptors, 4096 processes
@@ -137,7 +137,7 @@ fuser -k 8888/tcp
## Quick Debugging Commands
```bash
# Check relay status
ps aux | grep c_relay_ && netstat -tln | grep 8888
ps aux | grep c_relay_pg_ && netstat -tln | grep 8888
# View logs
tail -f relay.log
+9 -9
View File
@@ -1,6 +1,6 @@
# C-Relay API Documentation
# C-Relay-PG API Documentation
Complete API reference for the C-Relay event-based administration system and advanced features.
Complete API reference for the C-Relay-PG event-based administration system and advanced features.
## Table of Contents
@@ -21,7 +21,7 @@ Complete API reference for the C-Relay event-based administration system and adv
## Overview
C-Relay uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
C-Relay-PG uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
- **Cryptographic security**: All commands must be signed with the admin private key
- **Audit trail**: Complete history of all administrative actions
@@ -63,8 +63,8 @@ Store the admin private key securely:
export C_RELAY_ADMIN_KEY="nsec1abc123..."
# Secure file
echo "nsec1abc123..." > ~/.c-relay-admin
chmod 600 ~/.c-relay-admin
echo "nsec1abc123..." > ~/.c-relay-pg-admin
chmod 600 ~/.c-relay-pg-admin
# Password manager (recommended)
# Store in 1Password, Bitwarden, etc.
@@ -167,7 +167,7 @@ Examples:
"data": [
{
"key": "relay_name",
"value": "C-Relay",
"value": "C-Relay-PG",
"data_type": "string",
"category": "relay",
"description": "Relay name displayed in NIP-11"
@@ -506,10 +506,10 @@ ORDER BY count DESC
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `relay_name` | string | "C-Relay" | Relay name (NIP-11) |
| `relay_name` | string | "C-Relay-PG" | Relay name (NIP-11) |
| `relay_description` | string | "C Nostr Relay" | Relay description |
| `relay_contact` | string | "" | Admin contact info |
| `relay_software` | string | "c-relay" | Software identifier |
| `relay_software` | string | "c-relay-pg" | Software identifier |
| `relay_version` | string | auto | Software version |
| `supported_nips` | string | "1,9,11,13,15,20,33,40,42,45,50,70" | Supported NIPs |
| `language_tags` | string | "*" | Supported languages |
@@ -576,7 +576,7 @@ ORDER BY count DESC
## Real-time Monitoring
C-Relay provides subscription-based real-time monitoring using ephemeral events (kind 24567).
C-Relay-PG provides subscription-based real-time monitoring using ephemeral events (kind 24567).
### Activation
+43 -33
View File
@@ -1,12 +1,14 @@
# Alpine-based MUSL static binary builder for C-Relay
# Alpine-based MUSL static binary builder for C-Relay-PG
# Produces truly portable binaries with zero runtime dependencies
ARG DEBUG_BUILD=false
ARG DB_BACKEND=sqlite
FROM alpine:3.19 AS builder
# Re-declare build argument in this stage
# Re-declare build arguments in this stage
ARG DEBUG_BUILD=false
ARG DB_BACKEND=sqlite
# Install build dependencies
RUN apk add --no-cache \
@@ -26,9 +28,11 @@ RUN apk add --no-cache \
curl-static \
sqlite-dev \
sqlite-static \
postgresql-dev \
linux-headers \
wget \
bash
bash \
openssh-client
# Set working directory
WORKDIR /build
@@ -68,15 +72,8 @@ RUN cd /tmp && \
make install && \
rm -rf /tmp/libwebsockets
# Copy only submodule configuration and git directory
COPY .gitmodules /build/.gitmodules
COPY .git /build/.git
# Clean up any stale submodule references (nips directory is not a submodule)
RUN git rm --cached nips 2>/dev/null || true
# Initialize submodules (cached unless .gitmodules changes)
RUN git submodule update --init --recursive
# Submodules are provided in the local build context and copied explicitly below.
# Avoid network/SSH dependency inside Docker image build.
# Copy nostr_core_lib source files (cached unless nostr_core_lib changes)
COPY nostr_core_lib /build/nostr_core_lib/
@@ -84,11 +81,13 @@ COPY nostr_core_lib /build/nostr_core_lib/
# Copy c_utils_lib source files (cached unless c_utils_lib changes)
COPY c_utils_lib /build/c_utils_lib/
# Build c_utils_lib with MUSL-compatible flags (cached unless c_utils_lib changes)
# Build c_utils_lib static library for relay logging utilities
# (build only debug.c to avoid broken/missing version header surface in submodule revision)
RUN cd c_utils_lib && \
sed -i 's/CFLAGS = -Wall -Wextra -std=c99 -O2 -g/CFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g/' Makefile && \
make clean && \
make
mkdir -p build && \
gcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g \
-Isrc -Iinclude -c src/debug.c -o build/debug.o && \
ar rcs libc_utils.a build/debug.o
# Build nostr_core_lib with required NIPs (cached unless nostr_core_lib changes)
# Disable fortification in build.sh to prevent __*_chk symbol issues
@@ -99,44 +98,55 @@ RUN cd nostr_core_lib && \
rm -f *.o *.a 2>/dev/null || true && \
./build.sh --nips=1,6,13,17,19,44,59
# Copy c-relay source files LAST (only this layer rebuilds on source changes)
# Copy c-relay-pg source files LAST (only this layer rebuilds on source changes)
COPY src/ /build/src/
COPY Makefile /build/Makefile
# Build c-relay with full static linking (only rebuilds when src/ changes)
# Build c-relay-pg with full static linking (only rebuilds when src/ changes)
# Disable fortification to avoid __*_chk symbols that don't exist in MUSL
# Use conditional compilation flags based on DEBUG_BUILD argument
# Use conditional compilation flags based on DEBUG_BUILD and DB_BACKEND build args
RUN if [ "$DEBUG_BUILD" = "true" ]; then \
CFLAGS="-g -O0 -DDEBUG"; \
STRIP_CMD=""; \
echo "Building with DEBUG symbols enabled"; \
CFLAGS="-g -O2 -DDEBUG"; \
STRIP_CMD="echo 'Keeping debug symbols'"; \
echo "Building with DEBUG symbols enabled (optimized with -O2)"; \
else \
CFLAGS="-O2"; \
STRIP_CMD="strip /build/c_relay_static"; \
echo "Building optimized production binary"; \
STRIP_CMD="strip /build/c_relay_pg_static"; \
echo "Building optimized production binary (symbols stripped)"; \
fi && \
gcc -static $CFLAGS -Wall -Wextra -std=c99 \
if [ "$DB_BACKEND" = "postgres" ]; then \
DB_FLAGS="-DDB_BACKEND_POSTGRES -DHAVE_LIBPQ"; \
DB_LIBS="-lpq -lpgcommon -lpgport"; \
echo "Compiling with PostgreSQL backend"; \
else \
DB_FLAGS=""; \
DB_LIBS=""; \
echo "Compiling with SQLite backend"; \
fi && \
gcc -static $CFLAGS $DB_FLAGS -Wall -Wextra -std=c99 \
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
-I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core \
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket \
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql \
src/main.c src/config.c src/dm_admin.c src/request_validator.c \
src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c \
src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c \
-o /build/c_relay_static \
src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c \
src/caching_inbox_poller.c src/caching_service_launcher.c \
src/db_ops.c src/db_ops_sqlite.c src/db_ops_postgres.c src/thread_pool.c \
-o /build/c_relay_pg_static \
c_utils_lib/libc_utils.a \
nostr_core_lib/libnostr_core_x64.a \
-lwebsockets -lssl -lcrypto -lsqlite3 -lsecp256k1 \
-lcurl -lz -lpthread -lm -ldl && \
-lcurl -lz -lpthread -lm -ldl $DB_LIBS && \
eval "$STRIP_CMD"
# Verify it's truly static
RUN echo "=== Binary Information ===" && \
file /build/c_relay_static && \
ls -lh /build/c_relay_static && \
file /build/c_relay_pg_static && \
ls -lh /build/c_relay_pg_static && \
echo "=== Checking for dynamic dependencies ===" && \
(ldd /build/c_relay_static 2>&1 || echo "Binary is static") && \
(ldd /build/c_relay_pg_static 2>&1 || echo "Binary is static") && \
echo "=== Build complete ==="
# Output stage - just the binary
FROM scratch AS output
COPY --from=builder /build/c_relay_static /c_relay_static
COPY --from=builder /build/c_relay_pg_static /c_relay_pg_static
+38 -26
View File
@@ -1,28 +1,40 @@
# C-Relay Makefile
# C-Relay-PG Makefile
CC = gcc
CFLAGS = -Wall -Wextra -std=c99 -g -O2
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql
LIBS = -lsqlite3 -lwebsockets -lz -ldl -lpthread -lm -L/usr/local/lib -lsecp256k1 -lssl -lcrypto -L/usr/local/lib -lcurl -Lc_utils_lib -lc_utils
DB_BACKEND ?= sqlite
# Build directory
BUILD_DIR = build
# Source files
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c src/thread_pool.c src/caching_inbox_poller.c src/caching_service_launcher.c
DB_OPS_SRC = src/db_ops.c
ifeq ($(DB_BACKEND),postgres)
CFLAGS += -DDB_BACKEND_POSTGRES -DHAVE_LIBPQ
DB_OPS_SRC += src/db_ops_postgres.c
LIBS += -lpq
else
DB_OPS_SRC += src/db_ops_sqlite.c
endif
NOSTR_CORE_LIB = nostr_core_lib/libnostr_core_x64.a
C_UTILS_LIB = c_utils_lib/libc_utils.a
# Architecture detection
ARCH = $(shell uname -m)
ifeq ($(ARCH),x86_64)
TARGET = $(BUILD_DIR)/c_relay_x86
TARGET = $(BUILD_DIR)/c_relay_pg_x86
else ifeq ($(ARCH),aarch64)
TARGET = $(BUILD_DIR)/c_relay_arm64
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
else ifeq ($(ARCH),arm64)
TARGET = $(BUILD_DIR)/c_relay_arm64
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
else
TARGET = $(BUILD_DIR)/c_relay_$(ARCH)
TARGET = $(BUILD_DIR)/c_relay_pg_$(ARCH)
endif
# Default target
@@ -81,19 +93,19 @@ force-version:
@$(MAKE) src/main.h
# Build the relay
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Compiling C-Relay for architecture: $(ARCH)"
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Compiling C-Relay-PG for architecture: $(ARCH) (backend: $(DB_BACKEND))"
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(TARGET)"
# Build for specific architectures
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Building C-Relay for x86_64..."
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_x86"
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Building C-Relay-PG for x86_64 (backend: $(DB_BACKEND))..."
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_x86"
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Cross-compiling C-Relay for ARM64..."
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Cross-compiling C-Relay-PG for ARM64 (backend: $(DB_BACKEND))..."
@if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1; then \
echo "ERROR: ARM64 cross-compiler not found."; \
echo "Install with: make install-cross-tools"; \
@@ -116,9 +128,9 @@ arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(
fi
@echo "Using aarch64-linux-gnu-gcc with ARM64 libraries..."
PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
-L/usr/lib/aarch64-linux-gnu $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_arm64"
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_arm64"
# Install ARM64 cross-compilation dependencies
install-arm64-deps:
@@ -160,7 +172,7 @@ test: $(TARGET)
init-db:
@echo "Database initialization is now handled automatically when the server starts."
@echo "The schema is embedded in the binary - no external files needed."
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_x86"
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_pg_x86"
# Clean build artifacts
clean:
@@ -180,7 +192,7 @@ install-deps:
# Help
help:
@echo "C-Relay Build System"
@echo "C-Relay-PG Build System"
@echo ""
@echo "Targets:"
@echo " all Build the relay for current architecture (default)"
@@ -209,15 +221,15 @@ help:
# Build fully static MUSL binaries using Docker
static-musl-x86_64:
@echo "Building fully static MUSL binary for x86_64..."
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-x86_64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-x86_64 sh -c "cp /c_relay_static_musl_x86_64 /output/"
@echo "Static binary created: build/c_relay_static_musl_x86_64"
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-x86_64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-x86_64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/"
@echo "Static binary created: build/c_relay_pg_static_musl_x86_64"
static-musl-arm64:
@echo "Building fully static MUSL binary for ARM64..."
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-arm64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-arm64 sh -c "cp /c_relay_static_musl_x86_64 /output/c_relay_static_musl_arm64"
@echo "Static binary created: build/c_relay_static_musl_arm64"
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-arm64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-arm64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/c_relay_pg_static_musl_arm64"
@echo "Static binary created: build/c_relay_pg_static_musl_arm64"
static-musl: static-musl-x86_64 static-musl-arm64
@echo "Built static MUSL binaries for both architectures"
+6 -6
View File
@@ -45,21 +45,21 @@ Also included is a more standard administrative web front end. This front end co
## Screenshots
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/main.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/main.png)
Main page with real time updates.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/config.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/config.png)
Set your configuration preferences.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/subscriptions.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/subscriptions.png)
View current subscriptions
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/white-blacklists.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/white-blacklists.png)
Add npubs to white or black lists.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/sqlQuery.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/sqlQuery.png)
Run sql queries on the database.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/main-light.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/main-light.png)
Light mode.
+24 -24
View File
@@ -1,11 +1,11 @@
# C-Relay: High-Performance Nostr Relay
# C-Relay-PG: High-Performance Nostr Relay
A blazingly fast, production-ready Nostr relay implemented in C with an innovative event-based configuration system. Built for performance, security, and ease of deployment.
## 🚀 Why C-Relay?
## 🚀 Why C-Relay-PG?
### Event-Based Configuration
Unlike traditional relays that require config files, C-Relay uses **cryptographically signed Nostr events** for all configuration. This means:
Unlike traditional relays that require config files, C-Relay-PG uses **cryptographically signed Nostr events** for all configuration. This means:
- **Zero config files** - Everything stored in the database
- **Real-time updates** - Changes applied instantly without restart
- **Cryptographic security** - All changes must be signed by admin
@@ -36,7 +36,7 @@ Control your relay by sending direct messages from any Nostr client:
## 📋 Supported NIPs
C-Relay implements a comprehensive set of Nostr Improvement Proposals:
C-Relay-PG implements a comprehensive set of Nostr Improvement Proposals:
-**NIP-01**: Basic protocol flow implementation
-**NIP-09**: Event deletion
@@ -89,12 +89,12 @@ Download and run - no dependencies required:
```bash
# Download the latest static release
wget https://git.laantungir.net/laantungir/c-relay/releases/download/v0.6.0/c-relay-v0.6.0-linux-x86_64-static
chmod +x c-relay-v0.6.0-linux-x86_64-static
mv c-relay-v0.6.0-linux-x86_64-static c-relay
wget https://git.laantungir.net/laantungir/c-relay-pg/releases/download/v0.6.0/c-relay-pg-v0.6.0-linux-x86_64-static
chmod +x c-relay-pg-v0.6.0-linux-x86_64-static
mv c-relay-pg-v0.6.0-linux-x86_64-static c-relay-pg
# Run the relay
./c-relay
./c-relay-pg
```
**Important**: On first startup, save the **Admin Private Key** displayed in the console. You'll need it for all administrative operations.
@@ -107,8 +107,8 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
libwebsockets-dev libssl-dev libsecp256k1-dev libcurl4-openssl-dev zlib1g-dev
# Clone and build
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
./make_and_restart_relay.sh
```
@@ -136,8 +136,8 @@ The web interface provides:
```bash
# Clone repository
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
# Build
@@ -147,25 +147,25 @@ make clean && make
sudo systemd/install-service.sh
# Start and enable
sudo systemctl start c-relay
sudo systemctl enable c-relay
sudo systemctl start c-relay-pg
sudo systemctl enable c-relay-pg
# Capture admin keys from logs
sudo journalctl -u c-relay | grep "Admin Private Key"
sudo journalctl -u c-relay-pg | grep "Admin Private Key"
```
### Docker Deployment
```bash
# Build Docker image
docker build -f Dockerfile.alpine-musl -t c-relay .
docker build -f Dockerfile.alpine-musl -t c-relay-pg .
# Run container
docker run -d \
--name c-relay \
--name c-relay-pg \
-p 8888:8888 \
-v /path/to/data:/data \
c-relay
c-relay-pg
```
### Cloud Deployment
@@ -181,7 +181,7 @@ See [`docs/deployment_guide.md`](docs/deployment_guide.md) for detailed deployme
## 🔧 Configuration
C-Relay uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
C-Relay-PG uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
### Basic Configuration
@@ -239,8 +239,8 @@ The admin private key is displayed **only once** during first startup. Store it
```bash
# Save to secure location
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-admin
chmod 600 ~/.c-relay-admin
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-pg-admin
chmod 600 ~/.c-relay-pg-admin
```
### Production Security
@@ -269,9 +269,9 @@ Contributions are welcome! Please:
## 🔗 Links
- **Repository**: [https://github.com/your-org/c-relay](https://github.com/your-org/c-relay)
- **Releases**: [https://git.laantungir.net/laantungir/c-relay/releases](https://git.laantungir.net/laantungir/c-relay/releases)
- **Issues**: [https://github.com/your-org/c-relay/issues](https://github.com/your-org/c-relay/issues)
- **Repository**: [https://github.com/your-org/c-relay-pg](https://github.com/your-org/c-relay-pg)
- **Releases**: [https://git.laantungir.net/laantungir/c-relay-pg/releases](https://git.laantungir.net/laantungir/c-relay-pg/releases)
- **Issues**: [https://github.com/your-org/c-relay-pg/issues](https://github.com/your-org/c-relay-pg/issues)
- **Nostr Protocol**: [https://github.com/nostr-protocol/nostr](https://github.com/nostr-protocol/nostr)
## 💬 Support
+174
View File
@@ -0,0 +1,174 @@
# Real-Time Traffic Monitoring Commands (Direct Server Use)
Copy and paste these commands directly on your server.
## Quick Status Checks
### See IPs visiting in the last few minutes:
```bash
sudo tail -500 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
```
### See what status codes they're getting:
```bash
sudo tail -500 /var/log/nginx/access.log | awk '{print $1, $9}' | grep '216.73.216.38'
```
### Count status codes (200 vs 403):
```bash
sudo tail -500 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
```
## Real-Time Monitoring
### Watch live traffic (updates every 2 seconds):
```bash
watch -n 2 'sudo tail -200 /var/log/nginx/access.log | awk "{print \$1}" | sort | uniq -c | sort -rn | head -15'
```
### See live log entries as they happen:
```bash
sudo tail -f /var/log/nginx/access.log
```
### Live GoAccess dashboard:
```bash
sudo tail -f /var/log/nginx/access.log | goaccess -
```
## Active Connections
### See who's connected RIGHT NOW:
```bash
sudo netstat -tn | grep ':443' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
```
### Alternative (using ss command):
```bash
sudo ss -tn | grep ':443' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
```
## Detailed Analysis
### Last 100 requests with timestamps:
```bash
sudo tail -100 /var/log/nginx/access.log | awk '{print $4, $1}' | sed 's/\[//'
```
### See what blocked IPs are trying to access:
```bash
sudo tail -500 /var/log/nginx/access.log | grep '216.73.216.38' | awk '{print $7}' | head -10
```
### Show all 403 (blocked) requests:
```bash
sudo tail -500 /var/log/nginx/access.log | awk '$9==403 {print $1}' | sort | uniq -c | sort -rn
```
### Show all successful (200) requests:
```bash
sudo tail -500 /var/log/nginx/access.log | awk '$9==200 {print $1}' | sort | uniq -c | sort -rn | head -10
```
## Comprehensive Monitoring Script
### Create a monitoring script:
```bash
cat > /tmp/monitor-traffic.sh << 'EOF'
#!/bin/bash
echo "=== Traffic in last 5 minutes ==="
echo "Time: $(date)"
echo ""
echo "Top IPs:"
sudo tail -1000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
echo ""
echo "Blocked IPs (403 errors):"
sudo tail -1000 /var/log/nginx/access.log | awk '$9==403 {print $1}' | sort | uniq -c | sort -rn
echo ""
echo "Successful requests (200):"
sudo tail -1000 /var/log/nginx/access.log | awk '$9==200 {print $1}' | sort | uniq -c | sort -rn | head -5
echo ""
echo "Status Code Summary:"
sudo tail -1000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn
EOF
chmod +x /tmp/monitor-traffic.sh
```
### Run the monitoring script:
```bash
/tmp/monitor-traffic.sh
```
## Auto-Refreshing Dashboard
### Live dashboard (refreshes every 5 seconds):
```bash
watch -n 5 'echo "=== Last 5 minutes ==="
date
echo ""
echo "Top IPs:"
sudo tail -1000 /var/log/nginx/access.log | awk "{print \$1}" | sort | uniq -c | sort -rn | head -10
echo ""
echo "Status Codes:"
sudo tail -1000 /var/log/nginx/access.log | awk "{print \$9}" | sort | uniq -c | sort -rn'
```
Press `Ctrl+C` to exit.
## GoAccess HTML Report (Live Updating)
### Generate live HTML report:
```bash
sudo goaccess /var/log/nginx/access.log -o /var/www/html/live-stats.html --real-time-html --daemonize
```
Then visit: https://git.laantungir.net/live-stats.html
### Stop the live report:
```bash
sudo pkill -f "goaccess.*live-stats"
```
## Filter by Time
### Get timestamp from 5 minutes ago:
```bash
date -d '5 minutes ago' '+%d/%b/%Y:%H:%M'
```
### Analyze only recent logs (replace timestamp):
```bash
sudo awk '/01\/Feb\/2026:19:09/,0' /var/log/nginx/access.log | goaccess -
```
## Check Gitea CPU
### Current CPU usage:
```bash
ps aux | grep gitea | grep -v grep
```
### Watch CPU in real-time:
```bash
watch -n 2 'ps aux | grep gitea | grep -v grep'
```
## Most Useful Command for Quick Check
This one-liner shows everything you need:
```bash
echo "=== Quick Status ===" && \
echo "Time: $(date)" && \
echo "" && \
echo "Top 10 IPs (last 1000 requests):" && \
sudo tail -1000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10 && \
echo "" && \
echo "Status Codes:" && \
sudo tail -1000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c && \
echo "" && \
echo "Gitea CPU:" && \
ps aux | grep gitea | grep -v grep
```
Copy any of these commands and run them directly on your server!
+7 -7
View File
@@ -41,7 +41,7 @@ This guide is specifically tailored for C programs that use:
```bash
# 1. Copy the Dockerfile template (see below)
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
# 2. Customize for your project (see Customization section)
vim Dockerfile.static
@@ -466,13 +466,13 @@ docker build -t my-app:latest .
docker run --rm my-app:latest --help
```
## Reusing c-relay Files
## Reusing c-relay-pg Files
You can directly copy these files from c-relay:
You can directly copy these files from c-relay-pg:
### 1. Dockerfile.alpine-musl
```bash
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
```
Then customize:
@@ -482,7 +482,7 @@ Then customize:
### 2. build_static.sh
```bash
cp /path/to/c-relay/build_static.sh ./
cp /path/to/c-relay-pg/build_static.sh ./
```
Then customize:
@@ -492,7 +492,7 @@ Then customize:
### 3. .dockerignore (Optional)
```bash
cp /path/to/c-relay/.dockerignore ./
cp /path/to/c-relay-pg/.dockerignore ./
```
Helps speed up Docker builds by excluding unnecessary files.
@@ -522,7 +522,7 @@ Helps speed up Docker builds by excluding unnecessary files.
- [Alpine Linux](https://alpinelinux.org/)
- [nostr_core_lib](https://github.com/chebizarro/nostr_core_lib)
- [Static Linking Best Practices](https://www.musl-libc.org/faq.html)
- [c-relay Implementation](./docs/musl_static_build.md)
- [c-relay-pg Implementation](./docs/musl_static_build.md)
## Example: Minimal Nostr Client
+191
View File
@@ -1291,6 +1291,184 @@ body.dark-mode .sql-results-table tbody tr:nth-child(even) {
/* background-color: var(--secondary-color); */
}
/* ================================
WEB OF TRUST (WoT) STYLES
================================ */
.wot-status-row, .wot-stats-row {
display: flex;
justify-content: space-between;
align-items: center;
padding: 8px 0;
font-size: 14px;
}
.wot-indicator {
padding: 2px 10px;
border-radius: 4px;
font-weight: bold;
font-size: 12px;
}
.wot-indicator.wot-found { background: #28a745; color: white; }
.wot-indicator.wot-missing { background: #dc3545; color: white; }
.wot-indicator.wot-unknown { background: #6c757d; color: white; }
.wot-level-selector { padding: 10px 0; }
.wot-level-selector label { display: block; margin-bottom: 5px; font-weight: bold; }
.wot-level-btn { min-width: 100px; }
.wot-level-btn.active {
background: var(--accent-color, #ff0000);
color: white;
border-color: var(--accent-color, #ff0000);
}
.wot-level-description {
font-size: 12px;
color: var(--primary-color);
margin-top: 5px;
font-style: italic;
opacity: 0.8;
}
/* Dark mode adjustments for WoT */
body.dark-mode .wot-indicator.wot-found { background: #28a745; }
body.dark-mode .wot-indicator.wot-missing { background: #dc3545; }
body.dark-mode .wot-indicator.wot-unknown { background: #6c757d; }
/* ================================
ADMIN ACCESS GATE STYLES
================================ */
/* Access Denied Overlay */
.access-denied-overlay {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background: rgba(0, 0, 0, 0.85);
z-index: 9999;
display: none;
justify-content: center;
align-items: center;
}
.access-denied-content {
background: var(--card-bg);
border: 2px solid #dc3545;
border-radius: 12px;
padding: 40px 60px;
text-align: center;
max-width: 500px;
box-shadow: 0 10px 40px rgba(220, 53, 69, 0.3);
}
.access-denied-icon {
font-size: 64px;
margin-bottom: 20px;
}
.access-denied-content h2 {
color: #dc3545;
font-size: 32px;
margin-bottom: 20px;
letter-spacing: 2px;
}
.access-denied-message {
font-size: 16px;
color: var(--primary-color);
margin-bottom: 10px;
line-height: 1.5;
}
.access-denied-submessage {
font-size: 14px;
color: var(--muted-color);
margin-bottom: 30px;
font-style: italic;
}
.access-denied-logout-btn {
background: #dc3545;
color: white;
border: none;
padding: 12px 40px;
font-size: 16px;
font-weight: bold;
border-radius: 6px;
cursor: pointer;
transition: all 0.3s ease;
}
.access-denied-logout-btn:hover {
background: #c82333;
transform: translateY(-2px);
box-shadow: 0 4px 12px rgba(220, 53, 69, 0.4);
}
/* Admin Verification Loading Overlay */
.admin-verification-overlay {
position: fixed;
top: 0;
left: 0;
width: 100%;
height: 100%;
background: rgba(0, 0, 0, 0.8);
z-index: 9998;
display: none;
justify-content: center;
align-items: center;
}
.admin-verification-content {
background: var(--card-bg);
border: 1px solid var(--border-color);
border-radius: 12px;
padding: 40px 60px;
text-align: center;
max-width: 450px;
}
.admin-verification-content h3 {
color: var(--accent-color);
font-size: 20px;
margin-bottom: 15px;
}
.admin-verification-content p {
color: var(--muted-color);
font-size: 14px;
margin-top: 15px;
}
/* Spinner Animation */
.spinner {
width: 50px;
height: 50px;
border: 4px solid var(--border-color);
border-top: 4px solid var(--accent-color);
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 20px;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
/* Dark mode adjustments */
body.dark-mode .access-denied-content {
background: var(--card-bg);
}
body.dark-mode .admin-verification-content {
background: var(--card-bg);
}
.subscription-detail-row:hover {
background-color: var(--muted-color);
}
@@ -1308,3 +1486,16 @@ body.dark-mode .sql-results-table tbody tr:nth-child(even) {
font-size: 12px;
}
/* ================================
IP BANS TABLE - compact rows
================================ */
#ip-bans-table td, #ip-bans-table th {
padding: 4px 8px;
line-height: 1.3;
font-size: 13px;
}
#ip-bans-table button {
padding: 2px 8px;
font-size: 12px;
}
+336 -15
View File
@@ -4,7 +4,7 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>C-Relay Admin</title>
<title>C-Relay-PG Admin</title>
<link rel="stylesheet" href="/api/index.css">
</head>
@@ -16,7 +16,9 @@
<li><button class="nav-item" data-page="subscriptions">Subscriptions</button></li>
<li><button class="nav-item" data-page="configuration">Configuration</button></li>
<li><button class="nav-item" data-page="authorization">Authorization</button></li>
<li><button class="nav-item" data-page="ip-bans">IP BANS</button></li>
<li><button class="nav-item" data-page="relay-events">Relay Events</button></li>
<li><button class="nav-item" data-page="caching">Caching</button></li>
<li><button class="nav-item" data-page="dm">DM</button></li>
<li><button class="nav-item" data-page="database">Database Query</button></li>
</ul>
@@ -41,7 +43,7 @@
<span class="relay-letter" data-letter="Y">Y</span>
</div>
<div class="relay-info">
<div id="relay-name" class="relay-name">C-Relay</div>
<div id="relay-name" class="relay-name">C-Relay-PG</div>
<div id="relay-description" class="relay-description">Loading...</div>
<div id="relay-pubkey-container" class="relay-pubkey-container">
<div id="relay-pubkey" class="relay-pubkey">Loading...</div>
@@ -67,6 +69,17 @@
</div>
</div>
<!-- Access Denied Overlay (shown when non-admin user logs in) -->
<div id="access-denied-overlay" class="access-denied-overlay" style="display: none;">
<div class="access-denied-content">
<div class="access-denied-icon"></div>
<h2>ACCESS DENIED</h2>
<p class="access-denied-message">This interface is restricted to the relay administrator.</p>
<p class="access-denied-submessage">The logged-in account does not have admin privileges.</p>
<button type="button" class="access-denied-logout-btn" onclick="logout()">LOGOUT</button>
</div>
</div>
<!-- DATABASE STATISTICS Section -->
<!-- Subscribe to kind 24567 events to receive real-time monitoring data -->
<div class="section flex-section" id="databaseStatisticsSection" style="display: none;">
@@ -100,6 +113,10 @@
<td>Process ID</td>
<td id="process-id">-</td>
</tr>
<tr>
<td>WebSocket Connections</td>
<td id="websocket-connections">-</td>
</tr>
<tr>
<td>Active Subscriptions</td>
<td id="active-subscriptions">-</td>
@@ -108,14 +125,14 @@
<td>Memory Usage</td>
<td id="memory-usage">-</td>
</tr>
<tr>
<td>CPU Core</td>
<td id="cpu-core">-</td>
</tr>
<tr>
<td>CPU Usage</td>
<td id="cpu-usage">-</td>
</tr>
<tr>
<td>CPU Core</td>
<td id="cpu-core">-</td>
</tr>
<tr>
<td>Oldest Event</td>
<td id="oldest-event">-</td>
@@ -251,6 +268,7 @@
</div>
<!-- Auth Rules Management - Moved after configuration -->
<!-- AUTH RULES MANAGEMENT SECTION -->
<div class="section flex-section" id="authRulesSection" style="display: none;">
<div class="section-header">
AUTH RULES MANAGEMENT
@@ -273,16 +291,11 @@ AUTH RULES MANAGEMENT
</table>
</div>
<!-- Simplified Auth Rule Input Section -->
<!-- Auth Rule Input Section -->
<div id="authRuleInputSections" style="display: block;">
<!-- Combined Pubkey Auth Rule Section -->
<div class="input-group">
<label for="authRulePubkey">Pubkey (nsec or hex):</label>
<input type="text" id="authRulePubkey" placeholder="nsec1... or 64-character hex pubkey">
<label for="authRulePubkey">Public Key (npub or hex):</label>
<input type="text" id="authRulePubkey" placeholder="npub1... or 64-character hex pubkey">
</div>
<div id="whitelistWarning" class="warning-box" style="display: none;">
<strong>⚠️ WARNING:</strong> Adding whitelist rules changes relay behavior to whitelist-only
@@ -296,10 +309,51 @@ AUTH RULES MANAGEMENT
BLACKLIST</button>
<button type="button" id="refreshAuthRulesBtn">REFRESH</button>
</div>
</div>
</div>
<!-- WEB OF TRUST SECTION -->
<div class="section flex-section" id="wotSection" style="display: none;">
<div class="section-header">
WEB OF TRUST
</div>
<!-- Kind 3 Status Indicator -->
<div id="wotKind3Status" class="wot-status-row">
<span>Admin Contact List (kind 3):</span>
<span id="wotKind3Indicator" class="wot-indicator wot-unknown">Checking...</span>
</div>
<!-- WoT Level Selector -->
<div class="wot-level-selector">
<label>WoT Level:</label>
<div class="inline-buttons">
<button type="button" id="wotLevel0Btn" class="wot-level-btn" onclick="setWotLevel(0)">
OFF
</button>
<button type="button" id="wotLevel1Btn" class="wot-level-btn" onclick="setWotLevel(1)">
WRITE ONLY
</button>
<button type="button" id="wotLevel2Btn" class="wot-level-btn" onclick="setWotLevel(2)">
FULL
</button>
</div>
<div class="wot-level-description" id="wotLevelDescription">
Level 0: Open relay — anyone can read and write
</div>
</div>
<!-- WoT Stats -->
<div class="wot-stats-row">
<span>Whitelisted Pubkeys:</span>
<span id="wotWhitelistCount"></span>
</div>
<!-- Sync Button -->
<div class="inline-buttons">
<button type="button" id="wotSyncBtn" onclick="syncWot()">SYNC FROM KIND 3</button>
<button type="button" id="wotRefreshBtn" onclick="loadWotStatus()">REFRESH STATUS</button>
</div>
</div>
@@ -334,12 +388,138 @@ AUTH RULES MANAGEMENT
</div>
</div>
<!-- IP BANS Section -->
<div class="section" id="ipBansSection" style="display: none;">
<div class="section-header">
IP BAN MANAGEMENT
</div>
<!-- Statistics Cards -->
<div class="input-group">
<div class="config-table-container">
<table class="config-table" id="ip-bans-stats-table">
<thead>
<tr>
<th>Total IPs Tracked</th>
<th>Currently Banned</th>
<th>Total Bans Issued</th>
</tr>
</thead>
<tbody>
<tr>
<td id="ip-bans-total">-</td>
<td id="ip-bans-active">-</td>
<td id="ip-bans-issued">-</td>
</tr>
</tbody>
</table>
</div>
</div>
<!-- Add Ban Form -->
<div class="input-group">
<h3>Manually Ban IP Address</h3>
<div class="form-group">
<label for="ban-ip-input">IP Address:</label>
<input type="text" id="ban-ip-input" placeholder="192.168.1.100">
</div>
<div class="form-group">
<label for="ban-duration-select">Ban Duration:</label>
<select id="ban-duration-select">
<option value="3600">1 Hour</option>
<option value="86400" selected>24 Hours</option>
<option value="604800">7 Days</option>
<option value="2592000">30 Days</option>
<option value="31536000">1 Year</option>
<option value="999999999">Permanent</option>
</select>
</div>
<div class="inline-buttons">
<button type="button" id="add-ban-btn">BAN IP</button>
</div>
<div id="add-ban-status" class="status-message"></div>
</div>
<!-- Whitelist Management -->
<div class="input-group">
<h3>IP Whitelist (Never Banned)</h3>
<p style="font-size:13px;opacity:0.8;">IPs in this list are never idle-banned. Comma-separated.</p>
<div class="form-group">
<label for="whitelist-ip-input">Add IP to Whitelist:</label>
<input type="text" id="whitelist-ip-input" placeholder="103.81.231.220">
</div>
<div class="inline-buttons">
<button type="button" id="add-whitelist-btn">ADD TO WHITELIST</button>
</div>
<div id="whitelist-status" class="status-message"></div>
<div id="whitelist-current" style="margin-top:8px;font-size:13px;"></div>
</div>
<!-- Filter Controls -->
<div class="input-group">
<div class="inline-buttons">
<button type="button" id="ip-ban-filter-all" class="active">All IPs</button>
<button type="button" id="ip-ban-filter-banned">Currently Banned</button>
<button type="button" id="ip-ban-filter-expired">Expired</button>
<button type="button" id="refresh-ip-bans-btn">REFRESH</button>
</div>
</div>
<!-- IP Bans List -->
<div class="input-group">
<label>Banned IP Addresses:</label>
<div class="config-table-container">
<table class="config-table" id="ip-bans-table">
<thead>
<tr>
<th>IP Address</th>
<th>Status</th>
<th>Banned Until</th>
<th>Failures</th>
<th>Authed Successfully</th>
<th>Connection Attempts</th>
<th>Actions</th>
</tr>
</thead>
<tbody id="ip-bans-tbody">
<tr>
<td colspan="7" style="text-align: center;">Click REFRESH to load IP bans</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<!-- RELAY EVENTS Section -->
<div class="section" id="relayEventsSection" style="display: none;">
<div class="section-header">
RELAY EVENTS MANAGEMENT
</div>
<!-- Live Relay Event Feed -->
<div class="input-group">
<h3>Live Relay Event Feed (Normal Nostr Subscription)</h3>
<div class="config-table-container">
<table class="config-table" id="live-relay-events-table">
<thead>
<tr>
<th>Time</th>
<th>Kind</th>
<th>Pubkey</th>
<th>ID</th>
<th>Content Preview</th>
</tr>
</thead>
<tbody id="live-relay-events-table-body">
<tr>
<td colspan="5" style="text-align: center;">Waiting for live events...</td>
</tr>
</tbody>
</table>
</div>
</div>
<!-- Kind 0: User Metadata -->
<div class="input-group">
<h3>Kind 0: User Metadata</h3>
@@ -400,6 +580,143 @@ AUTH RULES MANAGEMENT
</div>
</div>
<!-- CACHING Section -->
<div class="section" id="cachingSection" style="display: none;">
<div class="section-header">CACHING</div>
<!-- Caching Service Status (read-only) -->
<div class="input-group">
<h3>Caching Service Status</h3>
<div id="caching-service-status" class="status-display">
<p>Service status will appear here after connecting.</p>
</div>
</div>
<!-- Relay Inbox Status (read-only) -->
<div class="input-group">
<h3>Relay Inbox Status</h3>
<div id="caching-inbox-status" class="status-display">
<p>Inbox status will appear here after connecting.</p>
</div>
</div>
<!-- Caching Configuration (editable) -->
<div class="input-group">
<h3>Caching Configuration</h3>
<div class="form-group">
<label for="caching-enabled">Enable Caching:</label>
<select id="caching-enabled">
<option value="false">Disabled</option>
<option value="true">Enabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-inbox-enabled">Enable Inbox Consumer:</label>
<select id="caching-inbox-enabled">
<option value="false">Disabled</option>
<option value="true">Enabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-root-npubs">Root Npubs (one per line):</label>
<textarea id="caching-root-npubs" rows="4" placeholder="npub1...">npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks</textarea>
</div>
<div class="form-group">
<label for="caching-bootstrap-relays">Bootstrap Relays (one per line):</label>
<textarea id="caching-bootstrap-relays" rows="4" placeholder="wss://relay.damus.io">wss://relay.damus.io
wss://nos.lol
wss://relay.primal.net
wss://laantungir.net/relay</textarea>
</div>
<div class="form-group">
<label for="caching-kinds">Kinds (comma-separated):</label>
<input type="text" id="caching-kinds" placeholder="1,3,6,10000,30023" value="0,1,3,6,10000,10002,30023">
</div>
<div class="form-group">
<label for="caching-admin-kinds">Admin Kinds (comma-separated, * for all):</label>
<input type="text" id="caching-admin-kinds" placeholder="*">
</div>
<div class="form-group">
<label for="caching-live-enabled">Live Subscriptions:</label>
<select id="caching-live-enabled">
<option value="true">Enabled</option>
<option value="false">Disabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-backfill-enabled">Backfill:</label>
<select id="caching-backfill-enabled">
<option value="true">Enabled</option>
<option value="false">Disabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-backfill-windows">Backfill Windows (seconds, comma-separated):</label>
<input type="text" id="caching-backfill-windows" placeholder="86400,604800,2592000,7776000,31536000">
</div>
<div class="form-group">
<label for="caching-backfill-page-size">Backfill Page Size:</label>
<input type="number" id="caching-backfill-page-size" placeholder="50">
</div>
<div class="form-group">
<label for="caching-backfill-tick-interval">Backfill Tick Interval (ms):</label>
<input type="number" id="caching-backfill-tick-interval" placeholder="5000">
</div>
<div class="form-group">
<label for="caching-inbox-batch-size">Inbox Batch Size:</label>
<input type="number" id="caching-inbox-batch-size" placeholder="25">
</div>
<div class="form-group">
<label for="caching-inbox-active-poll">Inbox Active Poll (ms):</label>
<input type="number" id="caching-inbox-active-poll" placeholder="200">
</div>
<div class="form-group">
<label for="caching-inbox-idle-poll">Inbox Idle Poll (ms):</label>
<input type="number" id="caching-inbox-idle-poll" placeholder="5000">
</div>
<div class="form-group">
<label for="caching-service-binary-path">Caching Service Binary Path:</label>
<input type="text" id="caching-service-binary-path" placeholder="./caching_relay" value="./caching_relay">
</div>
<div class="form-group">
<label for="caching-service-pg-conn">Caching Service PG Connection:</label>
<input type="text" id="caching-service-pg-conn" placeholder="host=localhost port=5432 dbname=crelay user=crelay password=crelay" value="host=localhost port=5432 dbname=crelay user=crelay password=crelay">
</div>
<div class="inline-buttons">
<button type="button" id="caching-apply-btn">APPLY CONFIGURATION</button>
<button type="button" id="caching-reset-progress-btn">RESET BACKFILL PROGRESS</button>
</div>
<div id="caching-config-status" class="status-message"></div>
</div>
<div class="input-group">
<h3>Caching Service Control</h3>
<p>Start or stop the external caching service process. Set <code>caching_service_binary_path</code> and <code>caching_service_pg_conn</code> above and click APPLY CONFIGURATION before starting.</p>
<div class="inline-buttons">
<button type="button" id="caching-start-service-btn">START CACHING SERVICE</button>
<button type="button" id="caching-stop-service-btn">STOP CACHING SERVICE</button>
</div>
<div id="caching-service-control-status" class="status-message"></div>
</div>
</div>
<!-- SQL QUERY Section -->
<div class="section" id="sqlQuerySection" style="display: none;">
<div class="section-header">
@@ -419,6 +736,10 @@ AUTH RULES MANAGEMENT
<option value="event_kinds">Event Kinds Distribution</option>
<option value="time_stats">Time-based Statistics</option>
</optgroup>
<optgroup label="IP Ban Queries">
<option value="banned_ips_summary">Banned IPs Summary</option>
<option value="banned_ips_list">All Banned IP Addresses</option>
</optgroup>
<optgroup label="Query History" id="history-group">
<!-- Dynamically populated from localStorage -->
</optgroup>
+1572 -104
View File
File diff suppressed because it is too large Load Diff
+95 -1
View File
@@ -3297,8 +3297,18 @@ var NostrTools = (() => {
this.enablePing = opts.enablePing;
}
async ensureRelay(url, params) {
const rawUrl = url;
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
url = normalizeURL(url);
let relay = this.relays.get(url);
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL ensureRelay START", {
rawUrl,
normalizedUrl: url,
hadRelay: !!relay,
relayMapKeysBefore: Array.from(this.relays.keys())
});
}
if (!relay) {
relay = new AbstractRelay(url, {
verifyEvent: this.trustedRelayURLs.has(url) ? alwaysTrue : this.verifyEvent,
@@ -3306,13 +3316,38 @@ var NostrTools = (() => {
enablePing: this.enablePing
});
relay.onclose = () => {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL ensureRelay onclose", {
normalizedUrl: url,
relayMapKeysBeforeDelete: Array.from(this.relays.keys())
});
}
this.relays.delete(url);
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL ensureRelay onclose complete", {
normalizedUrl: url,
relayMapKeysAfterDelete: Array.from(this.relays.keys())
});
}
};
if (params?.connectionTimeout)
relay.connectionTimeout = params.connectionTimeout;
this.relays.set(url, relay);
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL ensureRelay created relay", {
normalizedUrl: url,
relayMapKeysAfterCreate: Array.from(this.relays.keys())
});
}
}
await relay.connect();
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL ensureRelay connected", {
normalizedUrl: url,
relayConnected: relay.connected,
relayMapKeysAfterConnect: Array.from(this.relays.keys())
});
}
return relay;
}
close(relays) {
@@ -3334,16 +3369,26 @@ var NostrTools = (() => {
}
subscribeMany(relays, filters, params) {
params.onauth = params.onauth || params.doauth;
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
const request = [];
const uniqUrls = [];
for (let i2 = 0; i2 < relays.length; i2++) {
const url = normalizeURL(relays[i2]);
if (uniqUrls.indexOf(url) === -1) {
uniqUrls.push(url);
for (let f2 = 0; f2 < filters.length; f2++) {
request.push({ url, filter: filters[f2] });
}
}
}
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL subscribeMany", {
relaysInput: relays,
uniqUrls,
filtersCount: filters.length,
requestsCount: request.length
});
}
return this.subscribeMap(request, params);
}
subscribeMap(requests, params) {
@@ -3391,14 +3436,31 @@ var NostrTools = (() => {
_knownIds.add(id);
return have;
};
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
const allOpened = Promise.all(
requests.map(async ({ url, filter }, i2) => {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL subscribeMap request", {
index: i2,
url,
filterKinds: filter?.kinds,
hasAuthorFilter: !!filter?.authors,
hasPTagFilter: !!filter?.["#p"]
});
}
let relay;
try {
relay = await this.ensureRelay(url, {
connectionTimeout: params.maxWait ? Math.max(params.maxWait * 0.8, params.maxWait - 1e3) : void 0
});
} catch (err) {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL subscribeMap ensureRelay FAILED", {
index: i2,
url,
error: err?.message || String(err)
});
}
handleClose(i2, err?.message || String(err));
return;
}
@@ -3480,18 +3542,50 @@ var NostrTools = (() => {
return events[0] || null;
}
publish(relays, event, options) {
return relays.map(normalizeURL).map(async (url, i2, arr) => {
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
const normalizedRelays = relays.map(normalizeURL);
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL publish START", {
relaysInput: relays,
normalizedRelays,
eventKind: event?.kind,
eventId: event?.id
});
}
return normalizedRelays.map(async (url, i2, arr) => {
if (arr.indexOf(url) !== i2) {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL publish duplicate URL", { url, index: i2 });
}
return Promise.reject("duplicate url");
}
let r = await this.ensureRelay(url);
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL publish ensured relay", {
url,
relayConnected: r?.connected,
relayMapKeys: Array.from(this.relays.keys())
});
}
return r.publish(event).catch(async (err) => {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL publish ERROR", {
url,
error: err?.message || String(err)
});
}
if (err instanceof Error && err.message.startsWith("auth-required: ") && options?.onauth) {
await r.auth(options.onauth);
return r.publish(event);
}
throw err;
}).then((reason) => {
if (debugEnabled) {
console.log("🔎 SIMPLE_POOL publish SUCCESS", {
url,
reason
});
}
if (this.trackRelays) {
let set = this.seenOn.get(event.id);
if (!set) {
+813
View File
@@ -0,0 +1,813 @@
<!DOCTYPE html>
<?xml version="1.0" encoding="UTF-8"?>
<html lang="en" dir="ltr">
<head>
<meta charset="utf-8" />
<title>TEMPLATE</title>
<link rel="stylesheet" href="./css/client.css" />
<!-- Initialize theme BEFORE any components load -->
<script>
(function () {
const savedTheme = localStorage.getItem('theme');
if (savedTheme === 'dark') {
document.documentElement.classList.add('dark-mode');
if (document.body) {
document.body.classList.add('dark-mode');
}
}
})();
</script>
<link rel="shortcut icon" type="image/x-icon" href="./favicon/favicon-dots2.ico" />
<!-- SVG.js library (required by HamburgerMorphing) -->
<script src="https://cdn.jsdelivr.net/npm/@svgdotjs/svg.js@3.0/dist/svg.min.js"></script>
</head>
<body>
<!-- ================================================================
HAMBURGER BUTTON (Fixed, separate from header)
================================================================
The hamburger button is a fixed element outside the header
to ensure it stays visible above the sidenav (z-index: 10 > 3).
================================================================ -->
<div id="divSvgHam" class="divHeaderButtons">
<!-- HamburgerMorphing will be injected here -->
</div>
<!-- ================================================================
HEADER
================================================================
Standard header with title (center).
================================================================ -->
<div id="divHeader">
<div id="divHeaderFlexLeft">
<!-- Hamburger is now separate fixed element -->
</div>
<div id="divHeaderFlexCenter">
<div class="divHeaderText"></div>
</div>
<div id="divHeaderFlexRight">
<!-- No button in header right - logout is in sidenav footer -->
</div>
</div>
<!-- ================================================================
BODY
================================================================
Main content area. Add your page-specific content here.
================================================================ -->
<div id="divBody">
</div>
<!-- ================================================================
FOOTER
================================================================
Three-section footer layout:
- Left: Relay status animations (HamburgerMorphing instances)
- Center: General status information
- Right: Additional information
================================================================ -->
<div id="divFooter">
<div id="divFooterLeft" class="divFooterBox"></div>
<div id="divFooterCenter" class="divFooterBox"></div>
<div id="divFooterRight" class="divFooterBox"></div>
<div id="divFooterBalance" class="divFooterBox">0 sats</div>
</div>
<!-- ================================================================
SIDENAV
================================================================
Slide-out navigation panel. Opens from left when hamburger clicked.
Uses flexbox layout to pin version bar to bottom.
Includes a version bar footer with theme toggle and logout buttons.
================================================================ -->
<div id="divSideNav">
<div id="divSideNavHeader">
<!-- No close button - use main hamburger to close -->
</div>
<div id="divSideNavBody">
<div id="divFiles"></div>
</div>
<div id="divAiSection" class="sidenavSection">
<div id="divAiSectionTitle" class="sidenavSectionTitle">AI</div>
<div id="divAiList" class="sidenavSectionList">
<div id="divAiProvidersList">No saved providers yet.</div>
</div>
</div>
<div id="divRelaySection">
<div id="divRelaySectionTitle">
リレー
</div>
<div id="divRelayList">
Loading relays...
</div>
</div>
<div id="divBlossomSection">
<div id="divBlossomSectionTitle">ブロッサム</div>
<div id="divBlossomList">Loading blossom servers...</div>
</div>
<div id="divVersionBar">
<span id="versionDisplay">v0.0.1</span>
<div id="divVersionBarButtons">
<button id="themeToggleButton" title="Toggle Dark/Light Mode">
<div id="themeToggleHamburgerContainer"></div>
</button>
<button id="logoutButton" title="Logout">
<div id="logoutHamburgerContainer"></div>
</button>
</div>
</div>
</div>
<!-- ================================================================
REQUIRED SCRIPTS
================================================================
These scripts must be loaded in this order:
1. nostr.bundle.js - Nostr tools library
2. nostr-lite.js - Authentication modal (nostr-login-lite)
================================================================ -->
<script src="./nostr.bundle.js"></script>
<script src="./nostr-lite.js"></script>
<script type="module">
/* ================================================================
IMPORTS
================================================================
Import shared NDK functionality from init-ndk.mjs:
- initNDKPage() - Initialize authentication and worker
- getPubkey() - Get current user's pubkey
- subscribe() - Create NDK subscriptions
- publishEvent() - Publish events via NDK
- disconnect() - Disconnect from worker
- getRelayData() - Get relay connection data
- getRelayStats() - Get relay activity statistics
Import HamburgerMorphing for animated icons
================================================================ */
import {
initNDKPage,
getPubkey, injectHeaderAvatar,
subscribe,
publishEvent,
disconnect,
getVersion,
updateVersionDisplay,
getUserSettings,
patchUserSettings,
onUserSettings
} from './js/init-ndk.mjs';
import { HamburgerMorphing } from "./hamburger_morphing/hamburger.mjs";
import { initFooterRelayStatus, updateFooterRelayStatus, initSidenavRelaySection, updateSidenavRelaySection, setRelayActivityState } from './js/relay-ui.mjs';
import { initBlossomSection, updateBlossomSection } from './js/blossom-ui.mjs';
import { initAiSectionWithLocalConfig } from './js/ai-ui.mjs';
// Version will be loaded asynchronously
const versionInfo = await getVersion();
const VERSION = versionInfo.VERSION;
console.log(`[template.html ${VERSION}] Loading...`);
/* ================================================================
GLOBAL VARIABLES
================================================================
Track state for hamburger menu, relay status, and theme.
================================================================ */
let updateIntervalId = null;
let currentPubkey = null;
/*
AUTH STATE MODEL (Template reference)
------------------------------------------------------------------
This template now demonstrates three auth modes for standalone pages:
- required (default):
Behaves like existing pages: login is required immediately.
- optional:
Page can render public/read-only data without login, but can still
prompt login later for user actions (publish, settings, etc).
- none:
Never auto-login on load (pure public page).
URL behavior in this template:
- If ?auth=required|optional|none is present, it wins.
- Otherwise, if URL includes ?npub=... or ?pubkey=..., mode defaults
to optional because pages with explicit profile targets are commonly
public-readable.
- Otherwise, mode defaults to required.
*/
let isAuthenticated = false;
let authMode = 'required';
let authedPageInitialized = false;
let relayActivityListenersBound = false;
// Hamburger menu
let hamburgerInstance = null;
let isNavOpen = false;
// Version bar buttons
let logoutHamburger = null;
let themeToggleHamburger = null;
let isDarkMode = false;
// App-wide user settings (NIP-78 kind 30078, d:user-settings)
let pageSettings = {};
let unsubscribeUserSettings = null;
/* ================================================================
DOM VARIABLES
================================================================
Cache DOM element references for better performance.
================================================================ */
const divBody = document.getElementById("divBody");
const divSideNav = document.getElementById("divSideNav");
const divSideNavBody = document.getElementById("divSideNavBody");
const divFooterCenter = document.getElementById("divFooterCenter");
const divFooterRight = document.getElementById("divFooterRight");
/* ================================================================
HAMBURGER MENU
================================================================
Initialize and control the animated hamburger menu.
================================================================ */
function initHamburgerMenu() {
hamburgerInstance = new HamburgerMorphing('#divSvgHam', {
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
hamburgerInstance.animateTo('burger');
}
/* ================================================================
SIDENAV FUNCTIONS
================================================================
Open/close sidenav with hamburger morphing animation.
================================================================ */
function openNav() {
divSideNav.style.zIndex = 3;
divSideNav.style.width = "clamp(400px, 50vw, 600px)";
isNavOpen = true;
if (hamburgerInstance) {
hamburgerInstance.animateTo('arrow_left');
}
// Initialize version bar buttons when sidenav opens (lazy load)
if (!logoutHamburger) {
logoutHamburger = new HamburgerMorphing('#logoutHamburgerContainer', {
size: 24,
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
logoutHamburger.animateTo('x');
}
if (!themeToggleHamburger) {
themeToggleHamburger = new HamburgerMorphing('#themeToggleHamburgerContainer', {
size: 24,
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
// Determine current theme
const savedTheme = localStorage.getItem('theme');
isDarkMode = savedTheme === 'dark' || document.body.classList.contains('dark-mode');
const initialShape = isDarkMode ? 'moon' : 'circle';
themeToggleHamburger.animateTo(initialShape);
}
}
function closeNav() {
divSideNav.style.width = "0vw";
divSideNav.style.zIndex = -1;
isNavOpen = false;
if (hamburgerInstance) {
hamburgerInstance.animateTo('burger');
}
}
function toggleNav() {
if (isNavOpen) {
closeNav();
} else {
openNav();
}
}
/* ================================================================
AUTH MODE HELPERS
================================================================
These functions are meant as reusable guidance for future pages.
================================================================ */
function hasTargetPubkeyInUrl() {
const params = new URLSearchParams(window.location.search || '');
const npub = String(params.get('npub') || '').trim();
const pubkey = String(params.get('pubkey') || '').trim();
return Boolean(npub || pubkey);
}
function resolveAuthModeFromUrl() {
const params = new URLSearchParams(window.location.search || '');
const explicitAuth = String(params.get('auth') || '').trim().toLowerCase();
if (explicitAuth === 'required' || explicitAuth === 'optional' || explicitAuth === 'none') {
return explicitAuth;
}
// Convention: explicit target profiles are public-readable by default.
if (hasTargetPubkeyInUrl()) {
return 'optional';
}
return 'required';
}
function isAuthRequiredError(error) {
const message = String(error?.message || error || '').toLowerCase();
return message.includes('authentication required');
}
async function initializeAuthentication(mode) {
// required: existing behavior, throw if auth fails.
if (mode === 'required') {
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
return;
}
// none: public page, no login attempt on load.
if (mode === 'none') {
isAuthenticated = false;
currentPubkey = null;
return;
}
// optional: try silent/normal init; if auth required, continue public.
try {
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
} catch (error) {
if (isAuthRequiredError(error)) {
console.log('[template.html] Optional auth mode: continuing unauthenticated');
isAuthenticated = false;
currentPubkey = null;
return;
}
throw error;
}
}
async function initializeAuthenticatedPageFeatures() {
if (!isAuthenticated || authedPageInitialized) return;
await injectHeaderAvatar(currentPubkey);
console.log('[template.html] Authenticated as:', currentPubkey);
// Hydrate app-wide user settings for this page
try {
pageSettings = await getUserSettings();
} catch (error) {
console.warn('[template.html] getUserSettings failed:', error);
pageSettings = {};
}
// Subscribe to live user settings updates (cross-tab + publish echoes)
if (!unsubscribeUserSettings) {
unsubscribeUserSettings = onUserSettings((settings) => {
pageSettings = settings || {};
// TODO: Re-render page-specific UI from pageSettings here.
});
}
// Initialize relay-dependent UI only once authenticated.
initFooterRelayStatus();
initSidenavRelaySection();
await initBlossomSection();
initAiSectionWithLocalConfig();
await UpdateFooter();
if (!updateIntervalId) {
updateIntervalId = setInterval(UpdateFooter, 1000);
}
// Relay activity listeners only matter after worker init/auth.
if (!relayActivityListenersBound) {
window.addEventListener('ndkRelayActivity', (event) => {
const { relayUrl, activity, stats } = event.detail;
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`, stats);
setRelayActivityState(relayUrl, activity);
});
window.addEventListener('message', (event) => {
if (event.data && event.data.type === 'relayActivity') {
const { relayUrl, activity } = event.data;
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`);
setRelayActivityState(relayUrl, activity);
}
});
relayActivityListenersBound = true;
}
authedPageInitialized = true;
}
async function promptLoginIfNeeded() {
if (isAuthenticated) return true;
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
await initializeAuthenticatedPageFeatures();
return true;
}
/* ================================================================
UPDATE FOOTER
================================================================
Update footer sections with relay status, pubkey, and other info.
Called periodically by update loop.
================================================================ */
const UpdateFooter = async () => {
try {
// Update relay status visuals in footer and sidenav
await updateFooterRelayStatus();
await updateSidenavRelaySection();
await updateBlossomSection();
// Clear center and right sections
divFooterCenter.innerHTML = '';
divFooterRight.innerHTML = '';
} catch (error) {
console.error('[template.html] Error updating footer:', error);
}
};
/* ================================================================
LOGOUT
================================================================
Complete logout process:
1. Stop update loop
2. Disconnect from NDK worker
3. Logout from nostr-login-lite
4. Clear all storage (localStorage, sessionStorage, IndexedDB)
5. Reload page
================================================================ */
const Logout = async () => {
console.log("[template.html] Starting logout process...");
// Stop the update loop
if (updateIntervalId) {
clearInterval(updateIntervalId);
updateIntervalId = null;
}
// Disconnect from worker
disconnect();
// Logout from nostr-login-lite
if (window.NOSTR_LOGIN_LITE && window.NOSTR_LOGIN_LITE.logout) {
await window.NOSTR_LOGIN_LITE.logout();
}
// Clear all storage
localStorage.clear();
sessionStorage.clear();
// Clear IndexedDB
if (window.indexedDB) {
const databases = await window.indexedDB.databases();
for (const db of databases) {
if (db.name) {
window.indexedDB.deleteDatabase(db.name);
}
}
}
console.log("[template.html] Logged out, reloading page");
location.reload(true);
};
/* ================================================================
EVENT LISTENERS
================================================================
Wire up UI interactions.
Main hamburger button click handler is set up in main() after initialization.
================================================================ */
/* ================================================================
SUBSCRIPTION EXAMPLE
================================================================
Example of how to subscribe to Nostr events:
const sub = subscribe(
{ kinds: [1], authors: [pubkey], limit: 10 },
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
);
Cache usage options:
- 'CACHE_FIRST' - Check cache first, then relays
- 'ONLY_RELAY' - Only query relays
- 'ONLY_CACHE' - Only query cache
- 'PARALLEL' - Query cache and relays simultaneously
Listen for events via window events:
window.addEventListener('ndkEvent', (event) => {
const evt = event.detail;
console.log('Received event:', evt);
});
================================================================ */
/* ================================================================
PUBLISH EXAMPLE
================================================================
Example of how to publish a Nostr event:
const event = {
created_at: Math.floor(Date.now() / 1000),
kind: 1,
tags: [],
content: "Hello, Nostr!"
};
try {
const result = await publishEvent(event);
console.log("✅ Published to:", result.relayResults.successful);
console.log("❌ Failed:", result.relayResults.failed);
console.log("Total relays:", result.totalRelays);
} catch (error) {
console.error("Publish error:", error);
}
Note: Events are automatically signed by the NDK worker using
the message-based signer (which calls window.nostr.signEvent).
================================================================ */
/* ================================================================
USER SETTINGS EXAMPLE (NIP-78)
================================================================
Read/subscribe/write helper pattern for all pages:
// Read latest merged settings (cache + relay hydrated by worker)
const settings = await getUserSettings();
// Subscribe to cross-tab updates
const unsubscribe = onUserSettings((nextSettings) => {
// Re-render page from nextSettings
});
// Patch only your feature namespace
await patchUserSettings({
myFeature: {
someFlag: true
}
});
// On page teardown (if applicable)
// unsubscribe();
================================================================ */
/* ================================================================
INITIALIZATION
================================================================
Main initialization sequence:
1. Initialize hamburger menu
2. Set up hamburger click handler
3. Resolve auth mode from URL/query policy
4. Initialize authentication based on mode
5. Initialize authenticated-only features (if signed in)
6. Set up version bar button listeners
7. Restore sidenav state
8. Update version display
Notes:
- required mode = existing behavior (prompt login on load)
- optional mode = allow public load, login later on demand
- none mode = no auto-login on load
================================================================ */
(async function main() {
console.log("[template.html] Starting initialization...");
try {
// Initialize hamburger menu first
initHamburgerMenu();
// Add click handler to hamburger
const divSvgHam = document.getElementById('divSvgHam');
if (divSvgHam) {
divSvgHam.addEventListener('click', toggleNav);
}
// Initialize version bar buttons
const themeToggleButton = document.getElementById('themeToggleButton');
const logoutButton = document.getElementById('logoutButton');
if (themeToggleButton) {
themeToggleButton.addEventListener('click', () => {
isDarkMode = !isDarkMode;
if (isDarkMode) {
localStorage.setItem('theme', 'dark');
} else {
localStorage.setItem('theme', 'light');
}
// Save sidenav state before reload
localStorage.setItem('sidenavWasOpen', isNavOpen ? 'true' : 'false');
window.location.reload();
});
}
if (logoutButton) {
logoutButton.addEventListener('click', async () => {
try {
// In optional/none modes this doubles as a "Sign in" entry point.
if (!isAuthenticated) {
await promptLoginIfNeeded();
return;
}
await Logout();
} catch (error) {
console.error('Logout/login action failed:', error);
}
});
}
// Resolve and initialize page auth policy.
authMode = resolveAuthModeFromUrl();
console.log('[template.html] Resolved auth mode:', authMode);
await initializeAuthentication(authMode);
// Initialize authenticated features only when signed in.
await initializeAuthenticatedPageFeatures();
/* ============================================================
EXAMPLE: Subscribe to events
============================================================
Uncomment to subscribe to user's notes:
const notesSub = subscribe(
{ kinds: [1], authors: [currentPubkey], limit: 10 },
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
);
console.log("[template.html] Subscribed to kind 1");
============================================================ */
/* ============================================================
EXAMPLE: Listen for events from worker
============================================================
Uncomment to handle incoming events:
window.addEventListener('ndkEvent', (event) => {
const evt = event.detail;
console.log("[template.html] Received event:", evt.kind, evt.pubkey);
if (evt.pubkey === currentPubkey) {
if (evt.kind === 1) {
// Handle note event
console.log("Note:", evt.content);
}
}
});
============================================================ */
/* ============================================================
EXAMPLE: Listen for cached profile
============================================================
Uncomment to handle cached profile data:
window.addEventListener('ndkProfile', (event) => {
console.log("[template.html] Cached profile:", event.detail);
// event.detail contains profile object (name, about, etc.)
});
============================================================ */
// Restore sidenav state if it was open before theme toggle
const sidenavWasOpen = localStorage.getItem('sidenavWasOpen');
if (sidenavWasOpen === 'true') {
localStorage.removeItem('sidenavWasOpen');
openNav();
}
// Optional UX note for public mode pages.
if (!isAuthenticated && (authMode === 'optional' || authMode === 'none')) {
divFooterCenter.textContent = 'Public mode';
divFooterRight.textContent = 'Sign in from side menu for private features';
}
// Update version display
await updateVersionDisplay();
console.log('[template.html] Initialization complete');
} catch (error) {
console.error('[template.html] Initialization failed:', error);
divBody.innerHTML = `<div style="text-align: center; padding: 50px;">
<div style="font-size: 24px; margin-bottom: 20px; color: red;">❌ Authentication Error</div>
<div style="font-size: 16px; color: #666;">${error.message}</div>
<div style="margin-top: 20px;">
<button onclick="location.reload()" style="padding: 10px 20px; font-size: 16px;">Retry</button>
</div>
</div>`;
}
})();
/* ================================================================
WORKER MESSAGE TYPES
================================================================
The NDK worker can send these message types:
1. 'response' - Response to init/subscribe/publish requests
- data.profile - User profile (from init)
- data.relays - User relays (from init)
- data.success - Publish success status
- data.relayResults - Relay publish results
2. 'event' - Nostr event from subscription
- Dispatched as 'ndkEvent' window event
- event.detail contains the Nostr event
3. 'eose' - End of stored events for subscription
- Dispatched as 'ndkEose' window event
- event.detail.subId contains subscription ID
4. 'signRequest' - Request to sign event/encrypt/decrypt
- Handled automatically by init-ndk.mjs
- Calls window.nostr methods and sends response
5. 'error' - Error from worker
- Logged to console automatically
6. 'relayActivity' - Relay read/write activity notification
- Dispatched as 'ndkRelayActivity' window event
- Used to animate relay status icons in footer
================================================================ */
/* ================================================================
DISTRIBUTED ARCHITECTURE NOTES
================================================================
Each page is independently accessible and self-contained:
1. Authentication persists via nostr-login-lite localStorage
- Login once on any page
- All other pages automatically authenticated
2. NDK SharedWorker is shared across all tabs/pages
- Single NDK instance manages all connections
- Subscriptions from all pages handled by one worker
- Events broadcast to all connected pages
3. Dexie cache is shared across all pages
- IndexedDB persists across sessions
- Cache-first queries are fast
- Reduces relay load
4. User settings are centralized and shared
- Worker hydrates kind 30078 (`d:user-settings`) on init
- Pages read via getUserSettings()
- Pages patch via patchUserSettings({ featureNamespace: ... })
- Pages subscribe via onUserSettings() for live updates
5. Each page can be distributed independently
- Copy template.html and customize
- No dependencies on other pages
- Works standalone or as part of suite
6. Message-based signer bridges worker and page
- Worker's NDK uses MessageBasedSigner
- Signer sends sign requests to page
- Page calls window.nostr.signEvent()
- Response sent back to worker
- NDK completes signing and publishing
7. Relay status visualization
- Footer left section shows connected relays
- Each relay has animated icon (HamburgerMorphing)
- Icons morph based on activity (read/write)
- Temporary animations show real-time activity
================================================================ */
</script>
</body>
</html>
+54 -13
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Build fully static MUSL binaries for C-Relay using Alpine Docker
# Build fully static MUSL binaries for C-Relay-PG using Alpine Docker
# Produces truly portable binaries with zero runtime dependencies
set -e
@@ -11,19 +11,52 @@ DOCKERFILE="$SCRIPT_DIR/Dockerfile.alpine-musl"
# Parse command line arguments
DEBUG_BUILD=false
if [[ "$1" == "--debug" ]]; then
DEBUG_BUILD=true
DB_BACKEND="${DB_BACKEND:-postgres}"
while [[ $# -gt 0 ]]; do
case "$1" in
--debug)
DEBUG_BUILD=true
shift
;;
--db-backend)
if [[ -z "$2" ]]; then
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
exit 1
fi
DB_BACKEND="$2"
shift 2
;;
--db-backend=*)
DB_BACKEND="${1#*=}"
shift
;;
*)
echo "ERROR: Unknown argument: $1"
echo "Usage: $0 [--debug] [--db-backend postgres|sqlite]"
exit 1
;;
esac
done
if [[ "$DB_BACKEND" != "sqlite" && "$DB_BACKEND" != "postgres" ]]; then
echo "ERROR: Invalid DB backend '$DB_BACKEND'. Use sqlite or postgres."
exit 1
fi
if [[ "$DEBUG_BUILD" == "true" ]]; then
echo "=========================================="
echo "C-Relay MUSL Static Binary Builder (DEBUG MODE)"
echo "C-Relay-PG MUSL Static Binary Builder (DEBUG MODE)"
echo "=========================================="
else
echo "=========================================="
echo "C-Relay MUSL Static Binary Builder (PRODUCTION MODE)"
echo "C-Relay-PG MUSL Static Binary Builder (PRODUCTION MODE)"
echo "=========================================="
fi
echo "Project directory: $SCRIPT_DIR"
echo "Build directory: $BUILD_DIR"
echo "Debug build: $DEBUG_BUILD"
echo "DB backend: $DB_BACKEND"
echo ""
# Create build directory
@@ -63,20 +96,25 @@ ARCH=$(uname -m)
case "$ARCH" in
x86_64)
PLATFORM="linux/amd64"
OUTPUT_NAME="c_relay_static_x86_64"
OUTPUT_NAME="c_relay_pg_static_x86_64"
;;
aarch64|arm64)
PLATFORM="linux/arm64"
OUTPUT_NAME="c_relay_static_arm64"
OUTPUT_NAME="c_relay_pg_static_arm64"
;;
*)
echo "WARNING: Unknown architecture: $ARCH"
echo "Defaulting to linux/amd64"
PLATFORM="linux/amd64"
OUTPUT_NAME="c_relay_static_${ARCH}"
OUTPUT_NAME="c_relay_pg_static_${ARCH}"
;;
esac
# Append _debug suffix to output name for debug builds so production binary is never overwritten
if [ "$DEBUG_BUILD" = true ]; then
OUTPUT_NAME="${OUTPUT_NAME}_debug"
fi
echo "Building for platform: $PLATFORM"
echo "Output binary: $OUTPUT_NAME"
echo ""
@@ -111,14 +149,15 @@ echo "=========================================="
echo "This will:"
echo " - Use Alpine Linux (native MUSL)"
echo " - Build all dependencies statically"
echo " - Compile c-relay with full static linking"
echo " - Compile c-relay-pg with full static linking"
echo ""
$DOCKER_CMD build \
--platform "$PLATFORM" \
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
--build-arg DB_BACKEND=$DB_BACKEND \
-f "$DOCKERFILE" \
-t c-relay-musl-builder:latest \
-t c-relay-pg-musl-builder:latest \
--progress=plain \
. || {
echo ""
@@ -140,16 +179,17 @@ echo "=========================================="
$DOCKER_CMD build \
--platform "$PLATFORM" \
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
--build-arg DB_BACKEND=$DB_BACKEND \
--target builder \
-f "$DOCKERFILE" \
-t c-relay-static-builder-stage:latest \
-t c-relay-pg-static-builder-stage:latest \
. > /dev/null 2>&1
# Create a temporary container to copy the binary
CONTAINER_ID=$($DOCKER_CMD create c-relay-static-builder-stage:latest)
CONTAINER_ID=$($DOCKER_CMD create c-relay-pg-static-builder-stage:latest)
# Copy binary from container
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_static" "$BUILD_DIR/$OUTPUT_NAME" || {
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_pg_static" "$BUILD_DIR/$OUTPUT_NAME" || {
echo "ERROR: Failed to extract binary from container"
$DOCKER_CMD rm "$CONTAINER_ID" 2>/dev/null
exit 1
@@ -219,6 +259,7 @@ if [ "$DEBUG_BUILD" = true ]; then
else
echo "Build Type: PRODUCTION (optimized, stripped)"
fi
echo "DB Backend: $DB_BACKEND"
if [ "$TRULY_STATIC" = true ]; then
echo "Linkage: Fully static binary (Alpine MUSL-based)"
echo "Portability: Works on ANY Linux distribution"
Binary file not shown.
+31
View File
@@ -0,0 +1,31 @@
# Database files (from c-relay running in this directory)
*.db
*.db-shm
*.db-wal
# Build artifacts
*.o
*.a
# Binary (built into root, not version-controlled)
caching_relay
caching_relay_static_*
# Logs
*.log
crelay.log
# Test configs (not for version control)
test_config*.jsonc
# Note: caching_relay_config.jsonc IS version-controlled (it's the default
# config + state store). Don't ignore it.
# Docker build context temp (copied nostr_core_lib during static build)
nostr_core_lib/
# Editor
*.swp
*.swo
*~
.vscode/
+125
View File
@@ -0,0 +1,125 @@
# Alpine-based MUSL static binary builder for caching_relay
# Produces a truly portable binary with zero runtime dependencies.
# Adapted from c-relay's Dockerfile.alpine-musl.
ARG DEBUG_BUILD=false
FROM alpine:3.19 AS builder
ARG DEBUG_BUILD=false
# Install build dependencies
RUN apk add --no-cache \
build-base \
musl-dev \
git \
cmake \
pkgconfig \
autoconf \
automake \
libtool \
openssl-dev \
openssl-libs-static \
zlib-dev \
zlib-static \
curl-dev \
curl-static \
sqlite-dev \
linux-headers \
wget \
bash
WORKDIR /build
# Build libsecp256k1 static
RUN cd /tmp && \
git clone https://github.com/bitcoin-core/secp256k1.git && \
cd secp256k1 && \
./autogen.sh && \
./configure --enable-static --disable-shared --prefix=/usr \
CFLAGS="-fPIC" && \
make -j$(nproc) && \
make install && \
rm -rf /tmp/secp256k1
# Build libwebsockets static with minimal features
RUN cd /tmp && \
git clone --depth 1 --branch v4.3.3 https://github.com/warmcat/libwebsockets.git && \
cd libwebsockets && \
mkdir build && cd build && \
cmake .. \
-DLWS_WITH_STATIC=ON \
-DLWS_WITH_SHARED=OFF \
-DLWS_WITH_SSL=ON \
-DLWS_WITHOUT_TESTAPPS=ON \
-DLWS_WITHOUT_TEST_SERVER=ON \
-DLWS_WITHOUT_TEST_CLIENT=ON \
-DLWS_WITHOUT_TEST_PING=ON \
-DLWS_WITH_HTTP2=OFF \
-DLWS_WITH_LIBUV=OFF \
-DLWS_WITH_LIBEVENT=OFF \
-DLWS_IPV6=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_C_FLAGS="-fPIC" && \
make -j$(nproc) && \
make install && \
rm -rf /tmp/libwebsockets
# Copy nostr_core_lib source (sibling project)
COPY nostr_core_lib /build/nostr_core_lib/
# Build nostr_core_lib with the NIPs needed by the relay pool + signer:
# 1 (basic), 6 (keys), 19 (npub bech32), 4 (legacy encryption, signer dep),
# 42 (auth, relay pool dep), 44 (modern encryption, signer dep)
RUN cd nostr_core_lib && \
chmod +x build.sh && \
sed -i 's/CFLAGS="-Wall -Wextra -std=c99 -fPIC -O2"/CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"/' build.sh && \
rm -f *.o *.a 2>/dev/null || true && \
./build.sh --nips=1,4,6,19,42,44 && \
if [ -f libnostr_core_arm64.a ]; then \
cp libnostr_core_arm64.a libnostr_core.a; \
elif [ -f libnostr_core_x64.a ]; then \
cp libnostr_core_x64.a libnostr_core.a; \
else \
echo "ERROR: No supported nostr_core static library produced"; \
ls -la *.a 2>/dev/null || true; \
exit 1; \
fi
# Copy caching_relay source LAST (only this layer rebuilds on source changes)
COPY src/ /build/src/
# Build caching_relay with full static linking.
# No sqlite (the daemon doesn't use it), no c_utils (not needed).
RUN if [ "$DEBUG_BUILD" = "true" ]; then \
CFLAGS="-g -O2 -DDEBUG -fno-omit-frame-pointer"; \
STRIP_CMD="echo 'Keeping debug symbols'"; \
else \
CFLAGS="-O2"; \
STRIP_CMD="strip /build/caching_relay_static"; \
fi && \
gcc -static $CFLAGS -Wall -Wextra -std=c99 \
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
-I. -Isrc -Inostr_core_lib -Inostr_core_lib/nostr_core \
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket \
src/main.c src/debug.c src/jsonc_strip.c src/config.c src/state.c \
src/follow_graph.c src/relay_sink.c src/live_subscriber.c \
src/backfill.c src/relay_discovery.c \
-o /build/caching_relay_static \
nostr_core_lib/libnostr_core.a \
-lwebsockets -lssl -lcrypto -lsecp256k1 \
-lcurl -lz -lpthread -lm -ldl && \
eval "$STRIP_CMD"
# Verify it's truly static
RUN echo "=== Binary Information ===" && \
file /build/caching_relay_static && \
ls -lh /build/caching_relay_static && \
echo "=== Checking for dynamic dependencies ===" && \
(ldd /build/caching_relay_static 2>&1 || echo "Binary is static") && \
echo "=== Build complete ==="
# Output stage - just the binary
FROM scratch AS output
COPY --from=builder /build/caching_relay_static /caching_relay_static
+67
View File
@@ -0,0 +1,67 @@
# caching_relay Makefile - statically linked C99 binary, c-relay style
# Use bash for reliable glob expansion in recipes (dash handles globs differently).
SHELL := /bin/bash
CC = gcc
CFLAGS = -Wall -Wextra -std=c99 -g -O2
# nostr_core_lib is a sibling of c-relay-pg (one dir up from caching/).
NOSTR_CORE_DIR = ../nostr_core_lib
INCLUDES = -I. -Isrc -I$(NOSTR_CORE_DIR) -I$(NOSTR_CORE_DIR)/nostr_core \
-I$(NOSTR_CORE_DIR)/cjson -I$(NOSTR_CORE_DIR)/nostr_websocket \
-I/usr/include/postgresql
# -lsqlite3: nostr_core_lib's request_validator.x64.o references SQLite symbols;
# we use --whole-archive so the linker pulls in all objects (needed for NIP-42
# cross-references within the archive), which means SQLite must be satisfied.
# No c_utils_lib: nostr_core_lib does not depend on it for the NIPs we use.
# -lpq: PostgreSQL client library for the caching_event_inbox integration.
LIBS = -lwebsockets -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm -lpq -lsqlite3
MAIN_SRC = src/main.c src/debug.c src/jsonc_strip.c src/config.c src/state.c \
src/follow_graph.c src/relay_sink.c src/live_subscriber.c \
src/backfill.c src/relay_discovery.c \
src/pg_inbox.c src/pg_config.c
# Architecture detection
ARCH = $(shell uname -m)
ifeq ($(ARCH),x86_64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
else ifeq ($(ARCH),aarch64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
else ifeq ($(ARCH),arm64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
else
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
endif
# Binary goes in the c-relay-pg build/ directory so the launcher can find it.
TARGET = ../build/caching_relay
all: $(TARGET)
# Build nostr_core_lib with the NIPs needed by the relay pool + signer:
# 1 (basic), 6 (keys), 19 (npub bech32), 4 (legacy encryption, signer dep),
# 42 (auth, relay pool dep), 44 (modern encryption, signer dep)
$(NOSTR_CORE_LIB):
@echo "Building nostr_core_lib with required NIPs..."
cd $(NOSTR_CORE_DIR) && ./build.sh --nips=1,4,6,19,42,44
$(TARGET): $(MAIN_SRC) $(NOSTR_CORE_LIB)
@echo "Compiling caching_relay for architecture: $(ARCH)"
@# Extract all objects from the static library and link them directly.
@# This avoids archive symbol resolution ordering issues (NIP-42 cross-refs).
@# Use a fixed temp dir and chain all commands with && so failures stop the build.
@rm -rf /tmp/cr_lib && mkdir -p /tmp/cr_lib && \
ar x $(NOSTR_CORE_LIB) --output=/tmp/cr_lib && \
echo "Extracted $$(ls /tmp/cr_lib/*.o | wc -l) objects" && \
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) /tmp/cr_lib/*.o -o $(TARGET) $(LIBS) && \
rm -rf /tmp/cr_lib && \
echo "Build complete: $(TARGET)"
clean:
rm -f $(TARGET)
.PHONY: all clean
+297
View File
@@ -0,0 +1,297 @@
# caching_relay
A C99 daemon that caches Nostr events from people you follow into a local relay,
so your Nostr client can point at a single fast local relay instead of fanning
out to dozens of upstream relays.
## What it does
1. Reads a `.jsonc` config file listing your **root npub(s)**, upstream relays,
a local relay URL, and the event kinds to cache.
2. For each root npub, fetches its kind-3 contact list to discover **followed
pubkeys** (your follows + their follows).
3. **Live-subscribes** to new events of the configured kinds from the union of
followed pubkeys.
4. **Backfills** historical events using a progressive window-expansion strategy
(24h → 7d → 30d → 90d → 365d), round-robin per pubkey, throttled to be polite
to upstream relays.
5. **Re-publishes** every fetched event to the local relay via a plain WebSocket
`EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg,
or any Nostr relay).
6. Persists its backfill progress **in the config file itself** so a restart
resumes where it left off.
## Build
### Static binary (recommended, c-relay style)
Produces a truly portable statically-linked MUSL binary with zero runtime
dependencies:
```bash
./build_static.sh
# or with debug symbols:
./build_static.sh --debug
# or cross-compile for arm64:
./build_static.sh --arch arm64
```
Output: `caching_relay` (in the project root).
Requires Docker. The build runs in an Alpine container that compiles
libsecp256k1, libwebsockets, and nostr_core_lib from source, then statically
links everything.
### Local build (if you have the shared libs installed)
```bash
make
```
Output: `caching_relay` (in the project root).
Requires: `libwebsockets`, `openssl`, `libsecp256k1`, `libcurl`, `zlib` shared
libraries, and a pre-built `../nostr_core_lib/libnostr_core_x64.a`.
## Usage
```bash
./caching_relay -d 3
```
The daemon looks for `caching_relay_config.jsonc` in the current directory by default.
You can override with `-c`:
```bash
./caching_relay -c /path/to/my_config.jsonc -d 3
```
Options:
- `-c, --config <file>` - Path to `.jsonc` config file (default: `./caching_relay_config.jsonc`)
- `-d, --debug <level>` - Log level 1-4 (1=error, 2=warn, 3=info, 4=debug). Default 3.
- `-h, --help` - Show help
Signals:
- `SIGINT` / `SIGTERM` - Graceful shutdown (saves state, closes connections)
- `SIGHUP` - Reload config (preserves backfill state)
## Config file
The config file [`caching_relay_config.jsonc`](caching_relay_config.jsonc) lives in the
project root. It is JSONC (JSON with `//` and `/* */` comments). The daemon
rewrites it as plain JSON when saving state (comments are not preserved on
rewrite).
Key fields:
| Field | Description |
|-------|-------------|
| `root_npubs` | npubs whose kind-3 follows list we crawl |
| `upstream_relays` | bootstrap relays for initial discovery (kind-3, kind-10002). Outbox relays are discovered dynamically via NIP-65. |
| `local_relay` | relay to publish cached events into (never queried) |
| `kinds` | event kinds to cache for followed people (e.g. `[1, 3, 6, 10000, 30023]`) |
| `admin_kinds` | kinds to follow specifically for root (admin) npubs. Use `["*"]` for all kinds. If omitted, admin uses same `kinds` as everyone else. |
| `backfill.window_schedule_seconds` | progressive window sizes in seconds |
| `backfill.events_per_tick` | max events per pubkey per backfill tick |
| `backfill.tick_interval_seconds` | delay between pubkey backfills (throttle) |
| `live.enabled` | enable live subscription |
| `follow_graph_refresh_seconds` | how often to re-resolve the follow graph |
| `state.*` | managed by the daemon - do not hand-edit |
## Architecture
See [`plans/plan.md`](plans/plan.md) for the full architecture document with
Mermaid diagrams.
The daemon uses a **NIP-65 outbox model**: instead of querying a fixed set of
bootstrap relays for everything, it discovers which relays each followed pubkey
actually posts to (via kind 10002 relay lists) and connects to those relays
dynamically. The `upstream_relays` in the config serve only as **bootstrap
relays** -- the initial set used to discover kind-3 and kind-10002 events before
the outbox relay map is built.
### Relay pool selection: minimum covering set
After discovering each followed pubkey's outbox relays (from their kind 10002),
the daemon computes the **minimum set of relays that covers all followed
pubkeys**. This is the classic set cover problem, solved with a greedy
approximation:
1. Build a map: `{relay_url -> set of pubkeys that list it in their 10002}`
2. Greedily pick the relay that covers the most uncovered pubkeys
3. Repeat until all pubkeys are covered (or no more relays to pick)
4. Always include the bootstrap relays in the final set (they may have events
from pubkeys that don't publish a kind 10002)
This minimizes the number of WebSocket connections while ensuring every
followed pubkey is reachable. The daemon logs the selected relay set and which
pubkeys each relay covers.
Pubkeys that have no kind 10002 (or whose 10002 lists no relays) are covered by
the bootstrap relays as a fallback.
```
┌─────────────────────────────────────────┐
│ caching_relay │
│ │
upstream relays │ upstream_pool sink_pool │ local relay
(damus, nos.lol) │ (query + subscribe) (publish only) │ (c-relay)
│ │ │ │ │ │
│ │ follow_graph relay_sink │ │
└──────────►│ live_subscriber ──────►│ ├─────►│
│ backfill ──────────────►│ │ │
│ │ │ │ │
│ state + seen ring │ │ │
└──────────────────────────────────────────┘ │
```
Two `nostr_relay_pool_t` instances:
- **upstream_pool** - holds bootstrap relays initially, then dynamically adds
outbox relays discovered from kind 10002. Used for `query_sync` (kind-3 fetch,
kind-10002 fetch, backfill) and the long-lived live subscription.
- **sink_pool** - holds only the local relay; used exclusively for
`publish_async`. Never queried.
## Startup Flow
### First-time startup (empty local relay)
The daemon has never run before. The local relay has no cached events. The
daemon must bootstrap from the config's `upstream_relays` to discover the
outbox relays for each followed pubkey.
```
START
|
v
Load config (caching_relay_config.jsonc)
| - root_npubs, upstream_relays (bootstrap), kinds, admin_kinds
| - state.backfilled_until == 0 => first-time startup
|
v
Create upstream_pool with bootstrap relays only
Create sink_pool with local_relay
|
v
Phase 1: Resolve follow graph (from bootstrap relays)
| - For each root npub: query_sync kind=3 from bootstrap relays
| - Parse "p" tags => followed pubkey set
| - Log: "follow: resolved N followed pubkeys"
|
v
Phase 2: Discover outbox relays (NIP-65, kind 10002) from bootstrap relays
| - For each followed pubkey: query_sync kind=10002 from bootstrap relays
| - Parse "r" tags => per-pubkey relay list
| - Publish all kind-10002 events to local relay (cache them)
| - Dynamically add discovered relays to upstream_pool
| - Log: "relay_discovery: found N relays from M pubkeys"
| - Log: "upstream_pool: now connected to N relays" + list them
|
v
Phase 3: Open live subscriptions on upstream_pool
| - follows_sub: non-admin pubkeys + regular kinds
| - admin_sub: admin npubs + admin_kinds (or all kinds)
|
v
Phase 4: Begin progressive backfill
| - Window 0: 24h -> query each pubkey from their outbox relays
| - Window 1: 7d -> ...
| - Window 2: 30d -> ...
| - Publish all fetched events to local relay
| - Save state to config file as each window completes
|
v
Steady-state: live sub + periodic follow-graph refresh + periodic
kind-10002 refresh + backfill re-cycle
|
v
SHUTDOWN (SIGINT/SIGTERM) -> save state -> clean exit
```
### Subsequent startup (local relay already has cached events)
The daemon has run before. The local relay already has kind-3 and kind-10002
events cached from the previous run. The daemon can read these from the local
relay directly (fast, no network round-trip to bootstrap relays) and only falls
back to bootstrap relays for pubkeys it cannot find locally.
```
START
|
v
Load config (caching_relay_config.jsonc)
| - state.backfilled_until > 0 => subsequent startup
| - state.current_window_index, backfill_cursor preserved
|
v
Create upstream_pool with bootstrap relays
Create sink_pool with local_relay
|
v
Phase 1: Resolve follow graph (from LOCAL relay first, bootstrap fallback)
| - Query local relay for kind=3 per root npub
| - If found locally: use it (fast, no upstream query)
| - If not found: fall back to bootstrap relays
| - Parse "p" tags => followed pubkey set
| - Log: "follow: resolved N followed pubkeys (L local, B bootstrap)"
|
v
Phase 2: Discover outbox relays (from LOCAL relay first, bootstrap fallback)
| - Query local relay for kind=10002 per followed pubkey
| - If found locally: use it (fast)
| - If not found: fall back to bootstrap relays, cache result to local
| - Parse "r" tags => per-pubkey relay list
| - Dynamically add discovered relays to upstream_pool
| - Log: "relay_discovery: found N relays (L local, B bootstrap)"
| - Log: "upstream_pool: now connected to N relays" + list them
|
v
Phase 3: Open live subscriptions on upstream_pool
| - (same as first-time)
|
v
Phase 4: Resume backfill from saved state
| - Resume at state.current_window_index, state.backfill_cursor
| - No re-pull of already-backfilled windows
| - Continue progressive window expansion from where it left off
|
v
Steady-state (same as first-time)
|
v
SHUTDOWN -> save state -> clean exit
```
### Relay logging
The daemon logs all relay activity so you can see exactly which relays are
being used:
- `upstream: added wss://relay.damus.io` -- each relay added to the pool
- `relay_discovery: found 47 relays from 195 pubkeys` -- outbox discovery summary
- `upstream_pool: 50 relays connected:` -- full relay list at startup
- `relay_discovery: pubkey X -> wss://relay.example.com (local)` -- per-pubkey
relay source (local cache vs bootstrap)
- `backfill: pubkey[3/195] abc123 -> wss://relay.example.com (12 events)` --
which relay served each backfill query
## Dependencies
- [nostr_core_lib](../nostr_core_lib) - built with NIPs 1, 4, 6, 19, 42, 44
- libwebsockets, openssl, libsecp256k1, libcurl, zlib (all statically linked
in the Docker build)
## Testing
Start a local c-relay, then run the daemon:
```bash
# Start local relay (from c-relay project)
cd ../c-relay/build && ./c_relay_static_x86_64 -p 8888 &
# Run the caching daemon (uses ./caching_relay_config.jsonc by default)
./caching_relay -d 3
# Verify events are cached (from another terminal)
nak req -k 1 -l 10 ws://127.0.0.1:8888
```
+1
View File
@@ -0,0 +1 @@
0.0.2
+148
View File
@@ -0,0 +1,148 @@
#!/bin/bash
# Build fully static MUSL binary for caching_relay using Alpine Docker.
# Produces a truly portable binary with zero runtime dependencies.
# Adapted from c-relay's build_static.sh.
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DOCKERFILE="$SCRIPT_DIR/Dockerfile.alpine-musl"
# Parse command line arguments
DEBUG_BUILD=false
TARGET_ARCH=""
while [[ $# -gt 0 ]]; do
case "$1" in
--debug)
DEBUG_BUILD=true
shift
;;
--arch)
if [[ -z "$2" ]]; then
echo "ERROR: --arch requires a value"
echo "Usage: $0 [--debug] [--arch <arm64|x86_64>]"
exit 1
fi
case "$2" in
arm64|x86_64)
TARGET_ARCH="$2"
;;
*)
echo "ERROR: Unsupported architecture '$2'"
echo "Supported values: arm64, x86_64"
exit 1
;;
esac
shift 2
;;
*)
echo "ERROR: Unknown argument '$1'"
echo "Usage: $0 [--debug] [--arch <arm64|x86_64>]"
exit 1
;;
esac
done
if [ "$DEBUG_BUILD" = true ]; then
echo "=========================================="
echo "caching_relay MUSL Static Binary Builder (DEBUG MODE)"
echo "=========================================="
else
echo "=========================================="
echo "caching_relay MUSL Static Binary Builder (PRODUCTION MODE)"
echo "=========================================="
fi
echo "Project directory: $SCRIPT_DIR"
echo ""
if ! command -v docker &> /dev/null; then
echo "ERROR: Docker is not installed or not in PATH"
exit 1
fi
if ! docker info &> /dev/null; then
echo "ERROR: Docker daemon is not running or user not in docker group"
exit 1
fi
echo "Docker is available and running"
echo ""
# Detect host architecture
ARCH=$(uname -m)
case "$ARCH" in
x86_64) ARCH="x86_64";;
aarch64) ARCH="arm64";;
arm64) ARCH="arm64";;
esac
if [[ -n "$TARGET_ARCH" ]]; then
ARCH="$TARGET_ARCH"
echo "Using target architecture: $ARCH"
else
echo "Detected host architecture: $ARCH"
fi
echo ""
# nostr_core_lib is a sibling directory; copy it into the build context.
# Docker cannot follow symlinks outside the context, so we do a real copy
# (excluding .git and build artifacts) and clean up afterwards.
NOSTR_CORE_DIR="$SCRIPT_DIR/../nostr_core_lib"
if [ ! -d "$NOSTR_CORE_DIR" ]; then
echo "ERROR: nostr_core_lib not found at $NOSTR_CORE_DIR"
exit 1
fi
NEEDS_COPY=1
if [ -d "$SCRIPT_DIR/nostr_core_lib" ] && [ ! -L "$SCRIPT_DIR/nostr_core_lib" ]; then
# Already a real directory (e.g. from a previous run); assume it's good.
NEEDS_COPY=0
fi
if [ "$NEEDS_COPY" = "1" ]; then
echo "Copying nostr_core_lib into build context..."
rm -rf "$SCRIPT_DIR/nostr_core_lib"
mkdir -p "$SCRIPT_DIR/nostr_core_lib"
rsync -a --exclude='.git' --exclude='*.a' --exclude='*.o' \
--exclude='.venv*' --exclude='backups' --exclude='verify_*' \
--exclude='rewrite_mirror' --exclude='websocket_debug' \
"$NOSTR_CORE_DIR/" "$SCRIPT_DIR/nostr_core_lib/"
COPIED_NOSTR_CORE=1
fi
# Build args
BUILD_ARGS="--build-arg DEBUG_BUILD=$DEBUG_BUILD"
if [ "$ARCH" = "arm64" ] && [ "$(uname -m)" != "aarch64" ] && [ "$(uname -m)" != "arm64" ]; then
BUILD_ARGS="$BUILD_ARGS --platform linux/arm64"
fi
IMAGE_TAG="caching_relay_builder:latest"
echo "Building Docker image..."
docker build $BUILD_ARGS \
-f "$DOCKERFILE" \
-t "$IMAGE_TAG" \
"$SCRIPT_DIR" 2>&1
BUILD_RC=$?
# Clean up the copied nostr_core_lib to keep the project dir tidy.
if [ "${COPIED_NOSTR_CORE:-0}" = "1" ]; then
echo "Cleaning up copied nostr_core_lib from build context..."
rm -rf "$SCRIPT_DIR/nostr_core_lib"
fi
if [ $BUILD_RC -ne 0 ]; then
exit $BUILD_RC
fi
echo ""
echo "Extracting binary from image..."
# The output stage is FROM scratch with no CMD, so pass an empty command.
CONTAINER_ID=$(docker create "$IMAGE_TAG" "")
docker cp "$CONTAINER_ID:/caching_relay_static" "$SCRIPT_DIR/caching_relay"
docker rm "$CONTAINER_ID" >/dev/null
chmod +x "$SCRIPT_DIR/caching_relay"
echo ""
echo "=== Build complete ==="
ls -lh "$SCRIPT_DIR/caching_relay"
file "$SCRIPT_DIR/caching_relay"
+432
View File
@@ -0,0 +1,432 @@
/*
* caching_relay - progressive window-expansion backfill
*
* Uses until-based pagination so authors with more events than the page size
* are fully drained across multiple ticks instead of being skipped. In
* PostgreSQL mode, per-author/window progress is persisted to the
* caching_backfill_progress table so the service can resume after restart.
*/
#define _GNU_SOURCE
#include "backfill.h"
#include "follow_graph.h"
#include "pg_inbox.h"
#include "debug.h"
#include "../nostr_core_lib/cjson/cJSON.h"
#include <string.h>
#include <stdlib.h>
#include <time.h>
/* ------------------------------------------------------------------ */
/* Helpers */
/* ------------------------------------------------------------------ */
/* Returns 1 if the PostgreSQL inbox is initialized (PG mode active).
* Probes the config table; a non-NULL result means the connection is up. */
static int pg_mode_active(void) {
char *v = pg_inbox_get_config_value("caching_root_npubs");
if (v) { free(v); return 1; }
char *v2 = pg_inbox_get_config_value("caching_kinds");
if (v2) { free(v2); return 1; }
return 0;
}
/* Find the oldest (minimum) created_at among an array of event JSON objects.
* Returns 1 and sets *out_oldest, or 0 if no events / no valid created_at. */
static int find_oldest_created_at(cJSON **events, int count, long *out_oldest) {
long oldest = 0;
int found = 0;
for (int i = 0; i < count; i++) {
cJSON *ca = cJSON_GetObjectItem(events[i], "created_at");
if (!ca || !cJSON_IsNumber(ca)) continue;
long ts = (long)ca->valuedouble;
if (!found || ts < oldest) {
oldest = ts;
found = 1;
}
}
if (found && out_oldest) *out_oldest = oldest;
return found;
}
/* Count how many events in the array have created_at == ts. */
static int count_at_timestamp(cJSON **events, int count, long ts) {
int n = 0;
for (int i = 0; i < count; i++) {
cJSON *ca = cJSON_GetObjectItem(events[i], "created_at");
if (ca && cJSON_IsNumber(ca) && (long)ca->valuedouble == ts) n++;
}
return n;
}
/* Build the kinds array for a given pubkey (admin vs regular).
* Returns NULL if no kinds filter should be applied (admin_all_kinds). */
static cJSON *build_kinds(cr_config_t *cfg, const char *pk) {
int is_admin = cr_follow_is_root(cfg, pk);
if (is_admin && cfg->admin_all_kinds) {
return NULL;
}
cJSON *kinds = cJSON_CreateArray();
if (is_admin && cfg->admin_kind_count > 0) {
for (int i = 0; i < cfg->admin_kind_count; i++)
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
} else {
for (int i = 0; i < cfg->kind_count; i++)
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
}
return kinds;
}
/* Resolve the relay URL list to query for a given pubkey. Sets *out_urls and
* *out_n. Caller frees *out_urls (but not the strings, which alias internal
* storage). Returns 0 on success, -1 if no relays available. */
static int resolve_relays(nostr_relay_pool_t *upstream,
const cr_relay_map_t *relay_map,
const char *pk,
const char ***out_urls,
int *out_n) {
*out_urls = NULL;
*out_n = 0;
if (relay_map) {
const cr_outbox_entry_t *oe = cr_relay_map_get_outbox(relay_map, pk);
if (oe && oe->relay_count > 0) {
const char **urls = malloc(oe->relay_count * sizeof(char *));
if (!urls) return -1;
for (int j = 0; j < oe->relay_count; j++) urls[j] = oe->relays[j];
*out_urls = urls;
*out_n = oe->relay_count;
return 0;
}
}
/* Fallback: all upstream relays. */
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
if (n > 0) {
const char **urls = malloc(n * sizeof(char *));
if (!urls) { free(listed); free(statuses); return -1; }
for (int j = 0; j < n; j++) urls[j] = listed[j];
*out_urls = urls;
*out_n = n;
}
free(listed);
free(statuses);
return 0;
}
/* Issue a query_sync for one author with the given since/until/limit.
* Returns the events array (caller frees each event + the array) or NULL.
* Sets *out_count. */
static cJSON **query_author(nostr_relay_pool_t *upstream,
const cr_relay_map_t *relay_map,
cr_config_t *cfg, const char *pk,
long since, long until, int limit,
int *out_count) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(pk));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = build_kinds(cfg, pk);
if (kinds) cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)since));
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)until));
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)limit));
const char **urls = NULL;
int n = 0;
resolve_relays(upstream, relay_map, pk, &urls, &n);
cJSON **events = nostr_relay_pool_query_sync(upstream, urls, n, filter,
out_count, 15000);
free(urls);
cJSON_Delete(filter);
return events;
}
/* ------------------------------------------------------------------ */
/* Window advancement */
/* ------------------------------------------------------------------ */
static void advance_window(cr_backfill_t *bf, cr_config_t *cfg) {
/* Record that we've backfilled up to (now - current_window). */
cfg->state.backfilled_until = (long)time(NULL) - bf->current_window_s;
cfg->state.current_window_index = bf->window_index + 1;
cfg->state.backfill_cursor = 0;
if (!pg_mode_active()) {
cr_config_save_state(cfg);
}
DEBUG_INFO("backfill: window %d complete (%ld events). backfilled_until=%ld",
bf->window_index, bf->events_this_window, cfg->state.backfilled_until);
bf->window_index++;
if (bf->window_index >= cfg->backfill.window_count) {
DEBUG_INFO("backfill: all windows complete, entering steady-state");
bf->in_progress = 0;
return;
}
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
bf->cursor = 0;
bf->until_cursor = 0;
bf->author_complete = 0;
bf->blocked = 0;
bf->blocked_until_ts = 0;
bf->events_this_window = 0;
bf->window_started = time(NULL);
DEBUG_INFO("backfill: advancing to window %d (%ld seconds)",
bf->window_index, bf->current_window_s);
}
/* Persist the current cursor state. In PG mode we rely on the per-author
* progress table; in legacy mode we write the .jsonc state. */
static void persist_cursor(cr_backfill_t *bf, cr_config_t *cfg) {
cfg->state.backfill_cursor = bf->cursor;
if (!pg_mode_active()) {
cr_config_save_state(cfg);
}
}
/* Save progress for the current author to PG (no-op outside PG mode). */
static void pg_save_current(cr_backfill_t *bf, cr_config_t *cfg,
const char *pk, int complete) {
(void)cfg; /* window anchor is derived from bf + time(), cfg not needed */
if (!pg_mode_active() || !pk) return;
long anchor = (long)time(NULL) - bf->current_window_s;
pg_inbox_save_backfill_progress(pk, bf->window_index, anchor,
bf->until_cursor, complete);
}
/* ------------------------------------------------------------------ */
/* Init */
/* ------------------------------------------------------------------ */
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg) {
memset(bf, 0, sizeof(*bf));
bf->window_index = cfg->state.current_window_index;
if (bf->window_index >= cfg->backfill.window_count) {
/* Already done; steady state. */
bf->in_progress = 0;
DEBUG_INFO("backfill: already complete (window index %d >= %d), steady-state",
bf->window_index, cfg->backfill.window_count);
return;
}
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
bf->cursor = cfg->state.backfill_cursor;
bf->until_cursor = 0;
bf->author_complete = 0;
bf->blocked = 0;
bf->blocked_until_ts = 0;
bf->in_progress = 1;
bf->window_started = time(NULL);
/* In PG mode, per-author restore is deferred to the first tick (the
* followed set is not resolved yet at init time). The tick loop skips
* already-complete authors via the progress table. */
DEBUG_INFO("backfill: starting at window %d (%ld seconds), cursor %d",
bf->window_index, bf->current_window_s, bf->cursor);
}
/* ------------------------------------------------------------------ */
/* Tick */
/* ------------------------------------------------------------------ */
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
cr_sink_t *sink, const cr_relay_map_t *relay_map) {
if (!cfg->backfill.enabled) return -2;
if (!bf->in_progress) return -2;
if (followed->count == 0) return 0;
/* Throttle: only one query per tick_interval. */
time_t now = time(NULL);
if (bf->last_tick && (now - bf->last_tick) < cfg->backfill.tick_interval_seconds) {
return 0;
}
/* In PG mode, skip past authors already marked complete for this window
* (resume case). Restore the until_cursor for the first incomplete author. */
if (pg_mode_active()) {
while (bf->cursor < followed->count) {
const char *pk = followed->items[bf->cursor];
long uc = 0;
int complete = 0, found = 0;
if (pg_inbox_load_backfill_progress(pk, bf->window_index,
&uc, &complete, &found) == 0
&& found && complete) {
DEBUG_TRACE("backfill: author[%d] %s already complete, skipping",
bf->cursor, pk);
bf->cursor++;
cfg->state.backfill_cursor = bf->cursor;
continue;
}
if (found && uc > 0 && bf->until_cursor == 0) {
bf->until_cursor = uc;
DEBUG_TRACE("backfill: resumed author[%d] %s at until=%ld",
bf->cursor, pk, bf->until_cursor);
}
break;
}
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
return 1;
}
}
const char *pk = followed->items[bf->cursor];
/* Determine the until cursor for this query. First query for an author:
* until = now. Subsequent queries: until = oldest event created_at - 1
* from the previous page (set at the end of the previous tick). */
long until = bf->until_cursor;
if (until == 0) {
until = (long)now;
}
long since = (long)now - bf->current_window_s;
int page_size = cfg->backfill.events_per_tick;
if (page_size < 1) page_size = 50;
int max_cap = CR_BACKFILL_MAX_PAGE_CAP;
if (max_cap < page_size) max_cap = page_size;
bf->last_tick = now;
DEBUG_TRACE("backfill: querying %s (window %ld, since %ld, until %ld, limit %d)",
pk, bf->current_window_s, since, until, page_size);
int ev_count = 0;
cJSON **events = query_author(upstream, relay_map, cfg, pk,
since, until, page_size, &ev_count);
/* Saturation handling: if we got exactly page_size events, the page may
* be truncated. Bump the limit up to max_cap and retry once. */
int limit_used = page_size;
if (events && ev_count == page_size && page_size < max_cap) {
for (int k = 0; k < ev_count; k++) cJSON_Delete(events[k]);
free(events);
DEBUG_TRACE("backfill: saturated page for %s, retrying with limit %d",
pk, max_cap);
events = query_author(upstream, relay_map, cfg, pk,
since, until, max_cap, &ev_count);
limit_used = max_cap;
}
/* If still saturated at the cap, we cannot safely paginate past the
* oldest timestamp (the relay may have more events at that exact ts).
* Mark the author blocked at that timestamp and move on; it will be
* retried in a later window sweep. */
if (events && ev_count == max_cap && max_cap > page_size) {
long oldest = 0;
find_oldest_created_at(events, ev_count, &oldest);
int at_oldest = count_at_timestamp(events, ev_count, oldest);
DEBUG_WARN("backfill: author %s saturated at limit %d (oldest ts %ld, "
"%d events at that ts) - marking blocked, will retry next window",
pk, max_cap, oldest, at_oldest);
/* Publish what we have. */
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
for (int k = 0; k < ev_count; k++) {
cr_sink_publish(sink, events[k]);
cJSON_Delete(events[k]);
bf->events_this_window++;
bf->events_total++;
}
free(events);
bf->blocked = 1;
bf->blocked_until_ts = oldest;
bf->until_cursor = oldest; /* resume point for next window */
pg_save_current(bf, cfg, pk, 0); /* not complete - retry next window */
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
return 1;
}
/* Normal page processing. Capture the oldest timestamp BEFORE publishing
* so we can paginate if saturated. */
long oldest_ts = 0;
int have_oldest = 0;
if (events && ev_count > 0) {
have_oldest = find_oldest_created_at(events, ev_count, &oldest_ts);
}
if (events && ev_count > 0) {
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
for (int k = 0; k < ev_count; k++) {
cr_sink_publish(sink, events[k]);
cJSON_Delete(events[k]);
bf->events_this_window++;
bf->events_total++;
}
free(events);
}
DEBUG_LOG("backfill: pubkey[%d/%d] %s -> %d events (until=%ld)",
bf->cursor, followed->count, pk, ev_count, until);
int saturated = (ev_count == limit_used) && (ev_count > 0);
if (ev_count == 0) {
/* No more events for this author in the window. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else if (!saturated) {
/* Last page for this author. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else {
/* Saturated: paginate. Set until = oldest - 1 and keep the same
* author for the next tick. The `until` parameter is inclusive, so
* subtracting 1 avoids re-fetching the oldest event. */
if (have_oldest) {
bf->until_cursor = oldest_ts - 1;
if (bf->until_cursor < since) {
/* Walked back past the window start; author is done. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else {
pg_save_current(bf, cfg, pk, 0);
}
} else {
/* Could not determine oldest (no valid created_at); mark done. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
}
}
return 1;
}
+65
View File
@@ -0,0 +1,65 @@
/*
* caching_relay - progressive window-expansion backfill.
*
* Round-robin through the followed-pubkey set, issuing query_sync per pubkey
* for the current window (since = now - window_seconds). When a full pass
* completes, advance the window and persist state to the config file.
*
* Pagination: each author is queried with an `until` cursor that walks
* backwards in time. If a query returns exactly `page_size` events (saturated),
* the limit is bumped up to a configurable cap and retried; if still saturated
* the author is marked "blocked" at that timestamp and retried in a later
* window sweep. Otherwise the until cursor advances to
* (oldest_event_created_at - 1) and the same author is queried again on the
* next tick until no more events are returned.
*/
#ifndef CACHING_RELAY_BACKFILL_H
#define CACHING_RELAY_BACKFILL_H
#include "config.h"
#include "state.h"
#include "relay_sink.h"
#include "relay_discovery.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
/* Maximum limit we will bump a saturated page to before marking an author
* blocked at a timestamp. Keeps a single query bounded. */
#define CR_BACKFILL_MAX_PAGE_CAP 200
typedef struct {
int window_index;
long current_window_s;
int cursor; /* current index into followed set */
long until_cursor; /* current until timestamp for the current author */
int in_progress; /* 1 if a window pass is in progress */
long events_this_window; /* events pulled in current window */
long events_total; /* events pulled across all windows */
time_t last_tick; /* last time a pubkey was queried */
time_t window_started; /* when current window pass started */
int author_complete; /* 1 when current author is done, advance cursor */
int blocked; /* 1 when current author is blocked at a saturated timestamp */
long blocked_until_ts; /* timestamp where the author is blocked */
} cr_backfill_t;
/* Initialize backfill state from the persisted config state.
* In PostgreSQL mode (pg_inbox initialized), restores per-author progress
* for the current window from the caching_backfill_progress table. */
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg);
/* Perform one backfill tick. This queries ONE followed pubkey (the one at
* cfg->state.backfill_cursor) and publishes results to the sink. Uses
* until-based pagination so authors with more events than the page size are
* fully drained across multiple ticks. When a full round-robin pass completes,
* advances the window and saves state.
*
* Returns:
* 1 if a tick was performed (a pubkey was queried)
* 0 if throttled (tick_interval not elapsed) - caller should pump pools
* -1 on hard error
* -2 if backfill is complete (all windows done) - caller should go steady-state
*/
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
cr_sink_t *sink, const cr_relay_map_t *relay_map);
#endif /* CACHING_RELAY_BACKFILL_H */
+315
View File
@@ -0,0 +1,315 @@
/*
* caching_relay - config parsing + persistent state (in the .jsonc file itself)
*/
#define _GNU_SOURCE
#include "config.h"
#include "jsonc_strip.h"
#include "debug.h"
#include "../nostr_core_lib/cjson/cJSON.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>
static int read_file(const char *path, char **out, size_t *out_len) {
FILE *f = fopen(path, "rb");
if (!f) {
DEBUG_ERROR("cannot open config '%s': %s", path, strerror(errno));
return -1;
}
if (fseek(f, 0, SEEK_END) != 0) { fclose(f); return -1; }
long sz = ftell(f);
if (sz < 0) { fclose(f); return -1; }
rewind(f);
char *buf = malloc(sz + 1);
if (!buf) { fclose(f); return -1; }
size_t n = fread(buf, 1, sz, f);
fclose(f);
buf[n] = '\0';
*out = buf;
if (out_len) *out_len = n;
return 0;
}
static void copy_string_array(cJSON *arr, char *dst, int max, int *count_out,
int elem_len) {
int count = 0;
if (arr && cJSON_IsArray(arr)) {
cJSON *item;
cJSON_ArrayForEach(item, arr) {
if (count >= max) break;
if (!cJSON_IsString(item)) continue;
const char *s = cJSON_GetStringValue(item);
if (!s) continue;
strncpy(dst + count * elem_len, s, elem_len - 1);
dst[count * elem_len + (elem_len - 1)] = '\0';
count++;
}
}
*count_out = count;
}
static void copy_int_array(cJSON *arr, int *dst, int max, int *count_out) {
int count = 0;
if (arr && cJSON_IsArray(arr)) {
cJSON *item;
cJSON_ArrayForEach(item, arr) {
if (count >= max) break;
if (cJSON_IsNumber(item)) {
dst[count++] = (int)cJSON_GetNumberValue(item);
}
}
}
*count_out = count;
}
static void copy_long_array(cJSON *arr, long *dst, int max, int *count_out) {
int count = 0;
if (arr && cJSON_IsArray(arr)) {
cJSON *item;
cJSON_ArrayForEach(item, arr) {
if (count >= max) break;
if (cJSON_IsNumber(item)) {
dst[count++] = (long)cJSON_GetNumberValue(item);
}
}
}
*count_out = count;
}
int cr_config_load(cr_config_t *cfg, const char *path) {
memset(cfg, 0, sizeof(*cfg));
strncpy(cfg->path, path, sizeof(cfg->path) - 1);
char *raw = NULL;
size_t raw_len = 0;
if (read_file(path, &raw, &raw_len) != 0) return -1;
char *stripped = jsonc_strip_comments(raw, raw_len);
free(raw);
if (!stripped) {
DEBUG_ERROR("failed to strip comments from config");
return -1;
}
cJSON *root = cJSON_Parse(stripped);
free(stripped);
if (!root) {
DEBUG_ERROR("failed to parse config JSON");
return -1;
}
/* root_npubs */
copy_string_array(cJSON_GetObjectItem(root, "root_npubs"),
(char *)cfg->root_npubs, CR_MAX_ROOT_NPUBS,
&cfg->root_npub_count, CR_NPUB_LEN);
/* upstream_relays */
copy_string_array(cJSON_GetObjectItem(root, "upstream_relays"),
(char *)cfg->upstream_relays, CR_MAX_UPSTREAM,
&cfg->upstream_count, CR_URL_LEN);
/* local_relay */
cJSON *local = cJSON_GetObjectItem(root, "local_relay");
if (local && cJSON_IsString(local)) {
strncpy(cfg->local_relay, cJSON_GetStringValue(local), CR_URL_LEN - 1);
}
/* kinds */
copy_int_array(cJSON_GetObjectItem(root, "kinds"),
cfg->kinds, CR_MAX_KINDS, &cfg->kind_count);
/* admin_kinds - kinds to follow specifically for root (admin) npubs.
* Supports [*] to mean "all kinds" (no kind filter for admin). */
cJSON *ak = cJSON_GetObjectItem(root, "admin_kinds");
if (ak && cJSON_IsArray(ak)) {
cJSON *item;
cJSON_ArrayForEach(item, ak) {
if (cJSON_IsString(item)) {
const char *s = cJSON_GetStringValue(item);
if (s && strcmp(s, "*") == 0) {
cfg->admin_all_kinds = 1;
break;
}
}
}
if (!cfg->admin_all_kinds) {
copy_int_array(ak, cfg->admin_kinds, CR_MAX_KINDS, &cfg->admin_kind_count);
}
}
/* backfill */
cJSON *bf = cJSON_GetObjectItem(root, "backfill");
if (bf) {
cfg->backfill.enabled = cJSON_IsTrue(cJSON_GetObjectItem(bf, "enabled"));
copy_long_array(cJSON_GetObjectItem(bf, "window_schedule_seconds"),
cfg->backfill.window_schedule_seconds, CR_MAX_WINDOWS,
&cfg->backfill.window_count);
cJSON *ept = cJSON_GetObjectItem(bf, "events_per_tick");
if (ept) cfg->backfill.events_per_tick = (int)cJSON_GetNumberValue(ept);
cJSON *tis = cJSON_GetObjectItem(bf, "tick_interval_seconds");
if (tis) cfg->backfill.tick_interval_seconds = (int)cJSON_GetNumberValue(tis);
cJSON *wcs = cJSON_GetObjectItem(bf, "window_cooldown_seconds");
if (wcs) cfg->backfill.window_cooldown_seconds = (int)cJSON_GetNumberValue(wcs);
}
/* live */
cJSON *lv = cJSON_GetObjectItem(root, "live");
if (lv) {
cfg->live.enabled = cJSON_IsTrue(cJSON_GetObjectItem(lv, "enabled"));
cJSON *rsi = cJSON_GetObjectItem(lv, "resubscribe_interval_seconds");
if (rsi) cfg->live.resubscribe_interval_seconds = (int)cJSON_GetNumberValue(rsi);
}
/* follow_graph_refresh_seconds */
cJSON *fgr = cJSON_GetObjectItem(root, "follow_graph_refresh_seconds");
if (fgr) cfg->follow_graph_refresh_seconds = (int)cJSON_GetNumberValue(fgr);
/* state */
cJSON *st = cJSON_GetObjectItem(root, "state");
if (st) {
cJSON *bu = cJSON_GetObjectItem(st, "backfilled_until");
if (bu) cfg->state.backfilled_until = (long)cJSON_GetNumberValue(bu);
cJSON *cwi = cJSON_GetObjectItem(st, "current_window_index");
if (cwi) cfg->state.current_window_index = (int)cJSON_GetNumberValue(cwi);
cJSON *bc = cJSON_GetObjectItem(st, "backfill_cursor");
if (bc) cfg->state.backfill_cursor = (int)cJSON_GetNumberValue(bc);
}
cJSON_Delete(root);
/* Defaults if missing. */
if (cfg->backfill.window_count == 0) {
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
for (int i = 0; i < n; i++) cfg->backfill.window_schedule_seconds[i] = def_windows[i];
cfg->backfill.window_count = n;
}
if (cfg->backfill.events_per_tick == 0) cfg->backfill.events_per_tick = 50;
if (cfg->backfill.tick_interval_seconds == 0) cfg->backfill.tick_interval_seconds = 5;
if (cfg->backfill.window_cooldown_seconds == 0) cfg->backfill.window_cooldown_seconds = 60;
if (cfg->live.resubscribe_interval_seconds == 0) cfg->live.resubscribe_interval_seconds = 300;
if (cfg->follow_graph_refresh_seconds == 0) cfg->follow_graph_refresh_seconds = 600;
/* Validation. */
if (cfg->root_npub_count == 0) {
DEBUG_ERROR("config: at least one root_npub required");
return -1;
}
if (cfg->upstream_count == 0) {
DEBUG_ERROR("config: at least one upstream_relay required");
return -1;
}
if (cfg->local_relay[0] == '\0') {
DEBUG_ERROR("config: local_relay required");
return -1;
}
if (cfg->kind_count == 0) {
DEBUG_ERROR("config: at least one kind required");
return -1;
}
DEBUG_INFO("config loaded: %d root npubs, %d upstream relays, %d kinds",
cfg->root_npub_count, cfg->upstream_count, cfg->kind_count);
return 0;
}
/* Build a fresh cJSON tree from the in-memory config and write it out.
* We re-emit plain JSON (comments are not preserved on rewrite). */
int cr_config_save_state(cr_config_t *cfg) {
cJSON *root = cJSON_CreateObject();
cJSON *npubs = cJSON_CreateArray();
for (int i = 0; i < cfg->root_npub_count; i++)
cJSON_AddItemToArray(npubs, cJSON_CreateString(cfg->root_npubs[i]));
cJSON_AddItemToObject(root, "root_npubs", npubs);
cJSON *upstream = cJSON_CreateArray();
for (int i = 0; i < cfg->upstream_count; i++)
cJSON_AddItemToArray(upstream, cJSON_CreateString(cfg->upstream_relays[i]));
cJSON_AddItemToObject(root, "upstream_relays", upstream);
cJSON_AddItemToObject(root, "local_relay", cJSON_CreateString(cfg->local_relay));
cJSON *kinds = cJSON_CreateArray();
for (int i = 0; i < cfg->kind_count; i++)
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
cJSON_AddItemToObject(root, "kinds", kinds);
/* admin_kinds */
cJSON *akinds = cJSON_CreateArray();
if (cfg->admin_all_kinds) {
cJSON_AddItemToArray(akinds, cJSON_CreateString("*"));
} else {
for (int i = 0; i < cfg->admin_kind_count; i++)
cJSON_AddItemToArray(akinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
}
cJSON_AddItemToObject(root, "admin_kinds", akinds);
cJSON *bf = cJSON_CreateObject();
cJSON_AddItemToObject(bf, "enabled", cJSON_CreateBool(cfg->backfill.enabled));
cJSON *ws = cJSON_CreateArray();
for (int i = 0; i < cfg->backfill.window_count; i++)
cJSON_AddItemToArray(ws, cJSON_CreateNumber(cfg->backfill.window_schedule_seconds[i]));
cJSON_AddItemToObject(bf, "window_schedule_seconds", ws);
cJSON_AddItemToObject(bf, "events_per_tick", cJSON_CreateNumber(cfg->backfill.events_per_tick));
cJSON_AddItemToObject(bf, "tick_interval_seconds", cJSON_CreateNumber(cfg->backfill.tick_interval_seconds));
cJSON_AddItemToObject(bf, "window_cooldown_seconds", cJSON_CreateNumber(cfg->backfill.window_cooldown_seconds));
cJSON_AddItemToObject(root, "backfill", bf);
cJSON *lv = cJSON_CreateObject();
cJSON_AddItemToObject(lv, "enabled", cJSON_CreateBool(cfg->live.enabled));
cJSON_AddItemToObject(lv, "resubscribe_interval_seconds", cJSON_CreateNumber(cfg->live.resubscribe_interval_seconds));
cJSON_AddItemToObject(root, "live", lv);
cJSON_AddItemToObject(root, "follow_graph_refresh_seconds",
cJSON_CreateNumber(cfg->follow_graph_refresh_seconds));
cJSON *st = cJSON_CreateObject();
cJSON_AddItemToObject(st, "backfilled_until", cJSON_CreateNumber(cfg->state.backfilled_until));
cJSON_AddItemToObject(st, "current_window_index", cJSON_CreateNumber(cfg->state.current_window_index));
cJSON_AddItemToObject(st, "backfill_cursor", cJSON_CreateNumber(cfg->state.backfill_cursor));
cJSON_AddItemToObject(root, "state", st);
char *json = cJSON_Print(root);
cJSON_Delete(root);
if (!json) {
DEBUG_ERROR("config save: failed to serialize");
return -1;
}
/* Atomic write: temp file + rename. */
char tmp[1100];
snprintf(tmp, sizeof(tmp), "%s.tmp", cfg->path);
int fd = open(tmp, O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (fd < 0) {
DEBUG_ERROR("config save: cannot open tmp '%s': %s", tmp, strerror(errno));
free(json);
return -1;
}
size_t jlen = strlen(json);
ssize_t w = write(fd, json, jlen);
close(fd);
free(json);
if (w < 0 || (size_t)w != jlen) {
DEBUG_ERROR("config save: short write");
return -1;
}
if (rename(tmp, cfg->path) != 0) {
DEBUG_ERROR("config save: rename failed: %s", strerror(errno));
return -1;
}
DEBUG_LOG("config state saved: backfilled_until=%ld window=%d cursor=%d",
cfg->state.backfilled_until, cfg->state.current_window_index,
cfg->state.backfill_cursor);
return 0;
}
void cr_config_free(cr_config_t *cfg) {
(void)cfg;
}
+81
View File
@@ -0,0 +1,81 @@
/*
* caching_relay - config parsing + persistent state (in the .jsonc file itself)
*/
#ifndef CACHING_RELAY_CONFIG_H
#define CACHING_RELAY_CONFIG_H
#include <stddef.h>
#include <time.h>
/* Maximums to keep things statically sized and simple. */
#define CR_MAX_ROOT_NPUBS 16
#define CR_MAX_UPSTREAM 32
#define CR_MAX_KINDS 32
#define CR_MAX_WINDOWS 16
#define CR_NPUB_LEN 64 /* npub1... bech32, generous */
#define CR_URL_LEN 256
#define CR_HEX_PUBKEY_LEN 65 /* 64 hex chars + NUL */
typedef struct {
int enabled;
long window_schedule_seconds[CR_MAX_WINDOWS];
int window_count;
int events_per_tick;
int tick_interval_seconds;
int window_cooldown_seconds;
} cr_backfill_config_t;
typedef struct {
int enabled;
int resubscribe_interval_seconds;
} cr_live_config_t;
typedef struct {
long backfilled_until; /* unix ts; 0 = nothing backfilled yet */
int current_window_index; /* index into window_schedule */
int backfill_cursor; /* round-robin cursor over followed pubkeys */
} cr_state_t;
typedef struct {
char root_npubs[CR_MAX_ROOT_NPUBS][CR_NPUB_LEN];
int root_npub_count;
/* Decoded hex pubkeys for root npubs (filled by follow_graph). */
char root_hex[CR_MAX_ROOT_NPUBS][CR_HEX_PUBKEY_LEN];
int root_hex_ready;
char upstream_relays[CR_MAX_UPSTREAM][CR_URL_LEN];
int upstream_count;
char local_relay[CR_URL_LEN];
int kinds[CR_MAX_KINDS];
int kind_count;
/* Kinds to follow specifically for the root (admin) npubs.
* If admin_all_kinds is 1, grab everything (no kind filter). */
int admin_kinds[CR_MAX_KINDS];
int admin_kind_count;
int admin_all_kinds;
cr_backfill_config_t backfill;
cr_live_config_t live;
int follow_graph_refresh_seconds;
cr_state_t state;
/* Path the config was loaded from (for state write-back). */
char path[1024];
} cr_config_t;
/* Load + validate config from a .jsonc file. Returns 0 on success, -1 on error.
* On success cfg->path is set and cfg->state is populated from the file. */
int cr_config_load(cr_config_t *cfg, const char *path);
/* Persist the state sub-object back into the config file (temp + rename).
* Preserves all other config fields. Returns 0 on success, -1 on error. */
int cr_config_save_state(cr_config_t *cfg);
/* Free any heap resources held by cfg (currently none, but kept for future). */
void cr_config_free(cr_config_t *cfg);
#endif /* CACHING_RELAY_CONFIG_H */
+71
View File
@@ -0,0 +1,71 @@
#include "debug.h"
#include <stdarg.h>
#include <string.h>
/**
* @file debug.c
* @brief Debug and logging system implementation
*
* Provides a configurable logging system with timestamp formatting,
* level-based filtering, and optional file:line information.
*/
// Global debug level (default: no debug output)
debug_level_t g_debug_level = DEBUG_LEVEL_NONE;
/**
* @brief Initialize the debug system with a specific level
* @param level Debug level (0-5, clamped to valid range)
*/
void debug_init(int level) {
if (level < 0) level = 0;
if (level > 5) level = 5;
g_debug_level = (debug_level_t)level;
}
/**
* @brief Core logging function
* @param level Debug level for this message
* @param file Source file name (__FILE__)
* @param line Source line number (__LINE__)
* @param format printf-style format string
* @param ... Variable arguments for format string
*/
void debug_log(debug_level_t level, const char* file, int line, const char* format, ...) {
// Get timestamp
time_t now = time(NULL);
struct tm* tm_info = localtime(&now);
char timestamp[32];
strftime(timestamp, sizeof(timestamp), "%Y-%m-%d %H:%M:%S", tm_info);
// Get level string
const char* level_str = "UNKNOWN";
switch (level) {
case DEBUG_LEVEL_ERROR: level_str = "ERROR"; break;
case DEBUG_LEVEL_WARN: level_str = "WARN "; break;
case DEBUG_LEVEL_INFO: level_str = "INFO "; break;
case DEBUG_LEVEL_DEBUG: level_str = "DEBUG"; break;
case DEBUG_LEVEL_TRACE: level_str = "TRACE"; break;
default: break;
}
// Print prefix with timestamp and level
printf("[%s] [%s] ", timestamp, level_str);
// Print source location when debug level is TRACE (5) or higher
if (file && g_debug_level >= DEBUG_LEVEL_TRACE) {
// Extract just the filename (not full path)
const char* filename = strrchr(file, '/');
filename = filename ? filename + 1 : file;
printf("[%s:%d] ", filename, line);
}
// Print message
va_list args;
va_start(args, format);
vprintf(format, args);
va_end(args);
printf("\n");
fflush(stdout);
}
+65
View File
@@ -0,0 +1,65 @@
#ifndef C_UTILS_DEBUG_H
#define C_UTILS_DEBUG_H
/**
* @file debug.h
* @brief Debug and logging system with configurable verbosity levels
*
* Provides a simple, efficient logging system with 5 levels:
* - ERROR: Critical errors
* - WARN: Warnings
* - INFO: Informational messages
* - DEBUG: Debug messages
* - TRACE: Detailed trace with file:line info
*/
#include <stdio.h>
#include <time.h>
// Debug levels
typedef enum {
DEBUG_LEVEL_NONE = 0, /**< No debug output */
DEBUG_LEVEL_ERROR = 1, /**< Critical errors only */
DEBUG_LEVEL_WARN = 2, /**< Warnings and above */
DEBUG_LEVEL_INFO = 3, /**< Informational messages and above */
DEBUG_LEVEL_DEBUG = 4, /**< Debug messages and above */
DEBUG_LEVEL_TRACE = 5 /**< Detailed trace with file:line info */
} debug_level_t;
// Global debug level (set at runtime via CLI)
extern debug_level_t g_debug_level;
/**
* @brief Initialize the debug system
* @param level Debug level (0-5, clamped to valid range)
*/
void debug_init(int level);
/**
* @brief Core logging function
* @param level Debug level for this message
* @param file Source file name (__FILE__)
* @param line Source line number (__LINE__)
* @param format printf-style format string
* @param ... Variable arguments for format string
*/
void debug_log(debug_level_t level, const char* file, int line, const char* format, ...);
// Convenience macros that check level before calling
// Note: TRACE level (5) and above include file:line information for ALL messages
#define DEBUG_ERROR(...) \
do { if (g_debug_level >= DEBUG_LEVEL_ERROR) debug_log(DEBUG_LEVEL_ERROR, __FILE__, __LINE__, __VA_ARGS__); } while(0)
#define DEBUG_WARN(...) \
do { if (g_debug_level >= DEBUG_LEVEL_WARN) debug_log(DEBUG_LEVEL_WARN, __FILE__, __LINE__, __VA_ARGS__); } while(0)
#define DEBUG_INFO(...) \
do { if (g_debug_level >= DEBUG_LEVEL_INFO) debug_log(DEBUG_LEVEL_INFO, __FILE__, __LINE__, __VA_ARGS__); } while(0)
#define DEBUG_LOG(...) \
do { if (g_debug_level >= DEBUG_LEVEL_DEBUG) debug_log(DEBUG_LEVEL_DEBUG, __FILE__, __LINE__, __VA_ARGS__); } while(0)
#define DEBUG_TRACE(...) \
do { if (g_debug_level >= DEBUG_LEVEL_TRACE) debug_log(DEBUG_LEVEL_TRACE, __FILE__, __LINE__, __VA_ARGS__); } while(0)
#endif /* C_UTILS_DEBUG_H */
+110
View File
@@ -0,0 +1,110 @@
/*
* caching_relay - follow graph resolution
*/
#define _GNU_SOURCE
#include "follow_graph.h"
#include "debug.h"
#include <string.h>
#include <stdlib.h>
int cr_follow_is_root(const cr_config_t *cfg, const char *hex) {
if (!cfg->root_hex_ready) return 0;
for (int i = 0; i < cfg->root_npub_count; i++) {
if (strcmp(cfg->root_hex[i], hex) == 0) return 1;
}
return 0;
}
int cr_follow_decode_roots(cr_config_t *cfg) {
for (int i = 0; i < cfg->root_npub_count; i++) {
unsigned char pubkey[32];
if (nostr_decode_npub(cfg->root_npubs[i], pubkey) != NOSTR_SUCCESS) {
DEBUG_ERROR("follow: failed to decode npub '%s'", cfg->root_npubs[i]);
return -1;
}
nostr_bytes_to_hex(pubkey, 32, cfg->root_hex[i]);
}
cfg->root_hex_ready = 1;
DEBUG_INFO("follow: decoded %d root npubs", cfg->root_npub_count);
return 0;
}
/* Parse "p" tags from a kind-3 event and add pubkeys to the set. */
static int parse_p_tags(cJSON *event, cr_pubkey_set_t *followed) {
cJSON *tags = cJSON_GetObjectItem(event, "tags");
if (!tags || !cJSON_IsArray(tags)) return 0;
int added = 0;
cJSON *tag;
cJSON_ArrayForEach(tag, tags) {
if (!cJSON_IsArray(tag)) continue;
cJSON *name = cJSON_GetArrayItem(tag, 0);
if (!name || !cJSON_IsString(name)) continue;
if (strcmp(cJSON_GetStringValue(name), "p") != 0) continue;
cJSON *pk = cJSON_GetArrayItem(tag, 1);
if (!pk || !cJSON_IsString(pk)) continue;
const char *hex = cJSON_GetStringValue(pk);
if (strlen(hex) != 64) continue;
if (cr_pubkey_set_add(followed, hex) == 1) added++;
}
return added;
}
int cr_follow_resolve(cr_config_t *cfg, nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed) {
if (!cfg->root_hex_ready) {
if (cr_follow_decode_roots(cfg) != 0) return -1;
}
/* Seed the followed set with the root pubkeys themselves. */
for (int i = 0; i < cfg->root_npub_count; i++) {
cr_pubkey_set_add(followed, cfg->root_hex[i]);
}
int total_follows = 0;
for (int i = 0; i < cfg->root_npub_count; i++) {
/* Build filter: authors=[root], kinds=[3], limit=1 (most recent). */
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(cfg->root_hex[i]));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = cJSON_CreateArray();
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(3));
cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber(1));
const char **urls = NULL;
int n = 0;
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int listed_count = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
if (listed_count > 0) {
urls = malloc(listed_count * sizeof(char *));
for (int j = 0; j < listed_count; j++) { urls[n++] = listed[j]; }
}
free(listed);
free(statuses);
int ev_count = 0;
cJSON **events = nostr_relay_pool_query_sync(upstream, urls, n,
filter, &ev_count, 15000);
free(urls);
cJSON_Delete(filter);
if (events && ev_count > 0) {
int added = parse_p_tags(events[0], followed);
total_follows += added;
DEBUG_INFO("follow: root %d: kind-3 found, +%d follows (total set %d)",
i, added, followed->count);
for (int k = 0; k < ev_count; k++) cJSON_Delete(events[k]);
free(events);
} else {
DEBUG_WARN("follow: root %d: no kind-3 found", i);
free(events);
}
}
DEBUG_INFO("follow: resolved %d total followed pubkeys (set size %d)",
total_follows, followed->count);
return 0;
}
+27
View File
@@ -0,0 +1,27 @@
/*
* caching_relay - follow graph resolution.
*
* Decodes root npubs (NIP-19) to hex, queries their most recent kind-3
* contact list from the upstream pool, parses the "p" tags, and populates
* the in-memory followed-pubkey set (root pubkeys + their follows).
*/
#ifndef CACHING_RELAY_FOLLOW_GRAPH_H
#define CACHING_RELAY_FOLLOW_GRAPH_H
#include "config.h"
#include "state.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
/* Check if a hex pubkey is one of the root (admin) npubs. Returns 1 if yes. */
int cr_follow_is_root(const cr_config_t *cfg, const char *hex);
/* Decode all root_npubs in cfg into cfg->root_hex[]. Returns 0 on success. */
int cr_follow_decode_roots(cr_config_t *cfg);
/* Resolve the follow graph: for each root hex pubkey, query the most recent
* kind-3 from the upstream pool, parse "p" tags, and add root + follows to
* the followed set. Returns 0 on success, -1 on hard failure. */
int cr_follow_resolve(cr_config_t *cfg, nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed);
#endif /* CACHING_RELAY_FOLLOW_GRAPH_H */
+69
View File
@@ -0,0 +1,69 @@
/*
* caching_relay - strip JSONC comments so cJSON can parse it.
*/
#define _GNU_SOURCE
#include "jsonc_strip.h"
#include <stdlib.h>
#include <string.h>
char *jsonc_strip_comments(const char *input, size_t len) {
if (!input) return NULL;
if (len == 0) len = strlen(input);
/* Worst case: output is same size as input (no comments). +1 for NUL. */
char *out = malloc(len + 1);
if (!out) return NULL;
size_t i = 0, o = 0;
int in_string = 0;
int escape = 0;
while (i < len) {
char c = input[i];
if (in_string) {
out[o++] = c;
if (escape) {
escape = 0;
} else if (c == '\\') {
escape = 1;
} else if (c == '"') {
in_string = 0;
}
i++;
continue;
}
/* Not in a string. */
if (c == '"') {
in_string = 1;
out[o++] = c;
i++;
continue;
}
/* Line comment // */
if (c == '/' && i + 1 < len && input[i + 1] == '/') {
i += 2;
while (i < len && input[i] != '\n') i++;
continue;
}
/* Block comment / * ... * / */
if (c == '/' && i + 1 < len && input[i + 1] == '*') {
i += 2;
while (i < len && !(input[i] == '*' && i + 1 < len && input[i + 1] == '/')) i++;
if (i < len) i += 2; /* skip closing */
/* Preserve a space so tokens don't merge. */
out[o++] = ' ';
continue;
}
out[o++] = c;
i++;
}
out[o] = '\0';
return out;
}
+16
View File
@@ -0,0 +1,16 @@
/*
* caching_relay - strip JSONC comments so cJSON can parse it.
*
* cJSON does not natively understand JSONC. This produces a heap buffer the
* caller must free() containing the comment-stripped JSON. String literals are
* respected so a double-slash inside a string is preserved.
*/
#ifndef CACHING_RELAY_JSONC_STRIP_H
#define CACHING_RELAY_JSONC_STRIP_H
#include <stddef.h>
/* Returns malloc'd buffer (null-terminated) or NULL on alloc failure. */
char *jsonc_strip_comments(const char *input, size_t len);
#endif /* CACHING_RELAY_JSONC_STRIP_H */
+171
View File
@@ -0,0 +1,171 @@
/*
* caching_relay - live subscription on the upstream pool
*/
#define _GNU_SOURCE
#include "live_subscriber.h"
#include "follow_graph.h"
#include "debug.h"
#include <string.h>
#include <stdlib.h>
#include <time.h>
/* Context passed as user_data to the subscription callbacks. */
typedef struct {
cr_sink_t *sink;
cr_live_t *live;
} cr_live_ctx_t;
static cr_live_ctx_t g_live_ctx;
static void live_on_event(cJSON *event, const char *relay_url, void *user_data) {
cr_live_ctx_t *ctx = (cr_live_ctx_t *)user_data;
(void)relay_url;
ctx->live->events_received++;
cr_sink_publish(ctx->sink, event);
}
static void live_on_eose(cJSON **events, int event_count, void *user_data) {
(void)events; (void)event_count; (void)user_data;
DEBUG_LOG("live: EOSE (stored events flushed), staying open");
}
/* Build a filter for non-admin followed pubkeys with regular kinds. */
static cJSON *build_follows_filter(cr_config_t *cfg, cr_pubkey_set_t *followed) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
for (int i = 0; i < followed->count; i++) {
/* Skip admin pubkeys - they get their own subscription. */
if (cr_follow_is_root(cfg, followed->items[i])) continue;
cJSON_AddItemToArray(authors, cJSON_CreateString(followed->items[i]));
}
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = cJSON_CreateArray();
for (int i = 0; i < cfg->kind_count; i++)
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)time(NULL)));
return filter;
}
/* Build a filter for admin (root) pubkeys with admin_kinds (or all kinds). */
static cJSON *build_admin_filter(cr_config_t *cfg) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
for (int i = 0; i < cfg->root_npub_count; i++)
cJSON_AddItemToArray(authors, cJSON_CreateString(cfg->root_hex[i]));
cJSON_AddItemToObject(filter, "authors", authors);
/* If admin_all_kinds, omit the kinds filter entirely (grab everything). */
if (!cfg->admin_all_kinds && cfg->admin_kind_count > 0) {
cJSON *kinds = cJSON_CreateArray();
for (int i = 0; i < cfg->admin_kind_count; i++)
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
cJSON_AddItemToObject(filter, "kinds", kinds);
}
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)time(NULL)));
return filter;
}
static int get_relay_urls(nostr_relay_pool_t *upstream, const char ***urls_out) {
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
const char **urls = NULL;
if (n > 0) {
urls = malloc(n * sizeof(char *));
for (int j = 0; j < n; j++) urls[j] = listed[j];
}
free(listed);
free(statuses);
*urls_out = urls;
return n;
}
static nostr_pool_subscription_t *open_subscription(nostr_relay_pool_t *upstream,
cJSON *filter) {
const char **urls = NULL;
int n = get_relay_urls(upstream, &urls);
nostr_pool_subscription_t *sub = nostr_relay_pool_subscribe(
upstream, urls, n, filter,
live_on_event, live_on_eose, &g_live_ctx,
0, 1, NOSTR_POOL_EOSE_FULL_SET, 0, 0);
free(urls);
return sub;
}
static int open_subs(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed, cr_sink_t *sink) {
g_live_ctx.sink = sink;
g_live_ctx.live = live;
int admin_count = 0;
int follows_count = 0;
for (int i = 0; i < followed->count; i++) {
if (cr_follow_is_root(cfg, followed->items[i])) admin_count++;
else follows_count++;
}
/* Follows subscription (non-admin pubkeys, regular kinds). */
if (follows_count > 0) {
cJSON *filter = build_follows_filter(cfg, followed);
live->follows_sub = open_subscription(upstream, filter);
cJSON_Delete(filter);
if (!live->follows_sub) {
DEBUG_ERROR("live: follows subscribe failed");
} else {
DEBUG_INFO("live: follows sub - %d authors, %d kinds",
follows_count, cfg->kind_count);
}
}
/* Admin subscription (root npubs, admin_kinds or all kinds). */
if (admin_count > 0 || cfg->root_npub_count > 0) {
cJSON *filter = build_admin_filter(cfg);
live->admin_sub = open_subscription(upstream, filter);
cJSON_Delete(filter);
if (!live->admin_sub) {
DEBUG_ERROR("live: admin subscribe failed");
} else {
if (cfg->admin_all_kinds) {
DEBUG_INFO("live: admin sub - %d authors, ALL kinds",
cfg->root_npub_count);
} else {
DEBUG_INFO("live: admin sub - %d authors, %d kinds",
cfg->root_npub_count, cfg->admin_kind_count);
}
}
}
live->last_resubscribe = time(NULL);
return 0;
}
int cr_live_open(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed, cr_sink_t *sink) {
memset(live, 0, sizeof(*live));
return open_subs(live, cfg, upstream, followed, sink);
}
int cr_live_resubscribe(cr_live_t *live, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
cr_sink_t *sink) {
cr_live_close(live);
DEBUG_INFO("live: resubscribing (events so far: %ld)", live->events_received);
return open_subs(live, cfg, upstream, followed, sink);
}
void cr_live_close(cr_live_t *live) {
if (live->follows_sub) {
nostr_pool_subscription_close(live->follows_sub);
live->follows_sub = NULL;
}
if (live->admin_sub) {
nostr_pool_subscription_close(live->admin_sub);
live->admin_sub = NULL;
}
}
+37
View File
@@ -0,0 +1,37 @@
/*
* caching_relay - live subscription on the upstream pool.
*
* Builds filters from the followed-pubkey set + configured kinds, opens
* long-lived subscriptions, and routes on_event to the relay sink.
*
* Two subscriptions are opened:
* 1. "follows" sub: non-admin followed pubkeys + regular kinds
* 2. "admin" sub: root (admin) npubs + admin_kinds (or all kinds if [*])
*/
#ifndef CACHING_RELAY_LIVE_SUBSCRIBER_H
#define CACHING_RELAY_LIVE_SUBSCRIBER_H
#include "config.h"
#include "state.h"
#include "relay_sink.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
typedef struct {
nostr_pool_subscription_t *follows_sub; /* non-admin pubkeys, regular kinds */
nostr_pool_subscription_t *admin_sub; /* admin pubkeys, admin_kinds */
long events_received;
time_t last_resubscribe;
} cr_live_t;
/* Open the live subscription(s). Returns 0 on success. */
int cr_live_open(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed, cr_sink_t *sink);
/* Close + reopen the subscription(s). Returns 0 on success. */
int cr_live_resubscribe(cr_live_t *live, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
cr_sink_t *sink);
void cr_live_close(cr_live_t *live);
#endif /* CACHING_RELAY_LIVE_SUBSCRIBER_H */
+463
View File
@@ -0,0 +1,463 @@
/*
* caching_relay - a daemon that caches Nostr events from followed people
* into a local relay.
*
* Architecture: see plans/plan.md. Two nostr_relay_pool_t instances:
* - upstream_pool: query/subscribe to upstream relays
* - sink_pool (in cr_sink_t): publish-only to the local relay
*
* Build: see Makefile. Statically linked C99 binary, c-relay style.
*/
#define _GNU_SOURCE
#include "main.h"
#include "debug.h"
#include "config.h"
#include "state.h"
#include "follow_graph.h"
#include "relay_sink.h"
#include "live_subscriber.h"
#include "backfill.h"
#include "relay_discovery.h"
#include "pg_inbox.h"
#include "pg_config.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
#include "../nostr_core_lib/nostr_core/nostr_log.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <signal.h>
#include <time.h>
#include <unistd.h>
#include <getopt.h>
/* Forward nostr_core_lib's internal logging to our debug system. */
static void nostr_log_forwarder(int level, const char *component,
const char *message, void *user_data) {
(void)user_data;
/* Map nostr log levels to our debug levels (they're the same 1-5). */
if (level >= 5) {
DEBUG_TRACE("[nostr:%s] %s", component ? component : "?", message ? message : "");
} else if (level >= 4) {
DEBUG_LOG("[nostr:%s] %s", component ? component : "?", message ? message : "");
} else if (level >= 3) {
DEBUG_INFO("[nostr:%s] %s", component ? component : "?", message ? message : "");
} else if (level >= 2) {
DEBUG_WARN("[nostr:%s] %s", component ? component : "?", message ? message : "");
} else {
DEBUG_ERROR("[nostr:%s] %s", component ? component : "?", message ? message : "");
}
}
volatile sig_atomic_t g_shutdown = 0;
static volatile sig_atomic_t g_reload = 0;
static void on_signal(int sig) {
if (sig == SIGINT || sig == SIGTERM) g_shutdown = 1;
else if (sig == SIGHUP) g_reload = 1;
}
static void usage(const char *prog) {
fprintf(stderr,
"caching_relay %s - cache Nostr events from followed people into a local relay\n"
"\n"
"Usage: %s [-c <config.jsonc>] [-p <pg-conn>] [options]\n"
"\n"
"Options:\n"
" -c, --config <file> Path to .jsonc config file (default: ./caching_relay_config.jsonc)\n"
" -p, --pg-conn <str> PostgreSQL connection string (libpq format). When provided,\n"
" config is read from the c-relay-pg config table and fetched\n"
" events are inserted into the caching_event_inbox table instead\n"
" of being published via WebSocket to a local relay.\n"
" -d, --debug <level> Log level 0-5 (0=none, 1=error, 2=warn, 3=info, 4=debug, 5=trace). Default 3.\n"
" -r, --restart Reset state to first-time startup (ignore local relay cache,\n"
" re-discover all kind-10002 from bootstrap relays, reset backfill)\n"
" -h, --help Show this help\n"
"\n"
"Without -p, the daemon uses the .jsonc config file and publishes to a local relay\n"
"via WebSocket (legacy mode). With -p, it uses PostgreSQL for config and inbox.\n"
"The config file is also the persistent state store in legacy mode; the daemon\n"
"rewrites it as backfill progresses. See plans/plan.md and caching_relay_config.jsonc.\n",
CR_VERSION, prog);
}
int main(int argc, char **argv) {
const char *config_path = NULL;
const char *pg_conn = NULL;
int log_level = DEBUG_LEVEL_INFO;
int restart = 0;
static struct option longopts[] = {
{"config", required_argument, 0, 'c'},
{"pg-conn", required_argument, 0, 'p'},
{"debug", required_argument, 0, 'd'},
{"restart", no_argument, 0, 'r'},
{"help", no_argument, 0, 'h'},
{0, 0, 0, 0}
};
int opt;
while ((opt = getopt_long(argc, argv, "c:p:d:rh", longopts, NULL)) != -1) {
switch (opt) {
case 'c': config_path = optarg; break;
case 'p': pg_conn = optarg; break;
case 'd': log_level = atoi(optarg); break;
case 'r': restart = 1; break;
case 'h': usage(argv[0]); return 0;
default: usage(argv[0]); return 1;
}
}
/* In PostgreSQL mode, the .jsonc config file is optional. */
if (!config_path && !pg_conn) {
/* Default: look for caching_relay.jsonc in the current directory. */
config_path = "caching_relay_config.jsonc";
if (access(config_path, F_OK) != 0) {
fprintf(stderr, "ERROR: no config specified and default '%s' not found\n\n", config_path);
usage(argv[0]);
return 1;
}
}
if (log_level < 0) log_level = 0;
if (log_level > 5) log_level = 5;
debug_init(log_level);
/* Enable nostr_core_lib internal logging and forward to our debug system.
* This shows the actual WebSocket messages (REQ filters, EVENT responses)
* at debug level 5 (trace). */
nostr_set_log_callback(nostr_log_forwarder, NULL);
nostr_set_log_level((nostr_log_level_t)log_level);
DEBUG_INFO("caching_relay %s starting (config=%s, pg-conn=%s, loglevel=%d%s)",
CR_VERSION,
config_path ? config_path : "(none)",
pg_conn ? "yes" : "no",
log_level, restart ? ", RESTART" : "");
/* Signals. */
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_handler = on_signal;
sigaction(SIGINT, &sa, NULL);
sigaction(SIGTERM, &sa, NULL);
sigaction(SIGHUP, &sa, NULL);
/* Ignore SIGPIPE - relay pool handles its own socket errors. */
signal(SIGPIPE, SIG_IGN);
/* PostgreSQL inbox mode: connect and load config from the config table. */
long config_generation = -1;
if (pg_conn) {
if (pg_inbox_init(pg_conn) != 0) {
DEBUG_ERROR("failed to connect to PostgreSQL");
return 1;
}
config_generation = pg_inbox_get_config_generation();
if (config_generation < 0) {
DEBUG_WARN("could not read caching_config_generation (defaulting to 0)");
config_generation = 0;
}
DEBUG_INFO("config generation: %ld", config_generation);
}
/* Load config. */
cr_config_t cfg;
if (pg_conn) {
if (pg_config_load(&cfg) != 0) {
DEBUG_ERROR("failed to load config from PostgreSQL");
pg_inbox_shutdown();
return 1;
}
} else {
if (cr_config_load(&cfg, config_path) != 0) return 1;
}
/* --restart: reset state to first-time startup. */
if (restart) {
DEBUG_INFO("RESTART: resetting state to first-time startup");
cfg.state.backfilled_until = 0;
cfg.state.current_window_index = 0;
cfg.state.backfill_cursor = 0;
if (pg_conn) {
/* Clear per-author backfill progress so the next run starts fresh. */
pg_inbox_reset_backfill_progress();
} else {
cr_config_save_state(&cfg);
}
}
/* Init crypto. */
if (nostr_crypto_init() != NOSTR_SUCCESS) {
DEBUG_ERROR("nostr_crypto_init failed");
return 1;
}
/* In-memory state. */
cr_seen_ring_t seen;
cr_seen_ring_init(&seen);
cr_pubkey_set_t followed;
cr_pubkey_set_init(&followed);
/* Upstream pool. */
nostr_relay_pool_t *upstream = nostr_relay_pool_create(nostr_pool_reconnect_config_default());
if (!upstream) {
DEBUG_ERROR("failed to create upstream pool");
return 1;
}
for (int i = 0; i < cfg.upstream_count; i++) {
if (nostr_relay_pool_add_relay(upstream, cfg.upstream_relays[i]) != NOSTR_SUCCESS) {
DEBUG_WARN("failed to add upstream relay %s (continuing)", cfg.upstream_relays[i]);
} else {
DEBUG_INFO("upstream: added %s", cfg.upstream_relays[i]);
}
}
/* Sink: WebSocket pool (legacy) or PostgreSQL inbox. */
cr_sink_t sink;
if (pg_conn) {
if (cr_sink_init_pg(&sink, &seen) != 0) {
DEBUG_ERROR("failed to init sink (pg)");
return 1;
}
} else {
if (cr_sink_init(&sink, cfg.local_relay, &seen) != 0) {
DEBUG_ERROR("failed to init sink");
return 1;
}
}
/* Give the sink pool a moment to connect to the local relay before we
* start publishing (relay discovery caches kind-10002 events to local).
* No-op in PostgreSQL inbox mode (cr_sink_pump does nothing). */
if (!pg_conn) {
DEBUG_INFO("waiting for sink connection to establish...");
for (int i = 0; i < 30 && !g_shutdown; i++) {
cr_sink_pump(&sink, 100);
}
}
/* Resolve follow graph. */
if (g_shutdown) goto shutdown;
if (cr_follow_resolve(&cfg, upstream, &followed) != 0) {
DEBUG_ERROR("follow graph resolution failed");
return 1;
}
if (g_shutdown) goto shutdown;
/* Discover outbox relays (NIP-65) and compute minimum covering set. */
int is_first_time = (cfg.state.backfilled_until == 0);
cr_relay_map_t relay_map;
if (cr_relay_discovery_run(&relay_map, &cfg, upstream, &sink, &followed,
is_first_time) != 0) {
DEBUG_WARN("relay discovery failed, continuing with bootstrap relays only");
memset(&relay_map, 0, sizeof(relay_map));
}
/* Add discovered outbox relays to the upstream pool. */
for (int i = 0; i < relay_map.selected_count; i++) {
/* Check if already in the pool (bootstrap relays may already be there). */
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
int already = 0;
for (int j = 0; j < n; j++) {
if (strcmp(listed[j], relay_map.selected_relays[i]) == 0) {
already = 1; break;
}
}
free(listed);
free(statuses);
if (!already) {
if (nostr_relay_pool_add_relay(upstream, relay_map.selected_relays[i]) == NOSTR_SUCCESS) {
DEBUG_INFO("upstream: added outbox relay %s", relay_map.selected_relays[i]);
}
}
}
/* Log final upstream pool. */
{
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
DEBUG_INFO("upstream_pool: %d relays connected:", n);
for (int j = 0; j < n; j++) {
DEBUG_INFO(" %s", listed[j]);
}
free(listed);
free(statuses);
}
/* Open live subscription. */
cr_live_t live;
if (cfg.live.enabled) {
if (cr_live_open(&live, &cfg, upstream, &followed, &sink) != 0) {
DEBUG_WARN("live subscription failed to open (will retry on resubscribe)");
}
} else {
memset(&live, 0, sizeof(live));
}
/* Init backfill. */
cr_backfill_t bf;
cr_backfill_init(&bf, &cfg);
time_t last_follow_refresh = time(NULL);
time_t last_state_save = time(NULL);
time_t last_status_heartbeat = 0;
/* Initial status heartbeat in PostgreSQL mode. */
if (pg_conn) {
pg_inbox_update_status("starting", config_generation, (long)time(NULL),
followed.count, relay_map.selected_count, 0,
cfg.state.current_window_index,
cfg.state.backfill_cursor, 0, 0, NULL, 0);
}
DEBUG_INFO("entering main loop");
while (!g_shutdown) {
if (g_reload) {
g_reload = 0;
DEBUG_INFO("SIGHUP: reloading config (state preserved)");
cr_config_t newcfg;
int reload_ok = 0;
if (pg_conn) {
if (pg_config_load(&newcfg) == 0) reload_ok = 1;
} else if (config_path) {
if (cr_config_load(&newcfg, config_path) == 0) reload_ok = 1;
}
if (reload_ok) {
/* Preserve runtime state across reload. */
newcfg.state = cfg.state;
cr_config_free(&cfg);
cfg = newcfg;
DEBUG_INFO("config reloaded");
} else {
DEBUG_ERROR("config reload failed, keeping old config");
}
}
/* PostgreSQL: check for config generation change and reload. */
if (pg_conn) {
int changed = pg_config_generation_changed(config_generation);
if (changed == 1) {
long new_gen = pg_inbox_get_config_generation();
DEBUG_INFO("config generation changed (%ld -> %ld), reloading",
config_generation, new_gen);
cr_config_t newcfg;
if (pg_config_load(&newcfg) == 0) {
newcfg.state = cfg.state;
cr_config_free(&cfg);
cfg = newcfg;
config_generation = new_gen;
DEBUG_INFO("config reloaded from PostgreSQL");
/* Resubscribe live with new config. */
if (cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
} else {
DEBUG_ERROR("PostgreSQL config reload failed, keeping old config");
}
} else if (changed < 0) {
DEBUG_WARN("config generation check failed");
}
}
/* Pump upstream pool (drives live subscription callbacks). */
nostr_relay_pool_run(upstream, 100);
/* Pump sink pool (flush publish callbacks). */
cr_sink_pump(&sink, 50);
/* Backfill tick. */
int brc = cr_backfill_tick(&bf, &cfg, upstream, &followed, &sink, &relay_map);
(void)brc;
/* Periodic follow-graph refresh. */
time_t now = time(NULL);
if (cfg.follow_graph_refresh_seconds > 0 &&
(now - last_follow_refresh) >= cfg.follow_graph_refresh_seconds) {
DEBUG_INFO("refreshing follow graph");
cr_pubkey_set_t new_followed;
cr_pubkey_set_init(&new_followed);
if (cr_follow_resolve(&cfg, upstream, &new_followed) == 0) {
/* If the set changed, resubscribe live. */
int changed = (new_followed.count != followed.count);
if (!changed) {
for (int i = 0; i < followed.count; i++) {
if (!cr_pubkey_set_contains(&new_followed, followed.items[i])) {
changed = 1; break;
}
}
}
cr_pubkey_set_free(&followed);
followed = new_followed;
if (changed && cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
} else {
DEBUG_WARN("follow graph refresh failed");
cr_pubkey_set_free(&new_followed);
}
last_follow_refresh = now;
}
/* Periodic live resubscribe. */
if (cfg.live.enabled && cfg.live.resubscribe_interval_seconds > 0 &&
(now - live.last_resubscribe) >= cfg.live.resubscribe_interval_seconds) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
/* Periodic state save (in case backfill didn't just save). */
if ((now - last_state_save) >= 60) {
if (!pg_conn) cr_config_save_state(&cfg);
last_state_save = now;
}
/* PostgreSQL: periodic status heartbeat. */
if (pg_conn && (now - last_status_heartbeat) >= 15) {
long events_fetched = live.events_received + bf.events_total;
long inbox_inserts = sink.published_ok;
int connected = 0;
{
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
for (int j = 0; j < n; j++) {
if (statuses[j] == NOSTR_POOL_RELAY_CONNECTED) connected++;
}
free(listed);
free(statuses);
}
pg_inbox_update_status("running", config_generation, (long)now,
followed.count, relay_map.selected_count,
connected, cfg.state.current_window_index,
cfg.state.backfill_cursor,
events_fetched, inbox_inserts, NULL, 0);
last_status_heartbeat = now;
}
}
/* Graceful shutdown. */
shutdown:
DEBUG_INFO("shutting down...");
cr_live_close(&live);
if (!pg_conn) cr_config_save_state(&cfg);
if (pg_conn) {
pg_inbox_update_status("stopped", config_generation, (long)time(NULL),
followed.count, relay_map.selected_count, 0,
cfg.state.current_window_index,
cfg.state.backfill_cursor,
live.events_received + bf.events_total,
sink.published_ok, NULL, 0);
}
cr_sink_destroy(&sink);
nostr_relay_pool_destroy(upstream);
cr_pubkey_set_free(&followed);
cr_relay_map_free(&relay_map);
cr_config_free(&cfg);
if (pg_conn) pg_inbox_shutdown();
nostr_crypto_cleanup();
DEBUG_INFO("clean exit");
return 0;
}
+16
View File
@@ -0,0 +1,16 @@
/*
* caching_relay - main header
*
* Version information is auto-updated by increment_and_push.sh.
* Git tags are the source of truth for versioning.
*/
#ifndef CACHING_RELAY_MAIN_H
#define CACHING_RELAY_MAIN_H
// Version information (auto-updated by increment_and_push.sh)
#define CR_VERSION_MAJOR 0
#define CR_VERSION_MINOR 0
#define CR_VERSION_PATCH 2
#define CR_VERSION "v0.0.2"
#endif /* CACHING_RELAY_MAIN_H */
+236
View File
@@ -0,0 +1,236 @@
/*
* caching_relay - read caching configuration from the c-relay-pg PostgreSQL
* config table and populate the existing cr_config_t structure.
*/
#define _GNU_SOURCE
#include "pg_config.h"
#include "pg_inbox.h"
#include "debug.h"
#include <stdlib.h>
#include <string.h>
/* ------------------------------------------------------------------ */
/* Helpers */
/* ------------------------------------------------------------------ */
/* Split a comma-separated string into a string array (fixed-size rows).
* Trims leading/trailing whitespace from each token. */
static void split_csv_str(const char *s, char *dst, int max, int elem_len,
int *count_out) {
int count = 0;
if (!s) { *count_out = 0; return; }
/* Work on a mutable copy because strtok_r modifies its input. */
char *copy = strdup(s);
if (!copy) { *count_out = 0; return; }
char *saveptr = NULL;
char *tok = strtok_r(copy, ",", &saveptr);
while (tok && count < max) {
/* Trim leading spaces. */
while (*tok == ' ' || *tok == '\t') tok++;
/* Trim trailing spaces. */
char *end = tok + strlen(tok) - 1;
while (end > tok && (*end == ' ' || *end == '\t' || *end == '\r' || *end == '\n')) {
*end-- = '\0';
}
if (*tok != '\0') {
strncpy(dst + count * elem_len, tok, elem_len - 1);
dst[count * elem_len + (elem_len - 1)] = '\0';
count++;
}
tok = strtok_r(NULL, ",", &saveptr);
}
free(copy);
*count_out = count;
}
/* Split a comma-separated string of integers into an int array. */
static void split_csv_int(const char *s, int *dst, int max, int *count_out) {
int count = 0;
if (!s) { *count_out = 0; return; }
char *copy = strdup(s);
if (!copy) { *count_out = 0; return; }
char *saveptr = NULL;
char *tok = strtok_r(copy, ",", &saveptr);
while (tok && count < max) {
while (*tok == ' ' || *tok == '\t') tok++;
if (*tok != '\0') {
dst[count++] = atoi(tok);
}
tok = strtok_r(NULL, ",", &saveptr);
}
free(copy);
*count_out = count;
}
/* Split a comma-separated string of longs into a long array. */
static void split_csv_long(const char *s, long *dst, int max, int *count_out) {
int count = 0;
if (!s) { *count_out = 0; return; }
char *copy = strdup(s);
if (!copy) { *count_out = 0; return; }
char *saveptr = NULL;
char *tok = strtok_r(copy, ",", &saveptr);
while (tok && count < max) {
while (*tok == ' ' || *tok == '\t') tok++;
if (*tok != '\0') {
dst[count++] = strtol(tok, NULL, 10);
}
tok = strtok_r(NULL, ",", &saveptr);
}
free(copy);
*count_out = count;
}
/* Read a config key as a freshly-allocated string. Returns NULL if missing. */
static char *get_key(const char *key) {
return pg_inbox_get_config_value(key);
}
/* Read a config key as a boolean ("true"/"1"/"yes" -> 1). */
static int get_bool(const char *key, int default_val) {
char *v = get_key(key);
if (!v) return default_val;
int r = (strcmp(v, "true") == 0 || strcmp(v, "1") == 0 ||
strcmp(v, "yes") == 0 || strcmp(v, "on") == 0);
free(v);
return r;
}
/* Read a config key as a long. */
static long get_long(const char *key, long default_val) {
char *v = get_key(key);
if (!v) return default_val;
long r = strtol(v, NULL, 10);
free(v);
return r;
}
/* Read a config key as an int. */
static int get_int(const char *key, int default_val) {
return (int)get_long(key, (long)default_val);
}
/* ------------------------------------------------------------------ */
/* Public API */
/* ------------------------------------------------------------------ */
int pg_config_load(cr_config_t *cfg) {
if (!cfg) return -1;
memset(cfg, 0, sizeof(*cfg));
/* root_npubs */
char *npubs = get_key("caching_root_npubs");
split_csv_str(npubs, (char *)cfg->root_npubs, CR_MAX_ROOT_NPUBS,
CR_NPUB_LEN, &cfg->root_npub_count);
free(npubs);
/* bootstrap_relays -> upstream_relays */
char *relays = get_key("caching_bootstrap_relays");
split_csv_str(relays, (char *)cfg->upstream_relays, CR_MAX_UPSTREAM,
CR_URL_LEN, &cfg->upstream_count);
free(relays);
/* local_relay - no longer required for PostgreSQL mode, but keep a
* placeholder for compatibility. The caller may override it. */
cfg->local_relay[0] = '\0';
/* kinds */
char *kinds = get_key("caching_kinds");
split_csv_int(kinds, cfg->kinds, CR_MAX_KINDS, &cfg->kind_count);
free(kinds);
/* admin_kinds - supports "*" to mean all kinds */
char *akinds = get_key("caching_admin_kinds");
if (akinds) {
/* Check for "*" anywhere in the list. */
if (strstr(akinds, "*") != NULL) {
cfg->admin_all_kinds = 1;
} else {
split_csv_int(akinds, cfg->admin_kinds, CR_MAX_KINDS,
&cfg->admin_kind_count);
}
free(akinds);
}
/* live */
cfg->live.enabled = get_bool("caching_live_enabled", 1);
cfg->live.resubscribe_interval_seconds =
get_int("caching_live_resubscribe_seconds", 300);
/* backfill */
cfg->backfill.enabled = get_bool("caching_backfill_enabled", 1);
char *windows = get_key("caching_backfill_windows");
split_csv_long(windows, cfg->backfill.window_schedule_seconds,
CR_MAX_WINDOWS, &cfg->backfill.window_count);
free(windows);
cfg->backfill.events_per_tick =
get_int("caching_backfill_page_size", 50);
/* tick interval is stored in ms in the config table; convert to seconds. */
int tick_ms = get_int("caching_backfill_tick_interval_ms", 5000);
cfg->backfill.tick_interval_seconds = (tick_ms + 999) / 1000;
if (cfg->backfill.tick_interval_seconds < 1)
cfg->backfill.tick_interval_seconds = 1;
cfg->backfill.window_cooldown_seconds =
get_int("caching_backfill_window_cooldown_seconds", 60);
/* follow graph refresh */
cfg->follow_graph_refresh_seconds =
get_int("caching_follow_graph_refresh_seconds", 600);
/* Defaults if missing (mirror cr_config_load). */
if (cfg->backfill.window_count == 0) {
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
for (int i = 0; i < n; i++)
cfg->backfill.window_schedule_seconds[i] = def_windows[i];
cfg->backfill.window_count = n;
}
if (cfg->backfill.events_per_tick == 0)
cfg->backfill.events_per_tick = 50;
if (cfg->backfill.tick_interval_seconds == 0)
cfg->backfill.tick_interval_seconds = 5;
if (cfg->backfill.window_cooldown_seconds == 0)
cfg->backfill.window_cooldown_seconds = 60;
if (cfg->live.resubscribe_interval_seconds == 0)
cfg->live.resubscribe_interval_seconds = 300;
if (cfg->follow_graph_refresh_seconds == 0)
cfg->follow_graph_refresh_seconds = 600;
/* State: initialize to defaults for this phase. Phase 7 will read
* caching_backfill_progress / caching_service_state. */
cfg->state.backfilled_until = 0;
cfg->state.current_window_index = 0;
cfg->state.backfill_cursor = 0;
/* Validation. */
if (cfg->root_npub_count == 0) {
DEBUG_ERROR("pg_config: at least one caching_root_npubs required");
return -1;
}
if (cfg->upstream_count == 0) {
DEBUG_ERROR("pg_config: at least one caching_bootstrap_relays required");
return -1;
}
if (cfg->kind_count == 0) {
DEBUG_ERROR("pg_config: at least one caching_kinds required");
return -1;
}
DEBUG_INFO("pg_config loaded: %d root npubs, %d upstream relays, %d kinds",
cfg->root_npub_count, cfg->upstream_count, cfg->kind_count);
return 0;
}
int pg_config_generation_changed(long last_generation) {
long cur = pg_inbox_get_config_generation();
if (cur < 0) return -1;
return (cur != last_generation) ? 1 : 0;
}
+19
View File
@@ -0,0 +1,19 @@
/*
* caching_relay - read caching configuration from the c-relay-pg PostgreSQL
* config table and populate the existing cr_config_t structure.
*/
#ifndef CACHING_RELAY_PG_CONFIG_H
#define CACHING_RELAY_PG_CONFIG_H
#include "config.h" /* cr_config_t */
/* Load caching configuration from the c-relay-pg PostgreSQL config table.
* Populates the cr_config_t fields from the config table.
* Returns 0 on success, -1 on error. */
int pg_config_load(cr_config_t *cfg);
/* Check if the config generation has changed since the last load.
* Returns 1 if changed, 0 if not, -1 on error. */
int pg_config_generation_changed(long last_generation);
#endif /* CACHING_RELAY_PG_CONFIG_H */
+488
View File
@@ -0,0 +1,488 @@
/*
* caching_relay - PostgreSQL inbox connection and operations.
*/
#define _GNU_SOURCE
#include "pg_inbox.h"
#include "debug.h"
#include <libpq-fe.h>
#include <stdlib.h>
#include <string.h>
#include "../nostr_core_lib/cjson/cJSON.h"
static PGconn *g_pg = NULL;
/* ------------------------------------------------------------------ */
int pg_inbox_init(const char *connection_string) {
if (g_pg) {
PQfinish(g_pg);
g_pg = NULL;
}
g_pg = PQconnectdb(connection_string);
if (!g_pg) {
DEBUG_ERROR("pg_inbox: PQconnectdb returned NULL (out of memory)");
return -1;
}
if (PQstatus(g_pg) != CONNECTION_OK) {
DEBUG_ERROR("pg_inbox: connection failed: %s",
PQerrorMessage(g_pg));
PQfinish(g_pg);
g_pg = NULL;
return -1;
}
DEBUG_INFO("pg_inbox: connected to PostgreSQL (%s)",
PQdb(g_pg) ? PQdb(g_pg) : "?");
return 0;
}
void pg_inbox_shutdown(void) {
if (g_pg) {
PQfinish(g_pg);
g_pg = NULL;
DEBUG_INFO("pg_inbox: connection closed");
}
}
/* ------------------------------------------------------------------ */
/* Inbox insert */
/* ------------------------------------------------------------------ */
int pg_inbox_insert_event(const char *event_json, const char *source_relay,
const char *source_class, int priority) {
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return -1;
}
if (!event_json) {
DEBUG_WARN("pg_inbox: insert called with NULL event_json");
return -1;
}
/* Extract the event id from the JSON for the event_id column.
* The schema CHECK requires event_id = event_json->>'id'. */
cJSON *ev = cJSON_Parse(event_json);
if (!ev) {
DEBUG_WARN("pg_inbox: event_json is not valid JSON, skipping");
return -1;
}
cJSON *id_node = cJSON_GetObjectItem(ev, "id");
const char *event_id = NULL;
if (id_node && cJSON_IsString(id_node)) {
event_id = cJSON_GetStringValue(id_node);
}
if (!event_id || strlen(event_id) != 64) {
DEBUG_WARN("pg_inbox: event missing valid 64-char id, skipping");
cJSON_Delete(ev);
return -1;
}
/* priority is an integer; pass as a separate param via a small buffer. */
char prio_buf[16];
snprintf(prio_buf, sizeof(prio_buf), "%d", priority);
const char *vals[5];
int lens[5];
int fmts[5];
/* $1: event_id (text) */
vals[0] = event_id;
lens[0] = (int)strlen(event_id);
fmts[0] = 0;
/* $2: event_json (cast to jsonb in SQL) */
vals[1] = event_json;
lens[1] = (int)strlen(event_json);
fmts[1] = 0;
/* $3: source_relay (text, may be NULL) */
vals[2] = source_relay;
lens[2] = source_relay ? (int)strlen(source_relay) : 0;
fmts[2] = 0;
/* $4: source_class (text) */
vals[3] = source_class ? source_class : "backfill";
lens[3] = (int)strlen(vals[3]);
fmts[3] = 0;
/* $5: priority (smallint, passed as text) */
vals[4] = prio_buf;
lens[4] = (int)strlen(prio_buf);
fmts[4] = 0;
const char *sql =
"INSERT INTO caching_event_inbox "
"(event_id, event_json, source_relay, source_class, priority) "
"VALUES ($1, $2::jsonb, $3, $4, $5) "
"ON CONFLICT (event_id) DO NOTHING";
PGresult *res = PQexecParams(g_pg, sql, 5, NULL, vals, lens, fmts, 0);
cJSON_Delete(ev);
if (!res) {
DEBUG_ERROR("pg_inbox: insert returned NULL result for %s", event_id);
return -1;
}
ExecStatusType st = PQresultStatus(res);
if (st != PGRES_COMMAND_OK) {
DEBUG_WARN("pg_inbox: insert failed for %s: %s",
event_id, PQresultErrorMessage(res));
PQclear(res);
return -1;
}
/* ON CONFLICT DO NOTHING returns COMMAND_OK either way; the row count
* is not directly available without RETURNING, so we treat both
* inserted and conflict-skipped as success (0). */
PQclear(res);
return 0;
}
/* ------------------------------------------------------------------ */
/* Service state update */
/* ------------------------------------------------------------------ */
int pg_inbox_update_status(const char *service_state,
long config_generation,
long heartbeat_at,
int followed_author_count,
int selected_relay_count,
int connected_relay_count,
int current_window_index,
int backfill_cursor,
long events_fetched,
long inbox_inserts,
const char *last_error,
long last_error_at) {
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return -1;
}
char gen_buf[32], hb_buf[32], fac_buf[16], src_buf[16], crc_buf[16];
char cwi_buf[16], bc_buf[16], ef_buf[32], ii_buf[32], lea_buf[32];
snprintf(gen_buf, sizeof(gen_buf), "%ld", config_generation);
snprintf(hb_buf, sizeof(hb_buf), "%ld", heartbeat_at);
snprintf(fac_buf, sizeof(fac_buf), "%d", followed_author_count);
snprintf(src_buf, sizeof(src_buf), "%d", selected_relay_count);
snprintf(crc_buf, sizeof(crc_buf), "%d", connected_relay_count);
snprintf(cwi_buf, sizeof(cwi_buf), "%d", current_window_index);
snprintf(bc_buf, sizeof(bc_buf), "%d", backfill_cursor);
snprintf(ef_buf, sizeof(ef_buf), "%ld", events_fetched);
snprintf(ii_buf, sizeof(ii_buf), "%ld", inbox_inserts);
if (last_error_at > 0) {
snprintf(lea_buf, sizeof(lea_buf), "%ld", last_error_at);
} else {
lea_buf[0] = '\0';
}
const char *vals[12];
int lens[12];
int fmts[12];
int n = 0;
/* $1 service_state (may be NULL) */
vals[n] = service_state; lens[n] = service_state ? (int)strlen(service_state) : 0; fmts[n] = 0; n++;
/* $2 config_generation */
vals[n] = gen_buf; lens[n] = (int)strlen(gen_buf); fmts[n] = 0; n++;
/* $3 heartbeat_at */
vals[n] = hb_buf; lens[n] = (int)strlen(hb_buf); fmts[n] = 0; n++;
/* $4 followed_author_count */
vals[n] = fac_buf; lens[n] = (int)strlen(fac_buf); fmts[n] = 0; n++;
/* $5 selected_relay_count */
vals[n] = src_buf; lens[n] = (int)strlen(src_buf); fmts[n] = 0; n++;
/* $6 connected_relay_count */
vals[n] = crc_buf; lens[n] = (int)strlen(crc_buf); fmts[n] = 0; n++;
/* $7 current_window_index */
vals[n] = cwi_buf; lens[n] = (int)strlen(cwi_buf); fmts[n] = 0; n++;
/* $8 backfill_cursor */
vals[n] = bc_buf; lens[n] = (int)strlen(bc_buf); fmts[n] = 0; n++;
/* $9 events_fetched */
vals[n] = ef_buf; lens[n] = (int)strlen(ef_buf); fmts[n] = 0; n++;
/* $10 inbox_inserts */
vals[n] = ii_buf; lens[n] = (int)strlen(ii_buf); fmts[n] = 0; n++;
/* $11 last_error (may be NULL) */
vals[n] = last_error; lens[n] = last_error ? (int)strlen(last_error) : 0; fmts[n] = 0; n++;
/* $12 last_error_at (empty string -> NULL) */
vals[n] = lea_buf; lens[n] = (int)strlen(lea_buf); fmts[n] = 0; n++;
const char *sql =
"INSERT INTO caching_service_state "
"(id, service_state, config_generation, heartbeat_at, "
" followed_author_count, selected_relay_count, connected_relay_count, "
" current_window_index, backfill_cursor, events_fetched, inbox_inserts, "
" last_error, last_error_at, updated_at) "
"VALUES (1, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, "
" CASE WHEN $12 = '' THEN NULL ELSE $12::BIGINT END, "
" EXTRACT(EPOCH FROM NOW())::BIGINT) "
"ON CONFLICT (id) DO UPDATE SET "
" service_state = COALESCE(EXCLUDED.service_state, caching_service_state.service_state), "
" config_generation = EXCLUDED.config_generation, "
" heartbeat_at = EXCLUDED.heartbeat_at, "
" followed_author_count= EXCLUDED.followed_author_count, "
" selected_relay_count = EXCLUDED.selected_relay_count, "
" connected_relay_count= EXCLUDED.connected_relay_count, "
" current_window_index = EXCLUDED.current_window_index, "
" backfill_cursor = EXCLUDED.backfill_cursor, "
" events_fetched = EXCLUDED.events_fetched, "
" inbox_inserts = EXCLUDED.inbox_inserts, "
" last_error = COALESCE(EXCLUDED.last_error, caching_service_state.last_error), "
" last_error_at = CASE WHEN EXCLUDED.last_error_at IS NULL "
" THEN caching_service_state.last_error_at "
" ELSE EXCLUDED.last_error_at END, "
" updated_at = EXCLUDED.updated_at";
PGresult *res = PQexecParams(g_pg, sql, n, NULL, vals, lens, fmts, 0);
if (!res) {
DEBUG_ERROR("pg_inbox: update_status returned NULL result");
return -1;
}
ExecStatusType st = PQresultStatus(res);
if (st != PGRES_COMMAND_OK) {
DEBUG_ERROR("pg_inbox: update_status failed: %s",
PQresultErrorMessage(res));
PQclear(res);
return -1;
}
PQclear(res);
return 0;
}
/* ------------------------------------------------------------------ */
/* Config reads */
/* ------------------------------------------------------------------ */
char *pg_inbox_get_config_value(const char *key) {
if (!g_pg || !key) return NULL;
const char *vals[1] = { key };
int lens[1] = { (int)strlen(key) };
int fmts[1] = { 0 };
PGresult *res = PQexecParams(g_pg,
"SELECT value FROM config WHERE key = $1",
1, NULL, vals, lens, fmts, 0);
if (!res) return NULL;
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
DEBUG_WARN("pg_inbox: config select failed for '%s': %s",
key, PQresultErrorMessage(res));
PQclear(res);
return NULL;
}
if (PQntuples(res) < 1 || PQgetisnull(res, 0, 0)) {
PQclear(res);
return NULL;
}
char *v = strdup(PQgetvalue(res, 0, 0));
PQclear(res);
return v;
}
long pg_inbox_get_config_generation(void) {
char *v = pg_inbox_get_config_value("caching_config_generation");
if (!v) return -1;
long gen = strtol(v, NULL, 10);
free(v);
return gen;
}
/* ------------------------------------------------------------------ */
/* Backfill progress persistence */
/* ------------------------------------------------------------------ */
int pg_inbox_save_backfill_progress(const char *author_pubkey,
int window_index,
long window_anchor,
long until_cursor,
int complete) {
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return -1;
}
if (!author_pubkey) {
DEBUG_WARN("pg_inbox: save_backfill_progress called with NULL author");
return -1;
}
char wi_buf[16], wa_buf[32], uc_buf[32], cmp_buf[8];
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
snprintf(wa_buf, sizeof(wa_buf), "%ld", window_anchor);
snprintf(uc_buf, sizeof(uc_buf), "%ld", until_cursor);
snprintf(cmp_buf, sizeof(cmp_buf), "%d", complete ? 1 : 0);
const char *vals[5] = { author_pubkey, wi_buf, wa_buf, uc_buf, cmp_buf };
int lens[5] = {
(int)strlen(author_pubkey),
(int)strlen(wi_buf),
(int)strlen(wa_buf),
(int)strlen(uc_buf),
(int)strlen(cmp_buf),
};
int fmts[5] = { 0, 0, 0, 0, 0 };
const char *sql =
"INSERT INTO caching_backfill_progress "
"(author_pubkey, window_index, window_anchor, until_cursor, complete, updated_at) "
"VALUES ($1, $2, $3, $4, $5::BOOLEAN, EXTRACT(EPOCH FROM NOW())::BIGINT) "
"ON CONFLICT (author_pubkey, window_index) DO UPDATE SET "
" until_cursor = EXCLUDED.until_cursor, "
" complete = EXCLUDED.complete, "
" updated_at = EXCLUDED.updated_at";
PGresult *res = PQexecParams(g_pg, sql, 5, NULL, vals, lens, fmts, 0);
if (!res) {
DEBUG_ERROR("pg_inbox: save_backfill_progress returned NULL result");
return -1;
}
ExecStatusType st = PQresultStatus(res);
if (st != PGRES_COMMAND_OK) {
DEBUG_ERROR("pg_inbox: save_backfill_progress failed: %s",
PQresultErrorMessage(res));
PQclear(res);
return -1;
}
PQclear(res);
return 0;
}
int pg_inbox_load_backfill_progress(const char *author_pubkey,
int window_index,
long *out_until_cursor,
int *out_complete,
int *out_found) {
if (out_found) *out_found = 0;
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return -1;
}
if (!author_pubkey) return -1;
char wi_buf[16];
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
const char *vals[2] = { author_pubkey, wi_buf };
int lens[2] = { (int)strlen(author_pubkey), (int)strlen(wi_buf) };
int fmts[2] = { 0, 0 };
const char *sql =
"SELECT until_cursor, complete FROM caching_backfill_progress "
"WHERE author_pubkey = $1 AND window_index = $2";
PGresult *res = PQexecParams(g_pg, sql, 2, NULL, vals, lens, fmts, 0);
if (!res) {
DEBUG_ERROR("pg_inbox: load_backfill_progress returned NULL result");
return -1;
}
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
DEBUG_ERROR("pg_inbox: load_backfill_progress failed: %s",
PQresultErrorMessage(res));
PQclear(res);
return -1;
}
if (PQntuples(res) < 1) {
/* Not found - not an error. */
PQclear(res);
return 0;
}
if (out_until_cursor) {
if (PQgetisnull(res, 0, 0)) {
*out_until_cursor = 0;
} else {
*out_until_cursor = strtol(PQgetvalue(res, 0, 0), NULL, 10);
}
}
if (out_complete) {
if (PQgetisnull(res, 0, 1)) {
*out_complete = 0;
} else {
const char *v = PQgetvalue(res, 0, 1);
/* boolean comes back as 't'/'f' or '1'/'0' depending on output mode. */
*out_complete = (v && (v[0] == 't' || v[0] == 'T' || v[0] == '1')) ? 1 : 0;
}
}
if (out_found) *out_found = 1;
PQclear(res);
return 0;
}
cJSON *pg_inbox_load_incomplete_progress(int window_index) {
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return NULL;
}
char wi_buf[16];
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
const char *vals[1] = { wi_buf };
int lens[1] = { (int)strlen(wi_buf) };
int fmts[1] = { 0 };
const char *sql =
"SELECT author_pubkey, until_cursor, complete FROM caching_backfill_progress "
"WHERE window_index = $1 AND complete = FALSE";
PGresult *res = PQexecParams(g_pg, sql, 1, NULL, vals, lens, fmts, 0);
if (!res) {
DEBUG_ERROR("pg_inbox: load_incomplete_progress returned NULL result");
return NULL;
}
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
DEBUG_ERROR("pg_inbox: load_incomplete_progress failed: %s",
PQresultErrorMessage(res));
PQclear(res);
return NULL;
}
int n = PQntuples(res);
if (n <= 0) {
PQclear(res);
return NULL;
}
cJSON *arr = cJSON_CreateArray();
if (!arr) {
PQclear(res);
return NULL;
}
for (int i = 0; i < n; i++) {
cJSON *obj = cJSON_CreateObject();
if (!obj) continue;
const char *pk = PQgetisnull(res, i, 0) ? "" : PQgetvalue(res, i, 0);
long uc = PQgetisnull(res, i, 1) ? 0 : strtol(PQgetvalue(res, i, 1), NULL, 10);
int cmp = 0;
if (!PQgetisnull(res, i, 1)) {
const char *v = PQgetvalue(res, i, 2);
cmp = (v && (v[0] == 't' || v[0] == 'T' || v[0] == '1')) ? 1 : 0;
}
cJSON_AddStringToObject(obj, "author_pubkey", pk);
cJSON_AddNumberToObject(obj, "until_cursor", (double)uc);
cJSON_AddBoolToObject(obj, "complete", cmp ? 1 : 0);
cJSON_AddItemToArray(arr, obj);
}
PQclear(res);
return arr;
}
int pg_inbox_reset_backfill_progress(void) {
if (!g_pg) {
DEBUG_ERROR("pg_inbox: not initialized");
return -1;
}
PGresult *res = PQexec(g_pg, "DELETE FROM caching_backfill_progress");
if (!res) {
DEBUG_ERROR("pg_inbox: reset_backfill_progress returned NULL result");
return -1;
}
ExecStatusType st = PQresultStatus(res);
if (st != PGRES_COMMAND_OK) {
DEBUG_ERROR("pg_inbox: reset_backfill_progress failed: %s",
PQresultErrorMessage(res));
PQclear(res);
return -1;
}
PQclear(res);
return 0;
}
+88
View File
@@ -0,0 +1,88 @@
/*
* caching_relay - PostgreSQL inbox connection and operations.
*
* Manages the libpq connection to the c-relay-pg PostgreSQL database and
* provides functions to insert fetched events into the caching_event_inbox
* table, update the caching_service_state singleton, and read configuration
* from the shared config table.
*/
#ifndef CACHING_RELAY_PG_INBOX_H
#define CACHING_RELAY_PG_INBOX_H
#include <stddef.h>
/* Forward declaration so the header does not pull in cJSON everywhere. */
typedef struct cJSON cJSON;
/* Initialize PostgreSQL connection using the given connection string.
* Returns 0 on success, -1 on error. */
int pg_inbox_init(const char *connection_string);
/* Close the PostgreSQL connection. */
void pg_inbox_shutdown(void);
/* Insert an event into the caching_event_inbox table.
* event_json is the raw Nostr event JSON string.
* source_relay may be NULL. source_class is "live", "discovery", or "backfill".
* priority is 0 (live/discovery) or 1 (backfill).
* Returns 0 on success (including conflict-skip), -1 on error. */
int pg_inbox_insert_event(const char *event_json, const char *source_relay,
const char *source_class, int priority);
/* Update the caching_service_state singleton row.
* All string parameters may be NULL (left unchanged).
* Returns 0 on success, -1 on error. */
int pg_inbox_update_status(const char *service_state,
long config_generation,
long heartbeat_at,
int followed_author_count,
int selected_relay_count,
int connected_relay_count,
int current_window_index,
int backfill_cursor,
long events_fetched,
long inbox_inserts,
const char *last_error,
long last_error_at);
/* Read a config value from the c-relay-pg config table.
* Returns a malloc'd string (caller must free) or NULL if not found/error. */
char *pg_inbox_get_config_value(const char *key);
/* Read the current config generation.
* Returns the generation number, or -1 on error. */
long pg_inbox_get_config_generation(void);
/* ------------------------------------------------------------------ */
/* Backfill progress persistence */
/* ------------------------------------------------------------------ */
/* Save backfill progress for a specific author+window (UPSERT).
* Returns 0 on success, -1 on error. */
int pg_inbox_save_backfill_progress(const char *author_pubkey,
int window_index,
long window_anchor,
long until_cursor,
int complete);
/* Load backfill progress for a specific author+window.
* Returns 0 on success (found), -1 on error.
* Sets *out_until_cursor and *out_complete. If not found, sets *out_found = 0
* and returns 0. */
int pg_inbox_load_backfill_progress(const char *author_pubkey,
int window_index,
long *out_until_cursor,
int *out_complete,
int *out_found);
/* Load all incomplete backfill progress rows for a given window.
* Returns a cJSON array of objects with "author_pubkey", "until_cursor" and
* "complete" fields. Caller must cJSON_Delete the result.
* Returns NULL on error or empty. */
cJSON *pg_inbox_load_incomplete_progress(int window_index);
/* Reset all backfill progress (delete all rows).
* Returns 0 on success, -1 on error. */
int pg_inbox_reset_backfill_progress(void);
#endif /* CACHING_RELAY_PG_INBOX_H */
+471
View File
@@ -0,0 +1,471 @@
/*
* caching_relay - NIP-65 outbox relay discovery
*/
#define _GNU_SOURCE
#include "relay_discovery.h"
#include "debug.h"
#include <string.h>
#include <stdlib.h>
#include <signal.h>
/* Access the shutdown flag from main.c so we can abort discovery early. */
extern volatile sig_atomic_t g_shutdown;
/* Normalize a relay URL: strip trailing slash for consistency. */
static void normalize_url(char *url, size_t maxlen) {
size_t len = strlen(url);
while (len > 0 && url[len - 1] == '/') {
url[--len] = '\0';
}
(void)maxlen;
}
/* ---- helpers ---- */
static int get_pool_urls(nostr_relay_pool_t *pool, const char ***urls_out) {
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(pool, &listed, &statuses);
const char **urls = NULL;
if (n > 0) {
urls = malloc(n * sizeof(char *));
for (int j = 0; j < n; j++) urls[j] = listed[j];
}
free(listed);
free(statuses);
*urls_out = urls;
return n;
}
/* Query the most recent kind 10002 for a single pubkey from a pool.
* Returns the event (caller must cJSON_Delete) or NULL. */
static cJSON *query_kind10002(nostr_relay_pool_t *pool, const char *hex) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(hex));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = cJSON_CreateArray();
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(10002));
cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber(1));
const char **urls = NULL;
int n = get_pool_urls(pool, &urls);
int ev_count = 0;
cJSON **events = nostr_relay_pool_query_sync(pool, urls, n, filter, &ev_count, 10000);
free(urls);
cJSON_Delete(filter);
cJSON *result = NULL;
if (events && ev_count > 0) {
result = events[0]; /* take first (most recent) */
for (int k = 1; k < ev_count; k++) cJSON_Delete(events[k]);
}
free(events);
return result;
}
/* Parse "r" tags from a kind 10002 event into an outbox entry.
* Only includes relays with "read" marker or no marker (assume both).
* Skips "write" only relays. */
static int parse_r_tags(cJSON *event, cr_outbox_entry_t *entry) {
memset(entry, 0, sizeof(*entry));
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
if (pubkey && cJSON_IsString(pubkey)) {
strncpy(entry->pubkey, cJSON_GetStringValue(pubkey), CR_HEX_LEN - 1);
}
cJSON *tags = cJSON_GetObjectItem(event, "tags");
if (!tags || !cJSON_IsArray(tags)) return 0;
int added = 0;
cJSON *tag;
cJSON_ArrayForEach(tag, tags) {
if (!cJSON_IsArray(tag)) continue;
cJSON *name = cJSON_GetArrayItem(tag, 0);
if (!name || !cJSON_IsString(name)) continue;
if (strcmp(cJSON_GetStringValue(name), "r") != 0) continue;
cJSON *url = cJSON_GetArrayItem(tag, 1);
if (!url || !cJSON_IsString(url)) continue;
const char *relay_url = cJSON_GetStringValue(url);
if (!relay_url || !relay_url[0]) continue;
/* Check marker (3rd element): "read", "write", or absent. */
cJSON *marker = cJSON_GetArrayItem(tag, 2);
if (marker && cJSON_IsString(marker)) {
const char *m = cJSON_GetStringValue(marker);
if (strcmp(m, "write") == 0) continue; /* skip write-only relays */
}
/* No marker or "read" => include. */
if (entry->relay_count >= CR_MAX_RELAYS_PER_PUBKEY) break;
strncpy(entry->relays[entry->relay_count], relay_url, CR_URL_LEN - 1);
entry->relays[entry->relay_count][CR_URL_LEN - 1] = '\0';
normalize_url(entry->relays[entry->relay_count], CR_URL_LEN);
entry->relay_count++;
added++;
}
return added;
}
/* ---- greedy set cover ---- */
/* Build relay -> pubkey coverage map and compute minimum covering set. */
static void compute_covering_set(cr_relay_map_t *map, cr_config_t *cfg,
const cr_pubkey_set_t *followed) {
/* Build a temporary relay -> pubkeys map. */
typedef struct {
char url[CR_URL_LEN];
char pubkeys[CR_MAX_PUBKEYS_PER_RELAY][CR_HEX_LEN];
int pubkey_count;
} relay_cov_t;
relay_cov_t *relays = calloc(CR_MAX_DISCOVERED_RELAYS, sizeof(relay_cov_t));
int relay_count = 0;
/* Always include bootstrap relays in the coverage map. */
for (int i = 0; i < cfg->upstream_count && relay_count < CR_MAX_DISCOVERED_RELAYS; i++) {
strncpy(relays[relay_count].url, cfg->upstream_relays[i], CR_URL_LEN - 1);
relays[relay_count].url[CR_URL_LEN - 1] = '\0';
normalize_url(relays[relay_count].url, CR_URL_LEN);
relay_count++;
}
/* Add outbox relays and build coverage. */
for (int i = 0; i < map->outbox_count; i++) {
cr_outbox_entry_t *oe = &map->outboxes[i];
for (int r = 0; r < oe->relay_count; r++) {
/* Find or create relay entry. */
int found = -1;
for (int j = 0; j < relay_count; j++) {
if (strcmp(relays[j].url, oe->relays[r]) == 0) { found = j; break; }
}
if (found < 0) {
if (relay_count >= CR_MAX_DISCOVERED_RELAYS) break;
strncpy(relays[relay_count].url, oe->relays[r], CR_URL_LEN - 1);
found = relay_count++;
}
/* Add this pubkey to the relay's coverage. */
if (relays[found].pubkey_count < CR_MAX_PUBKEYS_PER_RELAY) {
strncpy(relays[found].pubkeys[relays[found].pubkey_count],
oe->pubkey, CR_HEX_LEN - 1);
relays[found].pubkey_count++;
}
}
}
/* Greedy set cover. */
char *covered = calloc(followed->count, 1); /* 1 if pubkey is covered */
map->selected_count = 0;
map->coverage = calloc(map->outbox_count, sizeof(int));
for (int i = 0; i < map->outbox_count; i++) map->coverage[i] = -1;
int uncovered_count = followed->count;
/* Helper: find index of a pubkey in the followed set. */
/* (linear search, fine for our scale) */
while (uncovered_count > 0 && map->selected_count < CR_MAX_DISCOVERED_RELAYS) {
int best_relay = -1;
int best_count = 0;
for (int r = 0; r < relay_count; r++) {
/* Count how many uncovered pubkeys this relay covers. */
int count = 0;
for (int p = 0; p < relays[r].pubkey_count; p++) {
/* Find this pubkey in followed set. */
for (int i = 0; i < followed->count; i++) {
if (strcmp(followed->items[i], relays[r].pubkeys[p]) == 0) {
if (!covered[i]) count++;
break;
}
}
}
if (count > best_count) {
best_count = count;
best_relay = r;
}
}
if (best_relay < 0 || best_count == 0) break; /* no more coverage possible */
/* Select this relay. */
strncpy(map->selected_relays[map->selected_count],
relays[best_relay].url, CR_URL_LEN - 1);
map->selected_count++;
/* Mark covered pubkeys and record coverage. */
for (int p = 0; p < relays[best_relay].pubkey_count; p++) {
for (int i = 0; i < followed->count; i++) {
if (strcmp(followed->items[i], relays[best_relay].pubkeys[p]) == 0) {
if (!covered[i]) {
covered[i] = 1;
uncovered_count--;
/* Record which selected relay covers this outbox entry. */
for (int oe_i = 0; oe_i < map->outbox_count; oe_i++) {
if (strcmp(map->outboxes[oe_i].pubkey,
relays[best_relay].pubkeys[p]) == 0) {
if (map->coverage[oe_i] < 0) {
map->coverage[oe_i] = map->selected_count - 1;
}
}
}
}
break;
}
}
}
}
/* Log results. */
DEBUG_INFO("relay_discovery: selected %d relays to cover %d/%d pubkeys",
map->selected_count, followed->count - uncovered_count, followed->count);
for (int i = 0; i < map->selected_count; i++) {
DEBUG_INFO(" relay[%d]: %s", i, map->selected_relays[i]);
}
if (uncovered_count > 0) {
DEBUG_WARN("relay_discovery: %d pubkeys have no outbox relays (covered by bootstrap)",
uncovered_count);
}
free(covered);
free(relays);
}
/* ---- batched kind-10002 query ---- */
/* Query kind 10002 for a batch of up to 100 pubkeys from a pool.
* Returns a cJSON array of events (caller must cJSON_Delete each + free).
* Sets *out_count to the number of events returned. */
static cJSON **query_kind10002_batch(nostr_relay_pool_t *pool,
const char **pubkeys, int pubkey_count,
int *out_count) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
for (int i = 0; i < pubkey_count; i++)
cJSON_AddItemToArray(authors, cJSON_CreateString(pubkeys[i]));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = cJSON_CreateArray();
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(10002));
cJSON_AddItemToObject(filter, "kinds", kinds);
const char **urls = NULL;
int n = get_pool_urls(pool, &urls);
/* TRACE: log the query being sent. */
if (g_debug_level >= DEBUG_LEVEL_TRACE) {
char *filter_str = cJSON_PrintUnformatted(filter);
DEBUG_TRACE("query_kind10002_batch: querying %d relays for %d pubkeys", n, pubkey_count);
if (filter_str) {
/* Only print first 200 chars of filter to avoid huge output */
if (strlen(filter_str) > 200) filter_str[200] = '\0';
DEBUG_TRACE(" filter: %s...", filter_str);
free(filter_str);
}
}
int ev_count = 0;
cJSON **events = nostr_relay_pool_query_sync(pool, urls, n, filter, &ev_count, 30000);
free(urls);
cJSON_Delete(filter);
DEBUG_TRACE("query_kind10002_batch: returned %d events", ev_count);
*out_count = ev_count;
return events;
}
/* Process a batch of kind-10002 events: parse r-tags, populate outbox entries,
* publish to local relay. Returns count of events processed. */
static int process_10002_batch(cJSON **events, int ev_count,
cr_relay_map_t *map, cr_sink_t *sink,
const char *source) {
int processed = 0;
for (int k = 0; k < ev_count; k++) {
cJSON *event = events[k];
if (!event) continue;
/* Get pubkey from event. */
cJSON *pk = cJSON_GetObjectItem(event, "pubkey");
if (!pk || !cJSON_IsString(pk)) { cJSON_Delete(event); continue; }
const char *hex = cJSON_GetStringValue(pk);
/* Find or create outbox entry for this pubkey. */
cr_outbox_entry_t *oe = NULL;
for (int i = 0; i < map->outbox_count; i++) {
if (strcmp(map->outboxes[i].pubkey, hex) == 0) { oe = &map->outboxes[i]; break; }
}
if (!oe) {
if (map->outbox_count >= /* followed->count passed in via map size */ 99999) {
cJSON_Delete(event); continue;
}
oe = &map->outboxes[map->outbox_count];
strncpy(oe->pubkey, hex, CR_HEX_LEN - 1);
map->outbox_count++;
}
int n = parse_r_tags(event, oe);
(void)n;
if (strcmp(source, "bootstrap") == 0) {
cr_sink_publish(sink, event);
/* Pump sink pool every 16 events to avoid pending publish overflow
* (NOSTR_POOL_MAX_PENDING_PUBLISHES = 32). */
if (processed > 0 && (processed % 16) == 0) {
cr_sink_pump(sink, 200);
}
}
processed++;
cJSON_Delete(event);
}
free(events);
/* Log sink stats after each batch. */
DEBUG_INFO("relay_discovery: batch processed %d events (sink: ok=%ld reject=%ld dup=%ld err=%ld)",
processed, sink->published_ok, sink->published_failed,
sink->published_dup, 0L);
return processed;
}
/* ---- main discovery function ---- */
#define CR_10002_BATCH_SIZE 100
int cr_relay_discovery_run(cr_relay_map_t *map, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_sink_t *sink,
const cr_pubkey_set_t *followed, int is_first_time) {
memset(map, 0, sizeof(*map));
map->outboxes = calloc(followed->count, sizeof(cr_outbox_entry_t));
if (!map->outboxes) return -1;
map->outbox_count = 0;
DEBUG_INFO("relay_discovery: discovering outbox relays for %d pubkeys (%s)",
followed->count, is_first_time ? "first-time" : "local-first");
int found_local = 0, found_bootstrap = 0, found_none = 0;
/* Pre-populate outbox entries for all followed pubkeys (empty, for coverage
* tracking of pubkeys with no 10002). */
for (int i = 0; i < followed->count; i++) {
strncpy(map->outboxes[i].pubkey, followed->items[i], CR_HEX_LEN - 1);
}
map->outbox_count = followed->count;
/* Track which pubkeys we've found 10002 for. */
char *found = calloc(followed->count, 1);
/* Phase 1: If subsequent startup, batch-query local relay first. */
if (!is_first_time && sink && sink->pool) {
DEBUG_INFO("relay_discovery: querying local relay for kind-10002 (batched)...");
for (int start = 0; start < followed->count && !g_shutdown; start += CR_10002_BATCH_SIZE) {
int batch = followed->count - start;
if (batch > CR_10002_BATCH_SIZE) batch = CR_10002_BATCH_SIZE;
int ev_count = 0;
cJSON **events = query_kind10002_batch(sink->pool,
(const char **)&followed->items[start],
batch, &ev_count);
if (events && ev_count > 0) {
for (int k = 0; k < ev_count; k++) {
cJSON *pk = cJSON_GetObjectItem(events[k], "pubkey");
if (pk && cJSON_IsString(pk)) {
const char *hex = cJSON_GetStringValue(pk);
for (int i = 0; i < followed->count; i++) {
if (strcmp(followed->items[i], hex) == 0) {
if (!found[i]) { found[i] = 1; found_local++; }
break;
}
}
}
}
process_10002_batch(events, ev_count, map, sink, "local");
/* Pump sink pool to flush published events. */
cr_sink_pump(sink, 500);
} else {
free(events);
}
}
DEBUG_INFO("relay_discovery: local relay provided %d kind-10002 events", found_local);
}
/* Phase 2: Batch-query bootstrap relays for pubkeys not found locally. */
int remaining = 0;
for (int i = 0; i < followed->count; i++) if (!found[i]) remaining++;
if (remaining > 0) {
DEBUG_INFO("relay_discovery: querying bootstrap relays for %d remaining pubkeys (batched)...",
remaining);
/* Build list of remaining pubkeys. */
const char **remaining_pk = malloc(remaining * sizeof(char *));
int ridx = 0;
for (int i = 0; i < followed->count; i++) {
if (!found[i]) remaining_pk[ridx++] = followed->items[i];
}
for (int start = 0; start < remaining && !g_shutdown; start += CR_10002_BATCH_SIZE) {
int batch = remaining - start;
if (batch > CR_10002_BATCH_SIZE) batch = CR_10002_BATCH_SIZE;
int ev_count = 0;
cJSON **events = query_kind10002_batch(upstream,
&remaining_pk[start],
batch, &ev_count);
if (events && ev_count > 0) {
for (int k = 0; k < ev_count; k++) {
cJSON *pk = cJSON_GetObjectItem(events[k], "pubkey");
if (pk && cJSON_IsString(pk)) {
const char *hex = cJSON_GetStringValue(pk);
for (int i = 0; i < followed->count; i++) {
if (strcmp(followed->items[i], hex) == 0) {
if (!found[i]) { found[i] = 1; found_bootstrap++; }
break;
}
}
}
}
process_10002_batch(events, ev_count, map, sink, "bootstrap");
/* Pump sink pool to flush published events. */
cr_sink_pump(sink, 500);
} else {
free(events);
}
}
free(remaining_pk);
}
found_none = followed->count - found_local - found_bootstrap;
DEBUG_INFO("relay_discovery: %d local, %d bootstrap, %d none",
found_local, found_bootstrap, found_none);
free(found);
/* Compute minimum covering set. */
compute_covering_set(map, cfg, followed);
return 0;
}
void cr_relay_map_free(cr_relay_map_t *map) {
if (map->outboxes) {
free(map->outboxes);
map->outboxes = NULL;
}
if (map->coverage) {
free(map->coverage);
map->coverage = NULL;
}
map->outbox_count = 0;
map->selected_count = 0;
}
const cr_outbox_entry_t *cr_relay_map_get_outbox(const cr_relay_map_t *map, const char *hex) {
for (int i = 0; i < map->outbox_count; i++) {
if (strcmp(map->outboxes[i].pubkey, hex) == 0) return &map->outboxes[i];
}
return NULL;
}
+67
View File
@@ -0,0 +1,67 @@
/*
* caching_relay - NIP-65 outbox relay discovery.
*
* For each followed pubkey, fetches their kind 10002 (relay list) and parses
* the "r" tags to build a per-pubkey outbox relay map. Then computes the
* minimum covering set of relays (greedy set cover) that covers all followed
* pubkeys.
*
* On first-time startup, queries bootstrap relays for kind 10002.
* On subsequent startups, queries the local relay first (fast), falls back
* to bootstrap relays for any pubkey not found locally.
*/
#ifndef CACHING_RELAY_RELAY_DISCOVERY_H
#define CACHING_RELAY_RELAY_DISCOVERY_H
#include "config.h"
#include "state.h"
#include "relay_sink.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
#define CR_MAX_RELAYS_PER_PUBKEY 8
#define CR_MAX_PUBKEYS_PER_RELAY 512
#define CR_MAX_DISCOVERED_RELAYS 128
/* Per-pubkey outbox relay list. */
typedef struct {
char pubkey[CR_HEX_LEN];
char relays[CR_MAX_RELAYS_PER_PUBKEY][CR_URL_LEN];
int relay_count;
} cr_outbox_entry_t;
/* Result of relay discovery. */
typedef struct {
cr_outbox_entry_t *outboxes; /* per-pubkey relay lists (heap) */
int outbox_count;
/* The selected minimum covering set of relay URLs. */
char selected_relays[CR_MAX_DISCOVERED_RELAYS][CR_URL_LEN];
int selected_count;
/* Per-pubkey: which selected relay covers this pubkey (index into selected_relays).
* -1 means covered by bootstrap relays only. */
int *coverage; /* heap, outbox_count entries */
} cr_relay_map_t;
/* Discover outbox relays for all followed pubkeys.
*
* cfg - config (provides bootstrap relays, local_relay, root_hex)
* upstream - upstream pool (has bootstrap relays; discovered relays will be added)
* sink - sink pool (local relay, for publishing 10002 events + querying on subsequent startup)
* followed - the followed pubkey set
* is_first_time - 1 if first startup (query bootstrap only), 0 if subsequent (local-first)
*
* Returns 0 on success, -1 on hard error. On success, map is populated.
* Caller must call cr_relay_map_free() when done.
*/
int cr_relay_discovery_run(cr_relay_map_t *map, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_sink_t *sink,
const cr_pubkey_set_t *followed, int is_first_time);
/* Free heap resources in a relay map. */
void cr_relay_map_free(cr_relay_map_t *map);
/* Get the outbox relays for a specific pubkey. Returns NULL if not found. */
const cr_outbox_entry_t *cr_relay_map_get_outbox(const cr_relay_map_t *map, const char *hex);
#endif /* CACHING_RELAY_RELAY_DISCOVERY_H */
+161
View File
@@ -0,0 +1,161 @@
/*
* caching_relay - relay sink: publishes fetched events to the local relay.
*
* See relay_sink.h for the dual-mode (WebSocket / PostgreSQL inbox) design.
*/
#define _GNU_SOURCE
#include "relay_sink.h"
#include "pg_inbox.h"
#include "debug.h"
#include <string.h>
#include <stdlib.h>
/* ------------------------------------------------------------------ */
/* WebSocket mode callback */
/* ------------------------------------------------------------------ */
static void sink_publish_cb(const char *relay_url, const char *event_id,
int success, const char *message, void *user_data) {
cr_sink_t *sink = (cr_sink_t *)user_data;
if (success) {
sink->published_ok++;
DEBUG_TRACE("sink: OK %s from %s", event_id ? event_id : "?",
relay_url ? relay_url : "?");
} else {
sink->published_failed++;
/* REJECT is important - show at INFO level so it's visible at level 3. */
DEBUG_INFO("sink: REJECT %s from %s: %s", event_id ? event_id : "?",
relay_url ? relay_url : "?",
message ? message : "(no message)");
}
}
/* ------------------------------------------------------------------ */
/* Init */
/* ------------------------------------------------------------------ */
int cr_sink_init(cr_sink_t *sink, const char *local_relay_url, cr_seen_ring_t *seen) {
memset(sink, 0, sizeof(*sink));
sink->seen = seen;
sink->source_class = CR_SINK_CLASS_LIVE;
strncpy(sink->url, local_relay_url, sizeof(sink->url) - 1);
sink->pool = nostr_relay_pool_create(nostr_pool_reconnect_config_default());
if (!sink->pool) {
DEBUG_ERROR("sink: failed to create pool");
return -1;
}
if (nostr_relay_pool_add_relay(sink->pool, local_relay_url) != NOSTR_SUCCESS) {
DEBUG_ERROR("sink: failed to add relay %s", local_relay_url);
nostr_relay_pool_destroy(sink->pool);
sink->pool = NULL;
return -1;
}
DEBUG_INFO("sink: initialized (WebSocket) for %s", local_relay_url);
return 0;
}
int cr_sink_init_pg(cr_sink_t *sink, cr_seen_ring_t *seen) {
memset(sink, 0, sizeof(*sink));
sink->seen = seen;
sink->source_class = CR_SINK_CLASS_LIVE;
sink->pool = NULL; /* PostgreSQL inbox mode - no WebSocket pool. */
DEBUG_INFO("sink: initialized (PostgreSQL inbox)");
return 0;
}
void cr_sink_set_source_class(cr_sink_t *sink, const char *class_ptr) {
if (sink) sink->source_class = class_ptr ? class_ptr : CR_SINK_CLASS_LIVE;
}
void cr_sink_destroy(cr_sink_t *sink) {
if (sink && sink->pool) {
nostr_relay_pool_destroy(sink->pool);
sink->pool = NULL;
}
}
/* ------------------------------------------------------------------ */
/* Publish */
/* ------------------------------------------------------------------ */
int cr_sink_publish_from(cr_sink_t *sink, cJSON *event, const char *source_relay) {
if (!sink || !event) return -1;
cJSON *id = cJSON_GetObjectItem(event, "id");
if (!id || !cJSON_IsString(id)) {
DEBUG_WARN("sink: event missing id, skipping");
return -1;
}
const char *eid = cJSON_GetStringValue(id);
if (cr_seen_ring_add(sink->seen, eid) == 0) {
sink->published_dup++;
return 0; /* already seen recently */
}
/* PostgreSQL inbox mode. */
if (!sink->pool) {
char *json = cJSON_PrintUnformatted(event);
if (!json) {
DEBUG_WARN("sink: failed to serialize event %s", eid);
sink->published_failed++;
return -1;
}
const char *klass = sink->source_class ? sink->source_class
: CR_SINK_CLASS_LIVE;
int priority = (strcmp(klass, CR_SINK_CLASS_BACKFILL) == 0) ? 1 : 0;
cJSON *kind = cJSON_GetObjectItem(event, "kind");
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
int kind_num = (kind && cJSON_IsNumber(kind)) ? (int)cJSON_GetNumberValue(kind) : -1;
DEBUG_TRACE("sink: INBOX event id=%s kind=%d pubkey=%s class=%s",
eid, kind_num,
pubkey && cJSON_IsString(pubkey) ? cJSON_GetStringValue(pubkey) : "?",
klass);
int rc = pg_inbox_insert_event(json, source_relay, klass, priority);
free(json);
if (rc != 0) {
sink->published_failed++;
return -1;
}
sink->published_ok++;
return 1;
}
/* WebSocket mode. */
const char *urls[1] = { sink->url };
cJSON *kind = cJSON_GetObjectItem(event, "kind");
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
int kind_num = (kind && cJSON_IsNumber(kind)) ? (int)cJSON_GetNumberValue(kind) : -1;
DEBUG_TRACE("sink: SEND event id=%s kind=%d pubkey=%s to %s",
eid, kind_num,
pubkey && cJSON_IsString(pubkey) ? cJSON_GetStringValue(pubkey) : "?",
sink->url);
int rc = nostr_relay_pool_publish_async(sink->pool, urls, 1, event,
sink_publish_cb, sink);
if (rc < 0) {
DEBUG_WARN("sink: publish_async error rc=%d for %s", rc, eid);
return -1;
}
if (rc == 0) {
DEBUG_INFO("sink: publish SKIPPED (not connected) for %s", eid);
return 0;
}
DEBUG_TRACE("sink: publish_async rc=%d for %s", rc, eid);
return 1;
}
int cr_sink_publish(cr_sink_t *sink, cJSON *event) {
return cr_sink_publish_from(sink, event, NULL);
}
void cr_sink_pump(cr_sink_t *sink, int timeout_ms) {
if (sink && sink->pool) {
nostr_relay_pool_run(sink->pool, timeout_ms);
}
/* No-op in PostgreSQL inbox mode. */
}
+75
View File
@@ -0,0 +1,75 @@
/*
* caching_relay - relay sink: publishes fetched events to the local relay.
*
* Two modes:
* - WebSocket mode (legacy): a dedicated single-relay pool publishes via
* nostr_relay_pool_publish_async(). Used when no PostgreSQL connection
* string is provided.
* - PostgreSQL inbox mode: events are inserted directly into the
* caching_event_inbox table via pg_inbox_insert_event(). Used when the
* daemon is started with --pg-conn.
*
* The public function signatures are identical in both modes so that callers
* (live_subscriber, backfill, relay_discovery) do not need to change.
*/
#ifndef CACHING_RELAY_RELAY_SINK_H
#define CACHING_RELAY_RELAY_SINK_H
#include "state.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
/* Source class labels for the inbox priority field. */
#define CR_SINK_CLASS_LIVE "live"
#define CR_SINK_CLASS_DISCOVERY "discovery"
#define CR_SINK_CLASS_BACKFILL "backfill"
typedef struct {
/* WebSocket mode only (NULL in PostgreSQL inbox mode). */
nostr_relay_pool_t *pool;
char url[256];
/* PostgreSQL inbox mode: source class used for priority assignment.
* Defaults to "live". Set to "backfill" by the backfill module.
* Ignored in WebSocket mode. */
const char *source_class;
cr_seen_ring_t *seen; /* shared seen ring for dedup */
long published_ok;
long published_failed;
long published_dup;
} cr_sink_t;
/* Create the sink.
*
* In WebSocket mode (pg_mode = 0): creates a relay pool and adds
* local_relay_url.
* In PostgreSQL inbox mode (pg_mode != 0): no pool is created; the URL is
* ignored. Events are inserted via pg_inbox_insert_event() (the pg_inbox
* module must be initialized beforehand).
*
* The original two-argument signature is preserved via cr_sink_init() below
* for backward compatibility; it defaults to WebSocket mode. */
int cr_sink_init(cr_sink_t *sink, const char *local_relay_url, cr_seen_ring_t *seen);
/* Initialize the sink in PostgreSQL inbox mode. */
int cr_sink_init_pg(cr_sink_t *sink, cr_seen_ring_t *seen);
/* Set the source class for priority assignment in PostgreSQL inbox mode.
* class_ptr must point to a static string literal (e.g. CR_SINK_CLASS_LIVE). */
void cr_sink_set_source_class(cr_sink_t *sink, const char *class_ptr);
void cr_sink_destroy(cr_sink_t *sink);
/* Publish a cJSON event. Dedups against the seen ring.
* Returns 1 if enqueued/inserted, 0 if dup (skipped), -1 on error. */
int cr_sink_publish(cr_sink_t *sink, cJSON *event);
/* Variant that records the originating relay URL in PostgreSQL inbox mode.
* In WebSocket mode the source_relay is ignored. */
int cr_sink_publish_from(cr_sink_t *sink, cJSON *event, const char *source_relay);
/* Pump the sink pool briefly to flush pending publish callbacks.
* No-op in PostgreSQL inbox mode. */
void cr_sink_pump(cr_sink_t *sink, int timeout_ms);
#endif /* CACHING_RELAY_RELAY_SINK_H */
+76
View File
@@ -0,0 +1,76 @@
/*
* caching_relay - in-memory runtime state
*/
#define _GNU_SOURCE
#include "state.h"
#include <stdlib.h>
#include <string.h>
/* ---- pubkey set ---- */
void cr_pubkey_set_init(cr_pubkey_set_t *s) {
s->items = NULL;
s->count = 0;
s->capacity = 0;
}
void cr_pubkey_set_free(cr_pubkey_set_t *s) {
for (int i = 0; i < s->count; i++) free(s->items[i]);
free(s->items);
s->items = NULL;
s->count = 0;
s->capacity = 0;
}
void cr_pubkey_set_clear(cr_pubkey_set_t *s) {
cr_pubkey_set_free(s);
cr_pubkey_set_init(s);
}
int cr_pubkey_set_contains(const cr_pubkey_set_t *s, const char *hex) {
for (int i = 0; i < s->count; i++) {
if (strcmp(s->items[i], hex) == 0) return 1;
}
return 0;
}
int cr_pubkey_set_add(cr_pubkey_set_t *s, const char *hex) {
if (!hex || !hex[0]) return 0;
if (cr_pubkey_set_contains(s, hex)) return 0;
if (s->count >= s->capacity) {
int newcap = s->capacity ? s->capacity * 2 : 64;
char **ni = realloc(s->items, newcap * sizeof(char *));
if (!ni) return -1;
s->items = ni;
s->capacity = newcap;
}
s->items[s->count] = strdup(hex);
if (!s->items[s->count]) return -1;
s->count++;
return 1;
}
/* ---- seen-event ring buffer ---- */
void cr_seen_ring_init(cr_seen_ring_t *r) {
memset(r, 0, sizeof(*r));
}
int cr_seen_ring_contains(const cr_seen_ring_t *r, const char *id) {
for (int i = 0; i < r->count; i++) {
int idx = (r->head - 1 - i + CR_SEEN_RING_SIZE) % CR_SEEN_RING_SIZE;
if (strcmp(r->ids[idx], id) == 0) return 1;
}
return 0;
}
int cr_seen_ring_add(cr_seen_ring_t *r, const char *id) {
if (!id || !id[0]) return 0;
if (cr_seen_ring_contains(r, id)) return 0;
strncpy(r->ids[r->head], id, CR_HEX_LEN - 1);
r->ids[r->head][CR_HEX_LEN - 1] = '\0';
r->head = (r->head + 1) % CR_SEEN_RING_SIZE;
if (r->count < CR_SEEN_RING_SIZE) r->count++;
return 1;
}
+41
View File
@@ -0,0 +1,41 @@
/*
* caching_relay - in-memory runtime state:
* - followed-pubkey set (hex pubkeys)
* - seen-event-id ring buffer (to avoid redundant publish attempts)
*/
#ifndef CACHING_RELAY_STATE_H
#define CACHING_RELAY_STATE_H
#include <stddef.h>
/* Hex pubkey is 64 chars + NUL. Event id is 64 hex chars + NUL. */
#define CR_HEX_LEN 65
/* A simple dynamic string set for followed pubkeys. */
typedef struct {
char **items;
int count;
int capacity;
} cr_pubkey_set_t;
void cr_pubkey_set_init(cr_pubkey_set_t *s);
void cr_pubkey_set_free(cr_pubkey_set_t *s);
int cr_pubkey_set_add(cr_pubkey_set_t *s, const char *hex); /* 1 if added, 0 if dup */
int cr_pubkey_set_contains(const cr_pubkey_set_t *s, const char *hex);
void cr_pubkey_set_clear(cr_pubkey_set_t *s);
/* Fixed-size ring buffer of event ids for dedup. */
#define CR_SEEN_RING_SIZE 4096
typedef struct {
char ids[CR_SEEN_RING_SIZE][CR_HEX_LEN];
int head;
int count;
} cr_seen_ring_t;
void cr_seen_ring_init(cr_seen_ring_t *r);
/* Returns 1 if the id was newly inserted, 0 if it was already present. */
int cr_seen_ring_add(cr_seen_ring_t *r, const char *id);
int cr_seen_ring_contains(const cr_seen_ring_t *r, const char *id);
#endif /* CACHING_RELAY_STATE_H */
+1
View File
@@ -0,0 +1 @@
key,value,data_type,description,category,requires_restart,created_at,updated_at
1 key value data_type description category requires_restart created_at updated_at
+5 -12
View File
@@ -1,19 +1,12 @@
#!/bin/bash
# Restart the service
sudo systemctl stop c-relay-pg.service
# Copy the binary to the deployment location
cp build/c_relay_x86 ~/Storage/c_relay/crelay
# Copy the service file to systemd (use the main service file)
sudo cp systemd/c-relay.service /etc/systemd/system/c-relay-local.service
# Reload systemd daemon to pick up the new service
sudo systemctl daemon-reload
# Enable the service (if not already enabled)
sudo systemctl enable c-relay-local.service
cp build/c_relay_pg_static_x86_64 ~/Storage/c_relay_pg/crelay
# Restart the service
sudo systemctl restart c-relay-local.service
sudo systemctl restart c-relay-pg.service
# Show service status
sudo systemctl status c-relay-local.service --no-pager -l
sudo systemctl status c-relay-pg.service --no-pager -l
+88 -18
View File
@@ -1,28 +1,98 @@
#!/bin/bash
# C-Relay Static Binary Deployment Script
# Deploys build/c_relay_static_x86_64 to server via ssh
# C-Relay-PG PostgreSQL Deployment Script
# Deploys static relay binary and configures PostgreSQL 18 + systemd service on remote server.
set -e
set -euo pipefail
# Configuration
LOCAL_BINARY="build/c_relay_static_x86_64"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay/c_relay"
SERVICE_NAME="c-relay"
REMOTE_HOST="ubuntu@laantungir.net"
LOCAL_BINARY="build/c_relay_pg_static_x86_64"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
SERVICE_NAME="c-relay-pg"
# Create backup
ssh ubuntu@laantungir.com "sudo cp '$REMOTE_BINARY_PATH' '${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)'" 2>/dev/null || true
LOCAL_SERVICE_FILE="systemd/c-relay.service"
LOCAL_PG_SETUP_SCRIPT="systemd/setup_postgres_18.sh"
# Upload binary to temp location
scp "$LOCAL_BINARY" "ubuntu@laantungir.com:/tmp/c_relay.tmp"
if [ ! -f "$LOCAL_BINARY" ]; then
echo "ERROR: Binary not found: $LOCAL_BINARY"
echo "Build it first (e.g. ./make_and_restart_relay.sh)"
exit 1
fi
# Install binary
ssh ubuntu@laantungir.com "sudo mv '/tmp/c_relay.tmp' '$REMOTE_BINARY_PATH'"
ssh ubuntu@laantungir.com "sudo chown c-relay:c-relay '$REMOTE_BINARY_PATH'"
ssh ubuntu@laantungir.com "sudo chmod +x '$REMOTE_BINARY_PATH'"
if [ ! -f "$LOCAL_SERVICE_FILE" ]; then
echo "ERROR: Service file not found: $LOCAL_SERVICE_FILE"
exit 1
fi
# Reload systemd and restart service
ssh ubuntu@laantungir.com "sudo systemctl daemon-reload"
ssh ubuntu@laantungir.com "sudo systemctl restart '$SERVICE_NAME'"
if [ ! -f "$LOCAL_PG_SETUP_SCRIPT" ]; then
echo "ERROR: PostgreSQL setup script not found: $LOCAL_PG_SETUP_SCRIPT"
exit 1
fi
echo "Deployment complete!"
echo "==> Uploading artifacts to $REMOTE_HOST"
scp "$LOCAL_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg.tmp"
scp "$LOCAL_SERVICE_FILE" "$REMOTE_HOST:/tmp/c-relay-pg.service"
scp "$LOCAL_PG_SETUP_SCRIPT" "$REMOTE_HOST:/tmp/setup_postgres_18.sh"
echo "==> Running remote install/configuration"
ssh "$REMOTE_HOST" 'bash -s' <<'EOF'
set -euo pipefail
SERVICE_NAME="c-relay-pg"
RELAY_USER="c-relay-pg"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
echo "[remote] Ensuring service user exists"
if ! id "$RELAY_USER" >/dev/null 2>&1; then
sudo useradd --system --home-dir /opt/c-relay-pg --shell /usr/sbin/nologin "$RELAY_USER"
fi
echo "[remote] Ensuring required directories exist"
sudo mkdir -p "$REMOTE_BINARY_DIR" /opt/c-relay-pg /etc/c-relay-pg
sudo chown "$RELAY_USER:$RELAY_USER" /opt/c-relay-pg
echo "[remote] Installing PostgreSQL 18 (PGDG) if missing"
if ! dpkg -s postgresql-18 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y curl ca-certificates lsb-release gnupg
sudo install -d /usr/share/postgresql-common/pgdg
sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null
sudo apt-get update
sudo apt-get install -y postgresql-18
fi
sudo systemctl enable postgresql
sudo systemctl restart postgresql
echo "[remote] Configuring PostgreSQL role/database"
sudo chmod +x /tmp/setup_postgres_18.sh
sudo /tmp/setup_postgres_18.sh
echo "[remote] Installing relay binary"
if [ -f "$REMOTE_BINARY_PATH" ]; then
sudo cp "$REMOTE_BINARY_PATH" "${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)"
fi
sudo mv /tmp/c_relay_pg.tmp "$REMOTE_BINARY_PATH"
sudo chown "$RELAY_USER:$RELAY_USER" "$REMOTE_BINARY_PATH"
sudo chmod +x "$REMOTE_BINARY_PATH"
echo "[remote] Installing systemd unit"
sudo mv /tmp/c-relay-pg.service /etc/systemd/system/c-relay-pg.service
sudo chown root:root /etc/systemd/system/c-relay-pg.service
sudo chmod 644 /etc/systemd/system/c-relay-pg.service
echo "[remote] Reloading and restarting service"
sudo systemctl daemon-reload
sudo systemctl enable "$SERVICE_NAME"
sudo systemctl restart "$SERVICE_NAME"
echo "[remote] Health checks"
sudo systemctl --no-pager --full status "$SERVICE_NAME" | sed -n '1,25p'
sudo -u "$RELAY_USER" psql -d crelay -c "SELECT current_user, current_database();"
EOF
echo "Deployment complete: $REMOTE_HOST"
+122
View File
@@ -0,0 +1,122 @@
#!/bin/bash
# C-Relay-PG Debug Binary Deployment Script
# Deploys build/c_relay_pg_static_x86_64_debug to server for CPU profiling
#
# Usage:
# ./deploy_lt_debug.sh -- deploy debug binary and restart
# ./deploy_lt_debug.sh --profile -- deploy, then run perf and fetch results
#
# After deploying, profile with:
# sudo perf record -g -p $(pgrep c_relay_pg) -- sleep 30
# sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -80
#
# Restore production binary:
# ./deploy_lt.sh
set -e
LOCAL_DEBUG_BINARY="build/c_relay_pg_static_x86_64_debug"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
REMOTE_PROD_BACKUP="/usr/local/bin/c_relay_pg/c_relay_pg.production"
SERVICE_NAME="c-relay-pg"
REMOTE_HOST="ubuntu@laantungir.com"
# Check debug binary exists
if [ ! -f "$LOCAL_DEBUG_BINARY" ]; then
echo "ERROR: Debug binary not found: $LOCAL_DEBUG_BINARY"
echo ""
echo "Build it first with:"
echo " ./build_static.sh --debug"
echo ""
exit 1
fi
echo "=========================================="
echo "C-Relay-PG Debug Deployment"
echo "=========================================="
echo "Binary: $LOCAL_DEBUG_BINARY ($(du -h "$LOCAL_DEBUG_BINARY" | cut -f1))"
echo "Target: $REMOTE_HOST:$REMOTE_BINARY_PATH"
echo ""
echo "WARNING: Debug binary has symbols and is larger than production."
echo " It is safe to run but should not be left deployed long-term."
echo ""
# Backup production binary (only if not already backed up)
echo "Backing up production binary..."
ssh "$REMOTE_HOST" "
if [ ! -f '$REMOTE_PROD_BACKUP' ]; then
sudo cp '$REMOTE_BINARY_PATH' '$REMOTE_PROD_BACKUP'
echo 'Production binary backed up to $REMOTE_PROD_BACKUP'
else
echo 'Production backup already exists, skipping'
fi
"
# Upload debug binary
echo "Uploading debug binary..."
scp "$LOCAL_DEBUG_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg_debug.tmp"
# Install debug binary
echo "Installing debug binary..."
ssh "$REMOTE_HOST" "
sudo mv '/tmp/c_relay_pg_debug.tmp' '$REMOTE_BINARY_PATH'
sudo chown c-relay-pg:c-relay-pg '$REMOTE_BINARY_PATH'
sudo chmod +x '$REMOTE_BINARY_PATH'
"
# Restart service
echo "Restarting c-relay-pg service..."
ssh "$REMOTE_HOST" "sudo systemctl daemon-reload && sudo systemctl restart '$SERVICE_NAME'"
echo ""
echo "✓ Debug binary deployed and service restarted"
echo ""
# If --profile flag, run perf automatically
if [ "$1" = "--profile" ]; then
echo "=========================================="
echo "Running perf profile (30 seconds)..."
echo "=========================================="
echo ""
# Wait for relay to start
sleep 3
ssh "$REMOTE_HOST" "
PID=\$(pgrep c_relay_pg)
if [ -z \"\$PID\" ]; then
echo 'ERROR: c_relay_pg not running'
exit 1
fi
echo \"Profiling PID \$PID for 30 seconds...\"
sudo perf record -g -p \$PID -- sleep 30
echo ''
echo '=== TOP FUNCTIONS BY CPU ==='
sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -60
"
else
echo "=========================================="
echo "Next Steps: Profile the relay"
echo "=========================================="
echo ""
echo "SSH to server and run:"
echo " sudo perf record -g -p \$(pgrep c_relay_pg) -- sleep 30"
echo " sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -60"
echo ""
echo "Or run with --profile to do it automatically:"
echo " ./deploy_lt_debug.sh --profile"
fi
echo ""
echo "=========================================="
echo "Restore Production Binary When Done"
echo "=========================================="
echo ""
echo "Run this to restore the production binary:"
echo " ./deploy_lt.sh"
echo ""
echo "Or manually on the server:"
echo " sudo cp '$REMOTE_PROD_BACKUP' '$REMOTE_BINARY_PATH'"
echo " sudo systemctl restart $SERVICE_NAME"
echo ""
+2 -2
View File
@@ -1,4 +1,4 @@
# C-Relay Administrator API Implementation Plan
# C-Relay-PG Administrator API Implementation Plan
## Problem Analysis
@@ -161,7 +161,7 @@ Would require changing schema, migration scripts, and storage logic.
#### README.md Documentation Format:
```markdown
# C-Relay Administrator API
# C-Relay-PG Administrator API
## Authentication
All admin commands require signing with the admin private key generated during first startup.
+190
View File
@@ -0,0 +1,190 @@
# Agent Browser Testing Guide for C-Relay-PG
This document explains how to use the `agent-browser` CLI tool to test the c-relay-pg admin web UI from an AI agent context (no physical display required).
## Prerequisites
- **agent-browser** installed globally: `npm install -g agent-browser`
- Binary location: `/home/user/.nvm/versions/node/v24.14.1/bin/agent-browser`
- The relay must be running locally (default port 8888)
- Test keys configured in `.test_keys`
## Starting the Relay for Local Testing
```bash
./make_and_restart_relay.sh -t
```
This reads `.test_keys` and starts the relay with:
- `ADMIN_PUBKEY` — the hex public key of the admin account
- `ADMIN_PRIVKEY` — the hex secret key (used for browser login, not by the relay itself)
- `SERVER_PRIVKEY` — the relay's own private key
## Key URLs
| URL | Purpose |
|-----|---------|
| `http://127.0.0.1:8888/api/index.html` | Admin web UI (the correct entry point) |
| `http://127.0.0.1:8888/` | Returns 406 without NIP-11 Accept header — **do not use for browser testing** |
| `http://127.0.0.1:8888/` with `Accept: application/nostr+json` | NIP-11 relay info JSON |
**Important:** The root URL `/` is a WebSocket/NIP-11 endpoint, not an HTML page. Always use `/api/index.html` for browser testing.
## Admin Login Credentials
The admin nsec for the current `.test_keys` configuration:
```
nsec: nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm
hex: 15a6fbfd759330c9948ba07d9ed77e49d414c4377a2516d47fc0ab0d54d411f5
npub: npub1tlyzk6slzt8d989f4pn3synk98fea64ea3jmrdc7dtd0kw8uxlas9a9fwr
hex pubkey: 5fc82b6a1f12ced29ca9a86718127629d39eeab9ec65b1b71e6adafb38fc37fb
```
## Complete Login Flow with agent-browser
### Step 1: Open the admin UI
```bash
agent-browser open http://127.0.0.1:8888/api/index.html
agent-browser wait --load networkidle
```
### Step 2: Take an interactive snapshot to see what is on screen
```bash
agent-browser snapshot -i
```
You will see a login modal with buttons like:
- `"Browser Extension"` — skip this
- `"Local Key"`**use this one**
- `"Seed Phrase"` — skip
- `"Nostr Connect"` — skip
- `"Read Only"` — skip
### Step 3: Click "Local Key"
```bash
agent-browser click @e15
```
(The ref number may vary — use the ref from the snapshot output for the "Local Key" button.)
After clicking, a text input appears asking for the secret key.
### Step 4: Enter the admin nsec
```bash
agent-browser fill @e14 "nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm"
```
(Use the ref from the snapshot for the textbox element.)
### Step 5: Click "Import Key"
```bash
agent-browser snapshot -i
```
Check the snapshot — the "Import Key" button should now be enabled. Click it:
```bash
agent-browser click @e15
```
### Step 6: Click "Continue" on the success screen
After import, a success screen appears with "Continue" button:
```bash
agent-browser wait 800
agent-browser snapshot -i
agent-browser click @e19
```
(Use the ref from the snapshot for the "Continue" button.)
### Step 7: Wait for admin UI to load
```bash
agent-browser wait 5000
agent-browser snapshot -i
```
You should now see the admin dashboard with:
- Statistics table (Database Size, Total Events, PID, etc.)
- Navigation buttons (Statistics, Subscriptions, Configuration, Authorization, etc.)
- Admin profile area showing "admin" label
## Navigating Admin Sections
After login, use the sidebar navigation buttons:
```bash
# View configuration
agent-browser click @e8 # Configuration button ref
# View authorization rules
agent-browser click @e9 # Authorization button ref
# View statistics
agent-browser click @e6 # Statistics button ref
# Always snapshot after navigation to see results
agent-browser wait 2500
agent-browser snapshot -i
```
## Checking for Errors
```bash
# View browser console logs
agent-browser console
# View JavaScript errors
agent-browser errors
# View relay server logs
tail -n 100 relay.log
```
## Chained Command Example (Full Login in One Shot)
```bash
agent-browser open http://127.0.0.1:8888/api/index.html && \
agent-browser wait --load networkidle && \
agent-browser snapshot -i
```
Then use refs from the snapshot to complete login steps.
## Tips for AI Agents
1. **Always use `/api/index.html`** — never the root URL
2. **Use `snapshot -i`** after every action to see the current interactive elements and their refs
3. **Refs change** between snapshots — always re-snapshot before clicking
4. **Wait after clicks** — use `agent-browser wait 2000` (milliseconds) between actions that trigger async operations
5. **The login flow has 3 screens**: method selection → key input → success confirmation
6. **Console logs are cumulative** — they show all logs since page load, which is useful for debugging admin API responses
7. **The relay log** at `relay.log` shows server-side processing of admin commands
8. **Command chaining** with `&&` works — the browser daemon persists between commands
## Verifying Admin API is Working
After login, the statistics page should show populated data:
- Database Size (e.g., "4 KB")
- Process ID (the relay PID)
- WebSocket Connections count
- Memory Usage
If these show "-" or "Loading...", check:
1. `relay.log` for errors
2. `agent-browser console` for JavaScript errors
3. `agent-browser errors` for page-level errors
## Closing the Browser
```bash
agent-browser close --all
```
+11 -11
View File
@@ -324,7 +324,7 @@ int main() {
}
```
## Migration Plan for c-relay
## Migration Plan for c-relay-pg
### Phase 1: Extract Debug System
1. Create `c_utils_lib` repository
@@ -333,16 +333,16 @@ int main() {
4. Add basic tests
### Phase 2: Add Versioning System
1. Extract version generation logic from c-relay
1. Extract version generation logic from c-relay-pg
2. Create reusable version utilities
3. Update c-relay to use new system
3. Update c-relay-pg to use new system
4. Update nostr_core_lib to use new system
### Phase 3: Add as Submodule
1. Add `c_utils_lib` as submodule to c-relay
2. Update c-relay Makefile
3. Update includes in c-relay source files
4. Remove old debug files from c-relay
1. Add `c_utils_lib` as submodule to c-relay-pg
2. Update c-relay-pg Makefile
3. Update includes in c-relay-pg source files
4. Remove old debug files from c-relay-pg
### Phase 4: Documentation & Examples
1. Create comprehensive README
@@ -352,7 +352,7 @@ int main() {
## Benefits
### For c-relay
### For c-relay-pg
- Cleaner separation of concerns
- Reusable utilities across projects
- Easier to maintain and test
@@ -379,7 +379,7 @@ int main() {
- Memory leak detection (valgrind)
### Integration Tests
- Test with real projects (c-relay, nostr_core_lib)
- Test with real projects (c-relay-pg, nostr_core_lib)
- Cross-platform testing
- Performance benchmarks
@@ -433,7 +433,7 @@ MIT License - permissive and suitable for learning and commercial use.
## Success Criteria
1. ✅ Successfully integrated into c-relay
1. ✅ Successfully integrated into c-relay-pg
2. ✅ Successfully integrated into nostr_core_lib
3. ✅ All tests passing
4. ✅ Documentation complete
@@ -449,7 +449,7 @@ MIT License - permissive and suitable for learning and commercial use.
4. Create build system
5. Write tests
6. Create documentation
7. Integrate into c-relay
7. Integrate into c-relay-pg
8. Publish to GitHub
---
+16 -16
View File
@@ -2,13 +2,13 @@
## Overview
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay project.
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay-pg project.
## Phase 1: Repository Setup & Structure
### Step 1.1: Create Repository Structure
**Location**: Create outside c-relay project (sibling directory)
**Location**: Create outside c-relay-pg project (sibling directory)
```bash
# Create directory structure
@@ -42,7 +42,7 @@ git branch -M main
### Step 2.1: Move Debug Files
**Source files** (from c-relay):
**Source files** (from c-relay-pg):
- `src/debug.c``c_utils_lib/src/debug.c`
- `src/debug.h``c_utils_lib/include/c_utils/debug.h`
@@ -484,20 +484,20 @@ How to integrate into projects:
3. Code examples
4. Migration from standalone utilities
## Phase 7: Integration with c-relay
## Phase 7: Integration with c-relay-pg
### Step 7.1: Add as Submodule
```bash
cd /path/to/c-relay
cd /path/to/c-relay-pg
git submodule add <repo-url> c_utils_lib
git submodule update --init --recursive
```
### Step 7.2: Update c-relay Makefile
### Step 7.2: Update c-relay-pg Makefile
```makefile
# Add to c-relay Makefile
# Add to c-relay-pg Makefile
C_UTILS_LIB = c_utils_lib/libc_utils.a
# Update includes
@@ -514,7 +514,7 @@ $(C_UTILS_LIB):
$(TARGET): $(C_UTILS_LIB) ...
```
### Step 7.3: Update c-relay Source Files
### Step 7.3: Update c-relay-pg Source Files
**Changes needed**:
@@ -543,7 +543,7 @@ $(TARGET): $(C_UTILS_LIB) ...
### Step 7.4: Test Integration
```bash
cd c-relay
cd c-relay-pg
make clean
make
./make_and_restart_relay.sh
@@ -556,7 +556,7 @@ Verify:
## Phase 8: Version System Integration
### Step 8.1: Update c-relay Makefile for Versioning
### Step 8.1: Update c-relay-pg Makefile for Versioning
```makefile
# Add version generation
@@ -567,7 +567,7 @@ src/version.h: .git/refs/tags/*
$(TARGET): src/version.h ...
```
### Step 8.2: Update c-relay to Use Generated Version
### Step 8.2: Update c-relay-pg to Use Generated Version
Replace hardcoded version in `src/main.h` with:
```c
@@ -583,7 +583,7 @@ Replace hardcoded version in `src/main.h` with:
- **Phase 4**: Build System - 2 hours
- **Phase 5**: Examples & Tests - 3 hours
- **Phase 6**: Documentation - 3 hours
- **Phase 7**: c-relay Integration - 2 hours
- **Phase 7**: c-relay-pg Integration - 2 hours
- **Phase 8**: Version Integration - 2 hours
**Total**: ~19 hours
@@ -593,11 +593,11 @@ Replace hardcoded version in `src/main.h` with:
- [ ] c_utils_lib builds successfully
- [ ] All tests pass
- [ ] Examples compile and run
- [ ] c-relay integrates successfully
- [ ] Debug output works in c-relay
- [ ] c-relay-pg integrates successfully
- [ ] Debug output works in c-relay-pg
- [ ] Version generation works
- [ ] Documentation complete
- [ ] No regressions in c-relay functionality
- [ ] No regressions in c-relay-pg functionality
## Next Steps
@@ -608,7 +608,7 @@ Replace hardcoded version in `src/main.h` with:
5. Create build system
6. Write tests and examples
7. Create documentation
8. Integrate into c-relay
8. Integrate into c-relay-pg
9. Test thoroughly
10. Publish to GitHub
+6 -6
View File
@@ -78,9 +78,9 @@ Configuration events follow the standard Nostr event format with kind 33334:
#### `relay_software`
- **Description**: Software identifier for NIP-11
- **Default**: `"c-relay"`
- **Default**: `"c-relay-pg"`
- **Format**: String, max 64 characters
- **Example**: `"c-relay v1.0.0"`
- **Example**: `"c-relay-pg v1.0.0"`
#### `relay_version`
- **Description**: Software version string
@@ -366,7 +366,7 @@ sqlite3 relay.nrdb "SELECT json_pretty(json_object(
#### Invalid Parameter Values
```bash
# Check relay logs for validation errors
journalctl -u c-relay | grep "Configuration.*invalid\|Invalid.*configuration"
journalctl -u c-relay-pg | grep "Configuration.*invalid\|Invalid.*configuration"
# Common issues:
# - Numeric values outside valid ranges
@@ -395,10 +395,10 @@ ORDER BY date DESC;"
#### Resource Usage After Changes
```bash
# Monitor system resources after configuration updates
top -p $(pgrep c_relay)
top -p $(pgrep c_relay_pg)
# Check for memory leaks
ps aux | grep c_relay | awk '{print $6}' # RSS memory
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory
```
### Emergency Recovery
@@ -425,7 +425,7 @@ nostrtool event \
# If database is corrupted, backup and recreate
cp relay.nrdb relay.nrdb.backup
rm relay.nrdb*
./build/c_relay_x86 # Creates fresh database with new keys
./build/c_relay_pg_x86 # Creates fresh database with new keys
```
---
+14 -14
View File
@@ -21,22 +21,22 @@ typedef enum {
```bash
# Production (default - no debug output)
./c_relay_x86
./c_relay_pg_x86
# Show errors only
./c_relay_x86 --debug-level=1
./c_relay_pg_x86 --debug-level=1
# Show errors and warnings
./c_relay_x86 --debug-level=2
./c_relay_pg_x86 --debug-level=2
# Show errors, warnings, and info (recommended for development)
./c_relay_x86 --debug-level=3
./c_relay_pg_x86 --debug-level=3
# Show all debug messages
./c_relay_x86 --debug-level=4
./c_relay_pg_x86 --debug-level=4
# Show everything including trace with file:line (very verbose)
./c_relay_x86 --debug-level=5
./c_relay_pg_x86 --debug-level=5
```
## Implementation
@@ -475,7 +475,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 3 (Errors + Warnings + Info)
```
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [INFO ] Loading configuration from database
[2025-01-12 14:30:15] [ERROR] Failed to open database: permission denied
[2025-01-12 14:30:16] [WARN ] Port 8888 unavailable, trying 8889
@@ -484,7 +484,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 4 (All Debug Messages)
```
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [DEBUG] Opening database: build/abc123...def.db
[2025-01-12 14:30:15] [DEBUG] Executing schema initialization
[2025-01-12 14:30:15] [INFO ] SQLite WAL mode enabled
@@ -496,7 +496,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 5 (Everything Including file:line for ALL messages)
```
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [DEBUG] [main.c:348] Opening database: build/abc123...def.db
[2025-01-12 14:30:15] [TRACE] [main.c:330] Entering init_database()
[2025-01-12 14:30:15] [ERROR] [config.c:125] Database locked
@@ -525,11 +525,11 @@ Update the existing `log_*` functions to use the new debug macros
make clean && make
# Test different levels
./build/c_relay_x86 # No output
./build/c_relay_x86 --debug-level=1 # Errors only
./build/c_relay_x86 --debug-level=3 # Info + warnings + errors
./build/c_relay_x86 --debug-level=4 # All debug messages
./build/c_relay_x86 --debug-level=5 # Everything with file:line on TRACE
./build/c_relay_pg_x86 # No output
./build/c_relay_pg_x86 --debug-level=1 # Errors only
./build/c_relay_pg_x86 --debug-level=3 # Info + warnings + errors
./build/c_relay_pg_x86 --debug-level=4 # All debug messages
./build/c_relay_pg_x86 --debug-level=5 # Everything with file:line on TRACE
```
### Step 5: Gradual Migration (Ongoing)
+1 -1
View File
@@ -34,7 +34,7 @@ static const struct {
// NIP-11 Relay Information (relay keys will be populated at runtime)
{"relay_description", "High-performance C Nostr relay with SQLite storage"},
{"relay_contact", ""},
{"relay_software", "https://git.laantungir.net/laantungir/c-relay.git"},
{"relay_software", "https://git.laantungir.net/laantungir/c-relay-pg.git"},
{"relay_version", "v1.0.0"},
// NIP-13 Proof of Work (pow_min_difficulty = 0 means PoW disabled)
+51 -51
View File
@@ -52,11 +52,11 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
#### User and Directory Setup
```bash
# Create dedicated system user
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
# Create application directory
sudo mkdir -p /opt/c-relay
sudo chown c-relay:c-relay /opt/c-relay
sudo mkdir -p /opt/c-relay-pg
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg
```
### Build and Installation
@@ -64,8 +64,8 @@ sudo chown c-relay:c-relay /opt/c-relay
#### Automated Installation (Recommended)
```bash
# Clone repository
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
# Build
@@ -81,12 +81,12 @@ sudo systemd/install-service.sh
make clean && make
# Install binary
sudo cp build/c_relay_x86 /opt/c-relay/
sudo chown c-relay:c-relay /opt/c-relay/c_relay_x86
sudo chmod +x /opt/c-relay/c_relay_x86
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
sudo chmod +x /opt/c-relay-pg/c_relay_pg_x86
# Install systemd service
sudo cp systemd/c-relay.service /etc/systemd/system/
sudo cp systemd/c-relay-pg.service /etc/systemd/system/
sudo systemctl daemon-reload
```
@@ -95,22 +95,22 @@ sudo systemctl daemon-reload
#### Start and Enable Service
```bash
# Start the service
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Enable auto-start on boot
sudo systemctl enable c-relay
sudo systemctl enable c-relay-pg
# Check status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
```
#### Capture Admin Keys (CRITICAL)
```bash
# View startup logs to get admin keys
sudo journalctl -u c-relay --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
# Or check the full log
sudo journalctl -u c-relay --no-pager | grep "Admin Private Key"
sudo journalctl -u c-relay-pg --no-pager | grep "Admin Private Key"
```
⚠️ **CRITICAL**: Save the admin private key immediately - it's only shown once and is needed for all configuration updates!
@@ -151,8 +151,8 @@ sudo iptables-save > /etc/iptables/rules.v4
ssh -i your-key.pem ubuntu@your-instance-ip
# Use the simple deployment script
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -168,10 +168,10 @@ sudo examples/deployment/simple-vps/deploy.sh
sudo mkfs.ext4 /dev/xvdf
sudo mkdir /data
sudo mount /dev/xvdf /data
sudo chown c-relay:c-relay /data
sudo chown c-relay-pg:c-relay-pg /data
# Update systemd service to use /data
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay/WorkingDirectory=\/data/' /etc/systemd/system/c-relay.service
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay-pg/WorkingDirectory=\/data/' /etc/systemd/system/c-relay-pg.service
sudo systemctl daemon-reload
```
@@ -180,7 +180,7 @@ sudo systemctl daemon-reload
#### Compute Engine Setup
```bash
# Create VM instance (e2-micro or larger)
gcloud compute instances create c-relay-instance \
gcloud compute instances create c-relay-pg-instance \
--image-family=ubuntu-2204-lts \
--image-project=ubuntu-os-cloud \
--machine-type=e2-micro \
@@ -193,9 +193,9 @@ gcloud compute firewall-rules create allow-nostr-relay \
--target-tags nostr-relay
# SSH and deploy
gcloud compute ssh c-relay-instance
git clone https://github.com/your-org/c-relay.git
cd c-relay
gcloud compute ssh c-relay-pg-instance
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -203,13 +203,13 @@ sudo examples/deployment/simple-vps/deploy.sh
```bash
# Create and attach persistent disk
gcloud compute disks create relay-data --size=50GB
gcloud compute instances attach-disk c-relay-instance --disk=relay-data
gcloud compute instances attach-disk c-relay-pg-instance --disk=relay-data
# Format and mount
sudo mkfs.ext4 /dev/sdb
sudo mkdir /data
sudo mount /dev/sdb /data
sudo chown c-relay:c-relay /data
sudo chown c-relay-pg:c-relay-pg /data
```
### DigitalOcean
@@ -223,8 +223,8 @@ sudo chown c-relay:c-relay /data
ssh root@your-droplet-ip
# Deploy relay
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
examples/deployment/simple-vps/deploy.sh
```
@@ -245,8 +245,8 @@ The `examples/deployment/` directory contains ready-to-use scripts:
### Simple VPS Deployment
```bash
# Clone repository and run automated deployment
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -417,9 +417,9 @@ LOG_FILE="/var/log/relay-monitor.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
# Check if relay is running
if ! pgrep -f "c_relay_x86" > /dev/null; then
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
echo "[$DATE] ERROR: Relay process not running" >> $LOG_FILE
systemctl restart c-relay
systemctl restart c-relay-pg
fi
# Check port availability
@@ -428,14 +428,14 @@ if ! netstat -tln | grep -q ":8888"; then
fi
# Check database file
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
if [[ -n "$RELAY_DB" ]]; then
DB_SIZE=$(du -h "$RELAY_DB" | cut -f1)
echo "[$DATE] INFO: Database size: $DB_SIZE" >> $LOG_FILE
fi
# Check memory usage
MEM_USAGE=$(ps aux | grep c_relay_x86 | grep -v grep | awk '{print $6}')
MEM_USAGE=$(ps aux | grep c_relay_pg_x86 | grep -v grep | awk '{print $6}')
if [[ -n "$MEM_USAGE" ]]; then
echo "[$DATE] INFO: Memory usage: ${MEM_USAGE}KB" >> $LOG_FILE
fi
@@ -454,8 +454,8 @@ sudo chmod +x /usr/local/bin/relay-monitor.sh
#### Centralized Logging with rsyslog
```bash
# /etc/rsyslog.d/50-c-relay.conf
if $programname == 'c-relay' then /var/log/c-relay.log
# /etc/rsyslog.d/50-c-relay-pg.conf
if $programname == 'c-relay-pg' then /var/log/c-relay-pg.log
& stop
```
@@ -465,7 +465,7 @@ if $programname == 'c-relay' then /var/log/c-relay.log
```yaml
# /etc/prometheus/prometheus.yml
scrape_configs:
- job_name: 'c-relay'
- job_name: 'c-relay-pg'
static_configs:
- targets: ['localhost:8888']
metrics_path: '/metrics' # If implemented
@@ -479,30 +479,30 @@ scrape_configs:
#### Service User Restrictions
```bash
# Restrict service user
sudo usermod -s /bin/false c-relay
sudo usermod -d /opt/c-relay c-relay
sudo usermod -s /bin/false c-relay-pg
sudo usermod -d /opt/c-relay-pg c-relay-pg
# Set proper permissions
sudo chmod 700 /opt/c-relay
sudo chown -R c-relay:c-relay /opt/c-relay
sudo chmod 700 /opt/c-relay-pg
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
```
#### File System Restrictions
```bash
# Mount data directory with appropriate options
echo "/dev/sdb /opt/c-relay ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
echo "/dev/sdb /opt/c-relay-pg ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
```
### Network Security
#### Fail2Ban Configuration
```ini
# /etc/fail2ban/jail.d/c-relay.conf
[c-relay-dos]
# /etc/fail2ban/jail.d/c-relay-pg.conf
[c-relay-pg-dos]
enabled = true
port = 8888
filter = c-relay-dos
logpath = /var/log/c-relay.log
filter = c-relay-pg-dos
logpath = /var/log/c-relay-pg.log
maxretry = 10
findtime = 60
bantime = 300
@@ -534,9 +534,9 @@ sudo mkfs.ext4 /dev/mapper/relay-data
#!/bin/bash
# /usr/local/bin/backup-relay.sh
BACKUP_DIR="/backup/c-relay"
BACKUP_DIR="/backup/c-relay-pg"
DATE=$(date +%Y%m%d_%H%M%S)
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
mkdir -p "$BACKUP_DIR"
@@ -574,7 +574,7 @@ sudo apt install -y awscli
aws configure
# Sync backups to S3
aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
aws s3 sync /backup/c-relay-pg/ s3://your-backup-bucket/c-relay-pg/ --delete
```
### Disaster Recovery
@@ -583,16 +583,16 @@ aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
```bash
# 1. Restore from backup
gunzip backup/relay_backup_20231201_020000.nrdb.gz
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay/
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay-pg/
# 2. Fix permissions
sudo chown c-relay:c-relay /opt/c-relay/*.nrdb
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/*.nrdb
# 3. Restart service
sudo systemctl restart c-relay
sudo systemctl restart c-relay-pg
# 4. Verify recovery
sudo journalctl -u c-relay --since "1 minute ago"
sudo journalctl -u c-relay-pg --since "1 minute ago"
```
---
+25 -25
View File
@@ -2,7 +2,7 @@
## Overview
This guide explains how to build truly portable MUSL-based static binaries of c-relay using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
This guide explains how to build truly portable MUSL-based static binaries of c-relay-pg using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
## Why MUSL?
@@ -36,8 +36,8 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
./build_static.sh
# The binary will be created at:
# build/c_relay_static_musl_x86_64 (on x86_64)
# build/c_relay_static_musl_arm64 (on ARM64)
# build/c_relay_pg_static_musl_x86_64 (on x86_64)
# build/c_relay_pg_static_musl_arm64 (on ARM64)
```
### What Happens During Build
@@ -56,7 +56,7 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
- Includes required NIPs: 001, 006, 013, 017, 019, 044, 059
- Produces static library (~316KB)
4. **c-relay Compilation**: Links everything statically:
4. **c-relay-pg Compilation**: Links everything statically:
- All source files compiled with `-static` flag
- Fortification disabled to avoid `__*_chk` symbols
- Results in ~7.6MB stripped binary
@@ -78,7 +78,7 @@ FROM alpine:3.19 AS builder
- Install build tools and static libraries
- Build dependencies from source
- Compile nostr_core_lib with MUSL flags
- Compile c-relay with full static linking
- Compile c-relay-pg with full static linking
- Strip binary to reduce size
# Stage 2: Output (scratch)
@@ -93,7 +93,7 @@ FROM scratch AS output
CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"
```
**For c-relay:**
**For c-relay-pg:**
```bash
gcc -static -O2 -Wall -Wextra -std=c99 \
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
@@ -110,7 +110,7 @@ gcc -static -O2 -Wall -Wextra -std=c99 \
- `-U_FORTIFY_SOURCE` (undefine any existing definition)
- `-D_FORTIFY_SOURCE=0` (set to 0)
This must be applied to **both** nostr_core_lib and c-relay compilation.
This must be applied to **both** nostr_core_lib and c-relay-pg compilation.
### NIP Dependencies
@@ -129,26 +129,26 @@ The build includes these NIPs in nostr_core_lib:
```bash
# Should show "statically linked"
file build/c_relay_static_musl_x86_64
file build/c_relay_pg_static_musl_x86_64
# Should show "not a dynamic executable"
ldd build/c_relay_static_musl_x86_64
ldd build/c_relay_pg_static_musl_x86_64
# Check size (should be ~7.6MB)
ls -lh build/c_relay_static_musl_x86_64
ls -lh build/c_relay_pg_static_musl_x86_64
```
### Test Execution
```bash
# Show help
./build/c_relay_static_musl_x86_64 --help
./build/c_relay_pg_static_musl_x86_64 --help
# Show version
./build/c_relay_static_musl_x86_64 --version
./build/c_relay_pg_static_musl_x86_64 --version
# Run relay
./build/c_relay_static_musl_x86_64 --port 8888
./build/c_relay_pg_static_musl_x86_64 --port 8888
```
### Cross-Distribution Testing
@@ -157,16 +157,16 @@ Test the binary on different distributions to verify portability:
```bash
# Alpine Linux
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_pg_static_musl_x86_64 --version
# Ubuntu
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_pg_static_musl_x86_64 --version
# Debian
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_pg_static_musl_x86_64 --version
# CentOS
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_pg_static_musl_x86_64 --version
```
## Troubleshooting
@@ -187,7 +187,7 @@ newgrp docker # Or logout and login again
**Solution**: Ensure fortification is disabled in both:
1. nostr_core_lib build.sh (line 534)
2. c-relay compilation flags in Dockerfile
2. c-relay-pg compilation flags in Dockerfile
### Binary Won't Execute
@@ -213,26 +213,26 @@ newgrp docker # Or logout and login again
```bash
# Copy binary to server
scp build/c_relay_static_musl_x86_64 user@server:/opt/c-relay/
scp build/c_relay_pg_static_musl_x86_64 user@server:/opt/c-relay-pg/
# Run on server (no dependencies needed!)
ssh user@server
cd /opt/c-relay
./c_relay_static_musl_x86_64 --port 8888
cd /opt/c-relay-pg
./c_relay_pg_static_musl_x86_64 --port 8888
```
### SystemD Service
```ini
[Unit]
Description=C-Relay Nostr Relay (MUSL Static)
Description=C-Relay-PG Nostr Relay (MUSL Static)
After=network.target
[Service]
Type=simple
User=c-relay
WorkingDirectory=/opt/c-relay
ExecStart=/opt/c-relay/c_relay_static_musl_x86_64
User=c-relay-pg
WorkingDirectory=/opt/c-relay-pg
ExecStart=/opt/c-relay-pg/c_relay_pg_static_musl_x86_64
Restart=always
RestartSec=5
+4 -4
View File
@@ -23,7 +23,7 @@ We chose Option B because:
## Detailed Implementation Steps
### Phase 1: Configuration Setup in c-relay
### Phase 1: Configuration Setup in c-relay-pg
#### 1.1 Add Configuration Parameter
**File:** `src/default_config_event.h`
@@ -293,7 +293,7 @@ int nostr_nip17_send_dm(cJSON* dm_event,
---
### Phase 4: Update c-relay Call Sites
### Phase 4: Update c-relay-pg Call Sites
#### 4.1 Update src/api.c
**Location:** Line 1319
@@ -470,7 +470,7 @@ causing compatibility issues.
- [ ] Update `nostr_nip17_send_dm()` signature and implementation
- [ ] Update `nip017.h` function declaration and documentation
### c-relay Changes
### c-relay-pg Changes
- [ ] Add `nip59_timestamp_max_delay_sec` to `default_config_event.h`
- [ ] Add validation in `config.c` for new parameter
- [ ] Update `src/api.c` call site to pass `max_delay_sec`
@@ -494,7 +494,7 @@ causing compatibility issues.
If issues arise:
1. Revert nostr_core_lib changes (git revert in submodule)
2. Revert c-relay changes
2. Revert c-relay-pg changes
3. Configuration parameter will be ignored if not used
4. Default behavior (0) provides maximum compatibility
+2 -2
View File
@@ -2,7 +2,7 @@
## Overview
This document describes the design for a general-purpose SQL query interface for the C-Relay admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
This document describes the design for a general-purpose SQL query interface for the C-Relay-PG admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
## Security Model
@@ -271,7 +271,7 @@ const SQL_QUERY_TEMPLATES = {
};
// Query history management (localStorage)
const QUERY_HISTORY_KEY = 'c_relay_sql_history';
const QUERY_HISTORY_KEY = 'c_relay_pg_sql_history';
const MAX_HISTORY_ITEMS = 20;
// Load query history from localStorage
+1 -1
View File
@@ -224,7 +224,7 @@ The script should:
- `curl` or `websocat` for WebSocket communication
- `jq` for JSON parsing
- Nostr CLI tools (optional, for event signing)
- Running c-relay instance
- Running c-relay-pg instance
## Example Output
+6 -6
View File
@@ -1,8 +1,8 @@
# C-Relay Complete Startup Flow Documentation
# C-Relay-PG Complete Startup Flow Documentation
## Overview
C-Relay has two distinct startup paths:
C-Relay-PG has two distinct startup paths:
1. **First-Time Startup**: No database exists, generates keys and initializes system
2. **Existing Relay Startup**: Database exists, loads configuration and resumes operation
@@ -220,7 +220,7 @@ int populate_default_config_values(void) {
struct config_default defaults[] = {
{"relay_port", "8888", "integer", "WebSocket port", "network", 1},
{"relay_name", "C-Relay", "string", "Relay name", "info", 0},
{"relay_name", "C-Relay-PG", "string", "Relay name", "info", 0},
{"relay_description", "High-performance C Nostr relay", "string", "Description", "info", 0},
{"max_subscriptions_per_client", "25", "integer", "Max subs per client", "limits", 0},
{"pow_min_difficulty", "0", "integer", "Minimum PoW difficulty", "security", 0},
@@ -983,7 +983,7 @@ if (current_version < LATEST_VERSION) {
**Solutions**:
- Use `--port <number>` to specify different port
- Kill existing process: `pkill -f c_relay_`
- Kill existing process: `pkill -f c_relay_pg_`
- Force kill port: `fuser -k 8888/tcp`
- Use `--strict-port` to fail fast instead of trying fallback ports
@@ -994,7 +994,7 @@ if (current_version < LATEST_VERSION) {
**Solutions**:
- Kill existing relay processes
- Remove WAL files: `rm build/*.db-wal build/*.db-shm`
- Check for stale processes: `ps aux | grep c_relay_`
- Check for stale processes: `ps aux | grep c_relay_pg_`
#### 3. Missing Admin Private Key
@@ -1071,7 +1071,7 @@ Typical startup times (on modern hardware):
## Summary
The c-relay startup system is designed for:
The c-relay-pg startup system is designed for:
1. **Security**: Admin keys never stored, relay keys encrypted
2. **Reliability**: Automatic port fallback, schema migrations
+9 -9
View File
@@ -18,10 +18,10 @@ The script now attempts to build with `musl-gcc` for truly portable static binar
SQLite is now built once and cached for future builds:
- **Cache location**: `~/.cache/c-relay-sqlite/`
- **Cache location**: `~/.cache/c-relay-pg-sqlite/`
- **Version-specific**: Each SQLite version gets its own cache directory
- **Significant speedup**: Subsequent builds skip the SQLite compilation step
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-sqlite` to clear cache
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-pg-sqlite` to clear cache
### 3. Smart Package Installation
@@ -51,13 +51,13 @@ The script will:
## Binary Types
### MUSL Static Binary (Ideal - Currently Not Achievable)
- **Filename**: `build/c_relay_static_musl_x86_64`
- **Filename**: `build/c_relay_pg_static_musl_x86_64`
- **Dependencies**: None (truly static)
- **Portability**: Works on any Linux distribution
- **Status**: Requires MUSL-compiled libwebsockets and other dependencies (not available by default)
### Glibc Static Binary (Current Output)
- **Filename**: `build/c_relay_static_x86_64` or `build/c_relay_static_glibc_x86_64`
- **Filename**: `build/c_relay_pg_static_x86_64` or `build/c_relay_pg_static_glibc_x86_64`
- **Dependencies**: None - fully statically linked with glibc
- **Portability**: Works on most Linux distributions (glibc is statically included)
- **Note**: Despite using glibc, this is a **fully static binary** with no runtime dependencies
@@ -68,11 +68,11 @@ The script automatically verifies binaries using `ldd` and `file`:
```bash
# For MUSL binary
ldd build/c_relay_static_musl_x86_64
ldd build/c_relay_pg_static_musl_x86_64
# Output: "not a dynamic executable" (good!)
# For glibc binary
ldd build/c_relay_static_glibc_x86_64
ldd build/c_relay_pg_static_glibc_x86_64
# Output: Shows glibc dependencies
```
@@ -112,7 +112,7 @@ The script attempts MUSL compilation but falls back to glibc:
### Clear SQLite Cache
```bash
rm -rf ~/.cache/c-relay-sqlite
rm -rf ~/.cache/c-relay-pg-sqlite
```
### Force Package Reinstall
@@ -127,8 +127,8 @@ cat /tmp/musl_build.log
### Verify Binary Type
```bash
file build/c_relay_static_*
ldd build/c_relay_static_* 2>&1
file build/c_relay_pg_static_*
ldd build/c_relay_pg_static_* 2>&1
```
## Performance Impact
+1 -1
View File
@@ -2,7 +2,7 @@
## Problem Summary
The c-relay Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
The c-relay-pg Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
## Solution: Two-Component Approach
+8 -8
View File
@@ -586,7 +586,7 @@ int add_pubkeys_to_config_table(void) {
rm -f *.db
# Start relay with defaults
./build/c_relay_x86
./build/c_relay_pg_x86
# Verify config table complete
sqlite3 <relay_pubkey>.db "SELECT COUNT(*) FROM config;"
@@ -602,7 +602,7 @@ int add_pubkeys_to_config_table(void) {
rm -f *.db
# Start relay with port override
./build/c_relay_x86 --port 9999
./build/c_relay_pg_x86 --port 9999
# Verify port override applied
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
@@ -612,13 +612,13 @@ int add_pubkeys_to_config_table(void) {
3. **Restart with Existing Database**
```bash
# Start relay (creates database)
./build/c_relay_x86
./build/c_relay_pg_x86
# Stop relay
pkill -f c_relay_
pkill -f c_relay_pg_
# Restart relay
./build/c_relay_x86
./build/c_relay_pg_x86
# Verify config unchanged
# Check relay.log for validation message
@@ -627,13 +627,13 @@ int add_pubkeys_to_config_table(void) {
4. **Restart with CLI Overrides**
```bash
# Start relay (creates database)
./build/c_relay_x86
./build/c_relay_pg_x86
# Stop relay
pkill -f c_relay_
pkill -f c_relay_pg_
# Restart with port override
./build/c_relay_x86 --port 9999
./build/c_relay_pg_x86 --port 9999
# Verify port override applied
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
+32 -32
View File
@@ -19,12 +19,12 @@ Complete guide for deploying, configuring, and managing the C Nostr Relay with e
```bash
# Clone and build
git clone <repository-url>
cd c-relay
cd c-relay-pg
git submodule update --init --recursive
make
# Start relay (zero configuration needed)
./build/c_relay_x86
./build/c_relay_pg_x86
```
### 2. First Startup - Save Keys
@@ -78,7 +78,7 @@ brew install git sqlite libwebsockets openssl libsecp256k1 curl zlib
```bash
# Clone repository
git clone <repository-url>
cd c-relay
cd c-relay-pg
# Initialize submodules
git submodule update --init --recursive
@@ -87,7 +87,7 @@ git submodule update --init --recursive
make clean && make
# Verify build
ls -la build/c_relay_x86
ls -la build/c_relay_pg_x86
```
### Production Deployment
@@ -98,27 +98,27 @@ ls -la build/c_relay_x86
sudo systemd/install-service.sh
# Start service
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Enable auto-start
sudo systemctl enable c-relay
sudo systemctl enable c-relay-pg
# Check status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
```
#### Manual Deployment
```bash
# Create dedicated user
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
# Install binary
sudo mkdir -p /opt/c-relay
sudo cp build/c_relay_x86 /opt/c-relay/
sudo chown -R c-relay:c-relay /opt/c-relay
sudo mkdir -p /opt/c-relay-pg
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
# Run as service user
sudo -u c-relay /opt/c-relay/c_relay_x86
sudo -u c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
```
## Configuration Management
@@ -188,7 +188,7 @@ Send this to your relay via WebSocket, and changes are applied immediately.
|-----------|-------------|---------|---------|
| `relay_description` | Relay description for NIP-11 | "C Nostr Relay" | "My awesome relay" |
| `relay_contact` | Admin contact information | "" | "admin@example.com" |
| `relay_software` | Software identifier | "c-relay" | "c-relay v1.0" |
| `relay_software` | Software identifier | "c-relay-pg" | "c-relay-pg v1.0" |
#### Client Limits
| Parameter | Description | Default | Range |
@@ -273,7 +273,7 @@ chmod 600 admin_keys_backup_*.txt
#### Key Recovery
If you lose your admin private key:
1. **Stop the relay**: `pkill c_relay` or `sudo systemctl stop c-relay`
1. **Stop the relay**: `pkill c_relay_pg` or `sudo systemctl stop c-relay-pg`
2. **Backup events**: `cp <relay_pubkey>.nrdb backup_$(date +%Y%m%d).nrdb`
3. **Remove database**: `rm <relay_pubkey>.nrdb*`
4. **Restart relay**: This creates new database with new keys
@@ -300,7 +300,7 @@ sudo ufw allow 8888/tcp
```bash
# Secure database file permissions
chmod 600 <relay_pubkey>.nrdb
chown c-relay:c-relay <relay_pubkey>.nrdb
chown c-relay-pg:c-relay-pg <relay_pubkey>.nrdb
# Regular backups
cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
@@ -311,10 +311,10 @@ cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
### Service Status
```bash
# Check if relay is running
ps aux | grep c_relay
ps aux | grep c_relay_pg
# SystemD status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
# Network connections
netstat -tln | grep 8888
@@ -324,16 +324,16 @@ sudo ss -tlpn | grep 8888
### Log Monitoring
```bash
# Real-time logs (systemd)
sudo journalctl -u c-relay -f
sudo journalctl -u c-relay-pg -f
# Recent logs
sudo journalctl -u c-relay --since "1 hour ago"
sudo journalctl -u c-relay-pg --since "1 hour ago"
# Error logs only
sudo journalctl -u c-relay -p err
sudo journalctl -u c-relay-pg -p err
# Configuration changes
sudo journalctl -u c-relay | grep "Configuration updated via kind 33334"
sudo journalctl -u c-relay-pg | grep "Configuration updated via kind 33334"
```
### Database Analytics
@@ -365,13 +365,13 @@ ORDER BY created_at DESC;
du -sh <relay_pubkey>.nrdb*
# Memory usage
ps aux | grep c_relay | awk '{print $6}' # RSS memory in KB
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory in KB
# Connection count (approximate)
netstat -an | grep :8888 | grep ESTABLISHED | wc -l
# System resources
top -p $(pgrep c_relay)
top -p $(pgrep c_relay_pg)
```
## Troubleshooting
@@ -385,11 +385,11 @@ netstat -tln | grep 8888
# If port in use, find process: sudo lsof -i :8888
# Check binary permissions
ls -la build/c_relay_x86
chmod +x build/c_relay_x86
ls -la build/c_relay_pg_x86
chmod +x build/c_relay_pg_x86
# Check dependencies
ldd build/c_relay_x86
ldd build/c_relay_pg_x86
```
#### Configuration Not Updating
@@ -442,7 +442,7 @@ sqlite3 recovered.nrdb < recovered.sql
# If repair fails, start fresh (loses all events)
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.corrupted
./build/c_relay_x86 # Creates new database
./build/c_relay_pg_x86 # Creates new database
```
#### Lost Configuration Recovery
@@ -455,12 +455,12 @@ If configuration is lost but database is intact:
#### Emergency Restart
```bash
# Quick restart with clean state
sudo systemctl stop c-relay
sudo systemctl stop c-relay-pg
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.backup
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Check logs for new admin keys
sudo journalctl -u c-relay --since "5 minutes ago" | grep "Admin Private Key"
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep "Admin Private Key"
```
## Advanced Usage
@@ -503,7 +503,7 @@ ws.on('open', function() {
# backup-relay.sh
DATE=$(date +%Y%m%d_%H%M%S)
DB_FILE=$(ls *.nrdb | head -1)
BACKUP_DIR="/backup/c-relay"
BACKUP_DIR="/backup/c-relay-pg"
mkdir -p $BACKUP_DIR
cp $DB_FILE $BACKUP_DIR/relay_backup_$DATE.nrdb
@@ -533,7 +533,7 @@ tar czf relay_migration.tar.gz *.nrdb* relay.log
# Target server
tar xzf relay_migration.tar.gz
./build/c_relay_x86 # Will detect existing database and continue
./build/c_relay_pg_x86 # Will detect existing database and continue
```
---
+1 -1
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Script to embed web files into C headers for the C-Relay admin interface
# Script to embed web files into C headers for the C-Relay-PG admin interface
# Converts HTML, CSS, and JS files from api/ directory into C byte arrays
set -e
+1 -1
View File
@@ -60,7 +60,7 @@ All examples assume the event-based configuration system where:
- **Save Admin Keys**: All deployment examples emphasize capturing the admin private key on first startup
- **Firewall Configuration**: Examples include proper firewall rules
- **SSL/TLS**: Production examples include HTTPS configuration
- **User Isolation**: Service runs as dedicated `c-relay` system user
- **User Isolation**: Service runs as dedicated `c-relay-pg` system user
## Support
+6 -6
View File
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Default configuration
RELAY_DIR="/opt/c-relay"
BACKUP_DIR="/backup/c-relay"
RELAY_DIR="/opt/c-relay-pg"
BACKUP_DIR="/backup/c-relay-pg"
RETENTION_DAYS="30"
COMPRESS="true"
REMOTE_BACKUP=""
@@ -47,8 +47,8 @@ show_help() {
echo "Usage: $0 [OPTIONS]"
echo
echo "Options:"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay)"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay-pg)"
echo " -r, --retention DAYS Retention period in days (default: 30)"
echo " -n, --no-compress Don't compress backups"
echo " -s, --s3-bucket BUCKET Upload to S3 bucket"
@@ -231,7 +231,7 @@ upload_to_s3() {
print_step "Uploading backup to S3..."
local s3_path="s3://$S3_BUCKET/c-relay/$(date +%Y)/$(date +%m)/"
local s3_path="s3://$S3_BUCKET/c-relay-pg/$(date +%Y)/$(date +%m)/"
if aws s3 cp "$BACKUP_FILE" "$s3_path" --storage-class STANDARD_IA; then
print_success "Backup uploaded to S3: $s3_path"
@@ -264,7 +264,7 @@ cleanup_old_backups() {
print_step "Cleaning S3 backups older than $cutoff_date..."
# Note: This is a simplified approach. In production, use S3 lifecycle policies
aws s3 ls "s3://$S3_BUCKET/c-relay/" --recursive | \
aws s3 ls "s3://$S3_BUCKET/c-relay-pg/" --recursive | \
awk '$1 < "'$cutoff_date'" {print $4}' | \
while read -r key; do
aws s3 rm "s3://$S3_BUCKET/$key"
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
RELAY_DIR="/opt/c-relay"
SERVICE_NAME="c-relay"
RELAY_DIR="/opt/c-relay-pg"
SERVICE_NAME="c-relay-pg"
RELAY_PORT="8888"
LOG_FILE="/var/log/relay-monitor.log"
ALERT_EMAIL=""
@@ -60,7 +60,7 @@ show_help() {
echo "Usage: $0 [OPTIONS]"
echo
echo "Options:"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
echo " -p, --port PORT Relay port (default: 8888)"
echo " -i, --interval SECONDS Check interval (default: 60)"
echo " -e, --email EMAIL Alert email address"
@@ -134,7 +134,7 @@ parse_args() {
check_process_running() {
print_step "Checking if relay process is running..."
if pgrep -f "c_relay_x86" > /dev/null; then
if pgrep -f "c_relay_pg_x86" > /dev/null; then
print_success "Relay process is running"
return 0
else
@@ -172,7 +172,7 @@ check_service_status() {
check_memory_usage() {
print_step "Checking memory usage..."
local memory_kb=$(ps aux | grep "c_relay_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
local memory_kb=$(ps aux | grep "c_relay_pg_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
if [[ -z "$memory_kb" ]]; then
print_warning "Could not determine memory usage"
+4 -4
View File
@@ -56,7 +56,7 @@ http {
}
# Upstream for the relay
upstream c_relay_backend {
upstream c_relay_pg_backend {
server 127.0.0.1:8888;
keepalive 32;
}
@@ -108,7 +108,7 @@ http {
# Main proxy location for WebSocket and HTTP
location / {
# Proxy settings
proxy_pass http://c_relay_backend;
proxy_pass http://c_relay_pg_backend;
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
@@ -144,7 +144,7 @@ http {
# Health check endpoint (if implemented)
location /health {
proxy_pass http://c_relay_backend/health;
proxy_pass http://c_relay_pg_backend/health;
access_log off;
}
@@ -157,7 +157,7 @@ http {
# Optional: Metrics endpoint (if implemented)
location /metrics {
proxy_pass http://c_relay_backend/metrics;
proxy_pass http://c_relay_pg_backend/metrics;
# Restrict access to monitoring systems
allow 10.0.0.0/8;
allow 172.16.0.0/12;
@@ -96,9 +96,9 @@ check_root() {
check_relay_running() {
print_step "Checking if C Nostr Relay is running..."
if ! pgrep -f "c_relay_x86" > /dev/null; then
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
print_error "C Nostr Relay is not running"
print_error "Please start the relay first with: sudo systemctl start c-relay"
print_error "Please start the relay first with: sudo systemctl start c-relay-pg"
exit 1
fi
+12 -12
View File
@@ -13,9 +13,9 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
RELAY_USER="c-relay"
INSTALL_DIR="/opt/c-relay"
SERVICE_NAME="c-relay"
RELAY_USER="c-relay-pg"
INSTALL_DIR="/opt/c-relay-pg"
SERVICE_NAME="c-relay-pg"
RELAY_PORT="8888"
# Functions
@@ -99,7 +99,7 @@ build_relay() {
# Check if we're in the source directory
if [[ ! -f "Makefile" ]]; then
print_error "Makefile not found. Please run this script from the c-relay source directory."
print_error "Makefile not found. Please run this script from the c-relay-pg source directory."
exit 1
fi
@@ -107,7 +107,7 @@ build_relay() {
make clean
make
if [[ ! -f "build/c_relay_x86" ]]; then
if [[ ! -f "build/c_relay_pg_x86" ]]; then
print_error "Build failed - binary not found"
exit 1
fi
@@ -118,9 +118,9 @@ build_relay() {
install_binary() {
print_step "Installing relay binary..."
cp build/c_relay_x86 "$INSTALL_DIR/"
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_x86"
chmod +x "$INSTALL_DIR/c_relay_x86"
cp build/c_relay_pg_x86 "$INSTALL_DIR/"
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_pg_x86"
chmod +x "$INSTALL_DIR/c_relay_pg_x86"
print_success "Binary installed to $INSTALL_DIR"
}
@@ -129,14 +129,14 @@ install_service() {
print_step "Installing systemd service..."
# Use the existing systemd service file
if [[ -f "systemd/c-relay.service" ]]; then
cp systemd/c-relay.service /etc/systemd/system/
if [[ -f "systemd/c-relay-pg.service" ]]; then
cp systemd/c-relay-pg.service /etc/systemd/system/
systemctl daemon-reload
print_success "Systemd service installed"
else
print_warning "Systemd service file not found, creating basic one..."
cat > /etc/systemd/system/c-relay.service << EOF
cat > /etc/systemd/system/c-relay-pg.service << EOF
[Unit]
Description=C Nostr Relay
After=network.target
@@ -146,7 +146,7 @@ Type=simple
User=$RELAY_USER
Group=$RELAY_USER
WorkingDirectory=$INSTALL_DIR
ExecStart=$INSTALL_DIR/c_relay_x86
ExecStart=$INSTALL_DIR/c_relay_pg_x86
Restart=always
RestartSec=5
@@ -1,4 +1,4 @@
# MUSL-based fully static C-Relay builder
# MUSL-based fully static C-Relay-PG builder
# Produces portable binaries with zero runtime dependencies
FROM alpine:latest AS builder
@@ -125,7 +125,7 @@ RUN cd /tmp && \
--prefix=/usr && \
make && make install
# Copy c-relay source
# Copy c-relay-pg source
COPY . /build/
# Initialize submodules
@@ -134,7 +134,7 @@ RUN git submodule update --init --recursive
# Build nostr_core_lib
RUN cd nostr_core_lib && ./build.sh
# Build c-relay static
# Build c-relay-pg static
RUN make clean && \
CC="musl-gcc -static" \
CFLAGS="-O2 -Wall -Wextra -std=c99 -g" \
@@ -143,8 +143,8 @@ RUN make clean && \
make
# Strip binary for size
RUN strip build/c_relay_x86
RUN strip build/c_relay_pg_x86
# Multi-stage build to produce minimal output
FROM scratch AS output
COPY --from=builder /build/build/c_relay_x86 /c_relay_static_musl_x86_64
COPY --from=builder /build/build/c_relay_pg_x86 /c_relay_pg_static_musl_x86_64
+16 -16
View File
@@ -19,7 +19,7 @@ RELEASE_MODE=false
VERSION_INCREMENT_TYPE="patch" # "patch", "minor", or "major"
show_usage() {
echo "C-Relay Increment and Push Script"
echo "C-Relay-PG Increment and Push Script"
echo ""
echo "USAGE:"
echo " $0 [OPTIONS] \"commit message\""
@@ -188,17 +188,17 @@ update_version_in_header() {
exit 1
fi
# Update VERSION macro
sed -i "s/#define VERSION \".*\"/#define VERSION \"$new_version\"/" src/main.h
# Update CRELAY_VERSION macro
sed -i "s/#define CRELAY_VERSION \".*\"/#define CRELAY_VERSION \"$new_version\"/" src/main.h
# Update VERSION_MAJOR macro
sed -i "s/#define VERSION_MAJOR [0-9]\+/#define VERSION_MAJOR $major/" src/main.h
# Update CRELAY_VERSION_MAJOR macro
sed -i "s/#define CRELAY_VERSION_MAJOR [0-9]\+/#define CRELAY_VERSION_MAJOR $major/" src/main.h
# Update VERSION_MINOR macro
sed -i "s/#define VERSION_MINOR .*/#define VERSION_MINOR $minor/" src/main.h
# Update CRELAY_VERSION_MINOR macro
sed -i "s/#define CRELAY_VERSION_MINOR .*/#define CRELAY_VERSION_MINOR $minor/" src/main.h
# Update VERSION_PATCH macro
sed -i "s/#define VERSION_PATCH [0-9]\+/#define VERSION_PATCH $patch/" src/main.h
# Update CRELAY_VERSION_PATCH macro
sed -i "s/#define CRELAY_VERSION_PATCH [0-9]\+/#define CRELAY_VERSION_PATCH $patch/" src/main.h
print_success "Updated version in src/main.h to $new_version"
}
@@ -330,7 +330,7 @@ build_release_binary() {
create_source_tarball() {
print_status "Creating source tarball..."
local tarball_name="c-relay-${NEW_VERSION#v}.tar.gz"
local tarball_name="c-relay-pg-${NEW_VERSION#v}.tar.gz"
# Create tarball excluding build artifacts and git files
if tar -czf "$tarball_name" \
@@ -365,7 +365,7 @@ upload_release_assets() {
fi
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
local assets_url="$api_url/releases/$release_id/assets"
print_status "Assets URL: $assets_url"
@@ -427,7 +427,7 @@ create_gitea_release() {
fi
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
# Create release
print_status "Creating release $NEW_VERSION..."
@@ -473,7 +473,7 @@ create_gitea_release() {
# Main execution
main() {
print_status "C-Relay Increment and Push Script"
print_status "C-Relay-PG Increment and Push Script"
# Check prerequisites
check_git_repo
@@ -505,13 +505,13 @@ main() {
# Build release binary
if build_release_binary; then
local binary_path="build/c_relay_static_x86_64"
local binary_path="build/c_relay_pg_static_x86_64"
else
print_warning "Binary build failed, continuing with release creation"
# Check if binary exists from previous build
if [[ -f "build/c_relay_static_x86_64" ]]; then
if [[ -f "build/c_relay_pg_static_x86_64" ]]; then
print_status "Using existing binary from previous build"
binary_path="build/c_relay_static_x86_64"
binary_path="build/c_relay_pg_static_x86_64"
else
binary_path=""
fi
+264 -15
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# C-Relay Build and Restart Script
# C-Relay-PG Build and Restart Script
# Builds the project first, then stops any running relay and starts a new one in the background
echo "=== C Nostr Relay Build and Restart Script ==="
@@ -13,6 +13,13 @@ ADMIN_KEY=""
RELAY_KEY=""
PORT_OVERRIDE=""
DEBUG_LEVEL="5"
DB_BACKEND="postgres"
DB_CONNSTRING=""
DB_HOST=""
DB_PORT=""
DB_NAME=""
DB_USER=""
DB_PASSWORD=""
# Key validation function
validate_hex_key() {
@@ -70,6 +77,24 @@ while [[ $# -gt 0 ]]; do
;;
--test-keys|-t)
USE_TEST_KEYS=true
# Read keys from .test_keys file
if [ -f ".test_keys" ]; then
echo "Reading test keys from .test_keys file..."
# Source the file to get the variables
source .test_keys
# Remove any single quotes from the values
# Note: -a flag expects ADMIN_PUBKEY (public key), not ADMIN_PRIVKEY
ADMIN_KEY=$(echo "$ADMIN_PUBKEY" | tr -d "'")
RELAY_KEY=$(echo "$SERVER_PRIVKEY" | tr -d "'")
echo "Using admin pubkey from .test_keys: ${ADMIN_KEY:0:16}..."
echo "Using relay privkey from .test_keys: ${RELAY_KEY:0:16}..."
else
echo "ERROR: .test_keys file not found"
echo "Please create a .test_keys file with the following format:"
echo " ADMIN_PUBKEY='your_admin_public_key_hex'"
echo " SERVER_PRIVKEY='your_relay_private_key_hex'"
exit 1
fi
shift
;;
--debug-level=*)
@@ -100,6 +125,84 @@ while [[ $# -gt 0 ]]; do
shift 2
fi
;;
--db-backend)
if [ -z "$2" ]; then
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
HELP=true
shift
else
DB_BACKEND="$2"
shift 2
fi
;;
--db-backend=*)
DB_BACKEND="${1#*=}"
shift
;;
--db-connstring)
if [ -z "$2" ]; then
echo "ERROR: --db-connstring requires a value"
HELP=true
shift
else
DB_CONNSTRING="$2"
shift 2
fi
;;
--db-connstring=*)
DB_CONNSTRING="${1#*=}"
shift
;;
--db-host)
if [ -z "$2" ]; then
echo "ERROR: --db-host requires a value"
HELP=true
shift
else
DB_HOST="$2"
shift 2
fi
;;
--db-port)
if [ -z "$2" ]; then
echo "ERROR: --db-port requires a value"
HELP=true
shift
else
DB_PORT="$2"
shift 2
fi
;;
--db-name)
if [ -z "$2" ]; then
echo "ERROR: --db-name requires a value"
HELP=true
shift
else
DB_NAME="$2"
shift 2
fi
;;
--db-user)
if [ -z "$2" ]; then
echo "ERROR: --db-user requires a value"
HELP=true
shift
else
DB_USER="$2"
shift 2
fi
;;
--db-password)
if [ -z "$2" ]; then
echo "ERROR: --db-password requires a value"
HELP=true
shift
else
DB_PASSWORD="$2"
shift 2
fi
;;
--help|-h)
HELP=true
shift
@@ -146,6 +249,98 @@ if [ -n "$DEBUG_LEVEL" ]; then
fi
fi
# Validate DB backend
if [ "$DB_BACKEND" != "sqlite" ] && [ "$DB_BACKEND" != "postgres" ]; then
echo "ERROR: Invalid --db-backend value '$DB_BACKEND'. Use sqlite or postgres."
exit 1
fi
# Validate DB port if provided
if [ -n "$DB_PORT" ]; then
if ! [[ "$DB_PORT" =~ ^[0-9]+$ ]] || [ "$DB_PORT" -lt 1 ] || [ "$DB_PORT" -gt 65535 ]; then
echo "ERROR: --db-port must be a number between 1 and 65535"
exit 1
fi
fi
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
[ -z "$DB_HOST" ] && DB_HOST="localhost"
[ -z "$DB_PORT" ] && DB_PORT="5432"
[ -z "$DB_NAME" ] && DB_NAME="crelay"
[ -z "$DB_USER" ] && DB_USER="crelay"
[ -z "$DB_PASSWORD" ] && DB_PASSWORD="crelay"
fi
ensure_postgres_database() {
if ! command -v psql >/dev/null 2>&1; then
echo "ERROR: psql not found. Install PostgreSQL client tools to continue."
return 1
fi
echo "Ensuring PostgreSQL database exists: host=$DB_HOST port=$DB_PORT dbname=$DB_NAME user=$DB_USER"
if [ -n "$DB_PASSWORD" ]; then
export PGPASSWORD="$DB_PASSWORD"
fi
# Prefer local postgres superuser for provisioning when available.
if sudo -n -u postgres psql -d postgres -tAc "SELECT 1" >/dev/null 2>&1; then
sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname = '$DB_USER'" | grep -q 1 || \
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "CREATE ROLE \"$DB_USER\" LOGIN PASSWORD '$DB_PASSWORD';" || return 1
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "ALTER ROLE \"$DB_USER\" WITH LOGIN PASSWORD '$DB_PASSWORD';" >/dev/null || return 1
if [ "$PRESERVE_DATABASE" = false ]; then
echo "Resetting PostgreSQL database '$DB_NAME' for fresh start..."
# Ensure no active sessions block DROP DATABASE (common when relay is still running).
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '$DB_NAME' AND pid <> pg_backend_pid();" >/dev/null || return 1
# Prefer force drop when supported; fallback to regular dropdb.
if ! sudo -u postgres dropdb --if-exists --force "$DB_NAME" >/dev/null 2>&1; then
sudo -u postgres dropdb --if-exists "$DB_NAME" >/dev/null 2>&1 || return 1
fi
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
echo "✓ Recreated PostgreSQL database '$DB_NAME' owned by '$DB_USER'"
else
DB_EXISTS=$(sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
if [ "$DB_EXISTS" != "1" ]; then
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
echo "✓ Created PostgreSQL database '$DB_NAME'"
else
echo "✓ PostgreSQL database '$DB_NAME' already exists"
fi
fi
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "GRANT ALL PRIVILEGES ON DATABASE \"$DB_NAME\" TO \"$DB_USER\";" >/dev/null || return 1
sudo -u postgres psql -d "$DB_NAME" -v ON_ERROR_STOP=1 -c "ALTER SCHEMA public OWNER TO \"$DB_USER\"; GRANT ALL ON SCHEMA public TO \"$DB_USER\";" >/dev/null || return 1
return 0
fi
# Fallback: use configured DB user directly.
if ! psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -v ON_ERROR_STOP=1 -tAc "SELECT 1" >/dev/null 2>&1; then
echo "ERROR: Cannot connect to PostgreSQL server/database 'postgres' with current settings"
echo " host=$DB_HOST port=$DB_PORT user=$DB_USER"
return 1
fi
DB_EXISTS=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
if [ "$DB_EXISTS" != "1" ]; then
echo "Database '$DB_NAME' not found. Creating..."
if ! createdb -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" "$DB_NAME" >/dev/null 2>&1; then
echo "ERROR: Failed to create PostgreSQL database '$DB_NAME'"
return 1
fi
echo "✓ Created PostgreSQL database '$DB_NAME'"
else
echo "✓ PostgreSQL database '$DB_NAME' already exists"
fi
return 0
}
# Show help
if [ "$HELP" = true ]; then
echo "Usage: $0 [OPTIONS]"
@@ -157,6 +352,13 @@ if [ "$HELP" = true ]; then
echo " -d, --debug-level <0-5> Set debug level: 0=none, 1=errors, 2=warnings, 3=info, 4=debug, 5=trace"
echo " --preserve-database Keep existing database files (don't delete for fresh start)"
echo " --test-keys, -t Use deterministic test keys for development (admin: all 'a's, relay: all '1's)"
echo " --db-backend <name> Database backend: postgres (default) or sqlite"
echo " --db-connstring <str> PostgreSQL libpq connection string"
echo " --db-host <host> PostgreSQL host"
echo " --db-port <port> PostgreSQL port"
echo " --db-name <name> PostgreSQL database name"
echo " --db-user <user> PostgreSQL database user"
echo " --db-password <pass> PostgreSQL database password"
echo " --help, -h Show this help message"
echo ""
echo "Event-Based Configuration:"
@@ -176,6 +378,9 @@ if [ "$HELP" = true ]; then
echo " $0 --test-keys # Use test keys for consistent development"
echo " $0 -t --preserve-database # Use test keys and preserve database"
echo ""
echo "Default PostgreSQL connection (when no DB flags provided):"
echo " host=localhost port=5432 dbname=crelay user=crelay password=crelay"
echo ""
echo "Key Format: Keys must be exactly 64 hexadecimal characters (0-9, a-f, A-F)"
echo "Default behavior: Deletes existing database files to start fresh with new keys"
echo " for development purposes"
@@ -199,13 +404,17 @@ fi
rm -rf dev-config/ 2>/dev/null
rm -f db/c_nostr_relay.db* 2>/dev/null
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
ensure_postgres_database || exit 1
fi
# Embed web files into C headers before building
echo "Embedding web files..."
./embed_web_files.sh
# Build the project - ONLY static build
echo "Building project (static binary with SQLite JSON1 extension)..."
./build_static.sh
echo "Building project (static binary, backend: $DB_BACKEND)..."
./build_static.sh --db-backend "$DB_BACKEND"
# Exit if static build fails - no fallback
if [ $? -ne 0 ]; then
@@ -224,13 +433,13 @@ fi
ARCH=$(uname -m)
case "$ARCH" in
x86_64)
BINARY_PATH="./build/c_relay_static_x86_64"
BINARY_PATH="./build/c_relay_pg_static_x86_64"
;;
aarch64|arm64)
BINARY_PATH="./build/c_relay_static_arm64"
BINARY_PATH="./build/c_relay_pg_static_arm64"
;;
*)
BINARY_PATH="./build/c_relay_static_$ARCH"
BINARY_PATH="./build/c_relay_pg_static_$ARCH"
;;
esac
@@ -246,13 +455,26 @@ fi
echo "Using static binary: $BINARY_PATH"
# Build the caching_relay binary (from caching/ directory)
# This is needed because the caching service launcher forks this binary.
if [ -d "caching" ] && [ -f "caching/Makefile" ]; then
echo "Building caching_relay binary..."
(cd caching && make) > /dev/null 2>&1
if [ $? -ne 0 ]; then
echo "WARNING: caching_relay build failed. The caching service will not be available."
echo "You can build it manually with: cd caching && make"
else
echo "✓ caching_relay binary built successfully"
fi
fi
echo "Build successful. Proceeding with relay restart..."
# Kill existing relay if running - start aggressive immediately
echo "Stopping any existing relay servers..."
# Get all relay processes and kill them immediately with -9
RELAY_PIDS=$(pgrep -f "c_relay_" || echo "")
RELAY_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$RELAY_PIDS" ]; then
echo "Force killing relay processes immediately: $RELAY_PIDS"
kill -9 $RELAY_PIDS 2>/dev/null
@@ -273,7 +495,7 @@ for attempt in {1..15}; do
fuser -k 8888/tcp 2>/dev/null || true
# Double-check for any remaining relay processes
REMAINING_PIDS=$(pgrep -f "c_relay_" || echo "")
REMAINING_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$REMAINING_PIDS" ]; then
echo "Killing remaining relay processes: $REMAINING_PIDS"
kill -9 $REMAINING_PIDS 2>/dev/null || true
@@ -291,7 +513,7 @@ for attempt in {1..15}; do
done
# Final safety check - ensure no relay processes remain
FINAL_PIDS=$(pgrep -f "c_relay_" || echo "")
FINAL_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$FINAL_PIDS" ]; then
echo "Final cleanup: killing processes $FINAL_PIDS"
kill -9 $FINAL_PIDS 2>/dev/null || true
@@ -307,7 +529,7 @@ echo "Database will be initialized automatically on startup if needed"
# Start relay in background with output redirection
echo "Starting relay server..."
echo "Debug: Current processes: $(ps aux | grep 'c_relay_' | grep -v grep || echo 'None')"
echo "Debug: Current processes: $(ps aux | grep 'c_relay_pg_' | grep -v grep || echo 'None')"
# Build command line arguments for relay binary
RELAY_ARGS=""
@@ -332,15 +554,42 @@ if [ -n "$DEBUG_LEVEL" ]; then
echo "Using debug level: $DEBUG_LEVEL"
fi
if [ -n "$DB_CONNSTRING" ]; then
RELAY_ARGS="$RELAY_ARGS --db-connstring '$DB_CONNSTRING'"
echo "Using PostgreSQL connection string override"
fi
if [ -n "$DB_HOST" ]; then
RELAY_ARGS="$RELAY_ARGS --db-host '$DB_HOST'"
fi
if [ -n "$DB_PORT" ]; then
RELAY_ARGS="$RELAY_ARGS --db-port '$DB_PORT'"
fi
if [ -n "$DB_NAME" ]; then
RELAY_ARGS="$RELAY_ARGS --db-name '$DB_NAME'"
fi
if [ -n "$DB_USER" ]; then
RELAY_ARGS="$RELAY_ARGS --db-user '$DB_USER'"
fi
if [ -n "$DB_PASSWORD" ]; then
RELAY_ARGS="$RELAY_ARGS --db-password '$DB_PASSWORD'"
fi
# Change to build directory before starting relay so database files are created there
cd build
# Start relay in background and capture its PID
if [ "$USE_TEST_KEYS" = true ]; then
echo "Using deterministic test keys for development..."
./$(basename $BINARY_PATH) -a 6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3 -r 1111111111111111111111111111111111111111111111111111111111111111 --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
echo "Using test keys from .test_keys file..."
# shellcheck disable=SC2086
eval ./$(basename $BINARY_PATH) -a "$ADMIN_KEY" -r "$RELAY_KEY" $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
elif [ -n "$RELAY_ARGS" ]; then
echo "Starting relay with custom configuration..."
./$(basename $BINARY_PATH) $RELAY_ARGS --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
# shellcheck disable=SC2086
eval ./$(basename $BINARY_PATH) $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
else
# No command line arguments needed for random key generation
echo "Starting relay with random key generation..."
@@ -410,8 +659,8 @@ if ps -p "$RELAY_PID" >/dev/null 2>&1; then
echo "=== Event-Based Relay Server Running ==="
echo "Configuration: Event-based (kind 33334 Nostr events)"
echo "Database: Automatically created with relay pubkey naming"
echo "To kill relay: pkill -f 'c_relay_'"
echo "To check status: ps aux | grep c_relay_"
echo "To kill relay: pkill -f 'c_relay_pg_'"
echo "To check status: ps aux | grep c_relay_pg_"
echo "To view logs: tail -f relay.log"
echo "Binary: $BINARY_PATH (zero configuration needed)"
echo "Ready for Nostr client connections!"
+1 -1
View File
@@ -1,3 +1,3 @@
#!/bin/bash
curl -H "Accept: application/nostr+json" http://localhost:8888/
curl -H "Accept: application/nostr+json" https://relay.laantungir.net
+12 -12
View File
@@ -12,11 +12,11 @@ After the next crash, analyze it:
# List all core dumps (most recent first)
sudo coredumpctl list
# View info about the most recent c-relay crash
sudo coredumpctl info c-relay
# View info about the most recent c-relay-pg crash
sudo coredumpctl info c-relay-pg
# Load the core dump in gdb for detailed analysis
sudo coredumpctl gdb c-relay
sudo coredumpctl gdb c-relay-pg
Inside gdb, run these commands:
(gdb) bt full # Full backtrace with all variables
@@ -36,8 +36,8 @@ DEBUGGING
Even simpler: Use this one-liner
# Start relay and immediately attach gdb
cd /usr/local/bin/c_relay
sudo -u c-relay ./c_relay --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay)
cd /usr/local/bin/c_relay_pg
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay_pg)
Inside gdb, after attaching:
@@ -48,17 +48,17 @@ Or shorter:
How to View the Logs
Check systemd journal:
# View all c-relay logs
sudo journalctl -u c-relay
# View all c-relay-pg logs
sudo journalctl -u c-relay-pg
# View recent logs (last 50 lines)
sudo journalctl -u c-relay -n 50
sudo journalctl -u c-relay-pg -n 50
# Follow logs in real-time
sudo journalctl -u c-relay -f
sudo journalctl -u c-relay-pg -f
# View logs since last boot
sudo journalctl -u c-relay -b
sudo journalctl -u c-relay-pg -b
Check if service is running:
@@ -78,9 +78,9 @@ sudo systemctl start rsyslog
sudo systemctl status rsyslog
sudo -u c-relay ./c_relay --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
./c_relay_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
./c_relay_pg_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
sudo ufw allow 8888/tcp
+1 -1
View File
@@ -1,5 +1,5 @@
{
"name": "c-relay",
"name": "c-relay-pg",
"lockfileVersion": 3,
"requires": true,
"packages": {
+178
View File
@@ -0,0 +1,178 @@
# API Upgrade Implementation Plan — Phases 1, 2, and 5
Goal: finish the remaining work from `plans/api_upgrade_plan.md`:
- **Phase 1 (design fix):** convert the `api-worker` thread from job-queue-driven to **timer + subscriber-check** driven.
- **Phase 2:** remove the per-event and per-subscription monitoring triggers so monitoring no longer runs on the event-processing path.
- **Phase 5:** add **PostgreSQL `LISTEN`/`NOTIFY`** so the worker wakes on real data changes instead of pure polling.
Target test relay: **port 7777** (local).
---
## Current State (verified in code)
- `api_worker_main()` ([`src/api.c:172`](src/api.c:172)) blocks on `api_worker_pop_job_blocking()` and reacts to three job types: `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `API_WORK_JOB_STATUS_POST`.
- Monitoring hooks `monitoring_on_event_stored()` ([`src/api.c:752`](src/api.c:752)) and `monitoring_on_subscription_change()` ([`src/api.c:759`](src/api.c:759)) are called from [`src/main.c:1169`](src/main.c:1169) and [`src/subscriptions.c:483`](src/subscriptions.c:483), [`src/subscriptions.c:536`](src/subscriptions.c:536). They enqueue jobs; sync fallback runs if the worker is down.
- Throttling lives in `monitoring_on_event_stored_sync()` / `monitoring_on_subscription_change_sync()` ([`src/api.c:711`](src/api.c:711), [`src/api.c:731`](src/api.c:731)).
- Round-robin selection in `generate_round_robin_monitoring_event()` ([`src/api.c:571`](src/api.c:571)); per-type build/sign/broadcast in `generate_monitoring_event_for_type()` ([`src/api.c:621`](src/api.c:621)).
- `has_subscriptions_for_kind(int)` ([`src/subscriptions.c:980`](src/subscriptions.c:980)) checks only subscriptions with an explicit `kinds` filter — it does **not** consider the `no_kind_filter_subs` list, so it alone is insufficient for a "is anyone listening to kind 24567" check.
- Worker DB connection is a `PGconn*` opened via `db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)). Schema is embedded in [`src/pg_schema.h`](src/pg_schema.h) (version `"4"`) and mirrored in [`src/pg_schema.sql`](src/pg_schema.sql).
- No `LISTEN`/`NOTIFY`/`pg_notify`/`PQnotifies` usage exists anywhere in `src/`.
---
## Design
### Phase 1 + 2 + 5 combined worker loop
The three phases are coupled, so they are implemented together in one rewrite of `api_worker_main()` and its helpers.
### CRITICAL BUG FIX (why the web API isn't getting updates today)
`postgres_db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)) calls
`postgres_db_set_thread_connection(conn)` at line 243 — but `g_thread_pg_conn` is declared `__thread`
([`src/db_ops_postgres.c:45`](src/db_ops_postgres.c:45)), so the binding is set in the **caller's thread**
(lws-main, which calls `start_api_worker`), NOT in the api-worker thread. When `api_worker_main` runs and
the monitoring query functions call `postgres_db_active_connection()`, `g_thread_pg_conn` is NULL in that
thread, so they fall back to the shared global `g_pg_conn` — concurrent access with the main thread. This
causes monitoring queries to fail or interleave, which is why the dashboard stops receiving kind 24567
events.
**Fix:** inside `api_worker_main()`, after opening `worker_db`, call
`db_set_thread_connection(worker_db)` so the `__thread` pointer is bound in the api-worker thread itself.
Call `db_clear_thread_connection()` before closing the connection on shutdown. This must be done in the
new rewrite regardless of the loop design.
```
api-worker thread:
1. Open own PG connection.
2. db_set_thread_connection(worker_db) // bind __thread pointer in THIS thread
3. If PG backend: LISTEN event_stored on this connection.
4. Loop while running:
a. Determine if anyone is listening to kind 24567
(has_subscriptions_for_kind(24567) OR no_kind_filter_subs non-empty).
b. If NO subscribers:
- condvar_timedwait(throttle_sec) // wakeable for shutdown / STATUS_POST job
- on wake: process any pending STATUS_POST job, then continue.
// Zero DB work, zero notify polling.
c. If subscribers exist:
- wait_for_notify_or_timeout(throttle_sec):
select() on { PQsocket, self-pipe read fd } with timeout = throttle_sec.
- If PQsocket readable: PQconsumeInput + drain PQnotifies.
- If self-pipe readable: drain (shutdown or STATUS_POST signal).
- If timeout: fall through.
- generate_round_robin_monitoring_event() // one d-tag per tick
- process any pending STATUS_POST job.
4. On shutdown: UNLISTEN, close PG connection.
```
Key properties:
- **Zero overhead when no subscribers** — condvar sleep, no `select`, no `LISTEN` polling.
- **Reactive when subscribers exist** — wakes within `throttle_sec` of an `event_stored` NOTIFY, but never more than once per `throttle_sec` (rate-limited by the select timeout + the round-robin cadence already in `generate_round_robin_monitoring_event`).
- **STATUS_POST preserved** — still enqueued from [`src/websockets.c:3470`](src/websockets.c:3470); the self-pipe wakes the worker's `select` so status posts are not delayed by a pending notify wait.
- **SQLite fallback**`db_worker_poll_notify()` returns "not supported" on SQLite, so the worker falls back to pure timer behavior (condvar timed wait) — identical to the no-PG path.
### Self-pipe for job/shutdown signalling
A self-pipe (or `eventfd` on Linux) is added to `api.c`:
- `api_worker_enqueue_job(STATUS_POST)` writes one byte to the pipe → wakes `select`.
- `stop_api_worker()` sets `g_api_worker_running = 0`, writes a byte, and broadcasts the condvar (covers both the subscriber and no-subscriber wait paths).
### Subscriber-check helper
Add `int has_any_subscription_for_kind(int event_kind)` in [`src/subscriptions.c`](src/subscriptions.c) that returns true if `has_subscriptions_for_kind(event_kind)` OR the `no_kind_filter_subs` list is non-empty. Export it in [`src/subscriptions.h`](src/subscriptions.h). The worker calls `has_any_subscription_for_kind(24567)`.
### DB abstraction for LISTEN/NOTIFY
Add to [`src/db_ops.h`](src/db_ops.h) / [`src/db_ops_postgres.c`](src/db_ops_postgres.c) / [`src/db_ops_sqlite.c`](src/db_ops_sqlite.c):
- `int db_worker_listen(void* conn, const char* channel);` — issues `LISTEN <channel>`.
- `int db_worker_poll_notify(void* conn, int timeout_ms);` — returns `1` if a notification was consumed, `0` on timeout, `-1` on error/unsupported. Uses `PQsocket()`, `select()`, `PQconsumeInput()`, `PQnotifies()`.
- SQLite stubs return `-1` (unsupported) so the worker falls back to timer mode.
### Schema change (Phase 5)
Add to both [`src/pg_schema.sql`](src/pg_schema.sql) and [`src/pg_schema.h`](src/pg_schema.h):
```sql
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('event_stored', json_build_object(
'kind', NEW.kind,
'pubkey', substring(NEW.pubkey, 1, 8)
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_notify_event_stored ON events;
CREATE TRIGGER trg_notify_event_stored
AFTER INSERT ON events
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
```
Bump `EMBEDDED_PG_SCHEMA_VERSION` from `"4"` to `"5"` in [`src/pg_schema.h`](src/pg_schema.h) and the matching `schema_info` version write in [`src/pg_schema.sql`](src/pg_schema.sql). The trigger is idempotent (`DROP TRIGGER IF EXISTS` + `CREATE`), so existing databases upgrade automatically on next startup via `postgres_db_apply_schema()` ([`src/db_ops_postgres.c:134`](src/db_ops_postgres.c:134)).
### Phase 2 removals
- Delete the `monitoring_on_event_stored()` call at [`src/main.c:1169`](src/main.c:1169) (and its forward decl at [`src/main.c:459`](src/main.c:459)).
- Delete the `monitoring_on_subscription_change()` calls at [`src/subscriptions.c:483`](src/subscriptions.c:483) and [`src/subscriptions.c:536`](src/subscriptions.c:536) (and its forward decl at [`src/subscriptions.c:58`](src/subscriptions.c:58)).
- Remove from [`src/api.h`](src/api.h): `monitoring_on_event_stored`, `monitoring_on_subscription_change`.
- Remove from [`src/api.c`](src/api.c): `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `monitoring_on_event_stored`, `monitoring_on_subscription_change`, `monitoring_on_event_stored_sync`, `monitoring_on_subscription_change_sync`. Keep `API_WORK_JOB_STATUS_POST`, `generate_round_robin_monitoring_event`, `generate_monitoring_event_for_type`, `generate_event_driven_monitoring`/`generate_subscription_driven_monitoring` (still used by `generate_monitoring_event` legacy path and tests).
- The `generate_event_driven_monitoring` / `generate_subscription_driven_monitoring` wrappers become unused by the worker but remain as public helpers for any external/test callers; the worker calls `generate_round_robin_monitoring_event()` directly.
### Thread layout (unchanged from plan target)
```
c_relay_pg process
├── lws-main — WebSocket event loop (no monitoring DB work)
├── db-read-1..N — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion (no monitoring hook)
├── db-write-1..N — Async sub logging, misc writes
└── api-worker — Timer + LISTEN/NOTIFY driven monitoring (own PG conn)
```
---
## Mermaid: api-worker state machine
```mermaid
stateDiagram-v2
[*] --> OpenConn
OpenConn --> BindThread: db_set_thread_connection
BindThread --> Listen: PG backend
BindThread --> Ready: SQLite/no-PG
Listen --> Ready
Ready --> CheckSubs
CheckSubs --> NoSubSleep: no kind-24567 subs
CheckSubs --> WaitNotify: subs present
NoSubSleep --> CheckSubs: throttle_sec or wake
WaitNotify --> Generate: NOTIFY or throttle_sec
Generate --> CheckSubs: after round-robin tick
WaitNotify --> NoSubSleep: subs dropped + wake
Ready --> [*]: stop_api_worker
```
---
## Implementation Todo List
1. Add `has_any_subscription_for_kind()` to `subscriptions.c`/`subscriptions.h` (checks kind index + no-kind-filter list).
2. Add `db_worker_listen()` and `db_worker_poll_notify()` to `db_ops.h`, `db_ops_postgres.c` (libpq), and `db_ops_sqlite.c` (stub).
3. Add `notify_event_stored()` function + `trg_notify_event_stored` trigger to `pg_schema.sql` and `pg_schema.h`; bump schema version to 5.
4. Rewrite `api_worker_main()` in `api.c`: bind the worker PG connection to the thread via `db_set_thread_connection()` (fixes the dashboard-not-updating bug), then run the timer + LISTEN/NOTIFY + subscriber-check loop; add self-pipe for STATUS_POST/shutdown wakeups; keep `start_api_worker`/`stop_api_worker`/`api_worker_process_completions`/`api_worker_enqueue_status_post` signatures.
5. Remove `API_WORK_JOB_EVENT_STORED`/`API_WORK_JOB_SUBSCRIPTION_CHANGE` job types and the `monitoring_on_event_stored`/`monitoring_on_subscription_change` functions + sync wrappers from `api.c` and `api.h`.
6. Remove the `monitoring_on_event_stored()` call from `main.c` (event storage path).
7. Remove the `monitoring_on_subscription_change()` calls from `subscriptions.c` (subscription create/close paths).
8. Build with `./make_and_restart_relay.sh` on port 7777; fix any compile errors.
9. Test: subscribe to kind 24567 on ws://localhost:7777 → confirm monitoring events arrive on throttle cadence (this validates the thread-connection fix).
10. Test: publish a kind-1 event → confirm a NOTIFY-driven monitoring event fires within throttle_sec.
11. Test: with no kind-24567 subscribers → confirm no monitoring events generated (zero overhead) and event storage latency unaffected.
12. Run `tests/quick_error_tests.sh` and `tests/subscribe_all.sh` against port 7777 to confirm no regressions.
---
## Risks & Mitigations
- **Self-pipe + condvar mix complexity.** Mitigation: the no-subscriber path uses only the condvar; the subscriber path uses `select` on the self-pipe + PQ socket. The condvar is only used to wake the no-subscriber sleep, so the two wait mechanisms never overlap.
- **`LISTEN` on a worker connection that also runs monitoring queries.** `LISTEN` is connection-local and persists; running `SELECT`s on the same connection does not cancel it. `PQconsumeInput` must be called before `PQnotifies`. Mitigation: encapsulate all of this in `db_worker_poll_notify()`.
- **Schema trigger on every insert adds per-event overhead.** `pg_notify` is cheap (in-memory queue) and the payload is tiny. The existing `sync_event_tags_from_events` AFTER INSERT trigger already does far more work per row, so this is negligible by comparison.
- **SQLite has no LISTEN/NOTIFY.** Mitigation: `db_worker_poll_notify` returns `-1` on SQLite and the worker falls back to pure timer mode — functionally identical to the no-notify path.
- **Removing the per-event monitoring hook changes dashboard freshness behavior.** Mitigation: with subscribers present, NOTIFY now drives near-real-time updates (better than before); with no subscribers, nothing is generated (matches the plan's "zero overhead" goal). Dashboard already handles event-driven updates via its kind-24567 subscription.
+224
View File
@@ -0,0 +1,224 @@
# API Upgrade Plan: Monitoring Thread + PostgreSQL Push
## Current API Architecture
### How Monitoring Works Today
The relay publishes monitoring data as **kind 24567 ephemeral Nostr events**. The admin dashboard is a standard Nostr client that subscribes to these events.
```
Dashboard connects → subscribes to kind 24567 with d-tags
Relay detects subscription → starts generating monitoring events
Relay queries DB → builds JSON → signs event → broadcasts to subscribers
Dashboard receives events → renders stats/charts
```
### Current Monitoring Event Types
| d-tag | Query Function | What It Contains | Triggered By |
|-------|---------------|-----------------|-------------|
| `event_kinds` | `query_event_kind_distribution()` | Event count per kind, total events | Event storage |
| `time_stats` | `query_time_based_statistics()` | Events in last 24h/7d/30d | Event storage |
| `top_pubkeys` | `query_top_pubkeys()` | Top 10 pubkeys by event count | Event storage |
| `subscription_details` | `query_subscription_details()` | Active subscriptions list | Subscription changes |
| `cpu_metrics` | `query_cpu_metrics()` | PID, memory, CPU, connections | Both triggers |
### Current Admin Command System
Two parallel admin interfaces exist:
**1. Kind 23456 Admin Events (WebSocket)**
- Admin sends NIP-44 encrypted commands as kind 23456 events
- Relay decrypts, processes, responds with kind 23457 events
- Commands: config get/set, auth rules, system commands, SQL queries
**2. NIP-17 DM Admin (Direct Messages)**
- Admin sends gift-wrapped DMs (kind 1059) to relay
- Relay unwraps, processes commands, responds via DM
- Commands: stats, config, help, SQL queries
### Current Problems
1. **All monitoring runs on lws-main thread** — DB queries block the event loop
2. **Monitoring is triggered by event storage**`generate_event_driven_monitoring()` runs synchronously after each event insert
3. **No dedicated monitoring thread** — monitoring piggybacks on the relay's main processing
4. **Throttle is time-based only**`kind_24567_reporting_throttle_sec` (default 5s) but queries still run on main thread
5. **Dashboard polls via subscription** — no way to get data without the relay actively generating events
---
## Upgrade Plan
### Phase 1: Dedicated API Thread (`api-worker`)
**What:** Create a new thread that owns all monitoring event generation. It runs on a timer, queries PostgreSQL with its own connection, builds and signs monitoring events, and pushes them into the broadcast system.
**Thread behavior:**
```
api-worker thread:
1. Open own PG connection
2. Loop:
a. Sleep for throttle_sec interval
b. Check if anyone is subscribed to kind 24567
c. If yes: run all monitoring queries
d. Build kind 24567 events with results
e. Sign events with relay key
f. Push completed events to lws-main for broadcast
g. Repeat
```
**Key design decisions:**
- The api-worker thread does ALL the heavy work (queries + signing)
- It only hands off pre-built, signed events to lws-main for broadcast
- lws-main never blocks on monitoring queries
- The thread sleeps when no one is subscribed (zero overhead)
**Files to change:**
- `src/api.c`: Refactor `generate_monitoring_event_for_type()` to be callable from any thread
- `src/websockets.c`: Add monitor thread lifecycle (start/stop), completion queue for broadcast
- `src/thread_pool.h`: Add api-worker thread config options
**New thread in the process:**
```
c_relay_pg process
├── lws-main — WebSocket event loop
├── db-read-1..N — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion
├── db-write-1..N — Async sub logging, misc writes
└── api-worker — NEW: periodic monitoring event generation
```
### Phase 2: Remove Monitoring from Main Thread Path
**What:** Remove `generate_event_driven_monitoring()` and `generate_subscription_driven_monitoring()` calls from the synchronous event processing path.
**Currently these are called from:**
- After event storage (event-driven monitoring)
- After subscription create/close (subscription-driven monitoring)
**After change:**
- The api-worker thread handles all monitoring on its own timer
- Event storage and subscription changes no longer trigger monitoring directly
- The throttle interval controls how fresh the data is (default 5 seconds)
**Benefit:** Event processing becomes faster — no monitoring overhead per event.
### Phase 3: Dashboard Uses Normal Subscriptions for Real-Time Events
**What:** Instead of the relay generating special monitoring events for "recent events," the dashboard simply subscribes to all events using standard Nostr REQ filters.
**Current approach:** Dashboard subscribes to kind 24567 monitoring events that contain aggregated stats.
**New approach for real-time event feed:**
```javascript
// Dashboard subscribes to ALL events on the relay
relayPool.subscribeMany([url], [
{ kinds: [1, 3, 5, 7, ...], limit: 50 } // Recent events
], {
onevent(event) {
// Render in real-time event feed
addEventToFeed(event);
}
});
```
**What this replaces:** The "Relay Events" page in the dashboard currently shows events from monitoring. Instead, it would show live events as they arrive via normal Nostr subscription.
**What stays as monitoring events:** Aggregated stats (event_kinds, time_stats, top_pubkeys, cpu_metrics, subscription_details) still need to be generated by the relay because they require SQL aggregation queries that a client can't do.
### Phase 4: Evaluate Which Monitoring Events Can Be Replaced
For each current monitoring event type, here's whether it should stay as a relay-generated event or be replaced by dashboard-side logic:
| d-tag | Current: Relay generates | Could dashboard do it? | Recommendation |
|-------|------------------------|----------------------|----------------|
| `event_kinds` | SQL: `SELECT kind, COUNT(*) FROM events GROUP BY kind` | No — requires DB aggregation | **Keep as monitoring event** |
| `time_stats` | SQL: `SELECT COUNT(*) FROM events WHERE created_at > ?` for 24h/7d/30d | No — requires DB aggregation | **Keep as monitoring event** |
| `top_pubkeys` | SQL: `SELECT pubkey, COUNT(*) FROM events GROUP BY pubkey ORDER BY count DESC LIMIT 10` | No — requires DB aggregation | **Keep as monitoring event** |
| `subscription_details` | SQL: `SELECT * FROM active_subscriptions_log` | No — requires DB access | **Keep as monitoring event** |
| `cpu_metrics` | Reads `/proc/self/stat`, memory info | No — requires server-side access | **Keep as monitoring event** |
| Real-time event feed | Currently via monitoring events | **Yes — normal Nostr subscription** | **Replace with REQ subscription** |
| Auth rules list | Currently via admin command | No — requires DB access | **Keep as admin command** |
| Config values | Currently via admin command | No — requires DB access | **Keep as admin command** |
**Conclusion:** All 5 monitoring event types should stay as relay-generated events (they require SQL aggregation or server-side data). The only thing that changes is the real-time event feed, which becomes a normal subscription.
### Phase 5: PostgreSQL LISTEN/NOTIFY for Monitoring Triggers (Future)
**What:** Instead of the monitor thread polling on a timer, PostgreSQL can push notifications when data changes.
**Example triggers:**
```sql
-- Notify when a new event is stored
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('event_stored', json_build_object(
'kind', NEW.kind,
'pubkey', substring(NEW.pubkey, 1, 8)
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_notify_event_stored
AFTER INSERT ON events
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
```
**How the monitor thread would use it:**
```
api-worker thread:
1. LISTEN event_stored
2. Wait for notification (blocks efficiently)
3. On notification: batch up for throttle_sec, then generate monitoring events
4. Repeat
```
**Benefit:** Zero-polling. The monitor thread only wakes up when data actually changes.
**This is optional and can be added later.** The timer-based approach in Phase 1 works well and is simpler.
---
## Updated Thread Layout (After All Phases)
```
c_relay_pg process
├── lws-main — WebSocket event loop (never blocks on DB for monitoring)
├── db-read-1..4 — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion (validate + store)
├── db-write-1..2 — Async sub logging, IP bans, misc writes
└── api-worker — Periodic monitoring event generation (own PG conn)
```
**Total: 9 threads, 9 PG connections** (with 4 readers + 2 writers)
---
## Implementation Priority
| Phase | Description | Complexity | Impact |
|-------|-------------|-----------|--------|
| 1 | Dedicated api-worker thread | Medium | High — unblocks main thread |
| 2 | Remove monitoring from event processing path | Low | Medium — faster event processing |
| 3 | Dashboard uses normal subscriptions for event feed | Low (frontend only) | Medium — simpler, more reliable |
| 4 | Evaluate monitoring event types | Analysis only | Confirms architecture |
| 5 | PostgreSQL LISTEN/NOTIFY triggers | Medium | Low (optimization) — add later |
**Recommended start:** Phase 1 + 2 together (they're coupled), then Phase 3 (frontend change), then Phase 5 when needed.
---
## Relationship to Thread Pool Plan
This plan complements `thread_pool_pg_optimization_plan.md`:
- Thread pool plan handles **reader/writer scaling** for client-facing operations
- This plan handles **monitoring/API operations** on a dedicated thread
- Both plans share the same PG connection model (one connection per thread)
- The api-worker thread is independent from the thread pool — it has its own lifecycle
The combined target is:
```
lws-main (1) + readers (4) + event-worker (1) + writers (2) + api-worker (1) = 9 threads
```
+240
View File
@@ -0,0 +1,240 @@
# C-Relay-PG Admin Page Migration Plan
## Overview
Migrate the relay admin dashboard from an embedded HTTP-served page (`api/index.html` + `api/index.js`) to a full client-ndk page (`c-relay-pg.html`) built on the client-ndk template. The page will live in the client-ndk project (`~/lt/client-ndk/www/`) alongside other pages and communicate with the relay purely through Nostr protocol subscriptions.
## Why This Migration
The current `/api` page is served directly by the relay's `lws-main` thread, which causes:
- **Slow page delivery** — HTTP file serving competes with WebSocket event processing on a single thread
- **Incomplete page loads** — 10-second `timeout_secs` kills connections when the main thread is busy
- **6+ sequential HTTP round-trips** — each asset requires a two-phase header/body delivery cycle
- **Unnecessary complexity** — embedded file generation, `lws_set_wsi_user()` overwriting, session data type dispatch
The admin page already communicates entirely via Nostr protocol:
- **Kind 23456** → Admin commands (encrypted, sent by admin)
- **Kind 23457** → Admin responses (encrypted, sent by relay)
- **Kind 24567** → Monitoring/stats events (ephemeral, broadcast by relay)
- **NIP-11** → Relay info (fetched via HTTP, independent of file serving)
- **NIP-17** → DM admin commands (gift-wrapped)
Moving to client-ndk means the relay serves zero HTML — it only speaks Nostr protocol.
## Architecture
```mermaid
graph LR
subgraph client-ndk project
A[c-relay-pg.html] --> B[init-ndk.mjs]
B --> C[NDK SharedWorker]
end
subgraph c-relay-pg
D[WebSocket Server]
E[api-worker thread]
end
C -->|kind 23456 admin cmds| D
D -->|kind 23457 responses| C
E -->|kind 24567 monitoring| D
D -->|kind 24567 events| C
C -->|kind 1059 NIP-17 DMs| D
```
## Current Admin Page Sections
| Section | Data Source | Communication |
|---------|-----------|---------------|
| Statistics | kind 24567 monitoring events with d-tags: event_kinds, time_stats, top_pubkeys, cpu_metrics | Subscribe to kind 24567 |
| Subscriptions | kind 24567 monitoring events with d-tag: subscription_details | Subscribe to kind 24567 |
| Configuration | kind 23457 admin responses | Send kind 23456 config_query, receive kind 23457 |
| Authorization | kind 23457 admin responses | Send kind 23456 auth rule commands, receive kind 23457 |
| IP Bans | kind 23457 admin responses | Send kind 23456 ip_ban_* commands, receive kind 23457 |
| Relay Events | Live subscription to all events + kind 0/10050/10002 management | Standard Nostr subscriptions |
| DM | NIP-17 gift-wrapped DMs kind 1059 | NIP-17 encrypt/wrap/publish |
| SQL Query | kind 23457 admin responses | Send kind 23456 sql_query, receive kind 23457 |
## Design Decisions
1. **Full client-ndk page** — Uses `client.css`, `init-ndk.mjs`, `HamburgerMorphing`, `relay-ui.mjs`, `blossom-ui.mjs`, `ai-ui.mjs` — all standard template infrastructure stays intact
2. **Sidenav preserved** — All existing sidenav sections (Relay, Blossom, AI) remain. Admin page navigation links are added to `divSideNavBody` as clickable items
3. **Page-specific CSS** — Follows client-ndk convention: page styles in inline `<style>` block, using only CSS variables from `client.css`
4. **Hardcoded relay URL**`wss://relay.laantungir.net` for now; future enhancement to support relay selection
5. **Auth mode: required** — Admin page requires authentication
## Migration Steps
### Step 1: Create `c-relay-pg.html` from Template
Start from `template.html` in client-ndk. Customize:
- **Title**: `C-Relay-PG`
- **Header text**: `C-Relay-PG Admin`
- **Auth mode**: `required` (default)
- **`divBody` content**: All 8 admin sections, each in its own container div, all `display:none` except Statistics (default)
- **`divSideNavBody`**: Add navigation links for each section:
```html
<div id="divAdminNav">
<div class="adminNavItem active" data-section="statistics">Statistics</div>
<div class="adminNavItem" data-section="subscriptions">Subscriptions</div>
<div class="adminNavItem" data-section="configuration">Configuration</div>
<div class="adminNavItem" data-section="authorization">Authorization</div>
<div class="adminNavItem" data-section="ip-bans">IP Bans</div>
<div class="adminNavItem" data-section="relay-events">Relay Events</div>
<div class="adminNavItem" data-section="dm">DM</div>
<div class="adminNavItem" data-section="database">Database Query</div>
</div>
```
### Step 2: Port Admin Communication Layer
Replace `SimplePool` + `nostr-lite.js` with NDK equivalents:
| Current index.js | New c-relay-pg.html |
|-----------------|---------------------|
| `new SimplePool()` | `subscribe()` from init-ndk.mjs |
| `relayPool.subscribeMany()` | `subscribe(filter, opts)` |
| `relayPool.publish()` | `publishEvent(event)` |
| `window.nostr.signEvent()` | Handled by NDK MessageBasedSigner |
| `window.NostrTools.nip42` | Handled by NDK NIP-42 support |
| `window.NostrTools.nip44` | `window.NostrTools.nip44` from nostr.bundle.js — same as before |
| `nlLite` nostr-login-lite | `initNDKPage()` + `getPubkey()` |
Key function — `sendAdminCommand(commandArray)`:
```javascript
async function sendAdminCommand(commandArray) {
const conversationKey = window.NostrTools.nip44.v2.utils.getConversationKey(
adminPrivateKey, relayPubkey
);
const encrypted = window.NostrTools.nip44.v2.encrypt(
JSON.stringify(commandArray), conversationKey
);
const event = {
created_at: Math.floor(Date.now() / 1000),
kind: 23456,
tags: [['p', relayPubkey]],
content: encrypted
};
return await publishEvent(event);
}
```
Note: NIP-44 encryption requires the admin's private key. The current page uses `window.nostr.nip44.encrypt()` which delegates to the browser extension. NDK's `publishEvent` handles signing, but encryption must be done before publishing. We'll use `window.nostr.nip44.encrypt()` if available (NIP-07 extension), same as the current page does.
### Step 3: Port Subscription Setup
Using NDK's `subscribe()` and `window.addEventListener('ndkEvent', ...)`:
```javascript
// Subscribe to admin responses + monitoring events from relay
const adminSub = subscribe(
{ kinds: [23457, 24567], authors: [RELAY_PUBKEY] },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Live event feed
const liveSub = subscribe(
{ limit: 50 },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Route events to handlers
window.addEventListener('ndkEvent', (e) => {
const evt = e.detail;
if (evt.kind === 23457) processAdminResponse(evt);
else if (evt.kind === 24567) processMonitoringEvent(evt);
else addEventToLiveFeed(evt);
});
```
### Step 4: Port NIP-44 Encryption
Admin commands require NIP-44 encryption. The template already loads `nostr.bundle.js` which includes `window.NostrTools.nip44`. Additionally, `window.nostr.nip44.encrypt()` is available from NIP-07 extensions. Use the same approach as the current page.
### Step 5: Port UI Sections into divBody
Each section's HTML goes into `divBody` as a container div. Page-specific styles go in the inline `<style>` block following client-ndk conventions:
- Use CSS variables only (no hardcoded colors)
- Tables use `var(--primary-color)`, `var(--muted-color)`, `var(--accent-color)`
- Buttons follow the `var(--button-*)` pattern
- Font is always `var(--font-family)`
Sections to port:
1. **Statistics** — DB overview table, event kinds table, time stats table, top pubkeys table, event rate chart area
2. **Subscriptions** — Active subscription details table
3. **Configuration** — Config key/value table with inline edit, refresh button
4. **Authorization** — Auth rules table, whitelist/blacklist inputs, WoT level selector
5. **IP Bans** — Ban stats, manual ban form, IP whitelist, ban list table with filters
6. **Relay Events** — Live event feed table, kind 0/10050/10002 management forms
7. **DM** — NIP-17 message textarea + send button, inbox display
8. **SQL Query** — Query dropdown, SQL textarea, execute button, results table
### Step 6: Page-Specific CSS
Following client-ndk rules from `client.css`:
- Override `#divBody` layout: `flex-direction: column; overflow-y: auto; padding: 20px;`
- Admin section containers: hidden by default, shown when nav item clicked
- Table styles: use `var(--primary-color)` for headers, `var(--muted-color)` for borders
- Button styles: border + transparent background, hover with `var(--accent-color)`
- All in inline `<style media="screen">` block in `<head>`
### Step 7: Relay URL — Hardcoded for Now
```javascript
const RELAY_WS_URL = 'wss://relay.laantungir.net';
const RELAY_HTTP_URL = 'https://relay.laantungir.net';
```
NIP-11 fetch to get relay pubkey:
```javascript
const nip11 = await fetch(RELAY_HTTP_URL, {
headers: { 'Accept': 'application/nostr+json' }
}).then(r => r.json());
const RELAY_PUBKEY = nip11.pubkey;
```
### Step 8: Admin Verification
Same flow as current page:
1. Get relay pubkey from NIP-11
2. Send kind 23456 `system_status` command
3. If relay responds with kind 23457, user is verified as admin
4. Show admin sections; otherwise show access denied
### Step 9 (Optional, Later): Remove Embedded HTTP Serving from Relay
After migration is complete and tested:
- Remove `handle_embedded_file_request()` from `src/api.c`
- Remove `handle_embedded_file_writeable()` from `src/api.c`
- Remove embedded file dispatch from `LWS_CALLBACK_HTTP` in `src/websockets.c`
- Remove `embed_web_files.sh` and `src/embedded_web_content.c`
- Keep NIP-11 serving (lightweight and required by protocol)
## File Changes Summary
### New File (in client-ndk project: ~/lt/client-ndk/www/)
- `c-relay-pg.html` — Complete admin page, self-contained with inline `<style>` and `<script type="module">`
### Files Unchanged
- All relay C code for kind 23456/23457/24567 processing — stays exactly the same
- `src/dm_admin.c` — NIP-17 DM handling stays the same
- `src/nip011.c` — NIP-11 stays the same
- All client-ndk shared files (client.css, init-ndk.mjs, relay-ui.mjs, etc.) — no modifications needed
### Files to Modify Later (optional relay cleanup)
- `src/api.c` — Remove embedded file serving functions
- `src/websockets.c` — Remove embedded file dispatch from HTTP callback
- `Makefile` — Remove embedded web content compilation step
## Risks and Mitigations
| Risk | Mitigation |
|------|-----------|
| NIP-44 encryption via window.nostr.nip44 | Same mechanism as current page — no change in encryption approach |
| NDK SharedWorker relay connection | Use ONLY_RELAY cache usage to ensure direct relay communication |
| Large page size with all sections inline | Sections are display:none by default — no performance impact |
| Event rate chart from text_graph.js | Port chart logic inline or skip initially |
| Relay pubkey needed before subscriptions | Fetch NIP-11 first in initialization sequence |
+632
View File
@@ -0,0 +1,632 @@
# c-relay-pg-pg Plan — PostgreSQL Backend + Multi-Instance + Dashboard
## Overview
**c-relay-pg-pg** is a new project (separate repository) forked from c-relay-pg after the `db_ops` abstraction layer is complete. It replaces the SQLite backend with PostgreSQL, enabling horizontal scaling, external dashboards, and production-grade deployments.
### Prerequisites
- The `db_ops.h` / `db_ops.c` abstraction layer must be complete in c-relay-pg first — see [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1.
- The fork happens after Phase 1 of that plan is done and tested.
### Why a Separate Project?
| | c-relay-pg | c-relay-pg-pg |
|---|---------|-----------|
| **Database** | SQLite (embedded) | PostgreSQL (client-server) |
| **Deployment** | Single binary + DB file | Binary + PostgreSQL server |
| **Scaling** | Single instance | Multiple instances + load balancer |
| **Dashboard** | Embedded web UI | Separate web server + Grafana |
| **Target user** | Personal relay, small community | Medium-large relay, production |
| **Complexity** | Minimal | More infrastructure |
For the full analysis of why PostgreSQL was chosen over MySQL/MariaDB and other databases, see the [database architecture analysis](database_architecture_analysis.md).
## Architecture
```mermaid
flowchart TD
subgraph c-relay-pg-pg - Production Deployment
LB[nginx - TLS + load balancing] -->|WebSocket| R1[c-relay-pg-pg Instance 1]
LB -->|WebSocket| R2[c-relay-pg-pg Instance 2]
LB -->|HTTP| DASH[Dashboard]
R1 -->|db_ops API| PGBACK[PostgreSQL Backend - db_ops_postgres.c]
R2 -->|db_ops API| PGBACK
PGBACK -->|libpq| PG[PostgreSQL Server]
DASH -->|SQL| PG
R1 <-->|LISTEN/NOTIFY| R2
end
```
---
## Phase 1: PostgreSQL Backend
### Goal
Fork c-relay-pg into a new repository called **c-relay-pg-pg**. Replace the SQLite `db_ops` implementation with PostgreSQL using `libpq`. The `db_ops.h` interface stays identical — only the backend changes.
### New Files
- `src/db_ops_sqlite.c` — Renamed from `db_ops.c` (the Phase 1 SQLite implementation from c-relay-pg)
- `src/db_ops_postgres.c` — New PostgreSQL implementation
- `src/db_ops.c` — Thin dispatcher that calls the active backend
### PostgreSQL Schema
```sql
-- PostgreSQL schema for c-relay-pg-pg
-- No event_tags table needed — JSONB + GIN handles everything
CREATE TABLE events (
id TEXT PRIMARY KEY,
pubkey TEXT NOT NULL,
created_at BIGINT NOT NULL,
kind INTEGER NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
content TEXT NOT NULL,
sig TEXT NOT NULL,
tags JSONB NOT NULL DEFAULT '[]',
event_json TEXT NOT NULL,
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Core indexes
CREATE INDEX idx_events_pubkey ON events(pubkey);
CREATE INDEX idx_events_kind ON events(kind);
CREATE INDEX idx_events_created_at ON events(created_at DESC);
CREATE INDEX idx_events_kind_created_at ON events(kind, created_at DESC);
CREATE INDEX idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
CREATE INDEX idx_events_pubkey_kind ON events(pubkey, kind);
-- THE KEY INDEX: GIN on JSONB tags — replaces the entire event_tags table
CREATE INDEX idx_events_tags ON events USING GIN (tags);
-- Partial index: only non-ephemeral events (what REQ queries actually filter)
CREATE INDEX idx_events_non_ephemeral ON events(created_at DESC)
WHERE kind < 20000 OR kind >= 30000;
-- Config table
CREATE TABLE config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
data_type TEXT NOT NULL CHECK (data_type IN ('string', 'integer', 'boolean', 'json')),
description TEXT,
category TEXT DEFAULT 'general',
requires_restart INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Auth rules table
CREATE TABLE auth_rules (
id SERIAL PRIMARY KEY,
rule_type TEXT NOT NULL CHECK (rule_type IN ('whitelist', 'blacklist', 'rate_limit', 'auth_required', 'wot_whitelist')),
pattern_type TEXT NOT NULL CHECK (pattern_type IN ('pubkey', 'kind', 'ip', 'global')),
pattern_value TEXT,
active INTEGER NOT NULL DEFAULT 1,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX idx_auth_rules_lookup ON auth_rules(rule_type, pattern_type, pattern_value) WHERE active = 1;
-- Relay private key storage
CREATE TABLE relay_seckey (
private_key_hex TEXT NOT NULL CHECK (length(private_key_hex) = 64),
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Subscription logging
CREATE TABLE subscriptions (
id SERIAL PRIMARY KEY,
subscription_id TEXT NOT NULL,
wsi_pointer TEXT NOT NULL,
client_ip TEXT NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('created', 'closed', 'expired', 'disconnected')),
filter_json TEXT,
events_sent INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
ended_at BIGINT,
duration INTEGER,
UNIQUE(subscription_id, wsi_pointer)
);
-- IP ban persistence
CREATE TABLE ip_bans (
ip TEXT PRIMARY KEY,
failure_count INTEGER NOT NULL DEFAULT 0,
ban_count INTEGER NOT NULL DEFAULT 0,
banned_until BIGINT NOT NULL DEFAULT 0,
first_failure BIGINT NOT NULL DEFAULT 0,
has_authed_successfully INTEGER NOT NULL DEFAULT 0,
last_success_at BIGINT NOT NULL DEFAULT 0,
total_connections INTEGER NOT NULL DEFAULT 0,
total_failures INTEGER NOT NULL DEFAULT 0,
total_successes INTEGER NOT NULL DEFAULT 0,
first_seen BIGINT NOT NULL DEFAULT 0,
idle_failure_count INTEGER NOT NULL DEFAULT 0,
idle_ban_count INTEGER NOT NULL DEFAULT 0,
idle_banned_until BIGINT NOT NULL DEFAULT 0,
idle_first_failure BIGINT NOT NULL DEFAULT 0
);
-- Materialized views for dashboard (refreshed periodically)
CREATE MATERIALIZED VIEW event_kinds_mv AS
SELECT kind, COUNT(*) as count,
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
FROM events GROUP BY kind;
CREATE MATERIALIZED VIEW time_stats_mv AS
SELECT 'total' as period, COUNT(*) as total_events, COUNT(DISTINCT pubkey) as unique_pubkeys
FROM events
UNION ALL
SELECT '24h', COUNT(*), COUNT(DISTINCT pubkey)
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 86400
UNION ALL
SELECT '7d', COUNT(*), COUNT(DISTINCT pubkey)
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 604800;
CREATE MATERIALIZED VIEW top_pubkeys_mv AS
SELECT pubkey, COUNT(*) as event_count,
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
FROM events GROUP BY pubkey ORDER BY event_count DESC LIMIT 20;
-- Unique indexes required for CONCURRENTLY refresh
CREATE UNIQUE INDEX idx_event_kinds_mv ON event_kinds_mv(kind);
CREATE UNIQUE INDEX idx_time_stats_mv ON time_stats_mv(period);
CREATE UNIQUE INDEX idx_top_pubkeys_mv ON top_pubkeys_mv(pubkey);
```
### Key Implementation Details
#### Tag Queries with JSONB
The biggest win — replacing the `event_tags` subquery with JSONB containment:
```c
// SQLite (current): subquery into event_tags table
// AND id IN (SELECT event_id FROM event_tags WHERE tag_name = ? AND tag_value IN (?))
// PostgreSQL: JSONB containment operator with GIN index
// AND tags @> '[["p", "pubkey_hex"]]'
// For multiple tag values:
// AND (tags @> '[["p", "val1"]]' OR tags @> '[["p", "val2"]]')
```
This eliminates the entire `event_tags` table (4 million rows, ~2 GB indexes in the current SQLite schema). The GIN index on JSONB handles everything in ~100200 MB.
#### LISTEN/NOTIFY Integration
For cross-instance event broadcasting:
```c
// In db_ops_postgres.c:
int db_notify_new_event(const char* event_id) {
char cmd[128];
snprintf(cmd, sizeof(cmd), "NOTIFY new_event, '%s'", event_id);
PGresult* res = PQexec(g_pg_conn, cmd);
PQclear(res);
return 0;
}
// Called from the lws event loop every iteration:
int db_check_notifications(char* event_id_out, size_t max_len) {
PQconsumeInput(g_pg_notify_conn);
PGnotify* notify = PQnotifies(g_pg_notify_conn);
if (notify) {
strncpy(event_id_out, notify->extra, max_len);
PQfreemem(notify);
return 1; // Got a notification
}
return 0; // No notifications
}
```
#### Connection Management
```c
// db_ops_postgres.c connection pool (simple version)
#define DB_POOL_SIZE 4
static PGconn* g_pg_read_pool[DB_POOL_SIZE]; // For REQ queries
static PGconn* g_pg_write_conn; // For EVENT inserts
static PGconn* g_pg_notify_conn; // For LISTEN/NOTIFY
static pthread_mutex_t g_pool_lock;
PGconn* db_get_read_connection(void) {
pthread_mutex_lock(&g_pool_lock);
// Round-robin or find idle connection
// ...
pthread_mutex_unlock(&g_pool_lock);
}
```
### Build System Changes
```makefile
# Makefile additions
DB_BACKEND ?= sqlite # Default to sqlite, override with: make DB_BACKEND=postgres
ifeq ($(DB_BACKEND),postgres)
MAIN_SRC += src/db_ops.c src/db_ops_postgres.c
LIBS += -lpq
CFLAGS += -DDB_BACKEND_POSTGRES
else
MAIN_SRC += src/db_ops.c src/db_ops_sqlite.c
CFLAGS += -DDB_BACKEND_SQLITE
endif
```
### Data Migration Tool
A standalone tool to migrate existing SQLite data to PostgreSQL:
```bash
# migrate_to_postgres.sh
# 1. Export events from SQLite
sqlite3 old_relay.db ".mode csv" "SELECT id,pubkey,created_at,kind,event_type,content,sig,tags,event_json,first_seen FROM events" > events.csv
# 2. Import into PostgreSQL
psql -d crelay -c "\COPY events FROM 'events.csv' WITH CSV"
# 3. Migrate config
sqlite3 old_relay.db ".mode csv" "SELECT key,value,data_type,description,category,requires_restart FROM config" > config.csv
psql -d crelay -c "\COPY config(key,value,data_type,description,category,requires_restart) FROM 'config.csv' WITH CSV"
# 4. Migrate auth rules
sqlite3 old_relay.db ".mode csv" "SELECT rule_type,pattern_type,pattern_value,active FROM auth_rules" > auth_rules.csv
psql -d crelay -c "\COPY auth_rules(rule_type,pattern_type,pattern_value,active) FROM 'auth_rules.csv' WITH CSV"
# 5. Refresh materialized views
psql -d crelay -c "REFRESH MATERIALIZED VIEW event_kinds_mv; REFRESH MATERIALIZED VIEW time_stats_mv; REFRESH MATERIALIZED VIEW top_pubkeys_mv;"
```
---
## Phase 2: Multi-Instance + Dashboard
### Goal
Run multiple c-relay-pg-pg instances behind a load balancer with a separate dashboard web server, all sharing the same PostgreSQL database.
### Components
1. **nginx config** — WebSocket load balancing with `ip_hash` stickiness
2. **systemd template**`c-relay-pg-pg@.service` for multiple instances
3. **LISTEN/NOTIFY integration** — Cross-instance event broadcasting in the `lws_service()` loop
4. **PgBouncer** — Connection pooling between c-relay-pg-pg instances and PostgreSQL
5. **Dashboard** — Separate web server querying PostgreSQL directly
6. **Materialized view refresh** — Background job to refresh analytics views
### Multi-Instance Architecture
```mermaid
flowchart TD
INTERNET[Internet - Nostr Clients] -->|wss://relay.example.com| NGINX[nginx reverse proxy - TLS termination + load balancing]
NGINX -->|ws://localhost:8888| R1[c-relay-pg-pg Instance 1 - port 8888]
NGINX -->|ws://localhost:8889| R2[c-relay-pg-pg Instance 2 - port 8889]
NGINX -->|ws://localhost:8890| R3[c-relay-pg-pg Instance 3 - port 8890]
NGINX -->|http://localhost:3000| DASHWEB[Dashboard Web Server]
R1 -->|libpq connection pool| PGPOOL[PgBouncer - connection pooler]
R2 -->|libpq connection pool| PGPOOL
R3 -->|libpq connection pool| PGPOOL
PGPOOL -->|pooled connections| PG[PostgreSQL Primary]
DASHWEB -->|read queries| PG
PG -->|streaming replication| REPLICA[Read Replica - optional]
DASHWEB -.->|heavy analytics| REPLICA
```
### nginx Load Balancing Config
```nginx
# nginx.conf - WebSocket load balancing for c-relay-pg-pg
upstream relay_backends {
# ip_hash ensures a client always hits the same instance
# (important for WebSocket session stickiness)
ip_hash;
server 127.0.0.1:8888;
server 127.0.0.1:8889;
server 127.0.0.1:8890;
}
server {
listen 443 ssl;
server_name relay.example.com;
# TLS config...
location / {
proxy_pass http://relay_backends;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 86400; # Keep WebSocket alive for 24h
}
# Dashboard on separate path
location /dashboard {
proxy_pass http://127.0.0.1:3000;
}
}
```
**Session stickiness** (`ip_hash`) ensures that once a client connects to Instance 2, all their subsequent WebSocket frames go to Instance 2. This is important because subscriptions are held in-memory per instance.
### Starting Multiple Instances
Each instance is the same binary, just on a different port, all pointing to the same PostgreSQL:
```bash
# Instance 1
./build/c_relay_pg_x86 --port 8888 --db-host localhost --db-name crelay &
# Instance 2
./build/c_relay_pg_x86 --port 8889 --db-host localhost --db-name crelay &
# Instance 3
./build/c_relay_pg_x86 --port 8890 --db-host localhost --db-name crelay &
```
Or with systemd template units:
```ini
# /etc/systemd/system/c-relay-pg-pg@.service
[Unit]
Description=C-Relay-PG-PG Nostr Instance %i
After=postgresql.service
[Service]
ExecStart=/opt/c-relay-pg-pg/c_relay_pg_x86 --port %i --db-host localhost --db-name crelay
Restart=always
User=c-relay-pg
[Install]
WantedBy=multi-user.target
```
```bash
systemctl enable c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
systemctl start c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
```
### Cross-Instance Event Broadcasting
When Instance 1 receives a new EVENT and stores it in PostgreSQL, Instance 2 and Instance 3 need to know about it so they can broadcast to their connected subscribers.
```mermaid
sequenceDiagram
participant Client_A as Client A - connected to Instance 1
participant I1 as Instance 1
participant PG as PostgreSQL
participant I2 as Instance 2
participant I3 as Instance 3
participant Client_B as Client B - connected to Instance 2
participant Client_C as Client C - connected to Instance 3
Client_A->>I1: EVENT - new kind:1 note
I1->>PG: INSERT INTO events...
PG-->>I1: OK
I1->>I1: broadcast to local subscribers
I1->>PG: NOTIFY new_event with event_id
Note over PG: PostgreSQL delivers notification to all listeners
PG-->>I2: NOTIFY: new_event event_id
PG-->>I3: NOTIFY: new_event event_id
I2->>PG: SELECT event_json FROM events WHERE id = event_id
PG-->>I2: event JSON
I2->>I2: match against local subscriptions
I2->>Client_B: EVENT message - if subscription matches
I3->>PG: SELECT event_json FROM events WHERE id = event_id
PG-->>I3: event JSON
I3->>I3: match against local subscriptions
I3->>Client_C: EVENT message - if subscription matches
```
#### PostgreSQL LISTEN/NOTIFY Implementation
Built into PostgreSQL — no additional infrastructure needed:
```c
// === In the relay's event loop (modified lws_service loop) ===
// Setup: create a dedicated connection for LISTEN
PGconn* notify_conn = PQconnectdb("host=localhost dbname=crelay");
PQexec(notify_conn, "LISTEN new_event");
int notify_fd = PQsocket(notify_conn); // Get the socket fd for poll()
// After storing an event:
void on_event_stored(PGconn* write_conn, const char* event_id) {
char notify_cmd[128];
snprintf(notify_cmd, sizeof(notify_cmd),
"NOTIFY new_event, '%s'", event_id);
PQexec(write_conn, notify_cmd);
}
// In the main event loop (runs every lws_service iteration):
void check_cross_instance_events(PGconn* notify_conn) {
// Non-blocking check for notifications
PQconsumeInput(notify_conn);
PGnotify* notify;
while ((notify = PQnotifies(notify_conn)) != NULL) {
// Another instance stored a new event
const char* event_id = notify->extra;
// Fetch the event and check against local subscriptions
cJSON* event = db_get_event_by_id(event_id);
if (event) {
broadcast_event_to_subscriptions(event);
cJSON_Delete(event);
}
PQfreemem(notify);
}
}
```
**Performance**: LISTEN/NOTIFY adds ~15ms latency for cross-instance delivery. For a Nostr relay, this is imperceptible — clients already expect network latency.
**Payload limit**: NOTIFY payloads are limited to 8000 bytes. For event IDs (64 hex chars), this is fine.
#### Alternative: Redis Pub/Sub
If you later need even lower latency or more sophisticated routing:
```c
// Using hiredis (Redis C client)
redisContext* redis = redisConnect("127.0.0.1", 6379);
// After storing event:
redisCommand(redis, "PUBLISH new_event %s", event_json);
// Subscriber (in each instance):
redisCommand(redis, "SUBSCRIBE new_event");
// Then poll for messages in the event loop
```
Redis adds sub-millisecond pub/sub but requires running a Redis server. For most relays, PostgreSQL LISTEN/NOTIFY is sufficient.
### Connection Pooling with PgBouncer
Each relay instance needs multiple database connections. Without pooling, 3 instances × 10 connections = 30 PostgreSQL backend processes. With PgBouncer:
```ini
# /etc/pgbouncer/pgbouncer.ini
[databases]
crelay = host=127.0.0.1 port=5432 dbname=crelay
[pgbouncer]
listen_port = 6432
listen_addr = 127.0.0.1
auth_type = md5
pool_mode = transaction # Return connection to pool after each transaction
max_client_conn = 200 # Total connections from all relay instances
default_pool_size = 20 # Actual PostgreSQL connections
```
Relay instances connect to PgBouncer (port 6432) instead of PostgreSQL directly (port 5432). PgBouncer multiplexes 200 client connections onto 20 actual PostgreSQL connections.
### Zero-Downtime Deployment
```mermaid
sequenceDiagram
participant LB as nginx
participant I1 as Instance 1 - v1.0
participant I2 as Instance 2 - v1.0
participant I3 as Instance 3 - v1.0
participant I1_NEW as Instance 1 - v1.1
Note over LB,I3: Normal operation: 3 instances serving traffic
LB->>I1: Mark upstream as down
Note over I1: Drain: wait for existing connections to close or timeout
I1->>I1: Graceful shutdown
Note over LB: Traffic now goes to I2 and I3 only
I1_NEW->>I1_NEW: Start with new binary
I1_NEW->>LB: Health check passes
LB->>I1_NEW: Mark upstream as up
Note over LB,I1_NEW: Instance 1 now running v1.1, repeat for I2 and I3
```
Rolling deploy script:
```bash
#!/bin/bash
# rolling_deploy.sh - Zero-downtime deployment
for port in 8888 8889 8890; do
echo "Deploying instance on port $port..."
# 1. Tell nginx to stop sending new connections
sed -i "s/server 127.0.0.1:$port;/server 127.0.0.1:$port down;/" /etc/nginx/nginx.conf
nginx -s reload
# 2. Wait for existing connections to drain (30 seconds)
sleep 30
# 3. Stop old instance
systemctl stop c-relay-pg-pg@$port
# 4. Deploy new binary
cp ./build/c_relay_pg_x86 /opt/c-relay-pg-pg/c_relay_pg_x86
# 5. Start new instance
systemctl start c-relay-pg-pg@$port
# 6. Wait for health check
sleep 5
# 7. Re-enable in nginx
sed -i "s/server 127.0.0.1:$port down;/server 127.0.0.1:$port;/" /etc/nginx/nginx.conf
nginx -s reload
echo "Instance on port $port deployed successfully"
done
```
### Dashboard Options
With PostgreSQL, the dashboard can be built with any technology:
- **Grafana** — Point directly at PostgreSQL, zero custom code
- **Custom web app** — React/Vue frontend + any backend (Node, Python, Go) querying PostgreSQL
- **Embedded in c-relay-pg-pg** — Keep current approach but queries go through `db_ops` to PostgreSQL (no longer blocks event loop since PostgreSQL handles concurrency)
### Scaling Scenarios
| Scenario | Instances | Connections | Events/hour | Setup |
|----------|-----------|-------------|-------------|-------|
| **Current** | 1 | ~1,200 | 56 | Single process + SQLite |
| **Small upgrade** | 2 | ~2,500 | 500 | 2 instances + PostgreSQL |
| **Medium relay** | 4 | ~5,000 | 5,000 | 4 instances + PostgreSQL + PgBouncer |
| **Large relay** | 8 | ~10,000 | 50,000 | 8 instances + PostgreSQL + read replica |
| **Multi-region** | 24 per region | ~50,000 | 500,000 | Multiple servers + PostgreSQL replication |
### Cost Perspective
Running multiple relay instances with PostgreSQL on a single VPS:
- **PostgreSQL**: ~200500 MB RAM baseline
- **PgBouncer**: ~10 MB RAM
- **Each relay instance**: ~50100 MB RAM
- **Dashboard web server**: ~50100 MB RAM
- **4 instances + PostgreSQL + PgBouncer + dashboard**: ~12 GB total RAM
- A **$20/month VPS** with 4 cores and 4 GB RAM handles this easily
- A **$40/month VPS** with 8 cores and 8 GB RAM handles 8 instances comfortably
---
## Risk Mitigation
| Risk | Mitigation |
|------|-----------|
| PostgreSQL connection failures | `db_ops` returns error codes. Relay logs errors but doesn't crash. Reconnection logic with exponential backoff. |
| Performance regression | Benchmark before/after. PostgreSQL should be faster for complex queries, similar for simple ones. |
| Data loss during migration | Migration tool is read-only on SQLite. PostgreSQL import is idempotent (INSERT ON CONFLICT). |
| SQLite fallback needed | Keep `db_ops_sqlite.c` working. Compile-time flag switches backends. Can always go back. |
| LISTEN/NOTIFY message loss | NOTIFY is transactional — if the INSERT commits, the NOTIFY is guaranteed. Missed notifications during reconnect handled by periodic full-sync. |
| PgBouncer adds latency | Transaction pooling mode adds <0.1ms. Negligible compared to query time. |
---
## Relationship to c-relay-pg
This project depends on the `db_ops.h` abstraction layer built in [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1. The interface is identical — only the backend implementation changes:
- **c-relay-pg**: `db_ops.c` → SQLite calls via `sqlite3_*`
- **c-relay-pg-pg**: `db_ops_postgres.c` → PostgreSQL calls via `libpq` (`PQexec`, `PQgetvalue`, etc.)
The `db_ops.h` header file is shared between both projects. Changes to the interface should be coordinated.
+332
View File
@@ -0,0 +1,332 @@
# c-relay-pg Thread Pool Plan — db_ops Abstraction + SQLite Thread Pool
## Overview
This plan covers improvements to **c-relay-pg** (this repo) — the lean, single-binary, embedded-SQLite Nostr relay. Two phases:
1. **Phase 1: Database Abstraction Layer** — Consolidate all scattered `sqlite3_*` calls into a single `db_ops.h` / `db_ops.c` module. Pure refactor, no behavior change.
2. **Phase 2: SQLite Thread Pool** — Add worker threads for concurrent reads, unblocking the `lws_service()` event loop.
This is the **final form of c-relay-pg**: an embedded-SQLite relay with clean architecture and non-blocking database access. The abstraction layer also enables a future fork to PostgreSQL — see [c-relay-pg-pg plan](c_relay_pg_pg_plan.md).
### Why This First?
The current architecture has a fundamental bottleneck documented in the [database architecture analysis](database_architecture_analysis.md): the single-threaded event loop blocks on every database call. A 672ms REQ query freezes every connected client. The thread pool fixes this without changing the database engine or deployment model.
| Metric | Current | After Thread Pool |
|--------|---------|-------------------|
| **Event loop blocking** | 0.1672ms per query | Near-zero |
| **Concurrent reads** | 1 | 48 |
| **Deployment** | Single binary + DB file | Same |
| **Database** | SQLite | Same |
| **Code risk** | N/A | Low — additive change |
## Architecture
```mermaid
flowchart TD
subgraph c-relay-pg - Final Form
RELAY[c-relay-pg binary] -->|db_ops API| DBOPS[db_ops.c - abstraction layer]
DBOPS -->|sqlite3 calls| SQLITE[SQLite WAL database]
end
```
```mermaid
flowchart TD
subgraph c-relay-pg with Thread Pool
LWS[lws_service event loop] -->|REQ arrives| Q[Thread-safe job queue]
LWS -->|EVENT arrives| WQ[Write queue - single writer]
Q --> T1[Reader Thread 1 - own sqlite3*]
Q --> T2[Reader Thread 2 - own sqlite3*]
Q --> T3[Reader Thread 3 - own sqlite3*]
Q --> T4[Reader Thread 4 - own sqlite3*]
WQ --> TW[Writer Thread - own sqlite3*]
T1 -->|results| CB[Callback to lws event loop]
T2 -->|results| CB
T3 -->|results| CB
T4 -->|results| CB
TW -->|OK/error| CB
CB -->|queue_message| LWS
end
```
---
## Phase 1: Database Abstraction Layer
### Goal
Consolidate all 258 scattered `sqlite3_*` calls across 8 files into a single `db_ops.h` / `db_ops.c` module with a backend-agnostic interface. SQLite continues to work — this is a pure refactor.
### Files That Currently Touch SQLite Directly
| File | `sqlite3_*` calls | Operations |
|------|-------------------|------------|
| [`src/main.c`](../src/main.c) | ~80 | Event store/retrieve, tag storage, REQ queries, COUNT queries, DB init, schema migration |
| [`src/config.c`](../src/config.c) | ~60 | Config CRUD, auth rules CRUD, WoT sync, relay key storage, startup sequence |
| [`src/api.c`](../src/api.c) | ~40 | Stats queries, monitoring views, admin SQL execution, config management |
| [`src/dm_admin.c`](../src/dm_admin.c) | ~20 | Auth rule management, WoT whitelist operations |
| [`src/websockets.c`](../src/websockets.c) | ~15 | COUNT queries, IP ban stats, connection tracking |
| [`src/subscriptions.c`](../src/subscriptions.c) | ~15 | Subscription logging — create/close/disconnect |
| [`src/nip009.c`](../src/nip009.c) | ~10 | Event deletion — by ID and by address |
| [`src/request_validator.c`](../src/request_validator.c) | ~8 | Blacklist/whitelist checks |
| [`src/ip_ban.c`](../src/ip_ban.c) | ~15 | IP ban table CRUD, persistence |
### Abstraction Layer Design
New files: `src/db_ops.h` and `src/db_ops.c`
```c
// src/db_ops.h — Database operations abstraction layer
#ifndef DB_OPS_H
#define DB_OPS_H
#include "../nostr_core_lib/cjson/cJSON.h"
// ============================================================
// Lifecycle
// ============================================================
int db_init(const char* connection_string); // SQLite: file path, PG: connection string
void db_close(void);
int db_is_available(void);
// ============================================================
// Schema / Migration
// ============================================================
int db_ensure_schema(void);
int db_get_schema_version(void);
int db_execute_raw(const char* sql); // For schema DDL only
// ============================================================
// Event Operations
// ============================================================
int db_store_event(cJSON* event);
int db_event_exists(const char* event_id);
char* db_get_event_json(const char* event_id); // Caller must free
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
int db_delete_events_by_address(const char* pubkey, int kind,
const char* d_tag, long before_timestamp);
// ============================================================
// Event Queries - REQ handling
// ============================================================
typedef struct {
int* kinds; int kind_count;
char** authors; int author_count;
char** ids; int id_count;
char** tag_names; // Parallel arrays: tag_names[i] has tag_values[i][]
char*** tag_values;
int* tag_value_counts;
int tag_filter_count;
long since; // 0 = not set
long until; // 0 = not set
int limit; // 0 = default 500
char* search; // NIP-50 search term, NULL = not set
} db_event_filter_t;
typedef struct db_result db_result_t;
db_result_t* db_query_events(const db_event_filter_t* filter);
const char* db_result_next_json(db_result_t* result); // Returns event_json, NULL when done
int db_result_row_count(db_result_t* result);
void db_result_free(db_result_t* result);
int db_count_events(const db_event_filter_t* filter);
// ============================================================
// Config Operations
// ============================================================
char* db_get_config(const char* key); // Caller must free, NULL if not found
int db_set_config(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart);
int db_update_config(const char* key, const char* value);
int db_config_exists(const char* key);
int db_get_config_count(void);
// ============================================================
// Auth Rules Operations
// ============================================================
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_auth_rule_exists(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_clear_auth_rules(const char* rule_type); // NULL = clear all
int db_is_blacklisted(const char* pubkey);
int db_is_whitelisted(const char* pubkey);
int db_has_any_whitelist(void);
// ============================================================
// Relay Key Storage
// ============================================================
int db_store_relay_private_key(const char* privkey_hex);
char* db_get_relay_private_key(void); // Caller must free
// ============================================================
// Subscription Logging
// ============================================================
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json);
int db_log_subscription_closed(const char* sub_id, const char* client_ip);
int db_log_subscription_disconnected(const char* client_ip);
int db_update_subscription_events_sent(const char* sub_id, int events_sent);
int db_cleanup_orphaned_subscriptions(void);
// ============================================================
// IP Ban Persistence
// ============================================================
int db_ensure_ip_ban_table(void);
int db_load_ip_bans(void* ban_table, int table_size); // Populates in-memory table
int db_save_ip_bans(const void* ban_table, int table_size);
// ============================================================
// Analytics / Stats - for dashboard
// ============================================================
cJSON* db_get_event_kind_distribution(void);
cJSON* db_get_time_based_stats(void);
cJSON* db_get_top_pubkeys(int limit);
cJSON* db_get_subscription_details(void);
int db_get_total_event_count(void);
// ============================================================
// Admin SQL Query
// ============================================================
cJSON* db_execute_admin_query(const char* sql, char* error_msg, size_t error_size);
#endif // DB_OPS_H
```
### Migration Strategy for Phase 1
The key principle: **change the interface, not the behavior**. Each function in `db_ops.c` initially just wraps the existing SQLite calls.
**Step-by-step for each file:**
1. **Create `db_ops.h` and `db_ops.c`** with the interface above
2. **Implement each `db_ops` function** by moving the existing SQLite code from the source files into `db_ops.c`
3. **Replace direct `sqlite3_*` calls** in each source file with `db_ops_*` calls
4. **Remove `extern sqlite3* g_db`** from each file — only `db_ops.c` knows about `g_db`
5. **Remove `#include <sqlite3.h>`** from each file — only `db_ops.c` includes it
6. **Update Makefile** to compile `db_ops.c`
### Order of Migration (by risk, lowest first)
1. **`src/ip_ban.c`** — Self-contained, simple CRUD. Good warmup.
2. **`src/subscriptions.c`** — Logging only, no critical path.
3. **`src/nip009.c`** — Event deletion, small file.
4. **`src/request_validator.c`** — Auth checks, small file.
5. **`src/api.c`** — Stats/monitoring queries. Larger but read-only.
6. **`src/dm_admin.c`** — Auth rules + WoT. Medium complexity.
7. **`src/config.c`** — Config CRUD. Large but well-structured.
8. **`src/websockets.c`** — COUNT queries, minimal DB usage.
9. **`src/main.c`** — Event store/retrieve, REQ queries. The big one — do last.
### Testing Phase 1
After Phase 1, the relay should behave **identically** to before. Run:
- `tests/run_all_tests.sh` — Full test suite
- `tests/performance_benchmarks.sh` — Verify no performance regression
- Manual testing with production-like traffic
---
## Phase 2: SQLite Thread Pool
### Goal
Add a pool of N worker threads, each with its own `sqlite3*` connection to the same database file. SQLite WAL mode (already enabled) supports **concurrent readers**. The event loop dispatches database work to the pool and remains responsive.
### Key Design Points
1. **Read path**: REQ queries dispatched to thread pool. Each worker opens its own `sqlite3*` connection. SQLite WAL allows unlimited concurrent readers.
2. **Write path**: EVENT inserts go through a single dedicated writer thread. SQLite only allows one writer at a time anyway — this serializes writes cleanly.
3. **Result delivery**: Worker threads cannot call `lws_write()` directly (libwebsockets is not thread-safe). Instead, they push results into a per-session message queue and call `lws_cancel_service()` to wake the event loop, which then drains the queue.
4. **Connection lifecycle**: Each thread opens its own connection with `PRAGMA journal_mode=WAL` and `PRAGMA busy_timeout=5000`.
### What Changes in the Codebase
| Component | Current | Thread Pool |
|-----------|---------|-------------|
| [`g_db`](../src/main.c:49) | Single global connection | One per thread + writer connection |
| [`handle_req_message()`](../src/main.c:1101) | Synchronous SQL in callback | Package filter into job, dispatch to pool |
| [`store_event()`](../src/main.c:787) | Synchronous INSERT in callback | Dispatch to writer thread |
| [`handle_count_message()`](../src/websockets.c:2863) | Synchronous COUNT in callback | Dispatch to pool |
| Result delivery | Direct `queue_message()` | Worker pushes to queue + `lws_cancel_service()` |
| Config reads | Direct `sqlite3_prepare` on `g_db` | Can stay synchronous with own connection or cache |
### New Files
- `src/thread_pool.h` — Thread pool interface
- `src/thread_pool.c` — Thread pool implementation (job queue, worker lifecycle, result delivery)
### Thread Pool Interface (sketch)
```c
// src/thread_pool.h
#ifndef THREAD_POOL_H
#define THREAD_POOL_H
typedef enum {
JOB_TYPE_REQ_QUERY,
JOB_TYPE_COUNT_QUERY,
JOB_TYPE_STORE_EVENT,
JOB_TYPE_DELETE_EVENT,
} job_type_t;
typedef struct {
job_type_t type;
void* session; // lws per-session data pointer
db_event_filter_t filter; // For REQ/COUNT jobs
cJSON* event; // For STORE jobs
char event_id[65]; // For DELETE jobs
} db_job_t;
int thread_pool_init(int num_readers, const char* db_path);
void thread_pool_shutdown(void);
int thread_pool_submit_read(db_job_t* job);
int thread_pool_submit_write(db_job_t* job);
#endif // THREAD_POOL_H
```
### Performance Characteristics
| Metric | Value |
|--------|-------|
| **Concurrent reads** | N readers in parallel (N = thread count, typically 48) |
| **Write throughput** | Same as current — SQLite serializes writes regardless |
| **Event loop latency** | Near-zero — REQ no longer blocks the loop |
| **Max theoretical read throughput** | ~48x current (limited by disk I/O, not CPU) |
| **Memory overhead** | ~50100 MB per connection (page cache) |
| **Latency per query** | Same as current per-query, but no head-of-line blocking |
### Limitations
- **Write contention**: SQLite still allows only ONE writer at a time. With WAL, readers don't block writers and writers don't block readers, but two simultaneous writes will serialize. At the current write rate (~56 events/hour), this is a non-issue.
- **Database size**: The 2.7 GB index bloat problem remains. Thread pool doesn't fix the schema — it fixes the concurrency.
- **Scaling ceiling**: Beyond ~8 reader threads, diminishing returns due to disk I/O contention on a single SQLite file.
- **Complexity**: Need a proper job queue, thread lifecycle management, and careful handling of the lws ↔ worker thread boundary.
---
## Risk Mitigation
| Risk | Mitigation |
|------|-----------|
| Phase 1 introduces bugs | Each file migrated independently, tested after each. Full test suite runs after each file. |
| Thread pool race conditions | Worker threads only touch their own `sqlite3*` connection. Result delivery uses a mutex-protected queue. `lws_cancel_service()` is documented as thread-safe. |
| Performance regression from abstraction | Abstraction layer is thin wrappers — no extra allocations or copies. Benchmark before/after. |
| lws thread safety issues | Workers never call `lws_write()` directly. They push to a queue and wake the event loop. This is the documented pattern for libwebsockets multi-threading. |
| Rollback needed | Phase 1 is a pure refactor — easy to revert. Phase 2 thread pool is additive — can be disabled with a compile flag. |
---
## Relationship to c-relay-pg-pg
After Phase 1 (abstraction layer) is complete, the codebase is ready to be forked into a separate **c-relay-pg-pg** project that replaces the SQLite backend with PostgreSQL. See [c-relay-pg-pg plan](c_relay_pg_pg_plan.md) for details.
The `db_ops.h` interface is designed to work with any backend:
- **SQLite** (this plan): `db_result_next_json()` copies from `sqlite3_column_text()`
- **PostgreSQL** (c-relay-pg-pg): `db_result_next_json()` returns from `PQgetvalue()`
- **LMDB** (future possibility): `db_result_next_json()` returns a zero-copy pointer into mmap'd memory
+72
View File
@@ -0,0 +1,72 @@
# Caching Code Consolidation Plan
## Goal
Move all caching_relay source code into a `caching/` directory inside c-relay-pg,
and switch the launcher to use PostgreSQL mode (`-p <pg-conn>`) so all config
comes from the c-relay-pg config table — no JSON config file needed.
## Key Finding
The caching_relay binary **already supports PG mode** via `-p <pg-conn>`. Its
`pg_config.c` reads the same config table keys the caching page sets
(`caching_root_npubs`, `caching_bootstrap_relays`, `caching_kinds`, etc.), and
`pg_inbox.c` writes fetched events to the `caching_event_inbox` table. The PG
schema tables already exist in `src/pg_schema.sql`.
## Steps
### Step 1: Create `caching/` directory and copy source files
Copy all `src/*.c` and `src/*.h` from `/home/user/lt/caching_relay/src/` into
`caching/src/` in this repo. Also copy the Makefile, build_static.sh,
Dockerfile.alpine-musl, and VERSION.
Files to copy:
- src/main.c, src/main.h
- src/config.c, src/config.h (legacy JSON config — kept for fallback)
- src/pg_config.c, src/pg_config.h (PG config reader)
- src/pg_inbox.c, src/pg_inbox.h (PG inbox writer)
- src/backfill.c, src/backfill.h
- src/follow_graph.c, src/follow_graph.h
- src/live_subscriber.c, src/live_subscriber.h
- src/relay_sink.c, src/relay_sink.h (legacy WebSocket sink — kept for fallback)
- src/relay_discovery.c, src/relay_discovery.h
- src/state.c, src/state.h
- src/debug.c, src/debug.h
- src/jsonc_strip.c, src/jsonc_strip.h
- Makefile
- VERSION
### Step 2: Adapt the Makefile
Update `caching/Makefile` to:
- Point `NOSTR_CORE_DIR` at `../nostr_core_lib` (sibling in this repo)
- Output binary to `caching/caching_relay` or `build/caching_relay`
### Step 3: Update the launcher to use PG mode
Update `src/caching_service_launcher.c`:
- Change `caching_service_start_fork()` to pass `-p <pg_conn>` instead of
`-c <config_path>`
- Update `caching_service_start()` to read `caching_service_pg_conn` from config
table instead of `caching_service_config_path`
### Step 4: Remove config_path from defaults and UI
- Remove `caching_service_config_path` from `src/default_config_event.h`
- Remove the "Caching Service Config Path" field from `api/index.html`
- Remove the field from `CACHING_CONFIG_FIELDS` in `api/index.js`
- Update `caching_service_binary_path` default to point at the new build
location (`./caching/caching_relay` or `./build/caching_relay`)
### Step 5: Build and test
- Build the caching_relay binary from the new `caching/` directory
- Build c-relay-pg with the updated launcher
- Test: set caching config on the page, apply, start service, verify it runs
in PG mode reading config from the database
### Step 6: Push
Run `./increment_and_push.sh` with a descriptive commit message.
+503
View File
@@ -0,0 +1,503 @@
# Caching Relay Daemon - Architecture Plan
> **Status:** Draft v3 - adds NIP-65 outbox model: bootstrap relays for
> discovery, kind-10002 per-pubkey relay lists, minimum covering set relay
> selection, local-relay-first on subsequent startups.
## Goal
A standalone C99 daemon that acts as a "caching relay feeder". It:
1. Reads a `.jsonc` config file listing one or more **root npubs** (e.g. your own npub), a set of upstream relays, a local relay URL, and configurable event kinds.
2. For each root npub, fetches its kind-3 contact list to discover **followed pubkeys**.
3. Subscribes **live** to events of the configured kinds from the union of followed pubkeys (root npubs + their follows).
4. Performs a **throttled backfill** of historical events per followed pubkey, spread out over time so upstream relays are not hammered.
5. Re-publishes every fetched event to the **local relay** via a plain WebSocket `EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg, or any Nostr relay).
6. Builds as a **statically linked C99 binary** following the c-relay model, linking against `nostr_core_lib` and `c_utils_lib`.
The user's Nostr client then points only at the local relay and gets a fast, pre-populated feed without doing any fan-out itself.
## Design Principles
- **Relay-agnostic on the sink side.** The daemon is just a Nostr client publishing `EVENT` messages over WebSocket. It does not touch the local relay's database directly. This keeps it decoupled and safe.
- **Reuse `nostr_core_lib`.** All Nostr protocol concerns (event validation, kind-3 parsing, relay pool, WebSocket client, NIP-19 npub decoding) come from `nostr_core_lib`. The daemon is orchestration logic only.
- **Single statically linked binary.** Built with the same Makefile pattern as c-relay: link `libnostr_core_x64.a` (or arm64) + `libc_utils.a` + system libs (`-lwebsockets -lsqlite3 -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm`).
- **C99, `-Wall -Wextra -std=c99 -g -O2`** matching c-relay's `CFLAGS`.
- **No config-file framework.** Hand-rolled `.jsonc` parser using cJSON (strip `//` and `/* */` comments before parsing, since cJSON does not natively support JSONC).
## Architecture Overview
```mermaid
flowchart TD
subgraph Config
CFG[caching_relay.jsonc]
end
subgraph Daemon
MAIN[main.c - lifecycle / signals]
CFGP[config.c - parse jsonc]
FOLLOW[follow_graph.c - root npubs + kind-3 resolution]
BACKFILL[backfill.c - throttled historical pull]
LIVE[live_subscriber.c - open subscriptions]
SINK[relay_sink.c - publish EVENT to local relay]
STATE[state.c - in-memory follow set + seen cache]
end
subgraph nostr_core_lib
UPOOL[core_relay_pool.c - upstream pool - query/subscribe]
SPOOL[core_relay_pool.c - sink pool - publish only]
NIP01[nip001.c - validate / parse]
NIP19[nip019.c - npub decode]
WS[nostr_websocket - client WS]
end
subgraph Upstream
R1[wss://relay.damus.io]
R2[wss://nos.lol]
R3[wss://...]
end
subgraph Local
LOCAL[wss://127.0.0.1:8888 - c-relay / c-relay-pg / any]
end
CFG --> CFGP
CFGP --> MAIN
MAIN --> FOLLOW
FOLLOW -->|query kind 3| UPOOL
UPOOL --> R1
UPOOL --> R2
UPOOL --> R3
FOLLOW --> STATE
MAIN --> LIVE
LIVE -->|subscribe authors + kinds| UPOOL
LIVE -->|on_event| SINK
MAIN --> BACKFILL
BACKFILL -->|query per pubkey since T| UPOOL
BACKFILL -->|on_event| SINK
SINK -->|publish_async| SPOOL
SPOOL -->|EVENT json| LOCAL
STATE --> LIVE
STATE --> BACKFILL
CFGP -->|read/write state| CFG
```
## Data Flow
```mermaid
sequenceDiagram
participant D as Daemon
participant UP as upstream_pool
participant SP as sink_pool
participant U as Upstream Relays
participant L as Local Relay
D->>D: parse caching_relay.jsonc + state
D->>UP: add_relay x N upstreams
D->>SP: add_relay local only
D->>UP: query_sync kind=3 authors=root_npubs
U-->>UP: kind-3 events
UP-->>D: followed pubkeys set
D->>D: merge root + follows into author set
D->>UP: subscribe live authors=author_set kinds=configured since=now
loop live loop - pump upstream_pool_run
U-->>UP: EVENT
UP-->>D: on_event callback
D->>SP: publish_async EVENT
SP->>L: EVENT json
D->>SP: pump sink_pool_run to flush callbacks
end
loop backfill - progressive window expansion
D->>UP: query_sync authors=pubkey_i kinds since=now-window limit=K
U-->>UP: historical events
UP-->>D: events
D->>SP: publish_async each EVENT
D->>D: sleep tick_interval_seconds
D->>D: on full round-robin pass - advance window, write state to jsonc
end
```
## Config File Format
`caching_relay.jsonc` (JSONC = JSON with comments). The config file is the
**single source of truth** and is also the daemon's persistent state store:
the daemon rewrites it to disk whenever the backfill window advances, so a
restart resumes exactly where it left off.
```jsonc
{
// Root npubs whose follows list we crawl
"root_npubs": [
"npub1...",
"npub1..."
],
// Upstream relays to pull events from
"upstream_relays": [
"wss://relay.damus.io",
"wss://nos.lol",
"wss://relay.nostr.band"
],
// Local relay to feed events into (publish-only, never queried)
"local_relay": "ws://127.0.0.1:8888",
// Event kinds to cache
"kinds": [1, 3, 6, 10000, 30023],
// ---- Backfill: progressive window expansion ----
"backfill": {
"enabled": true,
// Window schedule in seconds-from-now, applied in order.
// The daemon first pulls everything newer than (now - 24h).
// Once complete, it expands to (now - 7d), then (now - 30d), etc.
"window_schedule_seconds": [86400, 604800, 2592000, 7776000, 31536000],
// Per-pubkey query limit per tick (keeps individual queries light)
"events_per_tick": 50,
// Delay between pubkey backfill ticks (throttle to be polite)
"tick_interval_seconds": 5,
// Delay between completing one window and starting the next
"window_cooldown_seconds": 60
},
// ---- Live subscription ----
"live": {
"enabled": true,
"resubscribe_interval_seconds": 300
},
// Refresh the follow graph periodically
"follow_graph_refresh_seconds": 600,
// ---- Persistent state (managed by the daemon; do not hand-edit) ----
// The daemon writes these back to this file as backfill progresses.
// On restart it reads them to avoid re-pulling already-cached history.
"state": {
// How far back we have fully backfilled, as a unix timestamp.
// Starts at 0 / absent on first run. Advances as each window completes.
"backfilled_until": 0,
// Index into window_schedule_seconds we are currently working on.
"current_window_index": 0,
// Round-robin cursor: which followed pubkey we backfill next.
"backfill_cursor": 0
}
}
```
**State write-back rules:**
- The daemon rewrites the `.jsonc` file (preserving comments is *not* required
on rewrite - it may emit plain JSON once it has been modified at runtime;
the comments are only for the user's initial authoring convenience).
- Write-back happens: (a) when a window completes and `backfilled_until`
advances, (b) on graceful shutdown, (c) periodically (e.g. every 60s) so a
crash loses at most one minute of cursor progress.
- A write-ahead temp file + rename is used so the config is never left
half-written.
## Progressive Window-Expansion Backfill
Instead of a fixed per-pubkey cursor, the daemon uses a **global window** that
expands backwards in time in discrete steps. This is simpler, gives a natural
"recent first" experience, and is trivially resumable from the config file.
### Window schedule
`backfill.window_schedule_seconds` is an ordered list, e.g.
`[86400, 604800, 2592000, 7776000, 31536000]`
= 1 day, 1 week, 1 month, 3 months, 1 year.
### Algorithm
1. On startup, read `state.backfilled_until` (unix ts) and
`state.current_window_index` from the config.
2. If `backfilled_until == 0` (first run), set the current target window to
`window_schedule[0]` (e.g. 24h). The backfill `since` cutoff is
`now - window_schedule[0]`.
3. Round-robin through every followed pubkey. For each pubkey, issue
`nostr_relay_pool_query_sync` with:
- `authors = [pubkey]`
- `kinds = configured kinds`
- `since = now - current_window_seconds` (i.e. the *full* current window,
not an incremental slice - the local relay dedups, so re-overlap is free
and simpler than tracking per-pubkey cursors)
- `limit = events_per_tick`
4. Sleep `tick_interval_seconds` between pubkeys (throttle).
5. When one full round-robin pass over all followed pubkeys completes for the
current window:
- Set `state.backfilled_until = now - current_window_seconds`.
- Advance `state.current_window_index += 1`.
- Persist the config file (temp + rename).
- Sleep `window_cooldown_seconds` before starting the next (wider) window.
6. Repeat with the next (wider) window. The `since` cutoff moves further back
in time, so each window pulls the *additional* older slice. Because the
local relay dedups inserts, events that fall in the overlap with the
previous window are simply ignored on insert - no correctness issue.
7. After the last (widest) window completes, the daemon switches to
**steady-state**: it only keeps the live subscription running and
periodically re-runs the widest window to catch any events that migrated
into scope (e.g. a followed user backfilling their own old notes to a
different relay). `backfilled_until` stays pinned at the oldest window.
### Restart behavior
- On restart, the daemon reads `backfilled_until` and `current_window_index`.
- It resumes at the *current* window (the one that was in progress when it
stopped). Because each window re-pulls `since = now - window_seconds`, a
partial window just re-runs from the start of the round-robin - cheap and
correct.
- `state.backfill_cursor` records which pubkey in the round-robin was next;
this is a minor optimization and may be reset to 0 on restart without harm.
### Throttling summary
- `tick_interval_seconds` paces individual pubkey queries (e.g. 5s).
- `window_cooldown_seconds` paces window-to-window transitions (e.g. 60s).
- `events_per_tick` caps each query's result size (e.g. 50).
- Per-relay query latency from `nostr_relay_pool_get_relay_query_latency`
can be used to prefer fast relays for backfill; slow relays are still used
for the live subscription (where completeness matters more than speed).
## File Layout
```
caching_relay/
plans/plan.md (this file)
Makefile (c-relay-style, static link)
build_static.sh (Alpine MUSL static builder, adapted from c-relay)
Dockerfile.alpine-musl (adapted from c-relay)
src/
main.c (lifecycle, signal handling, main loop)
main.h
config.c / config.h (jsonc parse + validation)
follow_graph.c / .h (resolve root npubs -> kind-3 -> followed set)
live_subscriber.c / .h (open-ended live subscription on the pool)
backfill.c / .h (throttled historical pull worker)
relay_sink.c / .h (publish EVENT to local relay via dedicated sink pool)
state.c / .h (in-memory author set + seen-event ring buffer; config state read/write)
jsonc_strip.c / .h (strip // and /* */ comments before cJSON_Parse)
log.c / log.h (colored stderr logging, c-relay style)
examples/
caching_relay.jsonc (sample config)
README.md
```
## Build Model (mirrors c-relay)
- `Makefile` with:
- `CC = gcc`, `CFLAGS = -Wall -Wextra -std=c99 -g -O2`
- `INCLUDES = -I. -Isrc -I../nostr_core_lib -I../nostr_core_lib/nostr_core -I../nostr_core_lib/cjson -I../nostr_core_lib/nostr_websocket -I../c_utils_lib/src`
- `LIBS = -lwebsockets -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm -L../c_utils_lib -lc_utils`
- Note: no `-lsqlite3` - the daemon itself does not use SQLite. (c-relay uses
SQLite for its event store, but that is the local relay's concern, not the
daemon's.)
- `NOSTR_CORE_LIB = ../nostr_core_lib/libnostr_core_x64.a` (or arm64)
- Build `nostr_core_lib` with `--nips=1,6,19` (1 basic, 6 keys, 19 npub bech32).
NIP-42 is deferred to a later phase (public relays only for now). Kind-3
parsing is plain cJSON tag walking in NIP-01, no special NIP-02 module
needed.
- Single static link line: `$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)`
- `build_static.sh` adapted from c-relay's Alpine MUSL Docker builder to produce a truly static binary.
## Key nostr_core_lib APIs Used
- `nostr_relay_pool_create()` / `nostr_relay_pool_add_relay()` / `nostr_relay_pool_destroy()`
- `nostr_relay_pool_query_sync()` - one-shot historical/backfill queries (kind-3 fetch, per-pubkey backfill)
- `nostr_relay_pool_subscribe()` - long-lived live subscription with `on_event` / `on_eose` callbacks
- `nostr_relay_pool_run()` / `nostr_relay_pool_poll()` - event loop driving both live sub and backfill
- `nostr_relay_pool_publish_async()` - publish fetched events to the local relay
via a **dedicated single-relay sink pool** (separate from the upstream query
pool, so the local relay is never included in `query_sync` fan-out)
- `nostr_validate_event()` - validate before republishing (defensive; upstream events should already be valid)
- NIP-19: `nostr_nip19_decode` (or equivalent) to convert `npub1...` -> hex pubkey for filters
- cJSON for filter construction and kind-3 tag parsing (`["p", "<hex>", "<relay>", "<petname>"]`)
## Concurrency Model
Keep it simple and single-threaded with a cooperative event loop, matching c-relay's spirit:
- **Two `nostr_relay_pool_t` instances**, both driven from the main thread:
1. `upstream_pool` - holds all `upstream_relays`. Used for `query_sync`
(kind-3 fetch, backfill) and the long-lived live `subscribe`.
2. `sink_pool` - holds only `local_relay`. Used exclusively for
`publish_async` of fetched events. Never queried.
- The main loop alternates between:
- `nostr_relay_pool_run(upstream_pool, timeout_ms)` to pump live sub events,
- a time-sliced backfill step (one pubkey `query_sync` per
`tick_interval_seconds`),
- `nostr_relay_pool_run(sink_pool, 0)` to flush pending publish callbacks.
- Live subscription's `on_event` callback calls `relay_sink_publish()` which
enqueues an `publish_async` on the sink pool.
- Backfill `query_sync` is synchronous and blocks the upstream loop briefly -
acceptable since `events_per_tick` is small and `tick_interval_seconds`
provides pacing.
- If profiling later shows backfill blocking the live sub too much, backfill
can be moved to a second thread with its own upstream pool. Start without
that.
## State Persistence
- **No SQLite in the daemon.** The daemon's only persistent state is the
backfill window cursor, and that lives in the `.jsonc` config file itself
under the `state` object (see Config File Format above).
- The followed-pubkey set is re-derived from kind-3 on every
`follow_graph_refresh_seconds` tick and on startup - it is not persisted.
- Event dedup is delegated entirely to the local relay (c-relay's
`INSERT OR IGNORE` on event id). The daemon keeps a small in-memory ring
buffer of recently-published event ids only to avoid redundant publish
*attempts* within a single run; this is not persisted.
- Config write-back uses a temp file + atomic rename so the config is never
left half-written, even on crash.
## Signals & Lifecycle
- `SIGINT` / `SIGTERM` -> graceful shutdown: close subscriptions, destroy pool, close sink WS.
- `SIGHUP` -> reload config (re-read jsonc, refresh follow graph, adjust subscriptions).
- Logs to stderr with c-relay-style color prefixes.
## Decisions Resolved
1. **Sink pool.** Use a *dedicated* single-relay `nostr_relay_pool_t` for the
local sink, separate from the upstream pool. The local relay is never
queried, only published to. **Confirmed.**
- **Update (v3):** On subsequent startups, the local relay IS queried for
kind-3 and kind-10002 events (fast local cache lookup). It is still never
queried for general event backfill -- only for discovery metadata.
2. **NIP-42 auth.** Deferred to a later phase. Phase 1 targets public relays
only. **Confirmed.**
3. **Kind-3 freshness.** Use the most recent kind-3 per root npub
(`nostr_relay_pool_get_event` with `authors=[npub], kinds=[3]`). Re-resolve
on `follow_graph_refresh_seconds` interval. **Confirmed.**
4. **State persistence.** No SQLite in the daemon. The `.jsonc` config file is
the state store; the daemon writes back `state.backfilled_until`,
`state.current_window_index`, and `state.backfill_cursor` as backfill
progresses. **Confirmed.**
5. **Backfill strategy.** Progressive window expansion
(24h -> 7d -> 30d -> 90d -> 365d), round-robin per pubkey within each
window, full-window re-pull (local relay dedups). **Confirmed.**
6. **NIP-65 outbox model.** `upstream_relays` in config are **bootstrap
relays** only. The daemon discovers each followed pubkey's outbox relays
via kind 10002, then computes the **minimum covering set** of relays
(greedy set cover) that covers all followed pubkeys. Bootstrap relays are
always included in the final set. Pubkeys with no kind 10002 fall back to
bootstrap relays. **Confirmed.**
7. **Local-relay-first on subsequent startups.** On startup, if
`state.backfilled_until > 0`, the daemon queries the local relay first for
kind-3 and kind-10002 events (fast, no network). Falls back to bootstrap
relays for any pubkey not found locally. **Confirmed.**
8. **Admin kinds.** Root (admin) npubs get a separate kind list (`admin_kinds`)
with `["*"]` support for all kinds. Two live subscriptions: follows_sub
(non-admin, regular kinds) and admin_sub (admin, admin_kinds). **Confirmed.**
## NIP-65 Outbox Model Design (Phase 2)
### New module: `relay_discovery.c`
Responsibilities:
- For each followed pubkey, fetch their most recent kind 10002 (relay list).
- First-time: query from bootstrap relays.
- Subsequent: query from local relay first, bootstrap fallback.
- Parse `r` tags from kind 10002 events: `["r", "<url>"]` or
`["r", "<url>", "read"]` / `["r", "<url>", "write"]`.
- We care about "read" relays (we are reading events FROM them).
- If no read/write marker, assume both.
- Build a map: `pubkey -> list of outbox relay URLs`.
- Publish all kind-10002 events to the local relay (cache them for next startup).
### Greedy set cover algorithm
```
Input: pubkey_to_relays map {pubkey -> [relay1, relay2, ...]}
Output: minimal set of relay URLs covering all pubkeys
1. uncovered = set of all pubkeys
2. selected = empty set
3. Add all bootstrap relays to selected (always included)
4. For each bootstrap relay, remove its known pubkeys from uncovered
(bootstrap relays cover pubkeys with no 10002)
5. While uncovered is not empty:
a. Find relay R that covers the most pubkeys in uncovered
b. If no relay covers any uncovered pubkey, break (orphaned pubkeys)
c. Add R to selected
d. Remove all pubkeys covered by R from uncovered
6. Return selected
```
### Data structures
```c
/* Per-pubkey outbox relay list */
typedef struct {
char pubkey[CR_HEX_LEN];
char relays[CR_MAX_RELAYS_PER_PUBKEY][CR_URL_LEN];
int relay_count;
} cr_outbox_entry_t;
/* Relay-to-pubkeys coverage map (for set cover) */
typedef struct {
char url[CR_URL_LEN];
char pubkeys[CR_MAX_PUBKEYS_PER_RELAY][CR_HEX_LEN];
int pubkey_count;
} cr_relay_coverage_t;
/* Result of relay discovery */
typedef struct {
cr_outbox_entry_t *outboxes; /* per-pubkey relay lists */
int outbox_count;
char selected_relays[CR_MAX_UPSTREAM][CR_URL_LEN];
int selected_count;
} cr_relay_map_t;
```
### Startup sequence change
```
OLD:
load config -> create pools -> resolve follow graph -> open live sub -> backfill
NEW:
load config -> create pools ->
resolve follow graph (local-first on subsequent) ->
discover outbox relays (kind 10002, local-first on subsequent) ->
compute minimum covering set ->
add selected relays to upstream_pool ->
log selected relays + coverage ->
open live sub -> backfill (per-pubkey from their outbox relays)
```
### Backfill change
Instead of fanning out each backfill query to ALL upstream relays, backfill
queries each pubkey from **that pubkey's specific outbox relays** (or bootstrap
relays as fallback). This is more efficient and more polite to relays that
don't have that pubkey's events.
### New config fields
None required. `upstream_relays` is reinterpreted as bootstrap relays.
Optionally a `max_outbox_relays` cap could be added later if the covering set
grows too large.
## Implementation Todo List
### Phase 0 - Local relay sanity check (DONE)
0. **Start a local c-relay and verify read/write with `nak`.** DONE.
### Phase 1 - Daemon implementation (DONE)
1-12. All implemented, built, and tested. See git history. DONE.
### Phase 2 - NIP-65 outbox model
1. Implement `relay_discovery.c/.h`: fetch kind-10002 per pubkey, parse `r`
tags, build `pubkey -> relays` map, publish 10002 events to local relay.
2. Implement greedy set cover: compute minimum covering relay set from the
outbox map, always include bootstrap relays.
3. Update `follow_graph.c`: query local relay first for kind-3 on subsequent
startups, fall back to bootstrap relays.
4. Update `backfill.c`: query each pubkey from their specific outbox relays
instead of all upstream relays.
5. Update `main.c`: insert relay discovery phase between follow graph
resolution and live subscription. Add discovered relays to upstream_pool.
Log selected relays and coverage.
6. Update `caching_relay_config.jsonc`: update comments to say "bootstrap
relays" instead of "upstream relays".
7. Build and test NIP-65 outbox model end-to-end with a real npub.
8. Update `README.md` if any flowchart details change during implementation.
+543
View File
@@ -0,0 +1,543 @@
# Caching Relay PostgreSQL Inbox Integration Plan
> **Status:** Revised and simplified architecture.
>
> **Decision:** Keep caching as a separate application. The caching application
> focuses on polite upstream collection and writes raw event JSON to a minimal
> PostgreSQL inbox. c-relay-pg removes small priority-ordered batches and handles
> them through relay-owned event processing. The design deliberately accepts a
> small crash-loss window between dequeue and canonical storage; live overlap and
> backfill retries recover those events naturally.
## 1. Goal
Connect the standalone caching application to c-relay-pg without publishing
cached events through a loopback WebSocket and without allowing the caching
application to write directly to the canonical [`events`](../src/pg_schema.sql:12)
table.
The responsibilities are intentionally narrow:
- **Caching application:** sensibly and politely acquire the desired events.
- **PostgreSQL inbox:** temporarily hold structurally plausible event JSON.
- **c-relay-pg:** remain the sole authority for accepting, storing, and
broadcasting events.
This integration is PostgreSQL-only. The relay's normal SQLite operation remains
unchanged, but the external caching service is unavailable with that backend.
## 2. Simplified Architecture
```mermaid
flowchart LR
ROOT[Configured root npubs] --> CACHE[Caching application]
UP[Upstream relays] --> CACHE
CACHE --> INBOX[PostgreSQL caching event inbox]
INBOX --> POLLER[c-relay-pg bounded poller]
POLLER --> INGEST[Relay-owned event ingestion]
INGEST --> EVENTS[Canonical events table]
INGEST --> CLIENTS[Active subscriptions]
UI[Caching admin page] --> CFG[Shared caching configuration]
CFG --> CACHE
CACHE --> STATUS[Service heartbeat and progress]
STATUS --> UI
```
No loopback sink relay pool is used. No direct insert into the canonical event
store is allowed from the caching application.
## 3. Deliberate Simplicity Rules
The first implementation will not include:
- multiple caching workers;
- multiple inbox consumers;
- row leases or claim expiration;
- a queue state machine;
- retry rows;
- a dead-letter queue;
- PostgreSQL `LISTEN`/`NOTIFY`;
- persistent relay-health history;
- exactly-once delivery;
- direct process start/stop through the web page;
- upstream NIP-42 authentication;
- support for the SQLite backend.
The design prefers harmless duplicate acquisition over complicated delivery
coordination. Event IDs already provide natural deduplication in both the inbox
and canonical store.
## 4. Responsibility Boundaries
### 4.1 Caching application
The caching application is responsible for:
1. Reading caching configuration from PostgreSQL.
2. Decoding configured root npubs.
3. Resolving the roots' newest kind-3 follow lists.
4. Discovering useful read relays from kind 10002.
5. Computing a reasonable covering relay set.
6. Maintaining live subscriptions for roots and follows.
7. Performing polite, resumable historical backfill.
8. Inserting received event JSON into the inbox.
9. Persisting only the progress needed to resume discovery and backfill.
10. Publishing a lightweight heartbeat and status snapshot.
It is not responsible for:
- deciding whether an event belongs in the canonical relay database;
- applying c-relay-pg publication authorization;
- processing NIP-09 deletions;
- implementing replacement semantics;
- executing relay administrator commands;
- broadcasting to connected relay clients.
The caching callback may perform minimal application-side checks before insert:
- the callback supplied a JSON object;
- an event ID string exists;
- the serialized event is below a configured maximum size.
Cryptographic verification is optional in the fetcher and is not trusted by
c-relay-pg. The initial version should omit it unless the existing standalone
code already provides it at negligible integration cost.
### 4.2 PostgreSQL inbox
The inbox is deliberately dumb. It rejects obvious malformed data, deduplicates
events currently waiting in the inbox, and establishes live-over-backfill
priority. It does not attempt to verify a Nostr hash or Schnorr signature.
### 4.3 c-relay-pg
c-relay-pg is responsible for:
1. Removing a bounded batch from the inbox.
2. Parsing each event JSON object.
3. Applying authoritative structural, ID, and signature validation.
4. Applying relay-wide event rules such as expiration and PoW where applicable.
5. Bypassing client-session NIP-42 requirements because the source is internal.
6. Preventing imported kind 23456 events from executing administrator commands.
7. Applying duplicate, ephemeral, replaceable, addressable, and NIP-09 behavior.
8. Storing accepted events through its normal database abstraction.
9. Running post-store work and broadcasting newly accepted events from the main
libwebsockets thread.
10. Recording simple in-memory and cumulative import counters.
## 5. Minimal PostgreSQL Schema
Add the following objects to [`src/pg_schema.sql`](../src/pg_schema.sql) and the
embedded PostgreSQL schema header.
### 5.1 `caching_event_inbox`
Suggested logical schema:
```sql
CREATE TABLE IF NOT EXISTS caching_event_inbox (
queue_id BIGSERIAL PRIMARY KEY,
event_id TEXT NOT NULL UNIQUE,
event_json JSONB NOT NULL,
source_relay TEXT,
source_class TEXT NOT NULL DEFAULT 'backfill',
priority SMALLINT NOT NULL DEFAULT 1,
received_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
CHECK (jsonb_typeof(event_json) = 'object'),
CHECK (jsonb_typeof(event_json->'id') = 'string'),
CHECK (length(event_json->>'id') = 64),
CHECK (event_id = event_json->>'id'),
CHECK (jsonb_typeof(event_json->'pubkey') = 'string'),
CHECK (length(event_json->>'pubkey') = 64),
CHECK (jsonb_typeof(event_json->'sig') = 'string'),
CHECK (length(event_json->>'sig') = 128),
CHECK (jsonb_typeof(event_json->'created_at') = 'number'),
CHECK (jsonb_typeof(event_json->'kind') = 'number'),
CHECK (jsonb_typeof(event_json->'tags') = 'array'),
CHECK (jsonb_typeof(event_json->'content') = 'string'),
CHECK (source_class IN ('live', 'discovery', 'backfill')),
CHECK (priority IN (0, 1))
);
CREATE INDEX IF NOT EXISTS idx_caching_inbox_dequeue
ON caching_event_inbox(priority, received_at, queue_id);
```
Priority meanings:
- `0`: live and discovery metadata;
- `1`: historical backfill.
The caching application uses a parameterized insert with
`ON CONFLICT (event_id) DO NOTHING`. It must impose a serialized event-size
limit before sending the row. PostgreSQL schema constraints are structural
protection, not authoritative Nostr validation.
### 5.2 `caching_service_state`
Use one singleton row for inexpensive UI status:
- service version;
- service state: `starting`, `running`, `degraded`, or `stopped`;
- applied config generation;
- heartbeat timestamp;
- followed author count;
- selected and connected relay counts;
- current backfill window and author cursor;
- events fetched and inbox inserts;
- last error text and timestamp.
This is a status snapshot, not an audit log. The caching process overwrites the
same row periodically.
### 5.3 `caching_backfill_progress`
Persist only enough state for the caching application to resume politely:
- author pubkey;
- window index;
- immutable window anchor;
- current inclusive `until` cursor;
- completion flag;
- updated timestamp.
Use a composite primary key on author and window. Do not store individual fetched
event IDs here; inbox and canonical event IDs handle deduplication.
Timestamp-only Nostr pagination can saturate when many events share one second.
For the first version, increase the query limit up to a fixed safety ceiling when
a full page ends at one timestamp. If that ceiling remains saturated, leave the
author incomplete, log it, and retry later rather than falsely marking it done.
## 6. Simple Destructive Dequeue
c-relay-pg uses one PostgreSQL transaction to remove and return a small batch:
```sql
WITH selected AS (
SELECT queue_id
FROM caching_event_inbox
ORDER BY priority ASC, received_at ASC, queue_id ASC
LIMIT $1
FOR UPDATE
)
DELETE FROM caching_event_inbox AS inbox
USING selected
WHERE inbox.queue_id = selected.queue_id
RETURNING inbox.event_id,
inbox.event_json,
inbox.source_relay,
inbox.source_class,
inbox.received_at;
```
Only one c-relay-pg inbox consumer is supported. `SKIP LOCKED`, leases, and claim
owners are therefore unnecessary.
### 6.1 Accepted trade-off
There is a small loss window if c-relay-pg commits the delete and crashes before
storing the returned events. This is accepted deliberately:
- live subscriptions reconnect with overlap;
- backfill revisits incomplete and steady-state windows;
- duplicate reacquisition is safe;
- the reduced implementation complexity is worth the rare temporary loss.
The poller should keep batches small so the loss window and memory footprint are
small.
## 7. c-relay-pg Inbox Consumer
### 7.1 Polling
- Compile the consumer only for `DB_BACKEND_POSTGRES`.
- Poll a small batch at a configurable interval.
- Poll quickly while rows are found and back off to a slower interval when empty.
- Always order live/discovery rows before backfill rows.
- Allow only one batch in memory at a time.
- Stop polling immediately during relay shutdown.
Initial conservative defaults should be small, for example a few dozen rows per
batch and subsecond-to-multisecond active polling with a longer idle interval.
Exact defaults should be selected during integration testing rather than encoded
as architectural requirements.
### 7.2 Ingestion behavior
Do not send dequeued events through a loopback WebSocket. Refactor the existing
inbound event handling into a reusable internal ingestion entry point with an
explicit source mode:
- `CLIENT_EVENT`: normal client authentication and response behavior;
- `CACHING_INBOX_EVENT`: no client session or NIP-42 requirement, no OK response,
and no administrator command execution.
The shared path should preserve:
- event ID and signature verification;
- event limits;
- expiration and PoW behavior;
- NIP-09 authorization and deletion behavior;
- ephemeral handling;
- PostgreSQL replacement semantics in
[`postgres_db_insert_event_with_json()`](../src/db_ops_postgres.c:1596);
- duplicate detection;
- post-store monitoring;
- main-thread broadcast.
Avoid creating a second implementation of these rules solely for inbox events.
### 7.3 Main-thread broadcast
Database and cryptographic work may run off the libwebsockets thread, but
[`store_event_post_actions()`](../src/main.c:1161) and active subscription
broadcast must be queued to the main thread, following the existing asynchronous
completion pattern in
[`process_async_event_completions()`](../src/websockets.c:725).
Newly stored events are broadcast. Duplicates and stale replacements are not.
Validated ephemeral events are broadcast but not stored.
### 7.4 Failure behavior
Because rows have already been removed, malformed or invalid events are simply
counted and logged at a rate-limited level. They are not requeued.
A temporary canonical database failure should stop further inbox polling until
the relay database is healthy. The current in-memory batch may be lost under the
accepted simple-delivery model.
## 8. Caching Application Acquisition Strategy
The external application should spend most of its design effort here.
### 8.1 Follow graph
- Decode configured root npubs.
- Fetch the newest kind 3 for each root from bootstrap relays.
- Include roots themselves in the author set.
- Parse valid `p` tags, deduplicate authors, and enforce a configured author cap.
- Refresh periodically and replace live subscriptions when the set changes.
Local-first querying is optional in this architecture. PostgreSQL already holds
canonical kind-3 events, so the caching application may query the canonical event
store read-only for discovery metadata before contacting upstream relays. It must
not write canonical rows.
### 8.2 NIP-65 relay discovery
- Fetch newest kind 10002 events in bounded author batches.
- Use unmarked or `read` relay tags; ignore `write`-only tags.
- Normalize and deduplicate `wss://` URLs.
- Enforce per-author and global relay caps.
- Use the greedy covering-set logic from the standalone implementation.
- Retain configured bootstrap relays as fallback.
### 8.3 Friendly live subscriptions
- Separate root and followed-author subscriptions when their kind lists differ.
- Batch author lists into reasonable filter sizes.
- Reconnect with bounded exponential backoff and jitter.
- Use a small timestamp overlap when resubscribing.
- Insert callback event JSON into the inbox and continue; do not wait for
c-relay-pg processing.
### 8.4 Friendly backfill
- Use recent-first progressive windows.
- Persist an immutable anchor for each active window.
- Query one author at a time from that author's declared outbox relays, with
bootstrap fallback.
- Use configurable page size and delay between queries.
- Apply relay-specific backoff after errors or timeouts.
- Keep global and per-relay request rates low.
- Advance progress only after the returned page has been inserted into the
inbox or identified as already queued.
- Periodically repeat a bounded widest-window sweep after initial completion.
## 9. Shared Configuration
Keep administrator intent in the existing [`config`](../src/pg_schema.sql:192)
table with category `caching` and `requires_restart = 0`.
Suggested keys:
- `caching_enabled`;
- `caching_config_generation`;
- `caching_root_npubs`;
- `caching_bootstrap_relays`;
- `caching_kinds`;
- `caching_admin_kinds`;
- `caching_live_enabled`;
- `caching_live_resubscribe_seconds`;
- `caching_backfill_enabled`;
- `caching_backfill_windows`;
- `caching_backfill_page_size`;
- `caching_backfill_tick_interval_ms`;
- `caching_backfill_window_cooldown_seconds`;
- `caching_follow_graph_refresh_seconds`;
- `caching_relay_discovery_refresh_seconds`;
- `caching_max_followed_pubkeys`;
- `caching_max_upstream_relays`;
- `caching_max_relays_per_pubkey`;
- `caching_query_timeout_ms`;
- `caching_inbox_batch_size`;
- `caching_inbox_active_poll_ms`;
- `caching_inbox_idle_poll_ms`;
- `caching_max_event_json_bytes`.
The web UI updates these through the existing encrypted administrator API. After
a successful update, increment `caching_config_generation`. The caching
application polls the generation and reloads valid changes. No cross-process
command queue is needed.
`caching_enabled = false` tells the external application to close upstream
activity. systemd or the container runtime, not the web page, owns the process
lifecycle.
## 10. Admin UI
### 10.1 Sidenav placement
In [`api/index.html`](../api/index.html:14), insert **Caching** immediately after
**Relay Events** and before **DM**:
1. Statistics
2. Subscriptions
3. Configuration
4. Authorization
5. IP BAN
6. Relay Events
7. **Caching**
8. DM
9. Database Query
Register `cachingSection` in [`switchPage()`](../api/index.js:5349).
### 10.2 Page layout
Reuse the existing admin UI classes and divide the page into three blocks.
**Service status**
- enabled configuration;
- service heartbeat and state;
- desired and applied config generation;
- followed author count;
- selected and connected relay counts;
- current backfill window and cursor;
- fetched and inbox-inserted counters;
- last service error.
**Relay inbox status**
- pending live/discovery count;
- pending backfill count;
- oldest inbox row age;
- relay-consumed, accepted, duplicate, invalid, and failed counters;
- last successful inbox poll.
**Configuration**
- root npubs and bootstrap relays;
- normal and root kind lists;
- live and backfill toggles;
- backfill windows and pacing;
- discovery refresh intervals;
- author and relay safety caps;
- inbox batch and polling settings;
- **Apply Configuration** button;
- **Reset Backfill Progress** button with confirmation.
The UI reads service state from the singleton status row and lightweight inbox
aggregates. Poll only while the Caching page is visible and use a modest refresh
interval.
## 11. Process and Database Security
Use separate PostgreSQL roles:
- **Caching service role:** read caching configuration and permitted canonical
discovery metadata; insert/select its own status and progress; insert into the
inbox; no insert/update/delete permission on canonical events.
- **c-relay-pg role:** normal relay permissions plus dequeue permission on the
inbox and read access to caching status.
Use parameterized SQL throughout. Restrict public upstreams to normalized
`wss://` URLs. Apply a database statement timeout to caching queries so the
fetcher cannot hold resources indefinitely.
## 12. Lifecycle
### 12.1 Caching service
- Starts and stops independently under systemd or a container runtime.
- On startup, loads configuration and progress, updates heartbeat state, then
begins discovery/live/backfill.
- On shutdown, closes subscriptions, saves current progress, marks status
stopped, and disconnects from PostgreSQL.
- Its failure does not stop c-relay-pg; cached acquisition merely pauses.
### 12.2 c-relay-pg
- Starts the inbox poller only for PostgreSQL builds after database, writer pool,
and WebSocket systems are ready.
- Stops new polls before shutting down the writer pool or WebSocket context.
- Drains already-created main-thread completions before destroying those
dependencies.
- A missing inbox table should be logged as caching unavailable, not terminate
the relay, unless schema migration policy requires otherwise.
## 13. Implementation Sequence
1. Add `caching_event_inbox`, `caching_service_state`, and
`caching_backfill_progress` to the PostgreSQL schema and embedded schema.
2. Add PostgreSQL database abstraction functions for bounded destructive dequeue
and lightweight inbox counts.
3. Refactor c-relay-pg inbound event processing into a shared source-aware
ingestion entry point without changing normal client behavior.
4. Add the PostgreSQL-only inbox poller, bounded in-memory batch, and main-thread
post-action/broadcast completion path.
5. Add caching configuration defaults and validation to c-relay-pg.
6. Adapt the standalone caching application to read PostgreSQL configuration,
insert raw event JSON into the inbox, and update heartbeat/status.
7. Correct the standalone backfill implementation so limited queries paginate
and do not falsely complete an author/window.
8. Persist simple per-author/window progress and implement polite retry/backoff.
9. Add the Caching page after Relay Events and before DM, including service,
inbox, configuration, and reset-progress controls.
10. Add PostgreSQL role/grant documentation and systemd/container deployment
examples for the separate caching process.
11. Test live priority, duplicate acquisition, invalid inbox rows, replacement,
deletion, ephemeral broadcast, process restarts, destructive-dequeue crash
behavior, upstream outages, and graceful shutdown.
12. Build and validate c-relay-pg only through
[`make_and_restart_relay.sh`](../make_and_restart_relay.sh), using
`--preserve-database` where test state must survive.
## 14. Acceptance Criteria
- The caching application cannot write canonical event rows.
- Received upstream events require only minimal fetcher checks before inbox
insertion.
- Inbox constraints reject structurally implausible rows and deduplicate pending
event IDs.
- c-relay-pg removes small batches ordered live/discovery before backfill.
- c-relay-pg remains the authoritative ID/signature validator and storage owner.
- Imported events never execute relay administrator commands or require a client
NIP-42 session.
- Newly accepted events are broadcast to active subscriptions from the main
libwebsockets thread.
- Duplicate and stale replacement events are not rebroadcast.
- The caching service behaves politely through bounded filters, paced backfill,
targeted outbox queries, and retry backoff.
- Backfill progress survives caching-service restarts and does not falsely mark
saturated timestamps complete.
- A caching-service failure does not stop the relay.
- The accepted destructive-dequeue crash window is documented and recoverable
through live overlap and repeated backfill.
- The Caching UI appears after Relay Events and before DM and distinguishes
external service health from relay inbox consumption.
+113
View File
@@ -0,0 +1,113 @@
# Caching Status "Not Implemented" Fix Plan
## Root Cause
The caching page shows "Service status unavailable (caching_status command not
implemented on relay)" — but the command **is** implemented at
[`src/config.c:4128`](../src/config.c:4128). The real problem is **admin
authorization failure**: every admin command from the browser is rejected at
publish time with:
```
Unauthorized admin event attempt: invalid admin pubkey
```
This rejection happens at [`src/main.c:2249`](../src/main.c:2249) because the
browser's pubkey is not in the relay's `admin_pubkey` config list. The kind
23456 event never reaches the command handler, so `caching_status` never
executes, and the placeholder text persists.
### Confirmed admin key
- Relay admin pubkey (hex): `6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3`
- Relay admin npub: `npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks`
- Source: [`.relay.laantungir.net.keys`](../.relay.laantungir.net.keys:1)
The browser extension (nos2x) was using pubkey `8ff74724...`, which is not an
admin on the port 7777 relay.
## Secondary Issue (latent)
Even after auth is fixed, the caching status UI would render **empty blocks**
due to a schema mismatch between backend and frontend:
| Frontend expects ([`api/index.js:7046`](../api/index.js:7046)) | Backend emits ([`src/config.c:4135`](../src/config.c:4135)) |
|---|---|
| `data.service.enabled` | `data.caching_enabled` |
| `data.service.running` | *(not emitted)* |
| `data.inbox.enabled` | `data.caching_inbox_enabled` |
| `data.inbox.queue_depth` | `data.inbox_pending_live` + `data.inbox_pending_backfill` |
| *(not expected)* | `data.inbox_total_dequeued`, `data.inbox_total_accepted`, `data.inbox_total_rejected`, `data.inbox_oldest_age_seconds` |
## Fix Steps
```mermaid
flowchart TD
A[Step 1: Fix admin auth] --> B[Step 2: Align response schema]
B --> C[Step 3: Fix frontend handler]
C --> D[Step 4: Improve error messaging]
D --> E[Step 5: Test end-to-end]
```
### Step 1 — Fix admin authorization (config, user action)
Load the admin private key (corresponding to `6a04ab98...` /
`npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks`) into the
nos2x browser extension so admin commands authenticate. Use `nak` on the
command line to convert/derive the nsec if needed.
This unblocks **all** admin commands, not just caching.
### Step 2 — Align backend response schema (code, [`src/config.c:4128`](../src/config.c:4128))
Restructure the `data` object in the `caching_status` handler to emit nested
`service` and `inbox` objects matching the frontend handler:
```json
{
"command": "caching_status",
"status": "success",
"data": {
"service": {
"enabled": false,
"running": false,
"connected_relays": 0,
"events_cached": 0
},
"inbox": {
"enabled": false,
"running": true,
"queue_depth": 0,
"last_poll": 0,
"total_dequeued": 0,
"total_accepted": 0,
"total_rejected": 0,
"total_duplicates": 0,
"pending_live": 0,
"pending_backfill": 0,
"oldest_age_seconds": 0
}
}
}
```
Populate `service.running` / `connected_relays` / `events_cached` from the
caching service launcher state if available; otherwise emit zeros/defaults.
### Step 3 — Update frontend handler (code, [`api/index.js:7027`](../api/index.js:7027))
Update `handleCachingStatusResponse()` to render the inbox poller stats the
backend actually produces (dequeued/accepted/rejected/pending/oldest_age), and
map `data.inbox.pending_live + pending_backfill` to queue depth.
### Step 4 — Replace misleading placeholder (code, [`api/index.js:6889`](../api/index.js:6889))
- Change the pre-send placeholder from "caching_status command not implemented"
to a neutral "Loading..." message.
- On auth failure, show the actual error (e.g., "Not authorized: pubkey not
registered as admin") instead of the generic "unavailable" text.
### Step 5 — Test end-to-end on port 7777
Verify the caching page shows real service/inbox status after auth is fixed
and the schema is aligned.
File diff suppressed because it is too large Load Diff
+186
View File
@@ -0,0 +1,186 @@
# Plan: Eliminate g_db from the Main Thread
## Problem Statement
All SQLite calls go through `db_ops.c`, but `db_active_connection()` falls back to `g_db` when `g_thread_db` is NULL. Since the main thread never sets `g_thread_db`, every `db_*` call from lws-main executes synchronous SQLite on the main thread. The thread pool workers have their own connections and work correctly — the problem is the **fallback path**.
Current perf data shows lws-main at **67.6% avg CPU**, dominated by SQLite symbols (`sqlite3BtreeTableMoveto`, `sqlite3VdbeExec`, `pcache1Fetch`).
## Goal
Remove `g_db` as a runtime connection used by the main thread. After this change:
- The main thread has **no SQLite connection** and cannot execute synchronous queries
- All DB work routes through thread pool workers (which have their own connections)
- `g_db` is only used during startup/shutdown (before/after the event loop)
- Any accidental `db_*` call from lws-main returns an error instead of silently blocking
## Architecture After Change
```
lws-main thread:
- WebSocket protocol handling
- Message parsing/routing
- Completion queue draining
- NO SQLite access during event loop
db-read-0 thread:
- REQ queries
- COUNT queries
- Config cache miss reads
- Auth rule checks
- Monitoring/stats queries
db-write thread:
- EVENT inserts
- Subscription logging
- Config updates
- Auth rule modifications
event-worker thread:
- Signature validation
- Duplicate check
- Store via db-write
```
## Categorized g_db Call Sites
### Category A: Startup-only — keep using g_db
These run before the event loop starts. They are fine.
| Function | File | Purpose |
|----------|------|---------|
| `db_init()` | db_ops.c:27 | Open database |
| `db_exec_sql()` for PRAGMAs | main.c:854-882 | WAL, mmap, cache setup |
| `db_table_exists()` | main.c:750 | Schema check |
| `db_get_schema_version_dup()` | main.c:754 | Migration check |
| `db_exec_sql()` for schema | main.c:842 | Create tables |
| `populate_all_config_values_atomic()` | config.c:4290 | First-time config |
| `populate_default_config_values()` | config.c:1657 | Default config |
| `add_pubkeys_to_config_table()` | config.c:1778 | Pubkey storage |
| `apply_cli_overrides_atomic()` | config.c:4211 | CLI overrides |
| `db_store_relay_private_key_hex()` | config.c:495 | Key storage |
| `db_populate_event_tags_from_existing()` | main.c:1156 | Tag migration |
| `cleanup_all_subscriptions_on_startup()` | subscriptions.c:1092 | Orphan cleanup |
### Category B: Shutdown-only — keep using g_db
| Function | File | Purpose |
|----------|------|---------|
| `db_exec_sql()` WAL checkpoint | main.c:900 | Clean shutdown |
| `db_close()` | main.c:904 | Close connection |
### Category C: Runtime hot path — must move off main thread
These are called during the event loop from lws-main context.
| Function | File | Called from | Fix strategy |
|----------|------|------------|--------------|
| `db_get_config_value_dup()` | db_ops.c:777 | config cache miss | Already cached with 5s TTL; pre-warm cache at startup so misses are rare |
| `store_event()` for kind 14/1059 | main.c:983 | websockets.c sync path | Route through async event worker |
| `store_event_post_actions()` | main.c | completion handler | Already runs on main; its DB calls need routing |
| `db_log_subscription_created()` | db_ops.c:145 | subscriptions.c | Fire-and-forget via write queue |
| `db_log_subscription_closed()` | db_ops.c:165 | subscriptions.c | Fire-and-forget via write queue |
| `db_log_subscription_disconnected()` | db_ops.c:193 | subscriptions.c | Fire-and-forget via write queue |
| `db_update_subscription_events_sent()` | db_ops.c:225 | subscriptions.c | Fire-and-forget via write queue |
| `db_cleanup_orphaned_subscriptions()` | db_ops.c:244 | subscriptions.c | Move to startup only |
| `generate_and_post_status_event()` | websockets.c:3408 | periodic timer | Submit to write worker |
| `ip_ban_cleanup()` / `ip_ban_log_stats()` | websockets.c:3196 | periodic timer | Submit to write worker |
| `db_get_total_event_count_ll()` | db_ops.c:589 | api.c monitoring | Submit to read worker |
| `db_get_event_count_since()` | db_ops.c:606 | api.c monitoring | Submit to read worker |
| `db_get_event_kind_distribution_rows()` | db_ops.c:625 | api.c monitoring | Submit to read worker |
| `db_get_top_pubkeys_rows()` | db_ops.c:663 | api.c monitoring | Submit to read worker |
| `db_get_subscription_details_rows()` | db_ops.c:697 | api.c monitoring | Submit to read worker |
| `db_get_all_config_rows()` | db_ops.c:745 | api.c config query | Submit to read worker |
| `db_execute_readonly_query_json()` | db_ops.c:442 | api.c SQL query | Submit to read worker |
| `db_event_id_exists()` | db_ops.c:1041 | main.c event check | Already moved to event worker |
| `db_retrieve_event_by_id()` | db_ops.c:1056 | main.c | Submit to read worker |
| `db_get_event_pubkey()` | db_ops.c:262 | NIP-09 deletion | Submit to read worker |
| `db_delete_event_by_id()` | db_ops.c:285 | NIP-09 deletion | Submit to write worker |
| `db_delete_older_replaceable_events()` | db_ops.c:305 | store_event_core | Already on write worker |
| `db_store_config_event()` | db_ops.c:888 | config.c | Submit to write worker |
| `db_add_auth_rule()` | db_ops.c:1226 | config.c admin | Submit to write worker |
| `db_remove_auth_rule()` | db_ops.c:1242 | config.c admin | Submit to write worker |
| `db_delete_wot_whitelist_rules()` | db_ops.c:1258 | config.c WoT | Submit to write worker |
| `db_count_wot_whitelist_rules()` | db_ops.c:1266 | config.c | Cache or read worker |
| `db_store_event_tags_cjson()` | db_ops.c:1140 | main.c post-actions | Already on write worker path |
| `db_get_config_row_count()` | db_ops.c:1121 | config.c diagnostics | Cache or read worker |
| `db_set_config_value_full()` | db_ops.c:796 | config.c | Submit to write worker |
| `db_update_config_value_only()` | db_ops.c:821 | config.c | Submit to write worker |
| `db_upsert_config_value()` | db_ops.c:838 | api.c | Submit to write worker |
| `db_exec_sql()` for transactions | config.c | admin events | Submit to write worker |
| `db_count_with_sql()` | db_ops.c:415 | config.c admin | Submit to read worker |
| `is_config_table_ready()` | config.c:4595 | config.c hybrid | Cache result at startup |
| `generate_config_event_from_table()` | config.c:4933 | config.c | Cache or read worker |
### Category D: Auth rule checks — already use separate connections
These functions in `db_ops.c` open their own temporary read-only connections:
| Function | Line | Notes |
|----------|------|-------|
| `db_is_pubkey_blacklisted()` | 347 | Opens own connection |
| `db_is_hash_blacklisted()` | 362 | Opens own connection |
| `db_is_pubkey_whitelisted()` | 377 | Opens own connection |
| `db_count_active_whitelist_rules()` | 392 | Opens own connection |
These are already safe — they don't use `g_db`. No change needed.
## Implementation Strategy
### Phase 1: Pre-warm caches and eliminate cache-miss DB reads
1. Pre-warm the config cache at startup by loading all config values into the in-memory cache before the event loop starts
2. Pre-warm the NIP-11 cache at startup
3. Pre-warm the auth rules fast cache at startup
4. This eliminates the most frequent cache-miss `db_get_config_value_dup()` calls
### Phase 2: Route subscription logging through write worker
1. Add a `THREAD_POOL_JOB_FIRE_AND_FORGET` job type to thread_pool
2. Create async wrappers: `db_log_subscription_created_async()`, etc.
3. These submit SQL to the write worker queue and return immediately
4. No completion callback needed — fire and forget
### Phase 3: Route special-kind EVENT store through async worker
1. Remove the `event_is_async_eligible()` exclusion for kinds 14, 1059, 23456
2. For kind 23456: process admin command in completion handler on main thread after store
3. For kind 14/1059: process NIP-17 DM in completion handler after store
### Phase 4: Route periodic timer DB work through workers
1. `generate_and_post_status_event()` — submit event creation to write worker
2. `ip_ban_cleanup()` / `ip_ban_log_stats()` — submit to write worker
3. Monitoring queries — submit to read worker with completion callback
### Phase 5: Null out g_db before event loop
1. After startup is complete and thread pool is initialized, set `g_db = NULL`
2. Change `db_active_connection()` to assert/warn if both `g_thread_db` and `g_db` are NULL
3. Any accidental main-thread DB call will now fail loudly instead of silently blocking
4. Restore `g_db` briefly for shutdown checkpoint
## Implementation Order
1. Phase 1 — lowest risk, immediate benefit from eliminating cache misses
2. Phase 2 — straightforward fire-and-forget pattern
3. Phase 3 — moderate complexity, needs careful completion handler design
4. Phase 4 — moderate complexity, periodic tasks need async patterns
5. Phase 5 — the final enforcement step, only safe after phases 1-4
## Risk Assessment
- **Phase 1**: Very low risk — just pre-warming existing caches
- **Phase 2**: Low risk — subscription logging is non-critical, fire-and-forget is safe
- **Phase 3**: Medium risk — admin/DM event processing has complex state; needs careful testing
- **Phase 4**: Medium risk — periodic tasks have side effects that need to complete
- **Phase 5**: High risk if done prematurely — must verify ALL runtime paths are covered first
## Expected Impact
After all phases:
- lws-main CPU should drop from ~67% to near 0% SQLite overhead
- All SQLite work happens on db-read and db-write threads
- Main thread only does WebSocket I/O, JSON parsing, and completion queue draining
- Thread model becomes: lws-main = pure I/O, workers = all DB
@@ -0,0 +1,579 @@
# Event JSON Storage & Database Migration Plan
**Goal:** Store full event JSON in database for 2,500x faster retrieval + implement proper database migration system
---
## Decision: Fresh Start vs Migration
### Option A: Fresh Start (Recommended for This Change)
**Pros:**
- ✅ Clean implementation (no migration complexity)
- ✅ Fast deployment (no data conversion)
- ✅ No risk of migration bugs
- ✅ Opportunity to fix any schema issues
- ✅ Smaller database (no legacy data)
**Cons:**
- ❌ Lose existing events
- ❌ Relay starts "empty"
- ❌ Historical data lost
**Recommendation:** **Fresh start for this change** because:
1. Your relay is still in development/testing phase
2. The schema change is fundamental (affects every event)
3. Migration would require reconstructing JSON for every existing event (expensive)
4. You've been doing fresh starts anyway
### Option B: Implement Migration System
**Pros:**
- ✅ Preserve existing events
- ✅ No data loss
- ✅ Professional approach
- ✅ Reusable for future changes
**Cons:**
- ❌ Complex implementation
- ❌ Slow migration (reconstruct JSON for all events)
- ❌ Risk of bugs during migration
- ❌ Requires careful testing
**Recommendation:** **Implement migration system for FUTURE changes**, but start fresh for this one.
---
## Proposed Schema Change
### New Schema (v11)
```sql
CREATE TABLE events (
id TEXT PRIMARY KEY,
pubkey TEXT NOT NULL,
created_at INTEGER NOT NULL,
kind INTEGER NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
content TEXT NOT NULL,
sig TEXT NOT NULL,
tags JSON NOT NULL DEFAULT '[]',
event_json TEXT NOT NULL, -- NEW: Full event as JSON string
first_seen INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
);
-- Keep all existing indexes (they query the columns, not event_json)
CREATE INDEX idx_events_pubkey ON events(pubkey);
CREATE INDEX idx_events_kind ON events(kind);
CREATE INDEX idx_events_created_at ON events(created_at DESC);
CREATE INDEX idx_events_kind_created_at ON events(kind, created_at DESC);
CREATE INDEX idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
```
### Why Keep Both Columns AND event_json?
**Columns (id, pubkey, kind, etc.):**
- Used for **querying** (WHERE clauses, indexes)
- Fast filtering and sorting
- Required for SQL operations
**event_json:**
- Used for **retrieval** (SELECT results)
- Pre-serialized, ready to send
- Eliminates JSON reconstruction
**This is a common pattern** in high-performance systems (denormalization for read performance).
---
## Implementation Plan
### Phase 1: Schema Update (v11)
**File:** `src/sql_schema.h`
```c
#define EMBEDDED_SCHEMA_VERSION "11"
// In schema SQL:
"CREATE TABLE events (\n\
id TEXT PRIMARY KEY,\n\
pubkey TEXT NOT NULL,\n\
created_at INTEGER NOT NULL,\n\
kind INTEGER NOT NULL,\n\
event_type TEXT NOT NULL,\n\
content TEXT NOT NULL,\n\
sig TEXT NOT NULL,\n\
tags JSON NOT NULL DEFAULT '[]',\n\
event_json TEXT NOT NULL,\n\ -- NEW COLUMN
first_seen INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))\n\
);\n\
```
### Phase 2: Update store_event() Function
**File:** `src/main.c` (lines 660-773)
**Current:**
```c
int store_event(cJSON* event) {
// Extract fields
cJSON* id = cJSON_GetObjectItem(event, "id");
// ... extract other fields ...
// INSERT with individual columns
const char* sql = "INSERT INTO events (id, pubkey, ...) VALUES (?, ?, ...)";
}
```
**New:**
```c
int store_event(cJSON* event) {
// Serialize event to JSON string ONCE
char* event_json = cJSON_PrintUnformatted(event);
if (!event_json) {
return -1;
}
// Extract fields for indexed columns
cJSON* id = cJSON_GetObjectItem(event, "id");
// ... extract other fields ...
// INSERT with columns + event_json
const char* sql = "INSERT INTO events (id, pubkey, ..., event_json) VALUES (?, ?, ..., ?)";
// ... bind parameters ...
sqlite3_bind_text(stmt, 9, event_json, -1, SQLITE_TRANSIENT);
// ... execute ...
free(event_json);
}
```
### Phase 3: Update handle_req_message() Function
**File:** `src/main.c` (lines 1302-1361)
**Current:**
```c
while (sqlite3_step(stmt) == SQLITE_ROW) {
// Build event JSON from 7 columns
cJSON* event = cJSON_CreateObject();
cJSON_AddStringToObject(event, "id", (char*)sqlite3_column_text(stmt, 0));
// ... 6 more fields ...
cJSON* tags = cJSON_Parse(tags_json); // Parse tags
cJSON_AddItemToObject(event, "tags", tags);
// Create EVENT message
cJSON* event_msg = cJSON_CreateArray();
cJSON_AddItemToArray(event_msg, cJSON_CreateString("EVENT"));
cJSON_AddItemToArray(event_msg, cJSON_CreateString(sub_id));
cJSON_AddItemToArray(event_msg, event);
char* msg_str = cJSON_Print(event_msg);
queue_message(wsi, pss, msg_str, msg_len, LWS_WRITE_TEXT);
}
```
**New:**
```c
// Update SQL to select event_json
const char* sql = "SELECT event_json FROM events WHERE ...";
while (sqlite3_step(stmt) == SQLITE_ROW) {
const char* event_json = (char*)sqlite3_column_text(stmt, 0);
// Build EVENT message with pre-serialized event
// Format: ["EVENT","sub_id",{...event_json...}]
size_t msg_len = 12 + strlen(sub_id) + strlen(event_json); // ["EVENT","",""]
char* msg_str = malloc(msg_len + 1);
snprintf(msg_str, msg_len + 1, "[\"EVENT\",\"%s\",%s]", sub_id, event_json);
queue_message(wsi, pss, msg_str, strlen(msg_str), LWS_WRITE_TEXT);
free(msg_str);
}
```
**Speedup:** 366 × (cJSON operations) eliminated!
---
## Database Migration System Design
### For Future Schema Changes
**File:** `src/migrations.c` (new file)
```c
typedef struct {
int from_version;
int to_version;
const char* description;
int (*migrate_func)(sqlite3* db);
} migration_t;
// Migration from v10 to v11: Add event_json column
int migrate_v10_to_v11(sqlite3* db) {
// Step 1: Add column
const char* add_column_sql =
"ALTER TABLE events ADD COLUMN event_json TEXT";
if (sqlite3_exec(db, add_column_sql, NULL, NULL, NULL) != SQLITE_OK) {
return -1;
}
// Step 2: Populate event_json for existing events
const char* select_sql =
"SELECT id, pubkey, created_at, kind, content, sig, tags FROM events";
sqlite3_stmt* stmt;
if (sqlite3_prepare_v2(db, select_sql, -1, &stmt, NULL) != SQLITE_OK) {
return -1;
}
while (sqlite3_step(stmt) == SQLITE_ROW) {
// Reconstruct JSON
cJSON* event = cJSON_CreateObject();
cJSON_AddStringToObject(event, "id", (char*)sqlite3_column_text(stmt, 0));
// ... add other fields ...
char* event_json = cJSON_PrintUnformatted(event);
// Update row
const char* update_sql = "UPDATE events SET event_json = ? WHERE id = ?";
sqlite3_stmt* update_stmt;
sqlite3_prepare_v2(db, update_sql, -1, &update_stmt, NULL);
sqlite3_bind_text(update_stmt, 1, event_json, -1, SQLITE_TRANSIENT);
sqlite3_bind_text(update_stmt, 2, (char*)sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);
sqlite3_step(update_stmt);
sqlite3_finalize(update_stmt);
free(event_json);
cJSON_Delete(event);
}
sqlite3_finalize(stmt);
// Step 3: Make column NOT NULL
// (SQLite doesn't support ALTER COLUMN, so we'd need to recreate table)
return 0;
}
// Migration registry
static migration_t migrations[] = {
{10, 11, "Add event_json column for fast retrieval", migrate_v10_to_v11},
// Future migrations go here
};
int run_migrations(sqlite3* db, int current_version, int target_version) {
for (int i = 0; i < sizeof(migrations) / sizeof(migration_t); i++) {
if (migrations[i].from_version >= current_version &&
migrations[i].to_version <= target_version) {
printf("Running migration: %s\n", migrations[i].description);
if (migrations[i].migrate_func(db) != 0) {
fprintf(stderr, "Migration failed: %s\n", migrations[i].description);
return -1;
}
// Update schema version
char update_version_sql[256];
snprintf(update_version_sql, sizeof(update_version_sql),
"PRAGMA user_version = %d", migrations[i].to_version);
sqlite3_exec(db, update_version_sql, NULL, NULL, NULL);
}
}
return 0;
}
```
---
## Recommendation: Hybrid Approach
### For This Change (v10 → v11): Fresh Start
**Rationale:**
1. Your relay is still in development
2. Migration would be slow (reconstruct JSON for all events)
3. You've been doing fresh starts anyway
4. Clean slate for performance testing
**Steps:**
1. Update schema to v11 with event_json column
2. Update store_event() to populate event_json
3. Update handle_req_message() to use event_json
4. Deploy with fresh database
5. Test performance improvement
### For Future Changes: Use Migration System
**Rationale:**
1. Once relay is in production, data preservation matters
2. Migration system is reusable
3. Professional approach for production relay
**Steps:**
1. Create `src/migrations.c` and `src/migrations.h`
2. Implement migration framework
3. Add migration functions for each schema change
4. Test migrations thoroughly before deployment
---
## Migration System Features
### Core Features
1. **Version Detection**
- Read current schema version from database
- Compare with embedded schema version
- Determine which migrations to run
2. **Migration Chain**
- Run migrations in sequence (v8 → v9 → v10 → v11)
- Skip already-applied migrations
- Stop on first failure
3. **Backup Before Migration**
- Automatic database backup before migration
- Rollback capability if migration fails
- Backup retention policy
4. **Progress Reporting**
- Log migration progress
- Show estimated time remaining
- Report success/failure
### Safety Features
1. **Transaction Wrapping**
```c
sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, NULL);
int result = migrate_v10_to_v11(db);
if (result == 0) {
sqlite3_exec(db, "COMMIT", NULL, NULL, NULL);
} else {
sqlite3_exec(db, "ROLLBACK", NULL, NULL, NULL);
}
```
2. **Validation After Migration**
- Verify row counts match
- Check data integrity
- Validate indexes created
3. **Dry-Run Mode**
- Test migration without committing
- Report what would be changed
- Estimate migration time
---
## Implementation Timeline
### Immediate (Today): Fresh Start with event_json
**Changes:**
1. Update schema to v11 (add event_json column)
2. Update store_event() to populate event_json
3. Update handle_req_message() to use event_json
4. Deploy with fresh database
**Effort:** 4 hours
**Impact:** 2,500x faster event retrieval
### This Week: Build Migration Framework
**Changes:**
1. Create src/migrations.c and src/migrations.h
2. Implement migration runner
3. Add backup/rollback capability
4. Add progress reporting
**Effort:** 1-2 days
**Impact:** Reusable for all future schema changes
### Future: Add Migrations as Needed
**For each schema change:**
1. Write migration function
2. Add to migrations array
3. Test thoroughly
4. Deploy with automatic migration
---
## Code Structure
### File Organization
```
src/
├── migrations.c # NEW: Migration system
├── migrations.h # NEW: Migration API
├── sql_schema.h # Schema definition (v11)
├── main.c # Updated store_event() and handle_req_message()
└── ...
```
### Migration API
```c
// migrations.h
int init_migration_system(sqlite3* db);
int run_pending_migrations(sqlite3* db);
int backup_database(const char* db_path, char* backup_path, size_t backup_path_size);
int rollback_migration(sqlite3* db, const char* backup_path);
```
---
## Testing Strategy
### For Fresh Start (v11)
1. **Local testing:**
- Build with new schema
- Post test events
- Query events and measure performance
- Verify event_json is populated correctly
2. **Performance testing:**
- Query 366 events
- Measure time (should be <10ms instead of 18s)
- Check CPU usage (should be <20%)
3. **Production deployment:**
- Stop relay
- Delete old database
- Start relay with v11 schema
- Monitor performance
### For Migration System (Future)
1. **Unit tests:**
- Test each migration function
- Test rollback capability
- Test error handling
2. **Integration tests:**
- Create database with old schema
- Run migration
- Verify data integrity
- Test rollback
3. **Performance tests:**
- Measure migration time for large databases
- Test with 10K, 100K, 1M events
- Optimize slow migrations
---
## Migration Complexity Analysis
### For v10 → v11 Migration
**If we were to migrate existing data:**
```sql
-- Step 1: Add column (fast)
ALTER TABLE events ADD COLUMN event_json TEXT;
-- Step 2: Populate event_json (SLOW!)
-- For each of N events:
-- 1. SELECT 7 columns
-- 2. Reconstruct JSON (cJSON operations)
-- 3. Serialize to string (cJSON_Print)
-- 4. UPDATE event_json column
-- 5. Free memory
-- Estimated time:
-- - 1000 events: ~10 seconds
-- - 10000 events: ~100 seconds
-- - 100000 events: ~1000 seconds (16 minutes)
```
**Conclusion:** Migration is expensive for this change. Fresh start is better.
---
## Future Migration Examples
### Easy Migrations (Fast)
**Adding an index:**
```c
int migrate_add_index(sqlite3* db) {
return sqlite3_exec(db,
"CREATE INDEX idx_new ON events(new_column)",
NULL, NULL, NULL);
}
```
**Adding a column with default:**
```c
int migrate_add_column(sqlite3* db) {
return sqlite3_exec(db,
"ALTER TABLE events ADD COLUMN new_col TEXT DEFAULT ''",
NULL, NULL, NULL);
}
```
### Hard Migrations (Slow)
**Changing column type:**
- Requires table recreation
- Copy all data
- Recreate indexes
- Can take minutes for large databases
**Populating computed columns:**
- Requires row-by-row processing
- Can take minutes for large databases
---
## Recommendation Summary
### For This Change (event_json)
**Do:** Fresh start with v11 schema
- Fast deployment
- Clean implementation
- Immediate performance benefit
- No migration complexity
**Don't:** Migrate existing data
- Too slow (reconstruct JSON for all events)
- Too complex (first migration)
- Not worth it (relay still in development)
### For Future Changes
**Do:** Implement migration system
- Professional approach
- Data preservation
- Reusable framework
- Required for production relay
**Timeline:**
- **Today:** Deploy v11 with fresh start
- **This week:** Build migration framework
- **Future:** Use migrations for all schema changes
---
## Next Steps
1. ✅ Update schema to v11 (add event_json column)
2. ✅ Update store_event() to populate event_json
3. ✅ Update handle_req_message() to use event_json
4. ✅ Test locally with 366-event query
5. ✅ Deploy to production with fresh database
6. ✅ Measure performance improvement
7. ⏳ Build migration system for future use
**Expected result:** 366-event retrieval time drops from 18s to <10ms (2,500x speedup)
+312
View File
@@ -0,0 +1,312 @@
# Event Tags Denormalization Plan
## Problem Statement
The relay is at 99% CPU with 1,178 WebSocket connections and ~120 new REQ subscriptions per minute. The root cause is that every tag-filtered REQ query (e.g., `#g`, `#e`, `#p`) uses `json_each(json(tags))` with `json_extract()` — a correlated subquery that parses the JSON tags column for every candidate row. With geohash-based subscriptions from a location app generating constant connection churn, this creates unsustainable CPU load.
### Evidence
- `EXPLAIN QUERY PLAN` shows: `CORRELATED SCALAR SUBQUERY → SCAN json_each VIRTUAL TABLE`
- 8,077 subscription creates/hour but only 442 active at any time (connection churn)
- Only 56 events stored/hour — the load is entirely from **read queries**, not writes
- 14,819 events match the kind filter; each query parses JSON tags on candidate rows
## Solution: Denormalized `event_tags` Table
Replace `json_each()` queries with indexed lookups on a separate `event_tags` table. This is the standard approach used by strfry, nostream, and other production Nostr relays.
### Performance Impact
| Metric | Before | After |
|--------|--------|-------|
| Tag query type | `json_each()` correlated subquery | Indexed JOIN/subquery |
| Complexity per query | O(rows × tags_per_event) JSON parsing | O(log n) B-tree lookup |
| Expected CPU reduction | 99% | <10% for same traffic |
## Files to Modify
| File | Changes |
|------|---------|
| `src/sql_schema.h` | Add `event_tags` table, indexes, cascade triggers; bump to schema v12 |
| `src/main.c` | Add `store_event_tags()`, update `handle_req_message()` tag filter SQL, move config reads out of row loop, add startup tag population |
| `src/nip009.c` | Add `DELETE FROM event_tags` alongside event deletions |
| `src/dm_admin.c` | Downgrade NIP-17 decryption failure log level |
| `src/websockets.c` | Downgrade NIP-17 error log to DEBUG at lines 780 and 1488 |
## Detailed Changes
### 1. Schema: `src/sql_schema.h`
Add the `event_tags` table after the events table definition. Bump schema version to 12.
```sql
-- Denormalized event tags for fast indexed lookups
-- Replaces json_each(json(tags)) queries which cause full JSON parsing per row
CREATE TABLE event_tags (
event_id TEXT NOT NULL,
tag_name TEXT NOT NULL,
tag_value TEXT NOT NULL,
tag_index INTEGER NOT NULL DEFAULT 0,
FOREIGN KEY (event_id) REFERENCES events(id) ON DELETE CASCADE
);
-- Primary lookup index: find events by tag name + value
CREATE INDEX idx_event_tags_lookup ON event_tags(tag_name, tag_value);
-- Reverse lookup: find all tags for an event (for cleanup)
CREATE INDEX idx_event_tags_event ON event_tags(event_id);
-- Composite index for common query pattern: tag + kind (via join)
CREATE INDEX idx_event_tags_value_name ON event_tags(tag_value, tag_name);
```
**Key design decisions:**
- `ON DELETE CASCADE` handles cleanup when events are deleted (NIP-09, replaceable events)
- `tag_index` stores the position within the tags array (useful for ordered tag access)
- Only the first two elements of each tag are indexed (`tag_name` = `$[0]`, `tag_value` = `$[1]`) — this covers all standard Nostr tag filters (`#e`, `#p`, `#t`, `#g`, `#d`, etc.)
- `PRAGMA foreign_keys = ON` is already in the schema, so CASCADE will work
### 2. Store Tags: `src/main.c` — New `store_event_tags()` Function
Add a new function called after successful event INSERT in `store_event()`:
```c
// Insert denormalized tags into event_tags table for fast indexed lookups
int store_event_tags(const char* event_id, cJSON* tags) {
if (!g_db || !event_id || !tags || !cJSON_IsArray(tags)) {
return 0; // Not an error if no tags
}
const char* sql = "INSERT INTO event_tags (event_id, tag_name, tag_value, tag_index) VALUES (?, ?, ?, ?)";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc != SQLITE_OK) {
DEBUG_ERROR("Failed to prepare event_tags insert: %s", sqlite3_errmsg(g_db));
return -1;
}
int tag_index = 0;
cJSON* tag = NULL;
cJSON_ArrayForEach(tag, tags) {
if (cJSON_IsArray(tag) && cJSON_GetArraySize(tag) >= 2) {
cJSON* name = cJSON_GetArrayItem(tag, 0);
cJSON* value = cJSON_GetArrayItem(tag, 1);
if (cJSON_IsString(name) && cJSON_IsString(value)) {
sqlite3_reset(stmt);
sqlite3_bind_text(stmt, 1, event_id, -1, SQLITE_STATIC);
sqlite3_bind_text(stmt, 2, cJSON_GetStringValue(name), -1, SQLITE_STATIC);
sqlite3_bind_text(stmt, 3, cJSON_GetStringValue(value), -1, SQLITE_STATIC);
sqlite3_bind_int(stmt, 4, tag_index);
rc = sqlite3_step(stmt);
if (rc != SQLITE_DONE) {
DEBUG_ERROR("Failed to insert event tag: %s", sqlite3_errmsg(g_db));
}
}
}
tag_index++;
}
sqlite3_finalize(stmt);
return 0;
}
```
**Call site** — in `store_event()` at line ~784, after `monitoring_on_event_stored()`:
```c
// After successful event storage, insert denormalized tags
store_event_tags(cJSON_GetStringValue(id), tags);
```
### 3. Update REQ Query Builder: `src/main.c``handle_req_message()`
Replace the `json_each()` tag filter at lines 1243-1270 with an `event_tags` subquery:
**Before** (current code at line 1244):
```c
snprintf(sql_ptr, remaining,
" AND EXISTS (SELECT 1 FROM json_each(json(tags)) "
"WHERE json_extract(value, '$[0]') = ? "
"AND json_extract(value, '$[1]') IN (");
```
**After** (new code):
```c
snprintf(sql_ptr, remaining,
" AND id IN (SELECT event_id FROM event_tags "
"WHERE tag_name = ? AND tag_value IN (");
```
The rest of the parameter binding code (lines 1248-1270) stays the same — the bind params are identical (`tag_name` then `tag_value(s)`). Only the SQL template changes.
The closing parenthesis changes from `"))` to `"))` — same syntax, just different semantics.
### 4. NIP-09 Delete Cascade: `src/nip009.c`
The `ON DELETE CASCADE` foreign key handles this automatically. When `DELETE FROM events WHERE id = ?` executes, SQLite will automatically delete matching rows from `event_tags`. **No code changes needed in nip009.c** as long as `PRAGMA foreign_keys = ON` is set (it already is in the schema).
However, verify the replaceable/addressable event triggers in the schema also cascade properly. The existing triggers at schema lines 97-119 use `DELETE FROM events WHERE ...` which will trigger the CASCADE.
### 5. Performance Fix: Move Config Reads Out of Row Loop
In `handle_req_message()` at lines 1400-1401, `get_config_bool()` is called **inside the `while (sqlite3_step())` loop** — meaning it executes a SQLite query for every row returned. Move these before the loop:
**Before** (inside loop):
```c
while (sqlite3_step(stmt) == SQLITE_ROW) {
// ...
int expiration_enabled = get_config_bool("expiration_enabled", 1); // SQLite query per row!
int filter_responses = get_config_bool("expiration_filter", 1); // SQLite query per row!
```
**After** (before loop):
```c
int expiration_enabled = get_config_bool("expiration_enabled", 1);
int filter_responses = get_config_bool("expiration_filter", 1);
while (sqlite3_step(stmt) == SQLITE_ROW) {
// ... use cached values
```
### 6. Log Level Fixes
#### NIP-17 Decryption Failure — `src/websockets.c` lines 780 and 1488
Change from `DEBUG_ERROR` to `DEBUG_INFO`:
```c
// Before:
DEBUG_ERROR("NIP-17 admin message processing failed");
// After:
DEBUG_INFO("NIP-17 admin message processing failed");
```
This is expected behavior when non-admin gift wraps arrive — not an error condition.
#### Duplicate Event INSERT — `src/main.c` line 741
The `DEBUG_ERROR` at line 741 fires before the CONSTRAINT check at line 746. Suppress it for constraint violations:
```c
// Before (line 738-741):
if (rc != SQLITE_DONE) {
const char* err_msg = sqlite3_errmsg(g_db);
int extended_errcode = sqlite3_extended_errcode(g_db);
DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
}
// After:
if (rc != SQLITE_DONE) {
const char* err_msg = sqlite3_errmsg(g_db);
int extended_errcode = sqlite3_extended_errcode(g_db);
if (rc != SQLITE_CONSTRAINT) {
DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
}
}
```
### 7. Startup Tag Population
Since you said no migration code is needed (fresh deploy), the `event_tags` table will start empty and populate as new events arrive. Existing events won't have tags in the lookup table.
However, to avoid a period where old events don't appear in tag-filtered queries, add a one-time population function that runs at startup:
```c
// Populate event_tags from existing events (run once at startup)
int populate_event_tags_from_existing(void) {
if (!g_db) return -1;
// Check if event_tags is already populated
sqlite3_stmt* check_stmt;
sqlite3_prepare_v2(g_db, "SELECT COUNT(*) FROM event_tags", -1, &check_stmt, NULL);
if (sqlite3_step(check_stmt) == SQLITE_ROW && sqlite3_column_int(check_stmt, 0) > 0) {
sqlite3_finalize(check_stmt);
DEBUG_INFO("event_tags already populated, skipping");
return 0;
}
sqlite3_finalize(check_stmt);
DEBUG_INFO("Populating event_tags from existing events...");
const char* sql = "SELECT id, tags FROM events WHERE tags != '[]'";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc != SQLITE_OK) return -1;
// Use a transaction for bulk insert performance
sqlite3_exec(g_db, "BEGIN TRANSACTION", NULL, NULL, NULL);
int event_count = 0;
while (sqlite3_step(stmt) == SQLITE_ROW) {
const char* event_id = (const char*)sqlite3_column_text(stmt, 0);
const char* tags_json = (const char*)sqlite3_column_text(stmt, 1);
if (event_id && tags_json) {
cJSON* tags = cJSON_Parse(tags_json);
if (tags) {
store_event_tags(event_id, tags);
cJSON_Delete(tags);
event_count++;
}
}
}
sqlite3_finalize(stmt);
sqlite3_exec(g_db, "COMMIT", NULL, NULL, NULL);
DEBUG_INFO("Populated event_tags for %d events", event_count);
return 0;
}
```
Call this during startup, after database initialization but before accepting connections.
## Architecture Diagram
```mermaid
flowchart TD
subgraph "Current: O(n) per query"
A1["REQ with #g filter"] --> B1["SQL: SELECT ... WHERE EXISTS"]
B1 --> C1["json_each(json(tags))"]
C1 --> D1["json_extract per row"]
D1 --> E1["🔴 Full JSON parse per candidate row"]
end
subgraph "New: O(log n) per query"
A2["REQ with #g filter"] --> B2["SQL: SELECT ... WHERE id IN"]
B2 --> C2["event_tags table"]
C2 --> D2["idx_event_tags_lookup"]
D2 --> E2["🟢 B-tree index lookup"]
end
subgraph "Write Path (unchanged speed)"
F["EVENT arrives"] --> G["store_event()"]
G --> H["INSERT INTO events"]
H --> I["store_event_tags()"]
I --> J["INSERT INTO event_tags\n(one row per tag)"]
end
subgraph "Delete Path (automatic)"
K["NIP-09 DELETE"] --> L["DELETE FROM events"]
L --> M["ON DELETE CASCADE"]
M --> N["event_tags rows auto-deleted"]
end
```
## Testing Strategy
1. Run existing test suite: `tests/run_all_tests.sh`
2. Specifically run NIP tests that use tag filters: `tests/run_nip_tests.sh`
3. Verify tag-filtered REQ queries return correct results
4. Verify NIP-09 deletion cascades to event_tags
5. Verify replaceable event triggers cascade to event_tags
6. Monitor CPU usage after deployment with same traffic pattern
## Risk Assessment
- **Low risk**: The `event_tags` table is additive — it doesn't change the events table structure
- **Low risk**: `ON DELETE CASCADE` is a well-tested SQLite feature
- **Low risk**: The SQL change in `handle_req_message()` is a drop-in replacement (same bind params)
- **Medium risk**: Startup population on a large database could take a few seconds — but with only 55K events, this should complete in under 1 second
- **Write overhead**: Each event INSERT now also inserts ~5 rows into event_tags — negligible at 56 events/hour
+585
View File
@@ -0,0 +1,585 @@
# Plan: Ban Idle and Early-Disconnect Connections
## Problem Statement
The relay needs to defend against spam connections that:
1. Connect via WebSocket and sit idle doing nothing (the original problem)
2. Connect and immediately disconnect without subscribing or posting (the new requirement)
Both patterns indicate bot/scanner behavior that should result in IP bans.
## Goal
Implement a system that:
- Bans IPs that connect but never send a REQ or EVENT (regardless of how the connection ends)
- Works **independent of NIP-42 authentication** (no auth required)
- Allows legitimate users to connect and use the relay normally
- Uses existing IP ban infrastructure
## Architecture Overview
```mermaid
flowchart TD
A[Client connects] --> B{IP banned?}
B -->|Yes| C[Reject immediately]
B -->|No| D[Set idle timeout timer]
D --> E{Client sends REQ or EVENT?}
E -->|Yes| F[Mark session ACTIVE<br>Cancel idle timer]
E -->|No| G{Connection closes?}
G --> H{Session was ACTIVE?}
H -->|Yes| I[Clean close - no ban]
H -->|No| J["ip_ban_record_failure()<br>→ may trigger ban"]
F --> K[Normal operation]
K --> L[Connection closes]
L --> I
```
## Key Insight
The distinction between "idle timeout" and "early disconnect" is minimal:
- **Idle timeout**: Connection closed by server because client never became ACTIVE
- **Early disconnect**: Connection closed by client because client never became ACTIVE
Both result in the same check: `if (!session_was_active) ban_ip()`
## Implementation Plan
### 1. Add Session Activity Tracking
**File: `src/websockets.h`**
Add to `struct per_session_data`:
```c
// Session activity tracking for idle connection banning
int session_active; // 1 if client sent REQ or EVENT, 0 otherwise
int idle_timeout_sec; // Timeout value for this session (copied from config)
```
### 2. Set Idle Timeout on All Connections
**File: `src/websockets.c` - `LWS_CALLBACK_ESTABLISHED`**
Currently (lines 561-572):
```c
// Only set timeout if auth is required
if (pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) {
int auth_timeout = get_config_int("nip42_auth_timeout_sec", 10);
if (auth_timeout > 0) {
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, auth_timeout);
}
}
```
Change to:
```c
// Initialize session activity tracking
pss->session_active = 0;
pss->idle_timeout_sec = get_config_int("idle_connection_timeout_sec", 30);
// Set idle timeout for ALL connections (not just auth-required)
// This catches bots that connect and do nothing
if (pss->idle_timeout_sec > 0) {
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, pss->idle_timeout_sec);
DEBUG_TRACE("Idle timeout set: %d seconds for connection from %s",
pss->idle_timeout_sec, pss->client_ip);
}
// Also set auth timeout if auth is required (separate concern)
if (pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) {
int auth_timeout = get_config_int("nip42_auth_timeout_sec", 10);
// Use the shorter of the two timeouts
int effective_timeout = (pss->idle_timeout_sec > 0 && pss->idle_timeout_sec < auth_timeout)
? pss->idle_timeout_sec
: auth_timeout;
if (effective_timeout > 0) {
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, effective_timeout);
}
}
```
### 3. Mark Session as Active on Valid Activity
**File: `src/websockets.c` - `LWS_CALLBACK_RECEIVE`**
When processing REQ message (around line 1030):
```c
// Before handling REQ, mark session as active
pthread_mutex_lock(&pss->session_lock);
pss->session_active = 1;
pthread_mutex_unlock(&pss->session_lock);
// Cancel idle timeout - this is legitimate usage
lws_set_timeout(wsi, NO_PENDING_TIMEOUT, 0);
```
When processing EVENT message (around line 1380):
```c
// Before handling EVENT, mark session as active
pthread_mutex_lock(&pss->session_lock);
pss->session_active = 1;
pthread_mutex_unlock(&pss->session_lock);
// Cancel idle timeout - this is legitimate usage
lws_set_timeout(wsi, NO_PENDING_TIMEOUT, 0);
```
**Note**: We should also consider AUTH as "activity" since the client is engaging with the protocol:
```c
// In handle_nip42_auth_signed_event (nip042.c)
// After successful auth, also mark session active
pthread_mutex_lock(&pss->session_lock);
pss->session_active = 1;
pthread_mutex_unlock(&pss->session_lock);
```
### 3.5. Separate Rate Limiting for Idle Connections
Per user feedback, idle/early-disconnect failures have **separate rate limiting** from auth failures. This requires:
**New Config Keys:**
| Config Key | Type | Default | Range | Description |
|------------|------|---------|-------|-------------|
| `idle_ban_threshold` | int | 3 | 1-100 | Idle failures before ban |
| `idle_ban_window_sec` | int | 60 | 10-3600 | Window for counting idle failures |
| `idle_ban_duration_sec` | int | 300 | 60-86400 | Initial idle ban duration |
**Add to `ip_ban_entry_t` in `src/ip_ban.c`:**
```c
typedef struct {
int state;
char ip[46];
// Auth failure tracking (existing)
int failure_count;
time_t first_failure;
time_t banned_until;
int ban_count;
// NEW: Idle failure tracking (separate)
int idle_failure_count;
time_t idle_first_failure;
time_t idle_banned_until;
int idle_ban_count;
// Other existing fields...
int has_authed_successfully;
time_t last_success_at;
int total_connections;
int total_failures;
int total_successes;
time_t first_seen;
} ip_ban_entry_t;
```
**New function in `src/ip_ban.c`:**
```c
// Record an idle/early-disconnect failure for an IP
void ip_ban_record_idle_failure(const char* ip) {
if (!ip || !g_initialized) return;
if (!get_config_bool("idle_ban_enabled", 1)) return;
int threshold = get_config_int("idle_ban_threshold", 3);
int window_sec = get_config_int("idle_ban_window_sec", 60);
int ban_duration = get_config_int("idle_ban_duration_sec", 300);
pthread_mutex_lock(&g_ban_mutex);
ip_ban_entry_t* entry = get_or_create_entry(ip);
if (!entry) {
pthread_mutex_unlock(&g_ban_mutex);
return;
}
time_t now = time(NULL);
// Reset window if expired
if (entry->idle_first_failure > 0 && (now - entry->idle_first_failure) > window_sec) {
entry->idle_failure_count = 0;
entry->idle_first_failure = now;
}
if (entry->idle_first_failure == 0) {
entry->idle_first_failure = now;
}
entry->idle_failure_count++;
entry->total_failures++;
DEBUG_TRACE("IP %s idle failure count: %d/%d", ip, entry->idle_failure_count, threshold);
if (entry->idle_failure_count >= threshold) {
int duration = ban_duration;
for (int i = 0; i < entry->idle_ban_count && duration < 86400; i++) {
duration *= 2;
}
if (duration > 86400) duration = 86400;
entry->idle_banned_until = now + duration;
entry->idle_ban_count++;
entry->idle_failure_count = 0;
entry->idle_first_failure = 0;
DEBUG_WARN("IP %s banned for %d seconds (idle ban #%d) after %d idle failures",
ip, duration, entry->idle_ban_count, threshold);
}
pthread_mutex_unlock(&g_ban_mutex);
}
```
**Update `ip_ban_is_banned()` to check BOTH ban types:**
```c
int ip_ban_is_banned(const char* ip) {
if (!ip || !g_initialized) return 0;
pthread_mutex_lock(&g_ban_mutex);
int idx = find_slot(ip);
if (idx < 0 || g_ban_table[idx].state == IP_BAN_EMPTY) {
pthread_mutex_unlock(&g_ban_mutex);
return 0;
}
ip_ban_entry_t* entry = &g_ban_table[idx];
time_t now = time(NULL);
int banned = 0;
// Check auth ban (if enabled)
if (get_config_bool("auth_fail_ban_enabled", 1) &&
entry->banned_until > 0 && now < entry->banned_until) {
banned = 1;
}
// Check idle ban (if enabled)
if (get_config_bool("idle_ban_enabled", 1) &&
entry->idle_banned_until > 0 && now < entry->idle_banned_until) {
banned = 1;
}
// Clear expired bans
if (!banned) {
if (entry->banned_until > 0 && now >= entry->banned_until) {
entry->banned_until = 0;
entry->failure_count = 0;
entry->first_failure = 0;
}
if (entry->idle_banned_until > 0 && now >= entry->idle_banned_until) {
entry->idle_banned_until = 0;
entry->idle_failure_count = 0;
entry->idle_first_failure = 0;
}
}
pthread_mutex_unlock(&g_ban_mutex);
return banned;
}
```
**Update persistence:**
- `ip_ban_load_from_db()`: Load idle_* fields from new columns
- `ip_ban_save_to_db()`: Save idle_* fields to new columns
- Add migration SQL to create new columns if they don't exist
**Add to `src/ip_ban.h`:**
```c
// Record an idle/early-disconnect failure for an IP
void ip_ban_record_idle_failure(const char* ip);
```
### 4. Record Failure on Inactive Connection Close
**File: `src/websockets.c` - `LWS_CALLBACK_CLOSED`**
Currently (lines 2121-2128):
```c
// Record auth failure if connection closed while unauthenticated and auth was required
if (!pss->authenticated &&
(pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) &&
pss->auth_challenge_sent &&
strlen(pss->client_ip) > 0) {
ip_ban_record_failure(pss->client_ip);
}
```
Change to:
```c
// Record failure if connection closed without ever becoming active
// This catches:
// 1. Idle connections that timed out (never sent REQ/EVENT/AUTH)
// 2. Early disconnects (client closed before sending REQ/EVENT/AUTH)
if (!pss->session_active && strlen(pss->client_ip) > 0) {
// Use separate idle failure recording (has its own threshold/duration)
ip_ban_record_idle_failure(pss->client_ip);
DEBUG_LOG("Recording idle/early-disconnect failure for IP %s (connected %ld seconds)",
pss->client_ip,
time(NULL) - pss->connection_established);
}
// Legacy: record auth failure if auth was required but not completed
else if (!pss->authenticated &&
(pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) &&
pss->auth_challenge_sent &&
strlen(pss->client_ip) > 0) {
ip_ban_record_failure(pss->client_ip);
}
```
### 5. Add Configuration Keys
**File: `src/config.c` - Add validation for new keys**
In the config validation section, add:
```c
// Idle connection ban settings
if (strcmp(key, "idle_connection_timeout_sec") == 0) {
if (!is_valid_positive_integer(value)) {
snprintf(error_msg, error_size, "invalid idle_connection_timeout_sec '%s' (must be positive integer)", value);
return -1;
}
int timeout = atoi(value);
if (timeout < 5 || timeout > 300) {
snprintf(error_msg, error_size, "idle_connection_timeout_sec must be between 5 and 300 seconds");
return -1;
}
return 0;
}
if (strcmp(key, "idle_ban_enabled") == 0) {
if (!is_valid_boolean(value)) {
snprintf(error_msg, error_size, "invalid boolean value '%s' for idle_ban_enabled", value);
return -1;
}
return 0;
}
if (strcmp(key, "idle_ban_threshold") == 0) {
if (!is_valid_positive_integer(value)) {
snprintf(error_msg, error_size, "invalid idle_ban_threshold '%s' (must be positive integer)", value);
return -1;
}
int threshold = atoi(value);
if (threshold < 1 || threshold > 100) {
snprintf(error_msg, error_size, "idle_ban_threshold must be between 1 and 100");
return -1;
}
return 0;
}
if (strcmp(key, "idle_ban_window_sec") == 0) {
if (!is_valid_positive_integer(value)) {
snprintf(error_msg, error_size, "invalid idle_ban_window_sec '%s' (must be positive integer)", value);
return -1;
}
int window = atoi(value);
if (window < 10 || window > 3600) {
snprintf(error_msg, error_size, "idle_ban_window_sec must be between 10 and 3600");
return -1;
}
return 0;
}
if (strcmp(key, "idle_ban_duration_sec") == 0) {
if (!is_valid_positive_integer(value)) {
snprintf(error_msg, error_size, "invalid idle_ban_duration_sec '%s' (must be positive integer)", value);
return -1;
}
int duration = atoi(value);
if (duration < 60 || duration > 86400) {
snprintf(error_msg, error_size, "idle_ban_duration_sec must be between 60 and 86400");
return -1;
}
return 0;
}
```
Also update the config introspection/documentation functions to describe these keys.
### 6. Add API Documentation
**File: `src/api.c` - Add to config metadata**
```c
{"idle_connection_timeout_sec", "int", 5, 300},
{"idle_ban_enabled", "bool", 0, 1},
{"idle_ban_threshold", "int", 1, 100},
{"idle_ban_window_sec", "int", 10, 3600},
{"idle_ban_duration_sec", "int", 60, 86400},
```
Add descriptions in the config query handler:
```c
} else if (strcmp(key, "idle_connection_timeout_sec") == 0) {
description = "Seconds before an idle connection (no REQ/EVENT) is closed and IP flagged. 0 to disable.";
} else if (strcmp(key, "idle_ban_enabled") == 0) {
description = "Whether to ban IPs that repeatedly connect without sending REQ or EVENT.";
} else if (strcmp(key, "idle_ban_threshold") == 0) {
description = "Number of idle/early-disconnect failures before banning an IP.";
} else if (strcmp(key, "idle_ban_window_sec") == 0) {
description = "Time window in seconds for counting idle failures.";
} else if (strcmp(key, "idle_ban_duration_sec") == 0) {
description = "Initial ban duration in seconds for idle failures (doubles each time).";
}
```
### 7. Update ip_ban Persistence and Cleanup
**File: `src/ip_ban.c` - Update `ip_ban_load_from_db()`**
Add migration SQL to create new columns if they don't exist, then load them:
```sql
ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0;
```
**File: `src/ip_ban.c` - Update `ip_ban_save_to_db()`**
Add the idle_* fields to the INSERT/REPLACE statement.
**File: `src/ip_ban.c` - Update `ip_ban_cleanup()`**
Add cleanup logic for idle ban entries (same pattern as auth ban cleanup).
**File: `src/ip_ban.c` - Update `ip_ban_log_stats()`**
Add idle ban count to the periodic log summary:
```c
DEBUG_WARN("IP BAN SUMMARY: %d auth-banned, %d idle-banned, %d tracked, %d trusted",
auth_banned_count, idle_banned_count, tracked_count, trusted_count);
```
## Configuration Reference
### Idle Connection Settings
| Config Key | Type | Default | Range | Description |
|------------|------|---------|-------|-------------|
| `idle_connection_timeout_sec` | int | 30 | 5-300 | Seconds to wait for REQ/EVENT before closing connection. 0 = disable idle timeout. |
| `idle_ban_enabled` | bool | true | true/false | Whether to ban IPs with repeated idle/early-disconnect failures. |
| `idle_ban_threshold` | int | 3 | 1-100 | Idle failures before ban. |
| `idle_ban_window_sec` | int | 60 | 10-3600 | Window for counting idle failures. |
| `idle_ban_duration_sec` | int | 300 | 60-86400 | Initial idle ban duration. |
### Auth Failure Settings (Existing)
| Config Key | Type | Default | Range | Description |
|------------|------|---------|-------|-------------|
| `auth_fail_ban_enabled` | bool | true | true/false | Whether to ban IPs with failed auth. |
| `auth_fail_ban_threshold` | int | 3 | 1-100 | Auth failures before ban. |
| `auth_fail_window_sec` | int | 60 | 10-3600 | Window for counting auth failures. |
| `auth_fail_ban_duration_sec` | int | 300 | 60-86400 | Initial auth ban duration. |
## Behavior Matrix
| Scenario | idle_timeout_sec | idle_ban_enabled | Result |
|----------|------------------|------------------|--------|
| Connect → do nothing → timeout | >0 | true | Connection closed, IP failure recorded, may be banned |
| Connect → do nothing → timeout | >0 | false | Connection closed, no ban |
| Connect → immediate disconnect | any | true | IP failure recorded, may be banned |
| Connect → immediate disconnect | any | false | No ban |
| Connect → send REQ | any | any | Session active, no ban |
| Connect → send EVENT | any | any | Session active, no ban |
| Connect → send AUTH (NIP-42) | any | any | Session active, no ban |
## Testing Strategy
1. **Idle timeout test**: Connect via wscat, wait 30 seconds, verify disconnect and ban table entry
2. **Early disconnect test**: Connect via wscat, immediately Ctrl-C, verify ban table entry
3. **Active session test**: Connect, send REQ, disconnect immediately, verify NO ban
4. **Config disable test**: Set `idle_ban_enabled=false`, repeat test 1, verify no ban
5. **Timeout config test**: Set `idle_connection_timeout_sec=5`, verify faster disconnect
## Files to Modify
| File | Change Type |
|------|-------------|
| `src/websockets.h` | Add `session_active` and `idle_timeout_sec` fields to `per_session_data` |
| `src/websockets.c` ~565 | Set idle timeout on ALL connections in `LWS_CALLBACK_ESTABLISHED` |
| `src/websockets.c` ~1030 | Mark `session_active=1` and cancel idle timer on REQ |
| `src/websockets.c` ~1380 | Mark `session_active=1` and cancel idle timer on EVENT |
| `src/websockets.c` ~2121 | Call `ip_ban_record_idle_failure()` for inactive sessions in `LWS_CALLBACK_CLOSED` |
| `src/nip042.c` ~130 | Mark `session_active=1` on successful AUTH |
| `src/ip_ban.h` | Add `ip_ban_record_idle_failure()` declaration |
| `src/ip_ban.c` | Add idle_* fields to `ip_ban_entry_t`, new `ip_ban_record_idle_failure()` function, update `ip_ban_is_banned()` to check both ban types, update persistence and cleanup |
| `src/config.c` ~987+ | Add validation for 5 new idle_* config keys |
| `src/api.c` ~664 | Add config metadata entries for idle_* keys |
## Deployment on Existing Server
### Config Keys: No Manual SQL Required
All config keys use [`get_config_int()`](src/config.c:300) and [`get_config_bool()`](src/config.c:313) which return **hardcoded defaults** when a key doesn't exist in the config table. The new code will work immediately with these defaults:
- `idle_connection_timeout_sec` → defaults to 30
- `idle_ban_enabled` → defaults to true
- `idle_ban_threshold` → defaults to 3
- `idle_ban_window_sec` → defaults to 60
- `idle_ban_duration_sec` → defaults to 300
If you want to **override** any defaults, you can insert them into the config table. From the same directory as the `.db` file:
```bash
# Find your database file
DB_FILE=$(ls *.db | head -1)
# Optional: Insert custom values (only if you want non-default settings)
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_connection_timeout_sec', '30');"
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_enabled', 'true');"
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_threshold', '3');"
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_window_sec', '60');"
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_duration_sec', '300');"
```
Or change them via the admin API/web UI after deployment.
### ip_bans Table: Automatic Migration
The `ip_bans` table needs 4 new columns for idle tracking. The updated [`ip_ban_load_from_db()`](src/ip_ban.c:93) will run `ALTER TABLE` statements automatically on startup. These are safe because SQLite's `ALTER TABLE ADD COLUMN` is a no-op if the column already exists (we'll use error-tolerant execution).
If you prefer to run the migration manually before deploying:
```bash
DB_FILE=$(ls *.db | head -1)
sqlite3 "$DB_FILE" <<'SQL'
ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0;
ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0;
SQL
```
**Note:** SQLite will error on `ALTER TABLE ADD COLUMN` if the column already exists, but the code will handle this gracefully (ignore the error). Running it manually is optional — the relay will do it on startup.
### Deployment Steps
1. Build and deploy with `deploy_lt.sh` as usual
2. The relay starts, `ip_ban_load_from_db()` auto-migrates the `ip_bans` table
3. Config keys use defaults immediately — no manual SQL needed
4. Optionally tune settings via admin API or direct SQL
## Backward Compatibility
- Default `idle_connection_timeout_sec=30` provides immediate protection
- Default `idle_ban_enabled=true` maintains current spam protection behavior
- Setting `idle_connection_timeout_sec=0` restores old behavior (no idle timeout)
- Setting `idle_ban_enabled=false` allows connections without banning (for debugging)
- Existing auth-based banning continues to work independently with its own thresholds
- Database migration adds new columns with defaults — existing ban data preserved
## Summary
This plan implements a unified "session activity" tracking system with **separate rate limiting** for idle vs auth failures:
1. **Catches idle connections** via `lws_set_timeout()` on ALL connections (not just auth-required)
2. **Catches early disconnects** via the same `!session_active` check on close
3. **Respects legitimate users** who actually use the relay (REQ/EVENT/AUTH marks session active)
4. **Separate idle ban tracking** — idle failures have their own threshold, window, and duration independent of auth failures
5. **Extends existing ban infrastructure** — adds idle_* fields to `ip_ban_entry_t`, unified `ip_ban_is_banned()` checks both ban types
6. **Fully configurable** — 5 new config keys for idle banning, all tunable via admin events
The key insight is that there's no meaningful difference between "idle timeout" and "early disconnect" — both indicate a client that connected but never actually used the relay. The same `!session_active` check handles both cases, and the separate rate limiting ensures idle bots don't interfere with auth failure tracking for legitimate clients who fail NIP-42.
+194
View File
@@ -0,0 +1,194 @@
# Main Thread CPU Offload Plan
## Problem Statement
The main `c_relay_pg` thread consumes ~56% CPU while the DB worker threads (`db-read-1..4`, `db-write`) sit near 0%. All protocol handling, JSON parsing, event validation, subscription matching, and message queueing happens synchronously inside `nostr_relay_callback()` on the main libwebsockets event-loop thread.
## Current Architecture
```mermaid
flowchart TD
LWS[lws_service - main thread] --> CB[nostr_relay_callback]
CB --> PARSE[JSON parse - cJSON_Parse]
CB --> VALIDATE[Signature verify - nostr_validate_unified_request]
CB --> STORE[store_event - builds payload]
STORE --> SYNC_WRITE[thread_pool_execute_store_event_sync]
SYNC_WRITE --> |blocks main thread| DB_WRITE[db-write thread]
DB_WRITE --> |signal| SYNC_WRITE
CB --> REQ[handle_req_message - builds SQL]
REQ --> SYNC_READ[thread_pool_execute_req_sync]
SYNC_READ --> |blocks main thread| DB_READ[db-read-N thread]
DB_READ --> |signal| SYNC_READ
CB --> BROADCAST[broadcast_event_to_subscriptions]
BROADCAST --> QUEUE[queue_message per subscriber]
CB --> WRITEABLE[LWS_CALLBACK_SERVER_WRITEABLE]
WRITEABLE --> DRAIN[process_message_queue - lws_write]
```
### Why the main thread is hot
| Work item | Where | Cost |
|-----------|-------|------|
| JSON parsing | `cJSON_Parse` in `LWS_CALLBACK_RECEIVE` | Medium - per message |
| Signature verification | `nostr_validate_unified_request` - ed25519 crypto | **High** - per EVENT |
| SQL query building | `handle_req_message` filter-to-SQL loop | Low-Medium |
| Sync DB wait | `thread_pool_execute_*_sync` - pthread_cond_wait | Blocks but yields CPU |
| Result iteration + expiration check | Row loop in `handle_req_message` with `cJSON_Parse` per row | Medium-High for large result sets |
| Subscription matching | `broadcast_event_to_subscriptions` - filter matching | Medium - scales with subscriber count |
| Message serialization + queueing | `snprintf` + `queue_message_take_ownership` per subscriber | Medium |
| Config lookups | `get_config_int/bool` called repeatedly in hot paths | Low but frequent |
### Key constraint: libwebsockets is single-threaded
libwebsockets requires that **all `lws_write`, `lws_callback_on_writable`, and `lws_close_reason` calls happen from the service thread** (the thread running `lws_service`). This means we cannot directly write to WebSocket connections from worker threads. However, we **can** do computation on worker threads and post results back to the main thread for I/O.
## Offload Strategy
### Phase 1: Async EVENT Processing (Highest Impact)
Convert EVENT handling from synchronous to async. Currently the main thread does: parse → validate → store → broadcast, all blocking. Instead:
1. **Main thread**: Parse JSON (fast), extract event ID for dedup check, then submit a job to a new "event processing" worker thread
2. **Worker thread**: Signature verification (expensive crypto), store_event (already goes to DB thread), prepare broadcast payload
3. **Main thread callback**: Receive result via `wake_loop_cb` + `lws_cancel_service`, send OK response and broadcast to subscribers
```mermaid
flowchart TD
LWS[lws_service - main thread] --> RECEIVE[LWS_CALLBACK_RECEIVE]
RECEIVE --> PARSE[JSON parse + dedup check]
PARSE --> SUBMIT[Submit to event-worker queue]
SUBMIT --> LWS
WORKER[event-worker thread] --> VERIFY[Signature verification]
VERIFY --> STORE_DB[store_event via DB pool]
STORE_DB --> PREP[Prepare broadcast payload]
PREP --> RESULT_Q[Push result to completion queue]
RESULT_Q --> WAKE[lws_cancel_service]
LWS2[lws_service wakes] --> POLL[Poll completion queue]
POLL --> OK[Send OK response via queue_message]
POLL --> BCAST[broadcast_event_to_subscriptions]
```
**What this offloads**: ed25519 signature verification (~the most expensive per-event operation), event classification, tag serialization, and the synchronous DB store wait.
**What stays on main thread**: JSON parse (needed to extract event ID for dedup), OK response writing, broadcast fan-out (requires lws access).
### Phase 2: Async REQ Query Execution (Medium Impact)
Convert REQ handling from sync to async:
1. **Main thread**: Parse filters, build SQL, create subscription, submit query job
2. **DB reader thread**: Execute query (already happens, but currently blocks main thread via `_sync`)
3. **Main thread callback**: Iterate results, queue EVENT messages, send EOSE
This is simpler than Phase 1 because the thread pool already supports async submission via `thread_pool_submit_read` with a `result_cb`. The `_sync` wrappers just add a condvar wait on top. We need to:
- Use `thread_pool_submit_read` directly instead of `thread_pool_execute_req_sync`
- Store pending REQ context (sub_id, wsi, pss) so the callback can complete the work
- In the result callback, push results to a completion queue and call `lws_cancel_service`
- On the main thread, drain the completion queue and send EVENT + EOSE messages
### Phase 3: Reduce Per-Row Overhead in REQ Results (Low-Medium Impact)
Currently each row from a REQ query gets `cJSON_Parse` just to check NIP-40 expiration. This is wasteful:
- Option A: Add an `expiration` column to the events table so expiration filtering can be done in SQL
- Option B: Store expiration timestamp in a fast-parse format (extract during INSERT, store as integer column)
- Option C: Use string search on the raw JSON for the expiration tag instead of full parse
### Phase 4: Config Value Caching (Low Impact, Easy Win)
`get_config_int` and `get_config_bool` are called on every message in hot paths. These do SQLite queries. Cache config values in memory with a TTL or invalidation signal, so the main loop only refreshes them periodically (already done for `debug_level` every 60s — extend to all hot-path config values).
## Implementation Priority
| Phase | Impact | Risk | Complexity |
|-------|--------|------|------------|
| Phase 1: Async EVENT | **High** - removes crypto from main thread | Medium - async state management | Medium-High |
| Phase 2: Async REQ | **Medium** - unblocks main thread during queries | Low - infrastructure exists | Medium |
| Phase 3: Expiration optimization | **Low-Medium** - reduces per-row parse cost | Low | Low |
| Phase 4: Config caching | **Low** - reduces DB round-trips | Very Low | Low |
## Detailed Design: Phase 1 (Async EVENT Processing)
### New Components
#### Completion Queue (`src/completion_queue.h/.c`)
A thread-safe FIFO queue for posting results from worker threads back to the main thread:
```c
typedef struct {
int type; // COMPLETION_TYPE_EVENT_RESULT, COMPLETION_TYPE_REQ_RESULT, etc.
void* data; // Type-specific result data
struct lws* wsi; // Target WebSocket connection
void* pss; // Per-session data
} completion_item_t;
int completion_queue_init(void);
int completion_queue_push(completion_item_t* item);
completion_item_t* completion_queue_pop(void); // Non-blocking
void completion_queue_shutdown(void);
```
#### Event Worker Thread
A dedicated pthread that processes EVENT validation/storage:
```c
typedef struct {
cJSON* event; // Parsed event JSON - ownership transferred
cJSON* full_message; // Full message JSON for context
struct lws* wsi;
void* pss;
char sub_id[64]; // For response routing
} event_work_item_t;
```
#### Main Loop Integration
Add a completion queue drain step to the main event loop. After `lws_service` returns (either from timeout or `lws_cancel_service` wake), check the completion queue:
```c
while (g_server_running && !g_shutdown_flag) {
int result = lws_service(ws_context, 1000);
// NEW: Drain completion queue
completion_item_t* item;
while ((item = completion_queue_pop()) != NULL) {
process_completion(item); // Send OK, broadcast, etc.
free(item);
}
// ... existing periodic checks ...
}
```
### Changes to Existing Code
1. **`nostr_relay_callback` EVENT path**: After JSON parse and dedup check, instead of calling `nostr_validate_unified_request` + `store_event` + `broadcast_event_to_subscriptions` synchronously, submit an `event_work_item_t` to the event worker queue and return 0 immediately.
2. **`store_event`**: No changes needed — it already uses `thread_pool_execute_store_event_sync` which will run on the DB writer thread. The event worker thread will call it.
3. **`broadcast_event_to_subscriptions`**: No changes needed — it will be called from the main thread when processing the completion item, which is correct since it calls `queue_message_take_ownership` (requires lws thread).
### Thread Safety Considerations
- The `cJSON* event` object must be fully owned by the worker thread during processing. The main thread must not access it after submission.
- The `wsi` and `pss` pointers could become invalid if the client disconnects while the event is being processed. The completion handler must validate that the connection is still alive before sending the OK response.
- A generation counter or epoch on `pss` can detect stale references.
## What Cannot Be Offloaded
- **`lws_write` / `queue_message`**: Must happen on the lws service thread
- **`lws_callback_on_writable`**: Must happen on the lws service thread
- **`lws_close_reason`**: Must happen on the lws service thread
- **Subscription list iteration for broadcast**: Accesses `lws_wsi_user` which is lws-internal
These are fundamental libwebsockets constraints. The pattern is always: do computation off-thread, post result to completion queue, wake main thread, do I/O on main thread.
## Expected Impact
With Phase 1 alone, the main thread would no longer perform:
- ed25519 signature verification (~100-500μs per event depending on CPU)
- Synchronous DB store wait (~50-200μs per event)
- Event classification, tag serialization, JSON serialization for storage
This should reduce main-thread CPU by roughly 30-50% for EVENT-heavy workloads, shifting that work to the event worker thread and DB threads.

Some files were not shown because too many files have changed in this diff Show More