Compare commits

...
24 Commits
Author SHA1 Message Date
Laan Tungir 179b160cf3 v2.1.17 - Verified api-worker LISTEN/NOTIFY monitoring on v2.1.16 relay (port 7777): subscriber-cadence, NOTIFY-reactive, and zero-overhead idle tests all pass 2026-07-25 10:17:30 -04:00
Laan Tungir aedd22436e v2.1.16 - Implemented api-worker timer + LISTEN/NOTIFY monitoring (Phases 1,2,5) and fixed thread-connection binding bug that stopped dashboard updates 2026-07-25 10:07:08 -04:00
Laan Tungir 8fdc362a8b v2.1.15 - Fix PG addressable d-tag derivation and replaceable/addressable transactional replace semantics 2026-05-01 19:01:06 -04:00
Laan Tungir 67cf3973c5 v2.1.14 - Fix PostgreSQL replaceable/addressable upsert semantics with NIP-01 conflict ordering 2026-05-01 14:38:57 -04:00
Laan Tungir c02ae2be5b v2.1.13 - Fix PostgreSQL startup detection for existing relays and make admin_pubkey parsing whitespace-tolerant 2026-04-03 16:36:56 -04:00
Laan Tungir fd3efdd01f v2.1.12 - Implement multi-admin auth cache and checks, add admin management commands, and add multi-admin integration test 2026-04-03 15:46:33 -04:00
Laan Tungir 232f93c16f v2.1.11 - Fix api-worker DB lifecycle and NIP test reliability: add dedicated worker DB connection, guard PG expiration cast in REQ path, correct NIP-45 baseline-aware expectation, and harden run_all_tests skip handling 2026-04-03 07:32:14 -04:00
Laan Tungir 9a9365dcfa v2.1.10 - Fix NIP-50 PG search semantics and NIP-17 test compatibility 2026-04-02 13:00:34 -04:00
Laan Tungir 5577b57149 v2.1.9 - Rebrand repository, docs, scripts, and tooling from c-relay to c-relay-pg 2026-04-02 07:06:41 -04:00
Laan Tungir 22c943d495 v2.1.8 - Implement explicit WAL checkpoint scheduling and disable SQLite auto-checkpoint 2026-04-01 21:03:24 -04:00
Laan Tungir 7e3d9b6825 v2.1.7 - Refactor SQLite connection ownership into db_ops, remove direct sqlite usage from runtime modules, and document agent-browser admin testing flow 2026-04-01 19:48:29 -04:00
Laan Tungir 547d22d09f v2.1.6 - Fix strict-mode admin API key access by prewarming config and relay private key caches 2026-04-01 19:08:42 -04:00
Laan Tungir ce9ae5a488 v2.1.5 - Offload duplicate and COUNT paths, add config/NIP-11 caching, and reduce DB reader threads 2026-04-01 16:15:46 -04:00
Laan Tungir 9b248b740f v2.1.4 - Harden async EVENT fallback handling and verify relay/nginx 503 behavior 2026-04-01 15:19:28 -04:00
Laan Tungir 0f77aeb72b v2.1.3 - Fix async EVENT thread safety by splitting store core/post-actions and naming lws main thread 2026-04-01 11:29:14 -04:00
Laan Tungir bb64651a0c v2.1.2 - Fix async REQ callback leak and validate Phase 2+3 subscription offload path 2026-04-01 11:01:38 -04:00
Laan Tungir 17be9c2b03 v2.1.1 - Wire REQ/COUNT/EVENT paths through thread pool sync helpers and stabilize filter limit tests 2026-04-01 09:38:25 -04:00
Laan Tungir f396c622b6 v2.1.0 - Bump to v2.1.0 as SQLite baseline for PostgreSQL fork handoff 2026-04-01 09:14:23 -04:00
Laan Tungir cbd260c1ae v2.0.4 - Complete Phase 1 DB abstraction: opaque db_stmt API, ip_ban migration, and SQLite leakage cleanup 2026-04-01 09:10:42 -04:00
Laan Tungir bc9ac290ab v2.0.3 - Complete Phase 1 SQLite abstraction cleanup across config/api/websockets and add db path accessor 2026-04-01 07:47:21 -04:00
Laan Tungir 192eeb248d v2.0.2 - Stabilize websocket queue draining, add db config count helper, and align NIP-45/50 test expectations 2026-04-01 07:04:58 -04:00
Laan Tungir da6c505420 v2.0.1 - Fix test-mode build/restart pipeline and NIP-11 root path handling; validate core NIP tests 2026-04-01 05:43:57 -04:00
Laan Tungir c077c209e5 v2.0.0 - Major architecture refactor: db_ops abstraction, low-risk SQLite refactor, and thread-pool scaffolding 2026-04-01 05:25:03 -04:00
Laan Tungir 9bf02702c2 v1.2.56 - Update codebase and documentation 2026-04-01 04:51:44 -04:00
144 changed files with 24209 additions and 13037 deletions
+2
View File
@@ -0,0 +1,2 @@
ADMIN_PUBKEY='6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3'
SERVER_PRIVKEY='1111111111111111111111111111111111111111111111111111111111111111'
+11 -11
View File
@@ -1,12 +1,12 @@
# AGENTS.md - AI Agent Integration Guide for Architect Mode
**Project-Specific Information for AI Agents Working with C-Relay in Architect Mode**
**Project-Specific Information for AI Agents Working with C-Relay-PG in Architect Mode**
## Critical Architecture Understanding
### System Architecture Overview
C-Relay implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
C-Relay-PG implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
```
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
@@ -51,9 +51,9 @@ C-Relay implements a **unique event-based configuration architecture** that fund
## Architectural Decision Analysis
### Configuration System Design
**Traditional Approach vs C-Relay:**
**Traditional Approach vs C-Relay-PG:**
```
Traditional: C-Relay:
Traditional: C-Relay-PG:
config.json → kind 33334 events
ENV variables → cryptographically signed tags
File watching → database polling/restart
@@ -119,18 +119,18 @@ File watching → database polling/restart
```
Developer Machine:
├── ./make_and_restart_relay.sh
├── build/c_relay_x86
├── build/c_relay_pg_x86
├── build/<relay_pubkey>.db
└── relay.log
```
### Production SystemD Deployment
```
/opt/c-relay/:
├── c_relay_x86
/opt/c-relay-pg/:
├── c_relay_pg_x86
├── <relay_pubkey>.db
├── systemd service (c-relay.service)
└── c-relay user isolation
├── systemd service (c-relay-pg.service)
└── c-relay-pg user isolation
```
### Container Deployment Architecture
@@ -144,7 +144,7 @@ Container:
### Reverse Proxy Architecture
```
Internet → Nginx/HAProxy → C-Relay
Internet → Nginx/HAProxy → C-Relay-PG
├── WebSocket upgrade handling
├── SSL termination
└── Rate limiting
@@ -292,7 +292,7 @@ Admin Signs Event → WebSocket Submit → Validate → Store → Restart Requir
**Decision**: Same port serves both protocols
**Consequences**: Simplified deployment, protocol detection overhead, libwebsockets dependency
These architectural decisions form the foundation of C-Relay's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
These architectural decisions form the foundation of C-Relay-PG's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
**
[Response interrupted by a tool use result. Only one tool may be used at a time and should be placed at the end of the message.]
Executable
BIN
View File
Binary file not shown.
+8 -8
View File
@@ -1,6 +1,6 @@
# AGENTS.md - AI Agent Integration Guide
**Project-Specific Information for AI Agents Working with C-Relay**
**Project-Specific Information for AI Agents Working with C-Relay-PG**
## Critical Build Commands
@@ -15,9 +15,9 @@
- Starts relay in background with proper logging
### Architecture-Specific Binary Outputs
- **x86_64**: `./build/c_relay_x86`
- **ARM64**: `./build/c_relay_arm64`
- **Other**: `./build/c_relay_$(ARCH)`
- **x86_64**: `./build/c_relay_pg_x86`
- **ARM64**: `./build/c_relay_pg_arm64`
- **Other**: `./build/c_relay_pg_$(ARCH)`
### Database File Naming Convention
- **Format**: `<relay_pubkey>.db` (NOT `.nrdb` as shown in docs)
@@ -75,10 +75,10 @@
### Process Management
```bash
# Kill existing relay processes
pkill -f "c_relay_"
pkill -f "c_relay_pg_"
# Check running processes
ps aux | grep c_relay_
ps aux | grep c_relay_pg_
# Force kill port binding
fuser -k 8888/tcp
@@ -95,7 +95,7 @@ fuser -k 8888/tcp
- Event configuration tests: `tests/event_config_tests.sh`
### SystemD Integration Considerations
- Service runs as `c-relay` user in `/opt/c-relay`
- Service runs as `c-relay-pg` user in `/opt/c-relay-pg`
- Database files created in WorkingDirectory automatically
- No environment variables needed (event-based config)
- Resource limits: 65536 file descriptors, 4096 processes
@@ -137,7 +137,7 @@ fuser -k 8888/tcp
## Quick Debugging Commands
```bash
# Check relay status
ps aux | grep c_relay_ && netstat -tln | grep 8888
ps aux | grep c_relay_pg_ && netstat -tln | grep 8888
# View logs
tail -f relay.log
+9 -9
View File
@@ -1,6 +1,6 @@
# C-Relay API Documentation
# C-Relay-PG API Documentation
Complete API reference for the C-Relay event-based administration system and advanced features.
Complete API reference for the C-Relay-PG event-based administration system and advanced features.
## Table of Contents
@@ -21,7 +21,7 @@ Complete API reference for the C-Relay event-based administration system and adv
## Overview
C-Relay uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
C-Relay-PG uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
- **Cryptographic security**: All commands must be signed with the admin private key
- **Audit trail**: Complete history of all administrative actions
@@ -63,8 +63,8 @@ Store the admin private key securely:
export C_RELAY_ADMIN_KEY="nsec1abc123..."
# Secure file
echo "nsec1abc123..." > ~/.c-relay-admin
chmod 600 ~/.c-relay-admin
echo "nsec1abc123..." > ~/.c-relay-pg-admin
chmod 600 ~/.c-relay-pg-admin
# Password manager (recommended)
# Store in 1Password, Bitwarden, etc.
@@ -167,7 +167,7 @@ Examples:
"data": [
{
"key": "relay_name",
"value": "C-Relay",
"value": "C-Relay-PG",
"data_type": "string",
"category": "relay",
"description": "Relay name displayed in NIP-11"
@@ -506,10 +506,10 @@ ORDER BY count DESC
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `relay_name` | string | "C-Relay" | Relay name (NIP-11) |
| `relay_name` | string | "C-Relay-PG" | Relay name (NIP-11) |
| `relay_description` | string | "C Nostr Relay" | Relay description |
| `relay_contact` | string | "" | Admin contact info |
| `relay_software` | string | "c-relay" | Software identifier |
| `relay_software` | string | "c-relay-pg" | Software identifier |
| `relay_version` | string | auto | Software version |
| `supported_nips` | string | "1,9,11,13,15,20,33,40,42,45,50,70" | Supported NIPs |
| `language_tags` | string | "*" | Supported languages |
@@ -576,7 +576,7 @@ ORDER BY count DESC
## Real-time Monitoring
C-Relay provides subscription-based real-time monitoring using ephemeral events (kind 24567).
C-Relay-PG provides subscription-based real-time monitoring using ephemeral events (kind 24567).
### Activation
+38 -28
View File
@@ -1,12 +1,14 @@
# Alpine-based MUSL static binary builder for C-Relay
# Alpine-based MUSL static binary builder for C-Relay-PG
# Produces truly portable binaries with zero runtime dependencies
ARG DEBUG_BUILD=false
ARG DB_BACKEND=sqlite
FROM alpine:3.19 AS builder
# Re-declare build argument in this stage
# Re-declare build arguments in this stage
ARG DEBUG_BUILD=false
ARG DB_BACKEND=sqlite
# Install build dependencies
RUN apk add --no-cache \
@@ -26,9 +28,11 @@ RUN apk add --no-cache \
curl-static \
sqlite-dev \
sqlite-static \
postgresql-dev \
linux-headers \
wget \
bash
bash \
openssh-client
# Set working directory
WORKDIR /build
@@ -68,15 +72,8 @@ RUN cd /tmp && \
make install && \
rm -rf /tmp/libwebsockets
# Copy only submodule configuration and git directory
COPY .gitmodules /build/.gitmodules
COPY .git /build/.git
# Clean up any stale submodule references (nips directory is not a submodule)
RUN git rm --cached nips 2>/dev/null || true
# Initialize submodules (cached unless .gitmodules changes)
RUN git submodule update --init --recursive
# Submodules are provided in the local build context and copied explicitly below.
# Avoid network/SSH dependency inside Docker image build.
# Copy nostr_core_lib source files (cached unless nostr_core_lib changes)
COPY nostr_core_lib /build/nostr_core_lib/
@@ -84,11 +81,13 @@ COPY nostr_core_lib /build/nostr_core_lib/
# Copy c_utils_lib source files (cached unless c_utils_lib changes)
COPY c_utils_lib /build/c_utils_lib/
# Build c_utils_lib with MUSL-compatible flags (cached unless c_utils_lib changes)
# Build c_utils_lib static library for relay logging utilities
# (build only debug.c to avoid broken/missing version header surface in submodule revision)
RUN cd c_utils_lib && \
sed -i 's/CFLAGS = -Wall -Wextra -std=c99 -O2 -g/CFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g/' Makefile && \
make clean && \
make
mkdir -p build && \
gcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g \
-Isrc -Iinclude -c src/debug.c -o build/debug.o && \
ar rcs libc_utils.a build/debug.o
# Build nostr_core_lib with required NIPs (cached unless nostr_core_lib changes)
# Disable fortification in build.sh to prevent __*_chk symbol issues
@@ -99,44 +98,55 @@ RUN cd nostr_core_lib && \
rm -f *.o *.a 2>/dev/null || true && \
./build.sh --nips=1,6,13,17,19,44,59
# Copy c-relay source files LAST (only this layer rebuilds on source changes)
# Copy c-relay-pg source files LAST (only this layer rebuilds on source changes)
COPY src/ /build/src/
COPY Makefile /build/Makefile
# Build c-relay with full static linking (only rebuilds when src/ changes)
# Build c-relay-pg with full static linking (only rebuilds when src/ changes)
# Disable fortification to avoid __*_chk symbols that don't exist in MUSL
# Use conditional compilation flags based on DEBUG_BUILD argument
# Use conditional compilation flags based on DEBUG_BUILD and DB_BACKEND build args
RUN if [ "$DEBUG_BUILD" = "true" ]; then \
CFLAGS="-g -O2 -DDEBUG"; \
STRIP_CMD="echo 'Keeping debug symbols'"; \
echo "Building with DEBUG symbols enabled (optimized with -O2)"; \
else \
CFLAGS="-O2"; \
STRIP_CMD="strip /build/c_relay_static"; \
STRIP_CMD="strip /build/c_relay_pg_static"; \
echo "Building optimized production binary (symbols stripped)"; \
fi && \
gcc -static $CFLAGS -Wall -Wextra -std=c99 \
if [ "$DB_BACKEND" = "postgres" ]; then \
DB_FLAGS="-DDB_BACKEND_POSTGRES -DHAVE_LIBPQ"; \
DB_LIBS="-lpq -lpgcommon -lpgport"; \
echo "Compiling with PostgreSQL backend"; \
else \
DB_FLAGS=""; \
DB_LIBS=""; \
echo "Compiling with SQLite backend"; \
fi && \
gcc -static $CFLAGS $DB_FLAGS -Wall -Wextra -std=c99 \
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
-I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core \
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket \
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql \
src/main.c src/config.c src/dm_admin.c src/request_validator.c \
src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c \
src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c \
-o /build/c_relay_static \
src/caching_inbox_poller.c src/caching_service_launcher.c \
src/db_ops.c src/db_ops_sqlite.c src/db_ops_postgres.c src/thread_pool.c \
-o /build/c_relay_pg_static \
c_utils_lib/libc_utils.a \
nostr_core_lib/libnostr_core_x64.a \
-lwebsockets -lssl -lcrypto -lsqlite3 -lsecp256k1 \
-lcurl -lz -lpthread -lm -ldl && \
-lcurl -lz -lpthread -lm -ldl $DB_LIBS && \
eval "$STRIP_CMD"
# Verify it's truly static
RUN echo "=== Binary Information ===" && \
file /build/c_relay_static && \
ls -lh /build/c_relay_static && \
file /build/c_relay_pg_static && \
ls -lh /build/c_relay_pg_static && \
echo "=== Checking for dynamic dependencies ===" && \
(ldd /build/c_relay_static 2>&1 || echo "Binary is static") && \
(ldd /build/c_relay_pg_static 2>&1 || echo "Binary is static") && \
echo "=== Build complete ==="
# Output stage - just the binary
FROM scratch AS output
COPY --from=builder /build/c_relay_static /c_relay_static
COPY --from=builder /build/c_relay_pg_static /c_relay_pg_static
+38 -26
View File
@@ -1,28 +1,40 @@
# C-Relay Makefile
# C-Relay-PG Makefile
CC = gcc
CFLAGS = -Wall -Wextra -std=c99 -g -O2
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql
LIBS = -lsqlite3 -lwebsockets -lz -ldl -lpthread -lm -L/usr/local/lib -lsecp256k1 -lssl -lcrypto -L/usr/local/lib -lcurl -Lc_utils_lib -lc_utils
DB_BACKEND ?= sqlite
# Build directory
BUILD_DIR = build
# Source files
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c src/thread_pool.c src/caching_inbox_poller.c src/caching_service_launcher.c
DB_OPS_SRC = src/db_ops.c
ifeq ($(DB_BACKEND),postgres)
CFLAGS += -DDB_BACKEND_POSTGRES -DHAVE_LIBPQ
DB_OPS_SRC += src/db_ops_postgres.c
LIBS += -lpq
else
DB_OPS_SRC += src/db_ops_sqlite.c
endif
NOSTR_CORE_LIB = nostr_core_lib/libnostr_core_x64.a
C_UTILS_LIB = c_utils_lib/libc_utils.a
# Architecture detection
ARCH = $(shell uname -m)
ifeq ($(ARCH),x86_64)
TARGET = $(BUILD_DIR)/c_relay_x86
TARGET = $(BUILD_DIR)/c_relay_pg_x86
else ifeq ($(ARCH),aarch64)
TARGET = $(BUILD_DIR)/c_relay_arm64
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
else ifeq ($(ARCH),arm64)
TARGET = $(BUILD_DIR)/c_relay_arm64
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
else
TARGET = $(BUILD_DIR)/c_relay_$(ARCH)
TARGET = $(BUILD_DIR)/c_relay_pg_$(ARCH)
endif
# Default target
@@ -81,19 +93,19 @@ force-version:
@$(MAKE) src/main.h
# Build the relay
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Compiling C-Relay for architecture: $(ARCH)"
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Compiling C-Relay-PG for architecture: $(ARCH) (backend: $(DB_BACKEND))"
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(TARGET)"
# Build for specific architectures
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Building C-Relay for x86_64..."
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_x86"
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Building C-Relay-PG for x86_64 (backend: $(DB_BACKEND))..."
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_x86"
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Cross-compiling C-Relay for ARM64..."
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
@echo "Cross-compiling C-Relay-PG for ARM64 (backend: $(DB_BACKEND))..."
@if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1; then \
echo "ERROR: ARM64 cross-compiler not found."; \
echo "Install with: make install-cross-tools"; \
@@ -116,9 +128,9 @@ arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(
fi
@echo "Using aarch64-linux-gnu-gcc with ARM64 libraries..."
PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
-L/usr/lib/aarch64-linux-gnu $(LIBS)
@echo "Build complete: $(BUILD_DIR)/c_relay_arm64"
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_arm64"
# Install ARM64 cross-compilation dependencies
install-arm64-deps:
@@ -160,7 +172,7 @@ test: $(TARGET)
init-db:
@echo "Database initialization is now handled automatically when the server starts."
@echo "The schema is embedded in the binary - no external files needed."
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_x86"
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_pg_x86"
# Clean build artifacts
clean:
@@ -180,7 +192,7 @@ install-deps:
# Help
help:
@echo "C-Relay Build System"
@echo "C-Relay-PG Build System"
@echo ""
@echo "Targets:"
@echo " all Build the relay for current architecture (default)"
@@ -209,15 +221,15 @@ help:
# Build fully static MUSL binaries using Docker
static-musl-x86_64:
@echo "Building fully static MUSL binary for x86_64..."
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-x86_64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-x86_64 sh -c "cp /c_relay_static_musl_x86_64 /output/"
@echo "Static binary created: build/c_relay_static_musl_x86_64"
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-x86_64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-x86_64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/"
@echo "Static binary created: build/c_relay_pg_static_musl_x86_64"
static-musl-arm64:
@echo "Building fully static MUSL binary for ARM64..."
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-arm64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-arm64 sh -c "cp /c_relay_static_musl_x86_64 /output/c_relay_static_musl_arm64"
@echo "Static binary created: build/c_relay_static_musl_arm64"
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-arm64 --load .
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-arm64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/c_relay_pg_static_musl_arm64"
@echo "Static binary created: build/c_relay_pg_static_musl_arm64"
static-musl: static-musl-x86_64 static-musl-arm64
@echo "Built static MUSL binaries for both architectures"
+6 -6
View File
@@ -45,21 +45,21 @@ Also included is a more standard administrative web front end. This front end co
## Screenshots
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/main.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/main.png)
Main page with real time updates.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/config.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/config.png)
Set your configuration preferences.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/subscriptions.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/subscriptions.png)
View current subscriptions
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/white-blacklists.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/white-blacklists.png)
Add npubs to white or black lists.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/sqlQuery.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/sqlQuery.png)
Run sql queries on the database.
![](https://git.laantungir.net/laantungir/c-relay/raw/branch/master/screenshots/main-light.png)
![](https://git.laantungir.net/laantungir/c-relay-pg/raw/branch/master/screenshots/main-light.png)
Light mode.
+24 -24
View File
@@ -1,11 +1,11 @@
# C-Relay: High-Performance Nostr Relay
# C-Relay-PG: High-Performance Nostr Relay
A blazingly fast, production-ready Nostr relay implemented in C with an innovative event-based configuration system. Built for performance, security, and ease of deployment.
## 🚀 Why C-Relay?
## 🚀 Why C-Relay-PG?
### Event-Based Configuration
Unlike traditional relays that require config files, C-Relay uses **cryptographically signed Nostr events** for all configuration. This means:
Unlike traditional relays that require config files, C-Relay-PG uses **cryptographically signed Nostr events** for all configuration. This means:
- **Zero config files** - Everything stored in the database
- **Real-time updates** - Changes applied instantly without restart
- **Cryptographic security** - All changes must be signed by admin
@@ -36,7 +36,7 @@ Control your relay by sending direct messages from any Nostr client:
## 📋 Supported NIPs
C-Relay implements a comprehensive set of Nostr Improvement Proposals:
C-Relay-PG implements a comprehensive set of Nostr Improvement Proposals:
-**NIP-01**: Basic protocol flow implementation
-**NIP-09**: Event deletion
@@ -89,12 +89,12 @@ Download and run - no dependencies required:
```bash
# Download the latest static release
wget https://git.laantungir.net/laantungir/c-relay/releases/download/v0.6.0/c-relay-v0.6.0-linux-x86_64-static
chmod +x c-relay-v0.6.0-linux-x86_64-static
mv c-relay-v0.6.0-linux-x86_64-static c-relay
wget https://git.laantungir.net/laantungir/c-relay-pg/releases/download/v0.6.0/c-relay-pg-v0.6.0-linux-x86_64-static
chmod +x c-relay-pg-v0.6.0-linux-x86_64-static
mv c-relay-pg-v0.6.0-linux-x86_64-static c-relay-pg
# Run the relay
./c-relay
./c-relay-pg
```
**Important**: On first startup, save the **Admin Private Key** displayed in the console. You'll need it for all administrative operations.
@@ -107,8 +107,8 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
libwebsockets-dev libssl-dev libsecp256k1-dev libcurl4-openssl-dev zlib1g-dev
# Clone and build
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
./make_and_restart_relay.sh
```
@@ -136,8 +136,8 @@ The web interface provides:
```bash
# Clone repository
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
# Build
@@ -147,25 +147,25 @@ make clean && make
sudo systemd/install-service.sh
# Start and enable
sudo systemctl start c-relay
sudo systemctl enable c-relay
sudo systemctl start c-relay-pg
sudo systemctl enable c-relay-pg
# Capture admin keys from logs
sudo journalctl -u c-relay | grep "Admin Private Key"
sudo journalctl -u c-relay-pg | grep "Admin Private Key"
```
### Docker Deployment
```bash
# Build Docker image
docker build -f Dockerfile.alpine-musl -t c-relay .
docker build -f Dockerfile.alpine-musl -t c-relay-pg .
# Run container
docker run -d \
--name c-relay \
--name c-relay-pg \
-p 8888:8888 \
-v /path/to/data:/data \
c-relay
c-relay-pg
```
### Cloud Deployment
@@ -181,7 +181,7 @@ See [`docs/deployment_guide.md`](docs/deployment_guide.md) for detailed deployme
## 🔧 Configuration
C-Relay uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
C-Relay-PG uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
### Basic Configuration
@@ -239,8 +239,8 @@ The admin private key is displayed **only once** during first startup. Store it
```bash
# Save to secure location
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-admin
chmod 600 ~/.c-relay-admin
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-pg-admin
chmod 600 ~/.c-relay-pg-admin
```
### Production Security
@@ -269,9 +269,9 @@ Contributions are welcome! Please:
## 🔗 Links
- **Repository**: [https://github.com/your-org/c-relay](https://github.com/your-org/c-relay)
- **Releases**: [https://git.laantungir.net/laantungir/c-relay/releases](https://git.laantungir.net/laantungir/c-relay/releases)
- **Issues**: [https://github.com/your-org/c-relay/issues](https://github.com/your-org/c-relay/issues)
- **Repository**: [https://github.com/your-org/c-relay-pg](https://github.com/your-org/c-relay-pg)
- **Releases**: [https://git.laantungir.net/laantungir/c-relay-pg/releases](https://git.laantungir.net/laantungir/c-relay-pg/releases)
- **Issues**: [https://github.com/your-org/c-relay-pg/issues](https://github.com/your-org/c-relay-pg/issues)
- **Nostr Protocol**: [https://github.com/nostr-protocol/nostr](https://github.com/nostr-protocol/nostr)
## 💬 Support
+7 -7
View File
@@ -41,7 +41,7 @@ This guide is specifically tailored for C programs that use:
```bash
# 1. Copy the Dockerfile template (see below)
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
# 2. Customize for your project (see Customization section)
vim Dockerfile.static
@@ -466,13 +466,13 @@ docker build -t my-app:latest .
docker run --rm my-app:latest --help
```
## Reusing c-relay Files
## Reusing c-relay-pg Files
You can directly copy these files from c-relay:
You can directly copy these files from c-relay-pg:
### 1. Dockerfile.alpine-musl
```bash
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
```
Then customize:
@@ -482,7 +482,7 @@ Then customize:
### 2. build_static.sh
```bash
cp /path/to/c-relay/build_static.sh ./
cp /path/to/c-relay-pg/build_static.sh ./
```
Then customize:
@@ -492,7 +492,7 @@ Then customize:
### 3. .dockerignore (Optional)
```bash
cp /path/to/c-relay/.dockerignore ./
cp /path/to/c-relay-pg/.dockerignore ./
```
Helps speed up Docker builds by excluding unnecessary files.
@@ -522,7 +522,7 @@ Helps speed up Docker builds by excluding unnecessary files.
- [Alpine Linux](https://alpinelinux.org/)
- [nostr_core_lib](https://github.com/chebizarro/nostr_core_lib)
- [Static Linking Best Practices](https://www.musl-libc.org/faq.html)
- [c-relay Implementation](./docs/musl_static_build.md)
- [c-relay-pg Implementation](./docs/musl_static_build.md)
## Example: Minimal Nostr Client
+150 -2
View File
@@ -4,7 +4,7 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>C-Relay Admin</title>
<title>C-Relay-PG Admin</title>
<link rel="stylesheet" href="/api/index.css">
</head>
@@ -18,6 +18,7 @@
<li><button class="nav-item" data-page="authorization">Authorization</button></li>
<li><button class="nav-item" data-page="ip-bans">IP BANS</button></li>
<li><button class="nav-item" data-page="relay-events">Relay Events</button></li>
<li><button class="nav-item" data-page="caching">Caching</button></li>
<li><button class="nav-item" data-page="dm">DM</button></li>
<li><button class="nav-item" data-page="database">Database Query</button></li>
</ul>
@@ -42,7 +43,7 @@
<span class="relay-letter" data-letter="Y">Y</span>
</div>
<div class="relay-info">
<div id="relay-name" class="relay-name">C-Relay</div>
<div id="relay-name" class="relay-name">C-Relay-PG</div>
<div id="relay-description" class="relay-description">Loading...</div>
<div id="relay-pubkey-container" class="relay-pubkey-container">
<div id="relay-pubkey" class="relay-pubkey">Loading...</div>
@@ -496,6 +497,29 @@ WEB OF TRUST
RELAY EVENTS MANAGEMENT
</div>
<!-- Live Relay Event Feed -->
<div class="input-group">
<h3>Live Relay Event Feed (Normal Nostr Subscription)</h3>
<div class="config-table-container">
<table class="config-table" id="live-relay-events-table">
<thead>
<tr>
<th>Time</th>
<th>Kind</th>
<th>Pubkey</th>
<th>ID</th>
<th>Content Preview</th>
</tr>
</thead>
<tbody id="live-relay-events-table-body">
<tr>
<td colspan="5" style="text-align: center;">Waiting for live events...</td>
</tr>
</tbody>
</table>
</div>
</div>
<!-- Kind 0: User Metadata -->
<div class="input-group">
<h3>Kind 0: User Metadata</h3>
@@ -556,6 +580,130 @@ WEB OF TRUST
</div>
</div>
<!-- CACHING Section -->
<div class="section" id="cachingSection" style="display: none;">
<div class="section-header">CACHING</div>
<!-- Caching Service Status (read-only) -->
<div class="input-group">
<h3>Caching Service Status</h3>
<div id="caching-service-status" class="status-display">
<p>Service status will appear here after connecting.</p>
</div>
</div>
<!-- Relay Inbox Status (read-only) -->
<div class="input-group">
<h3>Relay Inbox Status</h3>
<div id="caching-inbox-status" class="status-display">
<p>Inbox status will appear here after connecting.</p>
</div>
</div>
<!-- Caching Configuration (editable) -->
<div class="input-group">
<h3>Caching Configuration</h3>
<div class="form-group">
<label for="caching-enabled">Enable Caching:</label>
<select id="caching-enabled">
<option value="false">Disabled</option>
<option value="true">Enabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-inbox-enabled">Enable Inbox Consumer:</label>
<select id="caching-inbox-enabled">
<option value="false">Disabled</option>
<option value="true">Enabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-root-npubs">Root Npubs (one per line):</label>
<textarea id="caching-root-npubs" rows="4" placeholder="npub1..."></textarea>
</div>
<div class="form-group">
<label for="caching-bootstrap-relays">Bootstrap Relays (one per line):</label>
<textarea id="caching-bootstrap-relays" rows="4" placeholder="wss://relay.damus.io"></textarea>
</div>
<div class="form-group">
<label for="caching-kinds">Kinds (comma-separated):</label>
<input type="text" id="caching-kinds" placeholder="1,3,6,10000,30023">
</div>
<div class="form-group">
<label for="caching-admin-kinds">Admin Kinds (comma-separated, * for all):</label>
<input type="text" id="caching-admin-kinds" placeholder="*">
</div>
<div class="form-group">
<label for="caching-live-enabled">Live Subscriptions:</label>
<select id="caching-live-enabled">
<option value="true">Enabled</option>
<option value="false">Disabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-backfill-enabled">Backfill:</label>
<select id="caching-backfill-enabled">
<option value="true">Enabled</option>
<option value="false">Disabled</option>
</select>
</div>
<div class="form-group">
<label for="caching-backfill-windows">Backfill Windows (seconds, comma-separated):</label>
<input type="text" id="caching-backfill-windows" placeholder="86400,604800,2592000,7776000,31536000">
</div>
<div class="form-group">
<label for="caching-backfill-page-size">Backfill Page Size:</label>
<input type="number" id="caching-backfill-page-size" placeholder="50">
</div>
<div class="form-group">
<label for="caching-backfill-tick-interval">Backfill Tick Interval (ms):</label>
<input type="number" id="caching-backfill-tick-interval" placeholder="5000">
</div>
<div class="form-group">
<label for="caching-inbox-batch-size">Inbox Batch Size:</label>
<input type="number" id="caching-inbox-batch-size" placeholder="25">
</div>
<div class="form-group">
<label for="caching-inbox-active-poll">Inbox Active Poll (ms):</label>
<input type="number" id="caching-inbox-active-poll" placeholder="200">
</div>
<div class="form-group">
<label for="caching-inbox-idle-poll">Inbox Idle Poll (ms):</label>
<input type="number" id="caching-inbox-idle-poll" placeholder="5000">
</div>
<div class="inline-buttons">
<button type="button" id="caching-apply-btn">APPLY CONFIGURATION</button>
<button type="button" id="caching-reset-progress-btn">RESET BACKFILL PROGRESS</button>
</div>
<div id="caching-config-status" class="status-message"></div>
</div>
<div class="input-group">
<h3>Caching Service Control</h3>
<p>Start or stop the external caching service process. Requires <code>caching_service_binary_path</code> and <code>caching_service_pg_conn</code> to be set in Configuration.</p>
<div class="inline-buttons">
<button type="button" id="caching-start-service-btn">START CACHING SERVICE</button>
<button type="button" id="caching-stop-service-btn">STOP CACHING SERVICE</button>
</div>
<div id="caching-service-control-status" class="status-message"></div>
</div>
</div>
<!-- SQL QUERY Section -->
<div class="section" id="sqlQuerySection" style="display: none;">
<div class="section-header">
+438 -12
View File
@@ -19,6 +19,8 @@ let nlLite = null;
let userPubkey = null;
let isLoggedIn = false;
let currentConfig = null;
// Caching page periodic refresh interval handle
let cachingRefreshInterval = null;
// Global subscription state
let relayPool = null;
let subscriptionId = null;
@@ -50,6 +52,7 @@ let pendingSqlQueries = new Map();
// Real-time event rate chart
let eventRateChart = null;
let previousTotalEvents = 0; // Track previous total for rate calculation
const liveRelayEventIds = new Set(); // Deduplicate live feed rows
// Temporary diagnostics toggle: enable deep SimplePool diagnostics in nostr.bundle.js
window.__SIMPLE_POOL_DEBUG__ = true;
@@ -185,7 +188,7 @@ async function fetchRelayInfo(relayUrl) {
method: 'GET',
headers: {
'Accept': 'application/nostr+json',
'User-Agent': 'C-Relay-Admin-API/1.0'
'User-Agent': 'C-Relay-PG-Admin-API/1.0'
},
timeout: 10000 // 10 second timeout
});
@@ -449,11 +452,12 @@ async function setupAutomaticRelayConnection(showSections = false) {
const relayInfo = await fetchRelayInfo(httpUrl);
const fetchedPubkey = relayInfo && relayInfo.pubkey ? relayInfo.pubkey : null;
relayPubkey = fetchedPubkey || '4f355bdcb7cc0af728ef3cceb9615d90684bb5b2ca5f859ab0f0b704075871aa';
// Fallback must match the actual running relay pubkey when NIP-11 doesn't expose pubkey.
relayPubkey = fetchedPubkey || 'e9aa50decff01f2cec1ec2b2e0b34332cf9c92cafdac5a7cc0881a6d26b59854';
// Keep relay metadata (including version) even when pubkey is missing/fallback
relayInfoData = {
name: relayInfo?.name || 'C-Relay',
name: relayInfo?.name || 'C-Relay-PG',
version: relayInfo?.version || '',
description: relayInfo?.description || 'Nostr Relay',
pubkey: relayPubkey
@@ -466,7 +470,7 @@ async function setupAutomaticRelayConnection(showSections = false) {
}
} catch (error) {
console.log('⚠️ Could not fetch relay info, using fallback pubkey:', error.message);
relayPubkey = '4f355bdcb7cc0af728ef3cceb9615d90684bb5b2ca5f859ab0f0b704075871aa';
relayPubkey = 'e9aa50decff01f2cec1ec2b2e0b34332cf9c92cafdac5a7cc0881a6d26b59854';
}
// Set up subscription to receive admin API responses
@@ -1328,6 +1332,10 @@ async function subscribeToConfiguration() {
kinds: [0, 10050, 10002], // Relay events: metadata, DM relays, relay list
authors: [getRelayPubkey()], // Only listen to relay's own events
limit: 10
}, {
// Live feed uses a normal Nostr subscription, not monitoring events
since: Math.floor(Date.now() / 1000),
limit: 200
}], {
async onevent(event) {
// Simplified logging - one line per event
@@ -1433,6 +1441,11 @@ async function subscribeToConfiguration() {
if ([0, 10050, 10002].includes(event.kind)) {
handleRelayEventReceived(event);
}
// Real-time live feed via normal subscription (exclude monitoring/admin traffic)
if (event.kind !== 24567 && event.kind !== 23457 && event.kind !== 23456) {
addEventToLiveFeed(event);
}
},
oneose() {
console.log('EOSE received - End of stored events');
@@ -2065,6 +2078,20 @@ function handleSystemCommandResponse(responseData) {
handleWotStatusResponse(responseData);
}
// Handle caching status response
if (responseData.command === 'caching_status') {
handleCachingStatusResponse(responseData);
}
// Handle caching reset progress response
if (responseData.command === 'caching_reset_progress') {
if (responseData.status === 'success') {
log('Backfill progress reset successfully', 'INFO');
} else {
log(`Failed to reset backfill progress: ${responseData.message || 'Unknown error'}`, 'ERROR');
}
}
if (typeof logTestEvent === 'function') {
logTestEvent('RECV', `System command response: ${responseData.command} - ${responseData.status}`, 'SYSTEM_CMD');
}
@@ -3962,9 +3989,9 @@ async function testPostEvent() {
created_at: Math.floor(Date.now() / 1000),
tags: [
["t", "test"],
["client", "c-relay-admin-api"]
["client", "c-relay-pg-admin-api"]
],
content: `Test event from C-Relay Admin API at ${new Date().toISOString()}`
content: `Test event from C-Relay-PG Admin API at ${new Date().toISOString()}`
};
logTestEvent('SENT', `Test event (before signing): ${JSON.stringify(testEvent)}`, 'EVENT');
@@ -4248,7 +4275,7 @@ function updateRelayInfoInHeader() {
// Get relay info from NIP-11/config data or use defaults
const relayInfo = getRelayInfo();
const relayName = relayInfo.name || 'C-Relay';
const relayName = relayInfo.name || 'C-Relay-PG';
const relayVersion = relayInfo.version || '';
const relayDescription = relayInfo.description || 'Nostr Relay';
@@ -4291,7 +4318,7 @@ function getRelayInfo() {
// Default values
return {
name: 'C-Relay',
name: 'C-Relay-PG',
version: '',
description: 'Nostr Relay',
pubkey: relayPubkey
@@ -4302,7 +4329,7 @@ function getRelayInfo() {
function updateStoredRelayInfo(configData) {
if (configData && configData.data) {
// Extract relay info from config data
const relayName = configData.data.find(item => item.key === 'relay_name')?.value || 'C-Relay';
const relayName = configData.data.find(item => item.key === 'relay_name')?.value || 'C-Relay-PG';
const relayVersion = configData.data.find(item => item.key === 'relay_version')?.value || '';
const relayDescription = configData.data.find(item => item.key === 'relay_description')?.value || 'Nostr Relay';
@@ -5357,6 +5384,7 @@ function switchPage(pageName) {
'wotSection',
'ipBansSection',
'relayEventsSection',
'cachingSection',
'nip17DMSection',
'sqlQuerySection'
];
@@ -5375,6 +5403,7 @@ function switchPage(pageName) {
'configuration': 'div_config',
'ip-bans': 'ipBansSection',
'relay-events': 'relayEventsSection',
'caching': 'cachingSection',
'dm': 'nip17DMSection',
'database': 'sqlQuerySection'
};
@@ -5412,6 +5441,30 @@ function switchPage(pageName) {
});
}
// Special handling for caching page - fetch status and start periodic refresh
if (pageName === 'caching') {
fetchCachingStatus().catch(error => {
console.log('Failed to refresh caching status on page switch: ' + error.message);
});
// Start periodic refresh every 10 seconds while the caching page is visible
if (cachingRefreshInterval) {
clearInterval(cachingRefreshInterval);
}
cachingRefreshInterval = setInterval(() => {
if (currentPage === 'caching') {
fetchCachingStatus().catch(error => {
console.log('Periodic caching status refresh failed: ' + error.message);
});
}
}, 10000);
} else {
// Clear periodic refresh when leaving the caching page
if (cachingRefreshInterval) {
clearInterval(cachingRefreshInterval);
cachingRefreshInterval = null;
}
}
// Close side navigation
closeSideNav();
@@ -5420,7 +5473,7 @@ function switchPage(pageName) {
// Initialize the app
document.addEventListener('DOMContentLoaded', () => {
console.log('C-Relay Admin API interface loaded');
console.log('C-Relay-PG Admin API interface loaded');
// Initialize dark mode
initializeDarkMode();
@@ -5522,7 +5575,7 @@ const SQL_QUERY_TEMPLATES = {
};
// Query history management (localStorage)
const QUERY_HISTORY_KEY = 'c_relay_sql_history';
const QUERY_HISTORY_KEY = 'c_relay_pg_sql_history';
const MAX_HISTORY_ITEMS = 20;
// Load query history from localStorage
@@ -6025,6 +6078,47 @@ function initializeToggleButtonsFromConfig(configData) {
// ================================
function addEventToLiveFeed(event) {
if (!event || !event.id) return;
if (liveRelayEventIds.has(event.id)) return;
liveRelayEventIds.add(event.id);
const tableBody = document.getElementById('live-relay-events-table-body');
if (!tableBody) return;
const preview = typeof event.content === 'string'
? event.content.slice(0, 120)
: JSON.stringify(event.content || '').slice(0, 120);
const row = document.createElement('tr');
row.innerHTML = `
<td>${new Date((event.created_at || Math.floor(Date.now() / 1000)) * 1000).toLocaleTimeString()}</td>
<td>${event.kind ?? '-'}</td>
<td title="${escapeHtml(event.pubkey || '')}">${escapeHtml((event.pubkey || '').slice(0, 16))}...</td>
<td title="${escapeHtml(event.id || '')}">${escapeHtml((event.id || '').slice(0, 16))}...</td>
<td>${escapeHtml(preview)}</td>
`;
const waitingRow = tableBody.querySelector('tr td[colspan="5"]');
if (waitingRow) {
tableBody.innerHTML = '';
}
tableBody.insertBefore(row, tableBody.firstChild);
while (tableBody.children.length > 100) {
const last = tableBody.lastChild;
if (last && last.querySelector('td:nth-child(4)')) {
const idText = last.querySelector('td:nth-child(4)').textContent || '';
if (idText.endsWith('...')) {
// keep set bounded even with truncated display
// actual id remains in Set until page reload (acceptable for UI scope)
}
}
tableBody.removeChild(last);
}
}
// Handle received relay events
function handleRelayEventReceived(event) {
console.log('Handling relay event:', event.kind, event);
@@ -6724,4 +6818,336 @@ function initIpBansEventListeners() {
}
// Initialize when DOM is ready
document.addEventListener('DOMContentLoaded', initIpBansEventListeners);
document.addEventListener('DOMContentLoaded', initIpBansEventListeners);
// ================================
// CACHING PAGE FUNCTIONS
// ================================
// Mapping of caching config keys to form field ids and metadata
const CACHING_CONFIG_FIELDS = [
{ key: 'caching_enabled', field: 'caching-enabled', type: 'select' },
{ key: 'caching_inbox_enabled', field: 'caching-inbox-enabled', type: 'select' },
{ key: 'caching_root_npubs', field: 'caching-root-npubs', type: 'textarea-list' },
{ key: 'caching_bootstrap_relays', field: 'caching-bootstrap-relays', type: 'textarea-list' },
{ key: 'caching_kinds', field: 'caching-kinds', type: 'string' },
{ key: 'caching_admin_kinds', field: 'caching-admin-kinds', type: 'string' },
{ key: 'caching_live_enabled', field: 'caching-live-enabled', type: 'select' },
{ key: 'caching_backfill_enabled', field: 'caching-backfill-enabled', type: 'select' },
{ key: 'caching_backfill_windows', field: 'caching-backfill-windows', type: 'string' },
{ key: 'caching_backfill_page_size', field: 'caching-backfill-page-size', type: 'integer' },
{ key: 'caching_backfill_tick_interval_ms',field: 'caching-backfill-tick-interval', type: 'integer' },
{ key: 'caching_inbox_batch_size', field: 'caching-inbox-batch-size', type: 'integer' },
{ key: 'caching_inbox_active_poll_ms', field: 'caching-inbox-active-poll', type: 'integer' },
{ key: 'caching_inbox_idle_poll_ms', field: 'caching-inbox-idle-poll', type: 'integer' }
];
// Helper: read a config value from currentConfig (which stores values in tags as [key, value])
function getCachingConfigValue(key) {
if (!currentConfig || !currentConfig.tags) return null;
for (const tag of currentConfig.tags) {
if (Array.isArray(tag) && tag.length >= 2 && tag[0] === key) {
return tag[1];
}
}
return null;
}
// Fetch caching status: populates form fields from config and queries service/inbox status
async function fetchCachingStatus() {
try {
if (!isLoggedIn || !userPubkey) {
throw new Error('Must be logged in to fetch caching status');
}
// Reuse the existing config query infrastructure to refresh currentConfig
await fetchConfiguration();
// Populate form fields from currentConfig
CACHING_CONFIG_FIELDS.forEach(field => {
const value = getCachingConfigValue(field.key);
const el = document.getElementById(field.field);
if (!el) return;
if (value === null || value === undefined) {
return;
}
if (field.type === 'textarea-list') {
// Config stores comma-separated; display as newline-separated
el.value = String(value).split(',').map(s => s.trim()).filter(s => s.length > 0).join('\n');
} else {
el.value = String(value);
}
});
// Attempt to fetch service/inbox status via the caching_status system command.
// If the backend has not implemented it yet, show a graceful "unavailable" message.
const serviceStatusEl = document.getElementById('caching-service-status');
const inboxStatusEl = document.getElementById('caching-inbox-status');
if (serviceStatusEl) {
serviceStatusEl.innerHTML = '<p>Service status unavailable (caching_status command not implemented on relay).</p>';
}
if (inboxStatusEl) {
inboxStatusEl.innerHTML = '<p>Inbox status unavailable (caching_status command not implemented on relay).</p>';
}
// Best-effort: send caching_status system command. Response handling is added in
// handleSystemCommandResponse() below; if unimplemented, the status blocks remain
// at the "unavailable" message set above.
try {
await sendAdminCommand(['system_command', 'caching_status']);
} catch (e) {
console.log('caching_status command not available: ' + e.message);
}
} catch (error) {
console.error('Failed to fetch caching status:', error);
const statusEl = document.getElementById('caching-config-status');
if (statusEl) {
statusEl.innerHTML = `<span style="color:red">Failed to load caching status: ${escapeHtml(error.message)}</span>`;
}
}
}
// Apply caching configuration: collect form values and send config_update
async function applyCachingConfig() {
const statusEl = document.getElementById('caching-config-status');
try {
if (!isLoggedIn || !userPubkey) {
throw new Error('Must be logged in to apply caching configuration');
}
const configObjects = [];
CACHING_CONFIG_FIELDS.forEach(field => {
const el = document.getElementById(field.field);
if (!el) return;
let value = el.value;
if (field.type === 'textarea-list') {
// Convert newlines back to comma-separated for storage
value = value.split('\n').map(s => s.trim()).filter(s => s.length > 0).join(',');
} else if (field.type === 'integer') {
// Normalize empty to 0
value = String(value).trim();
if (value === '') value = '0';
} else {
value = String(value).trim();
}
let dataType = 'string';
if (field.type === 'integer') {
dataType = 'integer';
} else if (field.type === 'select') {
dataType = 'boolean';
}
configObjects.push({
key: field.key,
value: value,
data_type: dataType,
category: 'caching'
});
});
if (statusEl) {
statusEl.innerHTML = '<span style="color:var(--accent-color,#ff0000)">Applying caching configuration...</span>';
}
// Send all caching config keys in a single config_update command
await sendConfigUpdateCommand(configObjects);
// Increment caching_config_generation so the caching service picks up changes
const currentGen = getCachingConfigValue('caching_config_generation');
let nextGen = 1;
if (currentGen !== null && !isNaN(parseInt(currentGen, 10))) {
nextGen = parseInt(currentGen, 10) + 1;
}
await sendConfigUpdateCommand([{
key: 'caching_config_generation',
value: String(nextGen),
data_type: 'integer',
category: 'caching'
}]);
if (statusEl) {
statusEl.innerHTML = '<span style="color:green">✓ Caching configuration applied successfully.</span>';
}
log('Caching configuration applied successfully', 'INFO');
// Refresh status to reflect applied values
setTimeout(() => fetchCachingStatus().catch(() => {}), 1500);
} catch (error) {
console.error('Failed to apply caching configuration:', error);
if (statusEl) {
statusEl.innerHTML = `<span style="color:red">Failed to apply configuration: ${escapeHtml(error.message)}</span>`;
}
log('Failed to apply caching configuration: ' + error.message, 'ERROR');
}
}
// Reset backfill progress via system_command
async function resetBackfillProgress() {
const statusEl = document.getElementById('caching-config-status');
try {
if (!isLoggedIn || !userPubkey) {
throw new Error('Must be logged in to reset backfill progress');
}
const confirmed = confirm('Reset all backfill progress? This will cause the caching service to re-fetch events for all backfill windows.');
if (!confirmed) {
return;
}
if (statusEl) {
statusEl.innerHTML = '<span style="color:var(--accent-color,#ff0000)">Resetting backfill progress...</span>';
}
await sendAdminCommand(['system_command', 'caching_reset_progress']);
if (statusEl) {
statusEl.innerHTML = '<span style="color:green">✓ Backfill progress reset command sent.</span>';
}
log('Backfill progress reset command sent', 'INFO');
} catch (error) {
console.error('Failed to reset backfill progress:', error);
if (statusEl) {
statusEl.innerHTML = `<span style="color:red">Failed to reset backfill progress: ${escapeHtml(error.message)}</span>`;
}
log('Failed to reset backfill progress: ' + error.message, 'ERROR');
}
}
// Handle caching_status system command responses (service + inbox status)
function handleCachingStatusResponse(responseData) {
const serviceStatusEl = document.getElementById('caching-service-status');
const inboxStatusEl = document.getElementById('caching-inbox-status');
if (responseData.status !== 'success') {
const msg = responseData.message || 'Unknown error';
if (serviceStatusEl) {
serviceStatusEl.innerHTML = `<p style="color:red">Service status error: ${escapeHtml(msg)}</p>`;
}
if (inboxStatusEl) {
inboxStatusEl.innerHTML = `<p style="color:red">Inbox status error: ${escapeHtml(msg)}</p>`;
}
return;
}
const data = responseData.data || responseData;
// Service status block
if (serviceStatusEl) {
const service = data.service || data.caching_service || {};
const enabled = service.enabled !== undefined ? service.enabled : (data.enabled !== undefined ? data.enabled : null);
const running = service.running !== undefined ? service.running : (data.running !== undefined ? data.running : null);
const connectedRelays = service.connected_relays !== undefined ? service.connected_relays : (data.connected_relays !== undefined ? data.connected_relays : null);
const eventsCached = service.events_cached !== undefined ? service.events_cached : (data.events_cached !== undefined ? data.events_cached : null);
let html = '<ul style="list-style:none;padding:0;margin:0;">';
if (enabled !== null) html += `<li><strong>Enabled:</strong> ${escapeHtml(String(enabled))}</li>`;
if (running !== null) html += `<li><strong>Running:</strong> ${escapeHtml(String(running))}</li>`;
if (connectedRelays !== null) html += `<li><strong>Connected Relays:</strong> ${escapeHtml(String(connectedRelays))}</li>`;
if (eventsCached !== null) html += `<li><strong>Events Cached:</strong> ${escapeHtml(String(eventsCached))}</li>`;
html += '</ul>';
serviceStatusEl.innerHTML = html;
}
// Inbox status block
if (inboxStatusEl) {
const inbox = data.inbox || data.caching_inbox || {};
const inboxEnabled = inbox.enabled !== undefined ? inbox.enabled : null;
const inboxRunning = inbox.running !== undefined ? inbox.running : null;
const queueDepth = inbox.queue_depth !== undefined ? inbox.queue_depth : null;
const lastPoll = inbox.last_poll !== undefined ? inbox.last_poll : null;
let html = '<ul style="list-style:none;padding:0;margin:0;">';
if (inboxEnabled !== null) html += `<li><strong>Inbox Enabled:</strong> ${escapeHtml(String(inboxEnabled))}</li>`;
if (inboxRunning !== null) html += `<li><strong>Inbox Running:</strong> ${escapeHtml(String(inboxRunning))}</li>`;
if (queueDepth !== null) html += `<li><strong>Queue Depth:</strong> ${escapeHtml(String(queueDepth))}</li>`;
if (lastPoll !== null) html += `<li><strong>Last Poll:</strong> ${escapeHtml(String(lastPoll))}</li>`;
html += '</ul>';
inboxStatusEl.innerHTML = html;
}
}
// Start the external caching service
async function startCachingService() {
const statusDiv = document.getElementById('caching-service-control-status');
if (!statusDiv) return;
if (!isLoggedIn || !userPubkey) {
statusDiv.innerHTML = '<span style="color:red">Must be logged in to start caching service</span>';
return;
}
statusDiv.innerHTML = '<span style="color:blue">Starting caching service...</span>';
try {
const command_array = ["system_command", "caching_start_service"];
await sendAdminCommand(command_array);
statusDiv.innerHTML = '<span style="color:green">✓ Caching service start initiated</span>';
log('Caching service start initiated', 'INFO');
} catch (error) {
statusDiv.innerHTML = `<span style="color:red">✗ Failed to start caching service: ${error.message}</span>`;
log(`Failed to start caching service: ${error.message}`, 'ERROR');
}
}
// Stop the external caching service
async function stopCachingService() {
const statusDiv = document.getElementById('caching-service-control-status');
if (!statusDiv) return;
if (!isLoggedIn || !userPubkey) {
statusDiv.innerHTML = '<span style="color:red">Must be logged in to stop caching service</span>';
return;
}
if (!confirm('Are you sure you want to stop the caching service?')) {
return;
}
statusDiv.innerHTML = '<span style="color:blue">Stopping caching service...</span>';
try {
const command_array = ["system_command", "caching_stop_service"];
await sendAdminCommand(command_array);
statusDiv.innerHTML = '<span style="color:green">✓ Caching service stop initiated</span>';
log('Caching service stop initiated', 'INFO');
} catch (error) {
statusDiv.innerHTML = `<span style="color:red">✗ Failed to stop caching service: ${error.message}</span>`;
log(`Failed to stop caching service: ${error.message}`, 'ERROR');
}
}
// Initialize Caching page event listeners
function initCachingEventListeners() {
const applyBtn = document.getElementById('caching-apply-btn');
const resetBtn = document.getElementById('caching-reset-progress-btn');
const startServiceBtn = document.getElementById('caching-start-service-btn');
const stopServiceBtn = document.getElementById('caching-stop-service-btn');
if (applyBtn) {
applyBtn.addEventListener('click', applyCachingConfig);
}
if (resetBtn) {
resetBtn.addEventListener('click', resetBackfillProgress);
}
if (startServiceBtn) {
startServiceBtn.addEventListener('click', startCachingService);
}
if (stopServiceBtn) {
stopServiceBtn.addEventListener('click', stopCachingService);
}
console.log('Caching page event listeners initialized');
}
// Initialize caching event listeners when DOM is ready
document.addEventListener('DOMContentLoaded', initCachingEventListeners);
+813
View File
@@ -0,0 +1,813 @@
<!DOCTYPE html>
<?xml version="1.0" encoding="UTF-8"?>
<html lang="en" dir="ltr">
<head>
<meta charset="utf-8" />
<title>TEMPLATE</title>
<link rel="stylesheet" href="./css/client.css" />
<!-- Initialize theme BEFORE any components load -->
<script>
(function () {
const savedTheme = localStorage.getItem('theme');
if (savedTheme === 'dark') {
document.documentElement.classList.add('dark-mode');
if (document.body) {
document.body.classList.add('dark-mode');
}
}
})();
</script>
<link rel="shortcut icon" type="image/x-icon" href="./favicon/favicon-dots2.ico" />
<!-- SVG.js library (required by HamburgerMorphing) -->
<script src="https://cdn.jsdelivr.net/npm/@svgdotjs/svg.js@3.0/dist/svg.min.js"></script>
</head>
<body>
<!-- ================================================================
HAMBURGER BUTTON (Fixed, separate from header)
================================================================
The hamburger button is a fixed element outside the header
to ensure it stays visible above the sidenav (z-index: 10 > 3).
================================================================ -->
<div id="divSvgHam" class="divHeaderButtons">
<!-- HamburgerMorphing will be injected here -->
</div>
<!-- ================================================================
HEADER
================================================================
Standard header with title (center).
================================================================ -->
<div id="divHeader">
<div id="divHeaderFlexLeft">
<!-- Hamburger is now separate fixed element -->
</div>
<div id="divHeaderFlexCenter">
<div class="divHeaderText"></div>
</div>
<div id="divHeaderFlexRight">
<!-- No button in header right - logout is in sidenav footer -->
</div>
</div>
<!-- ================================================================
BODY
================================================================
Main content area. Add your page-specific content here.
================================================================ -->
<div id="divBody">
</div>
<!-- ================================================================
FOOTER
================================================================
Three-section footer layout:
- Left: Relay status animations (HamburgerMorphing instances)
- Center: General status information
- Right: Additional information
================================================================ -->
<div id="divFooter">
<div id="divFooterLeft" class="divFooterBox"></div>
<div id="divFooterCenter" class="divFooterBox"></div>
<div id="divFooterRight" class="divFooterBox"></div>
<div id="divFooterBalance" class="divFooterBox">0 sats</div>
</div>
<!-- ================================================================
SIDENAV
================================================================
Slide-out navigation panel. Opens from left when hamburger clicked.
Uses flexbox layout to pin version bar to bottom.
Includes a version bar footer with theme toggle and logout buttons.
================================================================ -->
<div id="divSideNav">
<div id="divSideNavHeader">
<!-- No close button - use main hamburger to close -->
</div>
<div id="divSideNavBody">
<div id="divFiles"></div>
</div>
<div id="divAiSection" class="sidenavSection">
<div id="divAiSectionTitle" class="sidenavSectionTitle">AI</div>
<div id="divAiList" class="sidenavSectionList">
<div id="divAiProvidersList">No saved providers yet.</div>
</div>
</div>
<div id="divRelaySection">
<div id="divRelaySectionTitle">
リレー
</div>
<div id="divRelayList">
Loading relays...
</div>
</div>
<div id="divBlossomSection">
<div id="divBlossomSectionTitle">ブロッサム</div>
<div id="divBlossomList">Loading blossom servers...</div>
</div>
<div id="divVersionBar">
<span id="versionDisplay">v0.0.1</span>
<div id="divVersionBarButtons">
<button id="themeToggleButton" title="Toggle Dark/Light Mode">
<div id="themeToggleHamburgerContainer"></div>
</button>
<button id="logoutButton" title="Logout">
<div id="logoutHamburgerContainer"></div>
</button>
</div>
</div>
</div>
<!-- ================================================================
REQUIRED SCRIPTS
================================================================
These scripts must be loaded in this order:
1. nostr.bundle.js - Nostr tools library
2. nostr-lite.js - Authentication modal (nostr-login-lite)
================================================================ -->
<script src="./nostr.bundle.js"></script>
<script src="./nostr-lite.js"></script>
<script type="module">
/* ================================================================
IMPORTS
================================================================
Import shared NDK functionality from init-ndk.mjs:
- initNDKPage() - Initialize authentication and worker
- getPubkey() - Get current user's pubkey
- subscribe() - Create NDK subscriptions
- publishEvent() - Publish events via NDK
- disconnect() - Disconnect from worker
- getRelayData() - Get relay connection data
- getRelayStats() - Get relay activity statistics
Import HamburgerMorphing for animated icons
================================================================ */
import {
initNDKPage,
getPubkey, injectHeaderAvatar,
subscribe,
publishEvent,
disconnect,
getVersion,
updateVersionDisplay,
getUserSettings,
patchUserSettings,
onUserSettings
} from './js/init-ndk.mjs';
import { HamburgerMorphing } from "./hamburger_morphing/hamburger.mjs";
import { initFooterRelayStatus, updateFooterRelayStatus, initSidenavRelaySection, updateSidenavRelaySection, setRelayActivityState } from './js/relay-ui.mjs';
import { initBlossomSection, updateBlossomSection } from './js/blossom-ui.mjs';
import { initAiSectionWithLocalConfig } from './js/ai-ui.mjs';
// Version will be loaded asynchronously
const versionInfo = await getVersion();
const VERSION = versionInfo.VERSION;
console.log(`[template.html ${VERSION}] Loading...`);
/* ================================================================
GLOBAL VARIABLES
================================================================
Track state for hamburger menu, relay status, and theme.
================================================================ */
let updateIntervalId = null;
let currentPubkey = null;
/*
AUTH STATE MODEL (Template reference)
------------------------------------------------------------------
This template now demonstrates three auth modes for standalone pages:
- required (default):
Behaves like existing pages: login is required immediately.
- optional:
Page can render public/read-only data without login, but can still
prompt login later for user actions (publish, settings, etc).
- none:
Never auto-login on load (pure public page).
URL behavior in this template:
- If ?auth=required|optional|none is present, it wins.
- Otherwise, if URL includes ?npub=... or ?pubkey=..., mode defaults
to optional because pages with explicit profile targets are commonly
public-readable.
- Otherwise, mode defaults to required.
*/
let isAuthenticated = false;
let authMode = 'required';
let authedPageInitialized = false;
let relayActivityListenersBound = false;
// Hamburger menu
let hamburgerInstance = null;
let isNavOpen = false;
// Version bar buttons
let logoutHamburger = null;
let themeToggleHamburger = null;
let isDarkMode = false;
// App-wide user settings (NIP-78 kind 30078, d:user-settings)
let pageSettings = {};
let unsubscribeUserSettings = null;
/* ================================================================
DOM VARIABLES
================================================================
Cache DOM element references for better performance.
================================================================ */
const divBody = document.getElementById("divBody");
const divSideNav = document.getElementById("divSideNav");
const divSideNavBody = document.getElementById("divSideNavBody");
const divFooterCenter = document.getElementById("divFooterCenter");
const divFooterRight = document.getElementById("divFooterRight");
/* ================================================================
HAMBURGER MENU
================================================================
Initialize and control the animated hamburger menu.
================================================================ */
function initHamburgerMenu() {
hamburgerInstance = new HamburgerMorphing('#divSvgHam', {
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
hamburgerInstance.animateTo('burger');
}
/* ================================================================
SIDENAV FUNCTIONS
================================================================
Open/close sidenav with hamburger morphing animation.
================================================================ */
function openNav() {
divSideNav.style.zIndex = 3;
divSideNav.style.width = "clamp(400px, 50vw, 600px)";
isNavOpen = true;
if (hamburgerInstance) {
hamburgerInstance.animateTo('arrow_left');
}
// Initialize version bar buttons when sidenav opens (lazy load)
if (!logoutHamburger) {
logoutHamburger = new HamburgerMorphing('#logoutHamburgerContainer', {
size: 24,
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
logoutHamburger.animateTo('x');
}
if (!themeToggleHamburger) {
themeToggleHamburger = new HamburgerMorphing('#themeToggleHamburgerContainer', {
size: 24,
foreground: 'var(--primary-color)',
background: 'var(--secondary-color)',
hover: 'var(--accent-color)'
});
// Determine current theme
const savedTheme = localStorage.getItem('theme');
isDarkMode = savedTheme === 'dark' || document.body.classList.contains('dark-mode');
const initialShape = isDarkMode ? 'moon' : 'circle';
themeToggleHamburger.animateTo(initialShape);
}
}
function closeNav() {
divSideNav.style.width = "0vw";
divSideNav.style.zIndex = -1;
isNavOpen = false;
if (hamburgerInstance) {
hamburgerInstance.animateTo('burger');
}
}
function toggleNav() {
if (isNavOpen) {
closeNav();
} else {
openNav();
}
}
/* ================================================================
AUTH MODE HELPERS
================================================================
These functions are meant as reusable guidance for future pages.
================================================================ */
function hasTargetPubkeyInUrl() {
const params = new URLSearchParams(window.location.search || '');
const npub = String(params.get('npub') || '').trim();
const pubkey = String(params.get('pubkey') || '').trim();
return Boolean(npub || pubkey);
}
function resolveAuthModeFromUrl() {
const params = new URLSearchParams(window.location.search || '');
const explicitAuth = String(params.get('auth') || '').trim().toLowerCase();
if (explicitAuth === 'required' || explicitAuth === 'optional' || explicitAuth === 'none') {
return explicitAuth;
}
// Convention: explicit target profiles are public-readable by default.
if (hasTargetPubkeyInUrl()) {
return 'optional';
}
return 'required';
}
function isAuthRequiredError(error) {
const message = String(error?.message || error || '').toLowerCase();
return message.includes('authentication required');
}
async function initializeAuthentication(mode) {
// required: existing behavior, throw if auth fails.
if (mode === 'required') {
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
return;
}
// none: public page, no login attempt on load.
if (mode === 'none') {
isAuthenticated = false;
currentPubkey = null;
return;
}
// optional: try silent/normal init; if auth required, continue public.
try {
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
} catch (error) {
if (isAuthRequiredError(error)) {
console.log('[template.html] Optional auth mode: continuing unauthenticated');
isAuthenticated = false;
currentPubkey = null;
return;
}
throw error;
}
}
async function initializeAuthenticatedPageFeatures() {
if (!isAuthenticated || authedPageInitialized) return;
await injectHeaderAvatar(currentPubkey);
console.log('[template.html] Authenticated as:', currentPubkey);
// Hydrate app-wide user settings for this page
try {
pageSettings = await getUserSettings();
} catch (error) {
console.warn('[template.html] getUserSettings failed:', error);
pageSettings = {};
}
// Subscribe to live user settings updates (cross-tab + publish echoes)
if (!unsubscribeUserSettings) {
unsubscribeUserSettings = onUserSettings((settings) => {
pageSettings = settings || {};
// TODO: Re-render page-specific UI from pageSettings here.
});
}
// Initialize relay-dependent UI only once authenticated.
initFooterRelayStatus();
initSidenavRelaySection();
await initBlossomSection();
initAiSectionWithLocalConfig();
await UpdateFooter();
if (!updateIntervalId) {
updateIntervalId = setInterval(UpdateFooter, 1000);
}
// Relay activity listeners only matter after worker init/auth.
if (!relayActivityListenersBound) {
window.addEventListener('ndkRelayActivity', (event) => {
const { relayUrl, activity, stats } = event.detail;
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`, stats);
setRelayActivityState(relayUrl, activity);
});
window.addEventListener('message', (event) => {
if (event.data && event.data.type === 'relayActivity') {
const { relayUrl, activity } = event.data;
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`);
setRelayActivityState(relayUrl, activity);
}
});
relayActivityListenersBound = true;
}
authedPageInitialized = true;
}
async function promptLoginIfNeeded() {
if (isAuthenticated) return true;
await initNDKPage();
currentPubkey = await getPubkey();
isAuthenticated = true;
await initializeAuthenticatedPageFeatures();
return true;
}
/* ================================================================
UPDATE FOOTER
================================================================
Update footer sections with relay status, pubkey, and other info.
Called periodically by update loop.
================================================================ */
const UpdateFooter = async () => {
try {
// Update relay status visuals in footer and sidenav
await updateFooterRelayStatus();
await updateSidenavRelaySection();
await updateBlossomSection();
// Clear center and right sections
divFooterCenter.innerHTML = '';
divFooterRight.innerHTML = '';
} catch (error) {
console.error('[template.html] Error updating footer:', error);
}
};
/* ================================================================
LOGOUT
================================================================
Complete logout process:
1. Stop update loop
2. Disconnect from NDK worker
3. Logout from nostr-login-lite
4. Clear all storage (localStorage, sessionStorage, IndexedDB)
5. Reload page
================================================================ */
const Logout = async () => {
console.log("[template.html] Starting logout process...");
// Stop the update loop
if (updateIntervalId) {
clearInterval(updateIntervalId);
updateIntervalId = null;
}
// Disconnect from worker
disconnect();
// Logout from nostr-login-lite
if (window.NOSTR_LOGIN_LITE && window.NOSTR_LOGIN_LITE.logout) {
await window.NOSTR_LOGIN_LITE.logout();
}
// Clear all storage
localStorage.clear();
sessionStorage.clear();
// Clear IndexedDB
if (window.indexedDB) {
const databases = await window.indexedDB.databases();
for (const db of databases) {
if (db.name) {
window.indexedDB.deleteDatabase(db.name);
}
}
}
console.log("[template.html] Logged out, reloading page");
location.reload(true);
};
/* ================================================================
EVENT LISTENERS
================================================================
Wire up UI interactions.
Main hamburger button click handler is set up in main() after initialization.
================================================================ */
/* ================================================================
SUBSCRIPTION EXAMPLE
================================================================
Example of how to subscribe to Nostr events:
const sub = subscribe(
{ kinds: [1], authors: [pubkey], limit: 10 },
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
);
Cache usage options:
- 'CACHE_FIRST' - Check cache first, then relays
- 'ONLY_RELAY' - Only query relays
- 'ONLY_CACHE' - Only query cache
- 'PARALLEL' - Query cache and relays simultaneously
Listen for events via window events:
window.addEventListener('ndkEvent', (event) => {
const evt = event.detail;
console.log('Received event:', evt);
});
================================================================ */
/* ================================================================
PUBLISH EXAMPLE
================================================================
Example of how to publish a Nostr event:
const event = {
created_at: Math.floor(Date.now() / 1000),
kind: 1,
tags: [],
content: "Hello, Nostr!"
};
try {
const result = await publishEvent(event);
console.log("✅ Published to:", result.relayResults.successful);
console.log("❌ Failed:", result.relayResults.failed);
console.log("Total relays:", result.totalRelays);
} catch (error) {
console.error("Publish error:", error);
}
Note: Events are automatically signed by the NDK worker using
the message-based signer (which calls window.nostr.signEvent).
================================================================ */
/* ================================================================
USER SETTINGS EXAMPLE (NIP-78)
================================================================
Read/subscribe/write helper pattern for all pages:
// Read latest merged settings (cache + relay hydrated by worker)
const settings = await getUserSettings();
// Subscribe to cross-tab updates
const unsubscribe = onUserSettings((nextSettings) => {
// Re-render page from nextSettings
});
// Patch only your feature namespace
await patchUserSettings({
myFeature: {
someFlag: true
}
});
// On page teardown (if applicable)
// unsubscribe();
================================================================ */
/* ================================================================
INITIALIZATION
================================================================
Main initialization sequence:
1. Initialize hamburger menu
2. Set up hamburger click handler
3. Resolve auth mode from URL/query policy
4. Initialize authentication based on mode
5. Initialize authenticated-only features (if signed in)
6. Set up version bar button listeners
7. Restore sidenav state
8. Update version display
Notes:
- required mode = existing behavior (prompt login on load)
- optional mode = allow public load, login later on demand
- none mode = no auto-login on load
================================================================ */
(async function main() {
console.log("[template.html] Starting initialization...");
try {
// Initialize hamburger menu first
initHamburgerMenu();
// Add click handler to hamburger
const divSvgHam = document.getElementById('divSvgHam');
if (divSvgHam) {
divSvgHam.addEventListener('click', toggleNav);
}
// Initialize version bar buttons
const themeToggleButton = document.getElementById('themeToggleButton');
const logoutButton = document.getElementById('logoutButton');
if (themeToggleButton) {
themeToggleButton.addEventListener('click', () => {
isDarkMode = !isDarkMode;
if (isDarkMode) {
localStorage.setItem('theme', 'dark');
} else {
localStorage.setItem('theme', 'light');
}
// Save sidenav state before reload
localStorage.setItem('sidenavWasOpen', isNavOpen ? 'true' : 'false');
window.location.reload();
});
}
if (logoutButton) {
logoutButton.addEventListener('click', async () => {
try {
// In optional/none modes this doubles as a "Sign in" entry point.
if (!isAuthenticated) {
await promptLoginIfNeeded();
return;
}
await Logout();
} catch (error) {
console.error('Logout/login action failed:', error);
}
});
}
// Resolve and initialize page auth policy.
authMode = resolveAuthModeFromUrl();
console.log('[template.html] Resolved auth mode:', authMode);
await initializeAuthentication(authMode);
// Initialize authenticated features only when signed in.
await initializeAuthenticatedPageFeatures();
/* ============================================================
EXAMPLE: Subscribe to events
============================================================
Uncomment to subscribe to user's notes:
const notesSub = subscribe(
{ kinds: [1], authors: [currentPubkey], limit: 10 },
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
);
console.log("[template.html] Subscribed to kind 1");
============================================================ */
/* ============================================================
EXAMPLE: Listen for events from worker
============================================================
Uncomment to handle incoming events:
window.addEventListener('ndkEvent', (event) => {
const evt = event.detail;
console.log("[template.html] Received event:", evt.kind, evt.pubkey);
if (evt.pubkey === currentPubkey) {
if (evt.kind === 1) {
// Handle note event
console.log("Note:", evt.content);
}
}
});
============================================================ */
/* ============================================================
EXAMPLE: Listen for cached profile
============================================================
Uncomment to handle cached profile data:
window.addEventListener('ndkProfile', (event) => {
console.log("[template.html] Cached profile:", event.detail);
// event.detail contains profile object (name, about, etc.)
});
============================================================ */
// Restore sidenav state if it was open before theme toggle
const sidenavWasOpen = localStorage.getItem('sidenavWasOpen');
if (sidenavWasOpen === 'true') {
localStorage.removeItem('sidenavWasOpen');
openNav();
}
// Optional UX note for public mode pages.
if (!isAuthenticated && (authMode === 'optional' || authMode === 'none')) {
divFooterCenter.textContent = 'Public mode';
divFooterRight.textContent = 'Sign in from side menu for private features';
}
// Update version display
await updateVersionDisplay();
console.log('[template.html] Initialization complete');
} catch (error) {
console.error('[template.html] Initialization failed:', error);
divBody.innerHTML = `<div style="text-align: center; padding: 50px;">
<div style="font-size: 24px; margin-bottom: 20px; color: red;">❌ Authentication Error</div>
<div style="font-size: 16px; color: #666;">${error.message}</div>
<div style="margin-top: 20px;">
<button onclick="location.reload()" style="padding: 10px 20px; font-size: 16px;">Retry</button>
</div>
</div>`;
}
})();
/* ================================================================
WORKER MESSAGE TYPES
================================================================
The NDK worker can send these message types:
1. 'response' - Response to init/subscribe/publish requests
- data.profile - User profile (from init)
- data.relays - User relays (from init)
- data.success - Publish success status
- data.relayResults - Relay publish results
2. 'event' - Nostr event from subscription
- Dispatched as 'ndkEvent' window event
- event.detail contains the Nostr event
3. 'eose' - End of stored events for subscription
- Dispatched as 'ndkEose' window event
- event.detail.subId contains subscription ID
4. 'signRequest' - Request to sign event/encrypt/decrypt
- Handled automatically by init-ndk.mjs
- Calls window.nostr methods and sends response
5. 'error' - Error from worker
- Logged to console automatically
6. 'relayActivity' - Relay read/write activity notification
- Dispatched as 'ndkRelayActivity' window event
- Used to animate relay status icons in footer
================================================================ */
/* ================================================================
DISTRIBUTED ARCHITECTURE NOTES
================================================================
Each page is independently accessible and self-contained:
1. Authentication persists via nostr-login-lite localStorage
- Login once on any page
- All other pages automatically authenticated
2. NDK SharedWorker is shared across all tabs/pages
- Single NDK instance manages all connections
- Subscriptions from all pages handled by one worker
- Events broadcast to all connected pages
3. Dexie cache is shared across all pages
- IndexedDB persists across sessions
- Cache-first queries are fast
- Reduces relay load
4. User settings are centralized and shared
- Worker hydrates kind 30078 (`d:user-settings`) on init
- Pages read via getUserSettings()
- Pages patch via patchUserSettings({ featureNamespace: ... })
- Pages subscribe via onUserSettings() for live updates
5. Each page can be distributed independently
- Copy template.html and customize
- No dependencies on other pages
- Works standalone or as part of suite
6. Message-based signer bridges worker and page
- Worker's NDK uses MessageBasedSigner
- Signer sends sign requests to page
- Page calls window.nostr.signEvent()
- Response sent back to worker
- NDK completes signing and publishing
7. Relay status visualization
- Footer left section shows connected relays
- Each relay has animated icon (HamburgerMorphing)
- Icons morph based on activity (read/write)
- Temporary animations show real-time activity
================================================================ */
</script>
</body>
</html>
+49 -13
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Build fully static MUSL binaries for C-Relay using Alpine Docker
# Build fully static MUSL binaries for C-Relay-PG using Alpine Docker
# Produces truly portable binaries with zero runtime dependencies
set -e
@@ -11,19 +11,52 @@ DOCKERFILE="$SCRIPT_DIR/Dockerfile.alpine-musl"
# Parse command line arguments
DEBUG_BUILD=false
if [[ "$1" == "--debug" ]]; then
DEBUG_BUILD=true
DB_BACKEND="${DB_BACKEND:-postgres}"
while [[ $# -gt 0 ]]; do
case "$1" in
--debug)
DEBUG_BUILD=true
shift
;;
--db-backend)
if [[ -z "$2" ]]; then
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
exit 1
fi
DB_BACKEND="$2"
shift 2
;;
--db-backend=*)
DB_BACKEND="${1#*=}"
shift
;;
*)
echo "ERROR: Unknown argument: $1"
echo "Usage: $0 [--debug] [--db-backend postgres|sqlite]"
exit 1
;;
esac
done
if [[ "$DB_BACKEND" != "sqlite" && "$DB_BACKEND" != "postgres" ]]; then
echo "ERROR: Invalid DB backend '$DB_BACKEND'. Use sqlite or postgres."
exit 1
fi
if [[ "$DEBUG_BUILD" == "true" ]]; then
echo "=========================================="
echo "C-Relay MUSL Static Binary Builder (DEBUG MODE)"
echo "C-Relay-PG MUSL Static Binary Builder (DEBUG MODE)"
echo "=========================================="
else
echo "=========================================="
echo "C-Relay MUSL Static Binary Builder (PRODUCTION MODE)"
echo "C-Relay-PG MUSL Static Binary Builder (PRODUCTION MODE)"
echo "=========================================="
fi
echo "Project directory: $SCRIPT_DIR"
echo "Build directory: $BUILD_DIR"
echo "Debug build: $DEBUG_BUILD"
echo "DB backend: $DB_BACKEND"
echo ""
# Create build directory
@@ -63,17 +96,17 @@ ARCH=$(uname -m)
case "$ARCH" in
x86_64)
PLATFORM="linux/amd64"
OUTPUT_NAME="c_relay_static_x86_64"
OUTPUT_NAME="c_relay_pg_static_x86_64"
;;
aarch64|arm64)
PLATFORM="linux/arm64"
OUTPUT_NAME="c_relay_static_arm64"
OUTPUT_NAME="c_relay_pg_static_arm64"
;;
*)
echo "WARNING: Unknown architecture: $ARCH"
echo "Defaulting to linux/amd64"
PLATFORM="linux/amd64"
OUTPUT_NAME="c_relay_static_${ARCH}"
OUTPUT_NAME="c_relay_pg_static_${ARCH}"
;;
esac
@@ -116,14 +149,15 @@ echo "=========================================="
echo "This will:"
echo " - Use Alpine Linux (native MUSL)"
echo " - Build all dependencies statically"
echo " - Compile c-relay with full static linking"
echo " - Compile c-relay-pg with full static linking"
echo ""
$DOCKER_CMD build \
--platform "$PLATFORM" \
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
--build-arg DB_BACKEND=$DB_BACKEND \
-f "$DOCKERFILE" \
-t c-relay-musl-builder:latest \
-t c-relay-pg-musl-builder:latest \
--progress=plain \
. || {
echo ""
@@ -145,16 +179,17 @@ echo "=========================================="
$DOCKER_CMD build \
--platform "$PLATFORM" \
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
--build-arg DB_BACKEND=$DB_BACKEND \
--target builder \
-f "$DOCKERFILE" \
-t c-relay-static-builder-stage:latest \
-t c-relay-pg-static-builder-stage:latest \
. > /dev/null 2>&1
# Create a temporary container to copy the binary
CONTAINER_ID=$($DOCKER_CMD create c-relay-static-builder-stage:latest)
CONTAINER_ID=$($DOCKER_CMD create c-relay-pg-static-builder-stage:latest)
# Copy binary from container
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_static" "$BUILD_DIR/$OUTPUT_NAME" || {
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_pg_static" "$BUILD_DIR/$OUTPUT_NAME" || {
echo "ERROR: Failed to extract binary from container"
$DOCKER_CMD rm "$CONTAINER_ID" 2>/dev/null
exit 1
@@ -224,6 +259,7 @@ if [ "$DEBUG_BUILD" = true ]; then
else
echo "Build Type: PRODUCTION (optimized, stripped)"
fi
echo "DB Backend: $DB_BACKEND"
if [ "$TRULY_STATIC" = true ]; then
echo "Linkage: Fully static binary (Alpine MUSL-based)"
echo "Portability: Works on ANY Linux distribution"
-9970
View File
File diff suppressed because it is too large Load Diff
+4 -4
View File
@@ -1,12 +1,12 @@
#!/bin/bash
# Restart the service
sudo systemctl stop c-relay.service
sudo systemctl stop c-relay-pg.service
# Copy the binary to the deployment location
cp build/c_relay_static_x86_64 ~/Storage/c_relay/crelay
cp build/c_relay_pg_static_x86_64 ~/Storage/c_relay_pg/crelay
# Restart the service
sudo systemctl restart c-relay.service
sudo systemctl restart c-relay-pg.service
# Show service status
sudo systemctl status c-relay.service --no-pager -l
sudo systemctl status c-relay-pg.service --no-pager -l
+88 -18
View File
@@ -1,28 +1,98 @@
#!/bin/bash
# C-Relay Static Binary Deployment Script
# Deploys build/c_relay_static_x86_64 to server via ssh
# C-Relay-PG PostgreSQL Deployment Script
# Deploys static relay binary and configures PostgreSQL 18 + systemd service on remote server.
set -e
set -euo pipefail
# Configuration
LOCAL_BINARY="build/c_relay_static_x86_64"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay/c_relay"
SERVICE_NAME="c-relay"
REMOTE_HOST="ubuntu@laantungir.net"
LOCAL_BINARY="build/c_relay_pg_static_x86_64"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
SERVICE_NAME="c-relay-pg"
# Create backup
ssh ubuntu@laantungir.com "sudo cp '$REMOTE_BINARY_PATH' '${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)'" 2>/dev/null || true
LOCAL_SERVICE_FILE="systemd/c-relay.service"
LOCAL_PG_SETUP_SCRIPT="systemd/setup_postgres_18.sh"
# Upload binary to temp location
scp "$LOCAL_BINARY" "ubuntu@laantungir.com:/tmp/c_relay.tmp"
if [ ! -f "$LOCAL_BINARY" ]; then
echo "ERROR: Binary not found: $LOCAL_BINARY"
echo "Build it first (e.g. ./make_and_restart_relay.sh)"
exit 1
fi
# Install binary
ssh ubuntu@laantungir.com "sudo mv '/tmp/c_relay.tmp' '$REMOTE_BINARY_PATH'"
ssh ubuntu@laantungir.com "sudo chown c-relay:c-relay '$REMOTE_BINARY_PATH'"
ssh ubuntu@laantungir.com "sudo chmod +x '$REMOTE_BINARY_PATH'"
if [ ! -f "$LOCAL_SERVICE_FILE" ]; then
echo "ERROR: Service file not found: $LOCAL_SERVICE_FILE"
exit 1
fi
# Reload systemd and restart service
ssh ubuntu@laantungir.com "sudo systemctl daemon-reload"
ssh ubuntu@laantungir.com "sudo systemctl restart '$SERVICE_NAME'"
if [ ! -f "$LOCAL_PG_SETUP_SCRIPT" ]; then
echo "ERROR: PostgreSQL setup script not found: $LOCAL_PG_SETUP_SCRIPT"
exit 1
fi
echo "Deployment complete!"
echo "==> Uploading artifacts to $REMOTE_HOST"
scp "$LOCAL_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg.tmp"
scp "$LOCAL_SERVICE_FILE" "$REMOTE_HOST:/tmp/c-relay-pg.service"
scp "$LOCAL_PG_SETUP_SCRIPT" "$REMOTE_HOST:/tmp/setup_postgres_18.sh"
echo "==> Running remote install/configuration"
ssh "$REMOTE_HOST" 'bash -s' <<'EOF'
set -euo pipefail
SERVICE_NAME="c-relay-pg"
RELAY_USER="c-relay-pg"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
echo "[remote] Ensuring service user exists"
if ! id "$RELAY_USER" >/dev/null 2>&1; then
sudo useradd --system --home-dir /opt/c-relay-pg --shell /usr/sbin/nologin "$RELAY_USER"
fi
echo "[remote] Ensuring required directories exist"
sudo mkdir -p "$REMOTE_BINARY_DIR" /opt/c-relay-pg /etc/c-relay-pg
sudo chown "$RELAY_USER:$RELAY_USER" /opt/c-relay-pg
echo "[remote] Installing PostgreSQL 18 (PGDG) if missing"
if ! dpkg -s postgresql-18 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y curl ca-certificates lsb-release gnupg
sudo install -d /usr/share/postgresql-common/pgdg
sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null
sudo apt-get update
sudo apt-get install -y postgresql-18
fi
sudo systemctl enable postgresql
sudo systemctl restart postgresql
echo "[remote] Configuring PostgreSQL role/database"
sudo chmod +x /tmp/setup_postgres_18.sh
sudo /tmp/setup_postgres_18.sh
echo "[remote] Installing relay binary"
if [ -f "$REMOTE_BINARY_PATH" ]; then
sudo cp "$REMOTE_BINARY_PATH" "${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)"
fi
sudo mv /tmp/c_relay_pg.tmp "$REMOTE_BINARY_PATH"
sudo chown "$RELAY_USER:$RELAY_USER" "$REMOTE_BINARY_PATH"
sudo chmod +x "$REMOTE_BINARY_PATH"
echo "[remote] Installing systemd unit"
sudo mv /tmp/c-relay-pg.service /etc/systemd/system/c-relay-pg.service
sudo chown root:root /etc/systemd/system/c-relay-pg.service
sudo chmod 644 /etc/systemd/system/c-relay-pg.service
echo "[remote] Reloading and restarting service"
sudo systemctl daemon-reload
sudo systemctl enable "$SERVICE_NAME"
sudo systemctl restart "$SERVICE_NAME"
echo "[remote] Health checks"
sudo systemctl --no-pager --full status "$SERVICE_NAME" | sed -n '1,25p'
sudo -u "$RELAY_USER" psql -d crelay -c "SELECT current_user, current_database();"
EOF
echo "Deployment complete: $REMOTE_HOST"
+15 -15
View File
@@ -1,14 +1,14 @@
#!/bin/bash
# C-Relay Debug Binary Deployment Script
# Deploys build/c_relay_static_x86_64_debug to server for CPU profiling
# C-Relay-PG Debug Binary Deployment Script
# Deploys build/c_relay_pg_static_x86_64_debug to server for CPU profiling
#
# Usage:
# ./deploy_lt_debug.sh -- deploy debug binary and restart
# ./deploy_lt_debug.sh --profile -- deploy, then run perf and fetch results
#
# After deploying, profile with:
# sudo perf record -g -p $(pgrep c_relay) -- sleep 30
# sudo perf record -g -p $(pgrep c_relay_pg) -- sleep 30
# sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -80
#
# Restore production binary:
@@ -16,10 +16,10 @@
set -e
LOCAL_DEBUG_BINARY="build/c_relay_static_x86_64_debug"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay/c_relay"
REMOTE_PROD_BACKUP="/usr/local/bin/c_relay/c_relay.production"
SERVICE_NAME="c-relay"
LOCAL_DEBUG_BINARY="build/c_relay_pg_static_x86_64_debug"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
REMOTE_PROD_BACKUP="/usr/local/bin/c_relay_pg/c_relay_pg.production"
SERVICE_NAME="c-relay-pg"
REMOTE_HOST="ubuntu@laantungir.com"
# Check debug binary exists
@@ -33,7 +33,7 @@ if [ ! -f "$LOCAL_DEBUG_BINARY" ]; then
fi
echo "=========================================="
echo "C-Relay Debug Deployment"
echo "C-Relay-PG Debug Deployment"
echo "=========================================="
echo "Binary: $LOCAL_DEBUG_BINARY ($(du -h "$LOCAL_DEBUG_BINARY" | cut -f1))"
echo "Target: $REMOTE_HOST:$REMOTE_BINARY_PATH"
@@ -55,18 +55,18 @@ ssh "$REMOTE_HOST" "
# Upload debug binary
echo "Uploading debug binary..."
scp "$LOCAL_DEBUG_BINARY" "$REMOTE_HOST:/tmp/c_relay_debug.tmp"
scp "$LOCAL_DEBUG_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg_debug.tmp"
# Install debug binary
echo "Installing debug binary..."
ssh "$REMOTE_HOST" "
sudo mv '/tmp/c_relay_debug.tmp' '$REMOTE_BINARY_PATH'
sudo chown c-relay:c-relay '$REMOTE_BINARY_PATH'
sudo mv '/tmp/c_relay_pg_debug.tmp' '$REMOTE_BINARY_PATH'
sudo chown c-relay-pg:c-relay-pg '$REMOTE_BINARY_PATH'
sudo chmod +x '$REMOTE_BINARY_PATH'
"
# Restart service
echo "Restarting c-relay service..."
echo "Restarting c-relay-pg service..."
ssh "$REMOTE_HOST" "sudo systemctl daemon-reload && sudo systemctl restart '$SERVICE_NAME'"
echo ""
@@ -84,9 +84,9 @@ if [ "$1" = "--profile" ]; then
sleep 3
ssh "$REMOTE_HOST" "
PID=\$(pgrep c_relay)
PID=\$(pgrep c_relay_pg)
if [ -z \"\$PID\" ]; then
echo 'ERROR: c_relay not running'
echo 'ERROR: c_relay_pg not running'
exit 1
fi
echo \"Profiling PID \$PID for 30 seconds...\"
@@ -101,7 +101,7 @@ else
echo "=========================================="
echo ""
echo "SSH to server and run:"
echo " sudo perf record -g -p \$(pgrep c_relay) -- sleep 30"
echo " sudo perf record -g -p \$(pgrep c_relay_pg) -- sleep 30"
echo " sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -60"
echo ""
echo "Or run with --profile to do it automatically:"
+2 -2
View File
@@ -1,4 +1,4 @@
# C-Relay Administrator API Implementation Plan
# C-Relay-PG Administrator API Implementation Plan
## Problem Analysis
@@ -161,7 +161,7 @@ Would require changing schema, migration scripts, and storage logic.
#### README.md Documentation Format:
```markdown
# C-Relay Administrator API
# C-Relay-PG Administrator API
## Authentication
All admin commands require signing with the admin private key generated during first startup.
+190
View File
@@ -0,0 +1,190 @@
# Agent Browser Testing Guide for C-Relay-PG
This document explains how to use the `agent-browser` CLI tool to test the c-relay-pg admin web UI from an AI agent context (no physical display required).
## Prerequisites
- **agent-browser** installed globally: `npm install -g agent-browser`
- Binary location: `/home/user/.nvm/versions/node/v24.14.1/bin/agent-browser`
- The relay must be running locally (default port 8888)
- Test keys configured in `.test_keys`
## Starting the Relay for Local Testing
```bash
./make_and_restart_relay.sh -t
```
This reads `.test_keys` and starts the relay with:
- `ADMIN_PUBKEY` — the hex public key of the admin account
- `ADMIN_PRIVKEY` — the hex secret key (used for browser login, not by the relay itself)
- `SERVER_PRIVKEY` — the relay's own private key
## Key URLs
| URL | Purpose |
|-----|---------|
| `http://127.0.0.1:8888/api/index.html` | Admin web UI (the correct entry point) |
| `http://127.0.0.1:8888/` | Returns 406 without NIP-11 Accept header — **do not use for browser testing** |
| `http://127.0.0.1:8888/` with `Accept: application/nostr+json` | NIP-11 relay info JSON |
**Important:** The root URL `/` is a WebSocket/NIP-11 endpoint, not an HTML page. Always use `/api/index.html` for browser testing.
## Admin Login Credentials
The admin nsec for the current `.test_keys` configuration:
```
nsec: nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm
hex: 15a6fbfd759330c9948ba07d9ed77e49d414c4377a2516d47fc0ab0d54d411f5
npub: npub1tlyzk6slzt8d989f4pn3synk98fea64ea3jmrdc7dtd0kw8uxlas9a9fwr
hex pubkey: 5fc82b6a1f12ced29ca9a86718127629d39eeab9ec65b1b71e6adafb38fc37fb
```
## Complete Login Flow with agent-browser
### Step 1: Open the admin UI
```bash
agent-browser open http://127.0.0.1:8888/api/index.html
agent-browser wait --load networkidle
```
### Step 2: Take an interactive snapshot to see what is on screen
```bash
agent-browser snapshot -i
```
You will see a login modal with buttons like:
- `"Browser Extension"` — skip this
- `"Local Key"`**use this one**
- `"Seed Phrase"` — skip
- `"Nostr Connect"` — skip
- `"Read Only"` — skip
### Step 3: Click "Local Key"
```bash
agent-browser click @e15
```
(The ref number may vary — use the ref from the snapshot output for the "Local Key" button.)
After clicking, a text input appears asking for the secret key.
### Step 4: Enter the admin nsec
```bash
agent-browser fill @e14 "nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm"
```
(Use the ref from the snapshot for the textbox element.)
### Step 5: Click "Import Key"
```bash
agent-browser snapshot -i
```
Check the snapshot — the "Import Key" button should now be enabled. Click it:
```bash
agent-browser click @e15
```
### Step 6: Click "Continue" on the success screen
After import, a success screen appears with "Continue" button:
```bash
agent-browser wait 800
agent-browser snapshot -i
agent-browser click @e19
```
(Use the ref from the snapshot for the "Continue" button.)
### Step 7: Wait for admin UI to load
```bash
agent-browser wait 5000
agent-browser snapshot -i
```
You should now see the admin dashboard with:
- Statistics table (Database Size, Total Events, PID, etc.)
- Navigation buttons (Statistics, Subscriptions, Configuration, Authorization, etc.)
- Admin profile area showing "admin" label
## Navigating Admin Sections
After login, use the sidebar navigation buttons:
```bash
# View configuration
agent-browser click @e8 # Configuration button ref
# View authorization rules
agent-browser click @e9 # Authorization button ref
# View statistics
agent-browser click @e6 # Statistics button ref
# Always snapshot after navigation to see results
agent-browser wait 2500
agent-browser snapshot -i
```
## Checking for Errors
```bash
# View browser console logs
agent-browser console
# View JavaScript errors
agent-browser errors
# View relay server logs
tail -n 100 relay.log
```
## Chained Command Example (Full Login in One Shot)
```bash
agent-browser open http://127.0.0.1:8888/api/index.html && \
agent-browser wait --load networkidle && \
agent-browser snapshot -i
```
Then use refs from the snapshot to complete login steps.
## Tips for AI Agents
1. **Always use `/api/index.html`** — never the root URL
2. **Use `snapshot -i`** after every action to see the current interactive elements and their refs
3. **Refs change** between snapshots — always re-snapshot before clicking
4. **Wait after clicks** — use `agent-browser wait 2000` (milliseconds) between actions that trigger async operations
5. **The login flow has 3 screens**: method selection → key input → success confirmation
6. **Console logs are cumulative** — they show all logs since page load, which is useful for debugging admin API responses
7. **The relay log** at `relay.log` shows server-side processing of admin commands
8. **Command chaining** with `&&` works — the browser daemon persists between commands
## Verifying Admin API is Working
After login, the statistics page should show populated data:
- Database Size (e.g., "4 KB")
- Process ID (the relay PID)
- WebSocket Connections count
- Memory Usage
If these show "-" or "Loading...", check:
1. `relay.log` for errors
2. `agent-browser console` for JavaScript errors
3. `agent-browser errors` for page-level errors
## Closing the Browser
```bash
agent-browser close --all
```
+11 -11
View File
@@ -324,7 +324,7 @@ int main() {
}
```
## Migration Plan for c-relay
## Migration Plan for c-relay-pg
### Phase 1: Extract Debug System
1. Create `c_utils_lib` repository
@@ -333,16 +333,16 @@ int main() {
4. Add basic tests
### Phase 2: Add Versioning System
1. Extract version generation logic from c-relay
1. Extract version generation logic from c-relay-pg
2. Create reusable version utilities
3. Update c-relay to use new system
3. Update c-relay-pg to use new system
4. Update nostr_core_lib to use new system
### Phase 3: Add as Submodule
1. Add `c_utils_lib` as submodule to c-relay
2. Update c-relay Makefile
3. Update includes in c-relay source files
4. Remove old debug files from c-relay
1. Add `c_utils_lib` as submodule to c-relay-pg
2. Update c-relay-pg Makefile
3. Update includes in c-relay-pg source files
4. Remove old debug files from c-relay-pg
### Phase 4: Documentation & Examples
1. Create comprehensive README
@@ -352,7 +352,7 @@ int main() {
## Benefits
### For c-relay
### For c-relay-pg
- Cleaner separation of concerns
- Reusable utilities across projects
- Easier to maintain and test
@@ -379,7 +379,7 @@ int main() {
- Memory leak detection (valgrind)
### Integration Tests
- Test with real projects (c-relay, nostr_core_lib)
- Test with real projects (c-relay-pg, nostr_core_lib)
- Cross-platform testing
- Performance benchmarks
@@ -433,7 +433,7 @@ MIT License - permissive and suitable for learning and commercial use.
## Success Criteria
1. ✅ Successfully integrated into c-relay
1. ✅ Successfully integrated into c-relay-pg
2. ✅ Successfully integrated into nostr_core_lib
3. ✅ All tests passing
4. ✅ Documentation complete
@@ -449,7 +449,7 @@ MIT License - permissive and suitable for learning and commercial use.
4. Create build system
5. Write tests
6. Create documentation
7. Integrate into c-relay
7. Integrate into c-relay-pg
8. Publish to GitHub
---
+16 -16
View File
@@ -2,13 +2,13 @@
## Overview
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay project.
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay-pg project.
## Phase 1: Repository Setup & Structure
### Step 1.1: Create Repository Structure
**Location**: Create outside c-relay project (sibling directory)
**Location**: Create outside c-relay-pg project (sibling directory)
```bash
# Create directory structure
@@ -42,7 +42,7 @@ git branch -M main
### Step 2.1: Move Debug Files
**Source files** (from c-relay):
**Source files** (from c-relay-pg):
- `src/debug.c``c_utils_lib/src/debug.c`
- `src/debug.h``c_utils_lib/include/c_utils/debug.h`
@@ -484,20 +484,20 @@ How to integrate into projects:
3. Code examples
4. Migration from standalone utilities
## Phase 7: Integration with c-relay
## Phase 7: Integration with c-relay-pg
### Step 7.1: Add as Submodule
```bash
cd /path/to/c-relay
cd /path/to/c-relay-pg
git submodule add <repo-url> c_utils_lib
git submodule update --init --recursive
```
### Step 7.2: Update c-relay Makefile
### Step 7.2: Update c-relay-pg Makefile
```makefile
# Add to c-relay Makefile
# Add to c-relay-pg Makefile
C_UTILS_LIB = c_utils_lib/libc_utils.a
# Update includes
@@ -514,7 +514,7 @@ $(C_UTILS_LIB):
$(TARGET): $(C_UTILS_LIB) ...
```
### Step 7.3: Update c-relay Source Files
### Step 7.3: Update c-relay-pg Source Files
**Changes needed**:
@@ -543,7 +543,7 @@ $(TARGET): $(C_UTILS_LIB) ...
### Step 7.4: Test Integration
```bash
cd c-relay
cd c-relay-pg
make clean
make
./make_and_restart_relay.sh
@@ -556,7 +556,7 @@ Verify:
## Phase 8: Version System Integration
### Step 8.1: Update c-relay Makefile for Versioning
### Step 8.1: Update c-relay-pg Makefile for Versioning
```makefile
# Add version generation
@@ -567,7 +567,7 @@ src/version.h: .git/refs/tags/*
$(TARGET): src/version.h ...
```
### Step 8.2: Update c-relay to Use Generated Version
### Step 8.2: Update c-relay-pg to Use Generated Version
Replace hardcoded version in `src/main.h` with:
```c
@@ -583,7 +583,7 @@ Replace hardcoded version in `src/main.h` with:
- **Phase 4**: Build System - 2 hours
- **Phase 5**: Examples & Tests - 3 hours
- **Phase 6**: Documentation - 3 hours
- **Phase 7**: c-relay Integration - 2 hours
- **Phase 7**: c-relay-pg Integration - 2 hours
- **Phase 8**: Version Integration - 2 hours
**Total**: ~19 hours
@@ -593,11 +593,11 @@ Replace hardcoded version in `src/main.h` with:
- [ ] c_utils_lib builds successfully
- [ ] All tests pass
- [ ] Examples compile and run
- [ ] c-relay integrates successfully
- [ ] Debug output works in c-relay
- [ ] c-relay-pg integrates successfully
- [ ] Debug output works in c-relay-pg
- [ ] Version generation works
- [ ] Documentation complete
- [ ] No regressions in c-relay functionality
- [ ] No regressions in c-relay-pg functionality
## Next Steps
@@ -608,7 +608,7 @@ Replace hardcoded version in `src/main.h` with:
5. Create build system
6. Write tests and examples
7. Create documentation
8. Integrate into c-relay
8. Integrate into c-relay-pg
9. Test thoroughly
10. Publish to GitHub
+6 -6
View File
@@ -78,9 +78,9 @@ Configuration events follow the standard Nostr event format with kind 33334:
#### `relay_software`
- **Description**: Software identifier for NIP-11
- **Default**: `"c-relay"`
- **Default**: `"c-relay-pg"`
- **Format**: String, max 64 characters
- **Example**: `"c-relay v1.0.0"`
- **Example**: `"c-relay-pg v1.0.0"`
#### `relay_version`
- **Description**: Software version string
@@ -366,7 +366,7 @@ sqlite3 relay.nrdb "SELECT json_pretty(json_object(
#### Invalid Parameter Values
```bash
# Check relay logs for validation errors
journalctl -u c-relay | grep "Configuration.*invalid\|Invalid.*configuration"
journalctl -u c-relay-pg | grep "Configuration.*invalid\|Invalid.*configuration"
# Common issues:
# - Numeric values outside valid ranges
@@ -395,10 +395,10 @@ ORDER BY date DESC;"
#### Resource Usage After Changes
```bash
# Monitor system resources after configuration updates
top -p $(pgrep c_relay)
top -p $(pgrep c_relay_pg)
# Check for memory leaks
ps aux | grep c_relay | awk '{print $6}' # RSS memory
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory
```
### Emergency Recovery
@@ -425,7 +425,7 @@ nostrtool event \
# If database is corrupted, backup and recreate
cp relay.nrdb relay.nrdb.backup
rm relay.nrdb*
./build/c_relay_x86 # Creates fresh database with new keys
./build/c_relay_pg_x86 # Creates fresh database with new keys
```
---
+14 -14
View File
@@ -21,22 +21,22 @@ typedef enum {
```bash
# Production (default - no debug output)
./c_relay_x86
./c_relay_pg_x86
# Show errors only
./c_relay_x86 --debug-level=1
./c_relay_pg_x86 --debug-level=1
# Show errors and warnings
./c_relay_x86 --debug-level=2
./c_relay_pg_x86 --debug-level=2
# Show errors, warnings, and info (recommended for development)
./c_relay_x86 --debug-level=3
./c_relay_pg_x86 --debug-level=3
# Show all debug messages
./c_relay_x86 --debug-level=4
./c_relay_pg_x86 --debug-level=4
# Show everything including trace with file:line (very verbose)
./c_relay_x86 --debug-level=5
./c_relay_pg_x86 --debug-level=5
```
## Implementation
@@ -475,7 +475,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 3 (Errors + Warnings + Info)
```
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [INFO ] Loading configuration from database
[2025-01-12 14:30:15] [ERROR] Failed to open database: permission denied
[2025-01-12 14:30:16] [WARN ] Port 8888 unavailable, trying 8889
@@ -484,7 +484,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 4 (All Debug Messages)
```
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [DEBUG] Opening database: build/abc123...def.db
[2025-01-12 14:30:15] [DEBUG] Executing schema initialization
[2025-01-12 14:30:15] [INFO ] SQLite WAL mode enabled
@@ -496,7 +496,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
### Level 5 (Everything Including file:line for ALL messages)
```
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay v0.4.6
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay-PG v0.4.6
[2025-01-12 14:30:15] [DEBUG] [main.c:348] Opening database: build/abc123...def.db
[2025-01-12 14:30:15] [TRACE] [main.c:330] Entering init_database()
[2025-01-12 14:30:15] [ERROR] [config.c:125] Database locked
@@ -525,11 +525,11 @@ Update the existing `log_*` functions to use the new debug macros
make clean && make
# Test different levels
./build/c_relay_x86 # No output
./build/c_relay_x86 --debug-level=1 # Errors only
./build/c_relay_x86 --debug-level=3 # Info + warnings + errors
./build/c_relay_x86 --debug-level=4 # All debug messages
./build/c_relay_x86 --debug-level=5 # Everything with file:line on TRACE
./build/c_relay_pg_x86 # No output
./build/c_relay_pg_x86 --debug-level=1 # Errors only
./build/c_relay_pg_x86 --debug-level=3 # Info + warnings + errors
./build/c_relay_pg_x86 --debug-level=4 # All debug messages
./build/c_relay_pg_x86 --debug-level=5 # Everything with file:line on TRACE
```
### Step 5: Gradual Migration (Ongoing)
+1 -1
View File
@@ -34,7 +34,7 @@ static const struct {
// NIP-11 Relay Information (relay keys will be populated at runtime)
{"relay_description", "High-performance C Nostr relay with SQLite storage"},
{"relay_contact", ""},
{"relay_software", "https://git.laantungir.net/laantungir/c-relay.git"},
{"relay_software", "https://git.laantungir.net/laantungir/c-relay-pg.git"},
{"relay_version", "v1.0.0"},
// NIP-13 Proof of Work (pow_min_difficulty = 0 means PoW disabled)
+51 -51
View File
@@ -52,11 +52,11 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
#### User and Directory Setup
```bash
# Create dedicated system user
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
# Create application directory
sudo mkdir -p /opt/c-relay
sudo chown c-relay:c-relay /opt/c-relay
sudo mkdir -p /opt/c-relay-pg
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg
```
### Build and Installation
@@ -64,8 +64,8 @@ sudo chown c-relay:c-relay /opt/c-relay
#### Automated Installation (Recommended)
```bash
# Clone repository
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
git submodule update --init --recursive
# Build
@@ -81,12 +81,12 @@ sudo systemd/install-service.sh
make clean && make
# Install binary
sudo cp build/c_relay_x86 /opt/c-relay/
sudo chown c-relay:c-relay /opt/c-relay/c_relay_x86
sudo chmod +x /opt/c-relay/c_relay_x86
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
sudo chmod +x /opt/c-relay-pg/c_relay_pg_x86
# Install systemd service
sudo cp systemd/c-relay.service /etc/systemd/system/
sudo cp systemd/c-relay-pg.service /etc/systemd/system/
sudo systemctl daemon-reload
```
@@ -95,22 +95,22 @@ sudo systemctl daemon-reload
#### Start and Enable Service
```bash
# Start the service
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Enable auto-start on boot
sudo systemctl enable c-relay
sudo systemctl enable c-relay-pg
# Check status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
```
#### Capture Admin Keys (CRITICAL)
```bash
# View startup logs to get admin keys
sudo journalctl -u c-relay --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
# Or check the full log
sudo journalctl -u c-relay --no-pager | grep "Admin Private Key"
sudo journalctl -u c-relay-pg --no-pager | grep "Admin Private Key"
```
⚠️ **CRITICAL**: Save the admin private key immediately - it's only shown once and is needed for all configuration updates!
@@ -151,8 +151,8 @@ sudo iptables-save > /etc/iptables/rules.v4
ssh -i your-key.pem ubuntu@your-instance-ip
# Use the simple deployment script
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -168,10 +168,10 @@ sudo examples/deployment/simple-vps/deploy.sh
sudo mkfs.ext4 /dev/xvdf
sudo mkdir /data
sudo mount /dev/xvdf /data
sudo chown c-relay:c-relay /data
sudo chown c-relay-pg:c-relay-pg /data
# Update systemd service to use /data
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay/WorkingDirectory=\/data/' /etc/systemd/system/c-relay.service
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay-pg/WorkingDirectory=\/data/' /etc/systemd/system/c-relay-pg.service
sudo systemctl daemon-reload
```
@@ -180,7 +180,7 @@ sudo systemctl daemon-reload
#### Compute Engine Setup
```bash
# Create VM instance (e2-micro or larger)
gcloud compute instances create c-relay-instance \
gcloud compute instances create c-relay-pg-instance \
--image-family=ubuntu-2204-lts \
--image-project=ubuntu-os-cloud \
--machine-type=e2-micro \
@@ -193,9 +193,9 @@ gcloud compute firewall-rules create allow-nostr-relay \
--target-tags nostr-relay
# SSH and deploy
gcloud compute ssh c-relay-instance
git clone https://github.com/your-org/c-relay.git
cd c-relay
gcloud compute ssh c-relay-pg-instance
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -203,13 +203,13 @@ sudo examples/deployment/simple-vps/deploy.sh
```bash
# Create and attach persistent disk
gcloud compute disks create relay-data --size=50GB
gcloud compute instances attach-disk c-relay-instance --disk=relay-data
gcloud compute instances attach-disk c-relay-pg-instance --disk=relay-data
# Format and mount
sudo mkfs.ext4 /dev/sdb
sudo mkdir /data
sudo mount /dev/sdb /data
sudo chown c-relay:c-relay /data
sudo chown c-relay-pg:c-relay-pg /data
```
### DigitalOcean
@@ -223,8 +223,8 @@ sudo chown c-relay:c-relay /data
ssh root@your-droplet-ip
# Deploy relay
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
examples/deployment/simple-vps/deploy.sh
```
@@ -245,8 +245,8 @@ The `examples/deployment/` directory contains ready-to-use scripts:
### Simple VPS Deployment
```bash
# Clone repository and run automated deployment
git clone https://github.com/your-org/c-relay.git
cd c-relay
git clone https://github.com/your-org/c-relay-pg.git
cd c-relay-pg
sudo examples/deployment/simple-vps/deploy.sh
```
@@ -417,9 +417,9 @@ LOG_FILE="/var/log/relay-monitor.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
# Check if relay is running
if ! pgrep -f "c_relay_x86" > /dev/null; then
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
echo "[$DATE] ERROR: Relay process not running" >> $LOG_FILE
systemctl restart c-relay
systemctl restart c-relay-pg
fi
# Check port availability
@@ -428,14 +428,14 @@ if ! netstat -tln | grep -q ":8888"; then
fi
# Check database file
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
if [[ -n "$RELAY_DB" ]]; then
DB_SIZE=$(du -h "$RELAY_DB" | cut -f1)
echo "[$DATE] INFO: Database size: $DB_SIZE" >> $LOG_FILE
fi
# Check memory usage
MEM_USAGE=$(ps aux | grep c_relay_x86 | grep -v grep | awk '{print $6}')
MEM_USAGE=$(ps aux | grep c_relay_pg_x86 | grep -v grep | awk '{print $6}')
if [[ -n "$MEM_USAGE" ]]; then
echo "[$DATE] INFO: Memory usage: ${MEM_USAGE}KB" >> $LOG_FILE
fi
@@ -454,8 +454,8 @@ sudo chmod +x /usr/local/bin/relay-monitor.sh
#### Centralized Logging with rsyslog
```bash
# /etc/rsyslog.d/50-c-relay.conf
if $programname == 'c-relay' then /var/log/c-relay.log
# /etc/rsyslog.d/50-c-relay-pg.conf
if $programname == 'c-relay-pg' then /var/log/c-relay-pg.log
& stop
```
@@ -465,7 +465,7 @@ if $programname == 'c-relay' then /var/log/c-relay.log
```yaml
# /etc/prometheus/prometheus.yml
scrape_configs:
- job_name: 'c-relay'
- job_name: 'c-relay-pg'
static_configs:
- targets: ['localhost:8888']
metrics_path: '/metrics' # If implemented
@@ -479,30 +479,30 @@ scrape_configs:
#### Service User Restrictions
```bash
# Restrict service user
sudo usermod -s /bin/false c-relay
sudo usermod -d /opt/c-relay c-relay
sudo usermod -s /bin/false c-relay-pg
sudo usermod -d /opt/c-relay-pg c-relay-pg
# Set proper permissions
sudo chmod 700 /opt/c-relay
sudo chown -R c-relay:c-relay /opt/c-relay
sudo chmod 700 /opt/c-relay-pg
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
```
#### File System Restrictions
```bash
# Mount data directory with appropriate options
echo "/dev/sdb /opt/c-relay ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
echo "/dev/sdb /opt/c-relay-pg ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
```
### Network Security
#### Fail2Ban Configuration
```ini
# /etc/fail2ban/jail.d/c-relay.conf
[c-relay-dos]
# /etc/fail2ban/jail.d/c-relay-pg.conf
[c-relay-pg-dos]
enabled = true
port = 8888
filter = c-relay-dos
logpath = /var/log/c-relay.log
filter = c-relay-pg-dos
logpath = /var/log/c-relay-pg.log
maxretry = 10
findtime = 60
bantime = 300
@@ -534,9 +534,9 @@ sudo mkfs.ext4 /dev/mapper/relay-data
#!/bin/bash
# /usr/local/bin/backup-relay.sh
BACKUP_DIR="/backup/c-relay"
BACKUP_DIR="/backup/c-relay-pg"
DATE=$(date +%Y%m%d_%H%M%S)
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
mkdir -p "$BACKUP_DIR"
@@ -574,7 +574,7 @@ sudo apt install -y awscli
aws configure
# Sync backups to S3
aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
aws s3 sync /backup/c-relay-pg/ s3://your-backup-bucket/c-relay-pg/ --delete
```
### Disaster Recovery
@@ -583,16 +583,16 @@ aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
```bash
# 1. Restore from backup
gunzip backup/relay_backup_20231201_020000.nrdb.gz
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay/
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay-pg/
# 2. Fix permissions
sudo chown c-relay:c-relay /opt/c-relay/*.nrdb
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/*.nrdb
# 3. Restart service
sudo systemctl restart c-relay
sudo systemctl restart c-relay-pg
# 4. Verify recovery
sudo journalctl -u c-relay --since "1 minute ago"
sudo journalctl -u c-relay-pg --since "1 minute ago"
```
---
+25 -25
View File
@@ -2,7 +2,7 @@
## Overview
This guide explains how to build truly portable MUSL-based static binaries of c-relay using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
This guide explains how to build truly portable MUSL-based static binaries of c-relay-pg using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
## Why MUSL?
@@ -36,8 +36,8 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
./build_static.sh
# The binary will be created at:
# build/c_relay_static_musl_x86_64 (on x86_64)
# build/c_relay_static_musl_arm64 (on ARM64)
# build/c_relay_pg_static_musl_x86_64 (on x86_64)
# build/c_relay_pg_static_musl_arm64 (on ARM64)
```
### What Happens During Build
@@ -56,7 +56,7 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
- Includes required NIPs: 001, 006, 013, 017, 019, 044, 059
- Produces static library (~316KB)
4. **c-relay Compilation**: Links everything statically:
4. **c-relay-pg Compilation**: Links everything statically:
- All source files compiled with `-static` flag
- Fortification disabled to avoid `__*_chk` symbols
- Results in ~7.6MB stripped binary
@@ -78,7 +78,7 @@ FROM alpine:3.19 AS builder
- Install build tools and static libraries
- Build dependencies from source
- Compile nostr_core_lib with MUSL flags
- Compile c-relay with full static linking
- Compile c-relay-pg with full static linking
- Strip binary to reduce size
# Stage 2: Output (scratch)
@@ -93,7 +93,7 @@ FROM scratch AS output
CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"
```
**For c-relay:**
**For c-relay-pg:**
```bash
gcc -static -O2 -Wall -Wextra -std=c99 \
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
@@ -110,7 +110,7 @@ gcc -static -O2 -Wall -Wextra -std=c99 \
- `-U_FORTIFY_SOURCE` (undefine any existing definition)
- `-D_FORTIFY_SOURCE=0` (set to 0)
This must be applied to **both** nostr_core_lib and c-relay compilation.
This must be applied to **both** nostr_core_lib and c-relay-pg compilation.
### NIP Dependencies
@@ -129,26 +129,26 @@ The build includes these NIPs in nostr_core_lib:
```bash
# Should show "statically linked"
file build/c_relay_static_musl_x86_64
file build/c_relay_pg_static_musl_x86_64
# Should show "not a dynamic executable"
ldd build/c_relay_static_musl_x86_64
ldd build/c_relay_pg_static_musl_x86_64
# Check size (should be ~7.6MB)
ls -lh build/c_relay_static_musl_x86_64
ls -lh build/c_relay_pg_static_musl_x86_64
```
### Test Execution
```bash
# Show help
./build/c_relay_static_musl_x86_64 --help
./build/c_relay_pg_static_musl_x86_64 --help
# Show version
./build/c_relay_static_musl_x86_64 --version
./build/c_relay_pg_static_musl_x86_64 --version
# Run relay
./build/c_relay_static_musl_x86_64 --port 8888
./build/c_relay_pg_static_musl_x86_64 --port 8888
```
### Cross-Distribution Testing
@@ -157,16 +157,16 @@ Test the binary on different distributions to verify portability:
```bash
# Alpine Linux
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_pg_static_musl_x86_64 --version
# Ubuntu
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_pg_static_musl_x86_64 --version
# Debian
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_pg_static_musl_x86_64 --version
# CentOS
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_static_musl_x86_64 --version
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_pg_static_musl_x86_64 --version
```
## Troubleshooting
@@ -187,7 +187,7 @@ newgrp docker # Or logout and login again
**Solution**: Ensure fortification is disabled in both:
1. nostr_core_lib build.sh (line 534)
2. c-relay compilation flags in Dockerfile
2. c-relay-pg compilation flags in Dockerfile
### Binary Won't Execute
@@ -213,26 +213,26 @@ newgrp docker # Or logout and login again
```bash
# Copy binary to server
scp build/c_relay_static_musl_x86_64 user@server:/opt/c-relay/
scp build/c_relay_pg_static_musl_x86_64 user@server:/opt/c-relay-pg/
# Run on server (no dependencies needed!)
ssh user@server
cd /opt/c-relay
./c_relay_static_musl_x86_64 --port 8888
cd /opt/c-relay-pg
./c_relay_pg_static_musl_x86_64 --port 8888
```
### SystemD Service
```ini
[Unit]
Description=C-Relay Nostr Relay (MUSL Static)
Description=C-Relay-PG Nostr Relay (MUSL Static)
After=network.target
[Service]
Type=simple
User=c-relay
WorkingDirectory=/opt/c-relay
ExecStart=/opt/c-relay/c_relay_static_musl_x86_64
User=c-relay-pg
WorkingDirectory=/opt/c-relay-pg
ExecStart=/opt/c-relay-pg/c_relay_pg_static_musl_x86_64
Restart=always
RestartSec=5
+4 -4
View File
@@ -23,7 +23,7 @@ We chose Option B because:
## Detailed Implementation Steps
### Phase 1: Configuration Setup in c-relay
### Phase 1: Configuration Setup in c-relay-pg
#### 1.1 Add Configuration Parameter
**File:** `src/default_config_event.h`
@@ -293,7 +293,7 @@ int nostr_nip17_send_dm(cJSON* dm_event,
---
### Phase 4: Update c-relay Call Sites
### Phase 4: Update c-relay-pg Call Sites
#### 4.1 Update src/api.c
**Location:** Line 1319
@@ -470,7 +470,7 @@ causing compatibility issues.
- [ ] Update `nostr_nip17_send_dm()` signature and implementation
- [ ] Update `nip017.h` function declaration and documentation
### c-relay Changes
### c-relay-pg Changes
- [ ] Add `nip59_timestamp_max_delay_sec` to `default_config_event.h`
- [ ] Add validation in `config.c` for new parameter
- [ ] Update `src/api.c` call site to pass `max_delay_sec`
@@ -494,7 +494,7 @@ causing compatibility issues.
If issues arise:
1. Revert nostr_core_lib changes (git revert in submodule)
2. Revert c-relay changes
2. Revert c-relay-pg changes
3. Configuration parameter will be ignored if not used
4. Default behavior (0) provides maximum compatibility
+2 -2
View File
@@ -2,7 +2,7 @@
## Overview
This document describes the design for a general-purpose SQL query interface for the C-Relay admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
This document describes the design for a general-purpose SQL query interface for the C-Relay-PG admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
## Security Model
@@ -271,7 +271,7 @@ const SQL_QUERY_TEMPLATES = {
};
// Query history management (localStorage)
const QUERY_HISTORY_KEY = 'c_relay_sql_history';
const QUERY_HISTORY_KEY = 'c_relay_pg_sql_history';
const MAX_HISTORY_ITEMS = 20;
// Load query history from localStorage
+1 -1
View File
@@ -224,7 +224,7 @@ The script should:
- `curl` or `websocat` for WebSocket communication
- `jq` for JSON parsing
- Nostr CLI tools (optional, for event signing)
- Running c-relay instance
- Running c-relay-pg instance
## Example Output
+6 -6
View File
@@ -1,8 +1,8 @@
# C-Relay Complete Startup Flow Documentation
# C-Relay-PG Complete Startup Flow Documentation
## Overview
C-Relay has two distinct startup paths:
C-Relay-PG has two distinct startup paths:
1. **First-Time Startup**: No database exists, generates keys and initializes system
2. **Existing Relay Startup**: Database exists, loads configuration and resumes operation
@@ -220,7 +220,7 @@ int populate_default_config_values(void) {
struct config_default defaults[] = {
{"relay_port", "8888", "integer", "WebSocket port", "network", 1},
{"relay_name", "C-Relay", "string", "Relay name", "info", 0},
{"relay_name", "C-Relay-PG", "string", "Relay name", "info", 0},
{"relay_description", "High-performance C Nostr relay", "string", "Description", "info", 0},
{"max_subscriptions_per_client", "25", "integer", "Max subs per client", "limits", 0},
{"pow_min_difficulty", "0", "integer", "Minimum PoW difficulty", "security", 0},
@@ -983,7 +983,7 @@ if (current_version < LATEST_VERSION) {
**Solutions**:
- Use `--port <number>` to specify different port
- Kill existing process: `pkill -f c_relay_`
- Kill existing process: `pkill -f c_relay_pg_`
- Force kill port: `fuser -k 8888/tcp`
- Use `--strict-port` to fail fast instead of trying fallback ports
@@ -994,7 +994,7 @@ if (current_version < LATEST_VERSION) {
**Solutions**:
- Kill existing relay processes
- Remove WAL files: `rm build/*.db-wal build/*.db-shm`
- Check for stale processes: `ps aux | grep c_relay_`
- Check for stale processes: `ps aux | grep c_relay_pg_`
#### 3. Missing Admin Private Key
@@ -1071,7 +1071,7 @@ Typical startup times (on modern hardware):
## Summary
The c-relay startup system is designed for:
The c-relay-pg startup system is designed for:
1. **Security**: Admin keys never stored, relay keys encrypted
2. **Reliability**: Automatic port fallback, schema migrations
+9 -9
View File
@@ -18,10 +18,10 @@ The script now attempts to build with `musl-gcc` for truly portable static binar
SQLite is now built once and cached for future builds:
- **Cache location**: `~/.cache/c-relay-sqlite/`
- **Cache location**: `~/.cache/c-relay-pg-sqlite/`
- **Version-specific**: Each SQLite version gets its own cache directory
- **Significant speedup**: Subsequent builds skip the SQLite compilation step
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-sqlite` to clear cache
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-pg-sqlite` to clear cache
### 3. Smart Package Installation
@@ -51,13 +51,13 @@ The script will:
## Binary Types
### MUSL Static Binary (Ideal - Currently Not Achievable)
- **Filename**: `build/c_relay_static_musl_x86_64`
- **Filename**: `build/c_relay_pg_static_musl_x86_64`
- **Dependencies**: None (truly static)
- **Portability**: Works on any Linux distribution
- **Status**: Requires MUSL-compiled libwebsockets and other dependencies (not available by default)
### Glibc Static Binary (Current Output)
- **Filename**: `build/c_relay_static_x86_64` or `build/c_relay_static_glibc_x86_64`
- **Filename**: `build/c_relay_pg_static_x86_64` or `build/c_relay_pg_static_glibc_x86_64`
- **Dependencies**: None - fully statically linked with glibc
- **Portability**: Works on most Linux distributions (glibc is statically included)
- **Note**: Despite using glibc, this is a **fully static binary** with no runtime dependencies
@@ -68,11 +68,11 @@ The script automatically verifies binaries using `ldd` and `file`:
```bash
# For MUSL binary
ldd build/c_relay_static_musl_x86_64
ldd build/c_relay_pg_static_musl_x86_64
# Output: "not a dynamic executable" (good!)
# For glibc binary
ldd build/c_relay_static_glibc_x86_64
ldd build/c_relay_pg_static_glibc_x86_64
# Output: Shows glibc dependencies
```
@@ -112,7 +112,7 @@ The script attempts MUSL compilation but falls back to glibc:
### Clear SQLite Cache
```bash
rm -rf ~/.cache/c-relay-sqlite
rm -rf ~/.cache/c-relay-pg-sqlite
```
### Force Package Reinstall
@@ -127,8 +127,8 @@ cat /tmp/musl_build.log
### Verify Binary Type
```bash
file build/c_relay_static_*
ldd build/c_relay_static_* 2>&1
file build/c_relay_pg_static_*
ldd build/c_relay_pg_static_* 2>&1
```
## Performance Impact
+1 -1
View File
@@ -2,7 +2,7 @@
## Problem Summary
The c-relay Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
The c-relay-pg Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
## Solution: Two-Component Approach
+8 -8
View File
@@ -586,7 +586,7 @@ int add_pubkeys_to_config_table(void) {
rm -f *.db
# Start relay with defaults
./build/c_relay_x86
./build/c_relay_pg_x86
# Verify config table complete
sqlite3 <relay_pubkey>.db "SELECT COUNT(*) FROM config;"
@@ -602,7 +602,7 @@ int add_pubkeys_to_config_table(void) {
rm -f *.db
# Start relay with port override
./build/c_relay_x86 --port 9999
./build/c_relay_pg_x86 --port 9999
# Verify port override applied
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
@@ -612,13 +612,13 @@ int add_pubkeys_to_config_table(void) {
3. **Restart with Existing Database**
```bash
# Start relay (creates database)
./build/c_relay_x86
./build/c_relay_pg_x86
# Stop relay
pkill -f c_relay_
pkill -f c_relay_pg_
# Restart relay
./build/c_relay_x86
./build/c_relay_pg_x86
# Verify config unchanged
# Check relay.log for validation message
@@ -627,13 +627,13 @@ int add_pubkeys_to_config_table(void) {
4. **Restart with CLI Overrides**
```bash
# Start relay (creates database)
./build/c_relay_x86
./build/c_relay_pg_x86
# Stop relay
pkill -f c_relay_
pkill -f c_relay_pg_
# Restart with port override
./build/c_relay_x86 --port 9999
./build/c_relay_pg_x86 --port 9999
# Verify port override applied
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
+32 -32
View File
@@ -19,12 +19,12 @@ Complete guide for deploying, configuring, and managing the C Nostr Relay with e
```bash
# Clone and build
git clone <repository-url>
cd c-relay
cd c-relay-pg
git submodule update --init --recursive
make
# Start relay (zero configuration needed)
./build/c_relay_x86
./build/c_relay_pg_x86
```
### 2. First Startup - Save Keys
@@ -78,7 +78,7 @@ brew install git sqlite libwebsockets openssl libsecp256k1 curl zlib
```bash
# Clone repository
git clone <repository-url>
cd c-relay
cd c-relay-pg
# Initialize submodules
git submodule update --init --recursive
@@ -87,7 +87,7 @@ git submodule update --init --recursive
make clean && make
# Verify build
ls -la build/c_relay_x86
ls -la build/c_relay_pg_x86
```
### Production Deployment
@@ -98,27 +98,27 @@ ls -la build/c_relay_x86
sudo systemd/install-service.sh
# Start service
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Enable auto-start
sudo systemctl enable c-relay
sudo systemctl enable c-relay-pg
# Check status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
```
#### Manual Deployment
```bash
# Create dedicated user
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
# Install binary
sudo mkdir -p /opt/c-relay
sudo cp build/c_relay_x86 /opt/c-relay/
sudo chown -R c-relay:c-relay /opt/c-relay
sudo mkdir -p /opt/c-relay-pg
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
# Run as service user
sudo -u c-relay /opt/c-relay/c_relay_x86
sudo -u c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
```
## Configuration Management
@@ -188,7 +188,7 @@ Send this to your relay via WebSocket, and changes are applied immediately.
|-----------|-------------|---------|---------|
| `relay_description` | Relay description for NIP-11 | "C Nostr Relay" | "My awesome relay" |
| `relay_contact` | Admin contact information | "" | "admin@example.com" |
| `relay_software` | Software identifier | "c-relay" | "c-relay v1.0" |
| `relay_software` | Software identifier | "c-relay-pg" | "c-relay-pg v1.0" |
#### Client Limits
| Parameter | Description | Default | Range |
@@ -273,7 +273,7 @@ chmod 600 admin_keys_backup_*.txt
#### Key Recovery
If you lose your admin private key:
1. **Stop the relay**: `pkill c_relay` or `sudo systemctl stop c-relay`
1. **Stop the relay**: `pkill c_relay_pg` or `sudo systemctl stop c-relay-pg`
2. **Backup events**: `cp <relay_pubkey>.nrdb backup_$(date +%Y%m%d).nrdb`
3. **Remove database**: `rm <relay_pubkey>.nrdb*`
4. **Restart relay**: This creates new database with new keys
@@ -300,7 +300,7 @@ sudo ufw allow 8888/tcp
```bash
# Secure database file permissions
chmod 600 <relay_pubkey>.nrdb
chown c-relay:c-relay <relay_pubkey>.nrdb
chown c-relay-pg:c-relay-pg <relay_pubkey>.nrdb
# Regular backups
cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
@@ -311,10 +311,10 @@ cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
### Service Status
```bash
# Check if relay is running
ps aux | grep c_relay
ps aux | grep c_relay_pg
# SystemD status
sudo systemctl status c-relay
sudo systemctl status c-relay-pg
# Network connections
netstat -tln | grep 8888
@@ -324,16 +324,16 @@ sudo ss -tlpn | grep 8888
### Log Monitoring
```bash
# Real-time logs (systemd)
sudo journalctl -u c-relay -f
sudo journalctl -u c-relay-pg -f
# Recent logs
sudo journalctl -u c-relay --since "1 hour ago"
sudo journalctl -u c-relay-pg --since "1 hour ago"
# Error logs only
sudo journalctl -u c-relay -p err
sudo journalctl -u c-relay-pg -p err
# Configuration changes
sudo journalctl -u c-relay | grep "Configuration updated via kind 33334"
sudo journalctl -u c-relay-pg | grep "Configuration updated via kind 33334"
```
### Database Analytics
@@ -365,13 +365,13 @@ ORDER BY created_at DESC;
du -sh <relay_pubkey>.nrdb*
# Memory usage
ps aux | grep c_relay | awk '{print $6}' # RSS memory in KB
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory in KB
# Connection count (approximate)
netstat -an | grep :8888 | grep ESTABLISHED | wc -l
# System resources
top -p $(pgrep c_relay)
top -p $(pgrep c_relay_pg)
```
## Troubleshooting
@@ -385,11 +385,11 @@ netstat -tln | grep 8888
# If port in use, find process: sudo lsof -i :8888
# Check binary permissions
ls -la build/c_relay_x86
chmod +x build/c_relay_x86
ls -la build/c_relay_pg_x86
chmod +x build/c_relay_pg_x86
# Check dependencies
ldd build/c_relay_x86
ldd build/c_relay_pg_x86
```
#### Configuration Not Updating
@@ -442,7 +442,7 @@ sqlite3 recovered.nrdb < recovered.sql
# If repair fails, start fresh (loses all events)
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.corrupted
./build/c_relay_x86 # Creates new database
./build/c_relay_pg_x86 # Creates new database
```
#### Lost Configuration Recovery
@@ -455,12 +455,12 @@ If configuration is lost but database is intact:
#### Emergency Restart
```bash
# Quick restart with clean state
sudo systemctl stop c-relay
sudo systemctl stop c-relay-pg
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.backup
sudo systemctl start c-relay
sudo systemctl start c-relay-pg
# Check logs for new admin keys
sudo journalctl -u c-relay --since "5 minutes ago" | grep "Admin Private Key"
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep "Admin Private Key"
```
## Advanced Usage
@@ -503,7 +503,7 @@ ws.on('open', function() {
# backup-relay.sh
DATE=$(date +%Y%m%d_%H%M%S)
DB_FILE=$(ls *.nrdb | head -1)
BACKUP_DIR="/backup/c-relay"
BACKUP_DIR="/backup/c-relay-pg"
mkdir -p $BACKUP_DIR
cp $DB_FILE $BACKUP_DIR/relay_backup_$DATE.nrdb
@@ -533,7 +533,7 @@ tar czf relay_migration.tar.gz *.nrdb* relay.log
# Target server
tar xzf relay_migration.tar.gz
./build/c_relay_x86 # Will detect existing database and continue
./build/c_relay_pg_x86 # Will detect existing database and continue
```
---
+1 -1
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# Script to embed web files into C headers for the C-Relay admin interface
# Script to embed web files into C headers for the C-Relay-PG admin interface
# Converts HTML, CSS, and JS files from api/ directory into C byte arrays
set -e
+1 -1
View File
@@ -60,7 +60,7 @@ All examples assume the event-based configuration system where:
- **Save Admin Keys**: All deployment examples emphasize capturing the admin private key on first startup
- **Firewall Configuration**: Examples include proper firewall rules
- **SSL/TLS**: Production examples include HTTPS configuration
- **User Isolation**: Service runs as dedicated `c-relay` system user
- **User Isolation**: Service runs as dedicated `c-relay-pg` system user
## Support
+6 -6
View File
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Default configuration
RELAY_DIR="/opt/c-relay"
BACKUP_DIR="/backup/c-relay"
RELAY_DIR="/opt/c-relay-pg"
BACKUP_DIR="/backup/c-relay-pg"
RETENTION_DAYS="30"
COMPRESS="true"
REMOTE_BACKUP=""
@@ -47,8 +47,8 @@ show_help() {
echo "Usage: $0 [OPTIONS]"
echo
echo "Options:"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay)"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay-pg)"
echo " -r, --retention DAYS Retention period in days (default: 30)"
echo " -n, --no-compress Don't compress backups"
echo " -s, --s3-bucket BUCKET Upload to S3 bucket"
@@ -231,7 +231,7 @@ upload_to_s3() {
print_step "Uploading backup to S3..."
local s3_path="s3://$S3_BUCKET/c-relay/$(date +%Y)/$(date +%m)/"
local s3_path="s3://$S3_BUCKET/c-relay-pg/$(date +%Y)/$(date +%m)/"
if aws s3 cp "$BACKUP_FILE" "$s3_path" --storage-class STANDARD_IA; then
print_success "Backup uploaded to S3: $s3_path"
@@ -264,7 +264,7 @@ cleanup_old_backups() {
print_step "Cleaning S3 backups older than $cutoff_date..."
# Note: This is a simplified approach. In production, use S3 lifecycle policies
aws s3 ls "s3://$S3_BUCKET/c-relay/" --recursive | \
aws s3 ls "s3://$S3_BUCKET/c-relay-pg/" --recursive | \
awk '$1 < "'$cutoff_date'" {print $4}' | \
while read -r key; do
aws s3 rm "s3://$S3_BUCKET/$key"
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
RELAY_DIR="/opt/c-relay"
SERVICE_NAME="c-relay"
RELAY_DIR="/opt/c-relay-pg"
SERVICE_NAME="c-relay-pg"
RELAY_PORT="8888"
LOG_FILE="/var/log/relay-monitor.log"
ALERT_EMAIL=""
@@ -60,7 +60,7 @@ show_help() {
echo "Usage: $0 [OPTIONS]"
echo
echo "Options:"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
echo " -p, --port PORT Relay port (default: 8888)"
echo " -i, --interval SECONDS Check interval (default: 60)"
echo " -e, --email EMAIL Alert email address"
@@ -134,7 +134,7 @@ parse_args() {
check_process_running() {
print_step "Checking if relay process is running..."
if pgrep -f "c_relay_x86" > /dev/null; then
if pgrep -f "c_relay_pg_x86" > /dev/null; then
print_success "Relay process is running"
return 0
else
@@ -172,7 +172,7 @@ check_service_status() {
check_memory_usage() {
print_step "Checking memory usage..."
local memory_kb=$(ps aux | grep "c_relay_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
local memory_kb=$(ps aux | grep "c_relay_pg_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
if [[ -z "$memory_kb" ]]; then
print_warning "Could not determine memory usage"
+4 -4
View File
@@ -56,7 +56,7 @@ http {
}
# Upstream for the relay
upstream c_relay_backend {
upstream c_relay_pg_backend {
server 127.0.0.1:8888;
keepalive 32;
}
@@ -108,7 +108,7 @@ http {
# Main proxy location for WebSocket and HTTP
location / {
# Proxy settings
proxy_pass http://c_relay_backend;
proxy_pass http://c_relay_pg_backend;
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
@@ -144,7 +144,7 @@ http {
# Health check endpoint (if implemented)
location /health {
proxy_pass http://c_relay_backend/health;
proxy_pass http://c_relay_pg_backend/health;
access_log off;
}
@@ -157,7 +157,7 @@ http {
# Optional: Metrics endpoint (if implemented)
location /metrics {
proxy_pass http://c_relay_backend/metrics;
proxy_pass http://c_relay_pg_backend/metrics;
# Restrict access to monitoring systems
allow 10.0.0.0/8;
allow 172.16.0.0/12;
@@ -96,9 +96,9 @@ check_root() {
check_relay_running() {
print_step "Checking if C Nostr Relay is running..."
if ! pgrep -f "c_relay_x86" > /dev/null; then
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
print_error "C Nostr Relay is not running"
print_error "Please start the relay first with: sudo systemctl start c-relay"
print_error "Please start the relay first with: sudo systemctl start c-relay-pg"
exit 1
fi
+12 -12
View File
@@ -13,9 +13,9 @@ BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Configuration
RELAY_USER="c-relay"
INSTALL_DIR="/opt/c-relay"
SERVICE_NAME="c-relay"
RELAY_USER="c-relay-pg"
INSTALL_DIR="/opt/c-relay-pg"
SERVICE_NAME="c-relay-pg"
RELAY_PORT="8888"
# Functions
@@ -99,7 +99,7 @@ build_relay() {
# Check if we're in the source directory
if [[ ! -f "Makefile" ]]; then
print_error "Makefile not found. Please run this script from the c-relay source directory."
print_error "Makefile not found. Please run this script from the c-relay-pg source directory."
exit 1
fi
@@ -107,7 +107,7 @@ build_relay() {
make clean
make
if [[ ! -f "build/c_relay_x86" ]]; then
if [[ ! -f "build/c_relay_pg_x86" ]]; then
print_error "Build failed - binary not found"
exit 1
fi
@@ -118,9 +118,9 @@ build_relay() {
install_binary() {
print_step "Installing relay binary..."
cp build/c_relay_x86 "$INSTALL_DIR/"
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_x86"
chmod +x "$INSTALL_DIR/c_relay_x86"
cp build/c_relay_pg_x86 "$INSTALL_DIR/"
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_pg_x86"
chmod +x "$INSTALL_DIR/c_relay_pg_x86"
print_success "Binary installed to $INSTALL_DIR"
}
@@ -129,14 +129,14 @@ install_service() {
print_step "Installing systemd service..."
# Use the existing systemd service file
if [[ -f "systemd/c-relay.service" ]]; then
cp systemd/c-relay.service /etc/systemd/system/
if [[ -f "systemd/c-relay-pg.service" ]]; then
cp systemd/c-relay-pg.service /etc/systemd/system/
systemctl daemon-reload
print_success "Systemd service installed"
else
print_warning "Systemd service file not found, creating basic one..."
cat > /etc/systemd/system/c-relay.service << EOF
cat > /etc/systemd/system/c-relay-pg.service << EOF
[Unit]
Description=C Nostr Relay
After=network.target
@@ -146,7 +146,7 @@ Type=simple
User=$RELAY_USER
Group=$RELAY_USER
WorkingDirectory=$INSTALL_DIR
ExecStart=$INSTALL_DIR/c_relay_x86
ExecStart=$INSTALL_DIR/c_relay_pg_x86
Restart=always
RestartSec=5
@@ -1,4 +1,4 @@
# MUSL-based fully static C-Relay builder
# MUSL-based fully static C-Relay-PG builder
# Produces portable binaries with zero runtime dependencies
FROM alpine:latest AS builder
@@ -125,7 +125,7 @@ RUN cd /tmp && \
--prefix=/usr && \
make && make install
# Copy c-relay source
# Copy c-relay-pg source
COPY . /build/
# Initialize submodules
@@ -134,7 +134,7 @@ RUN git submodule update --init --recursive
# Build nostr_core_lib
RUN cd nostr_core_lib && ./build.sh
# Build c-relay static
# Build c-relay-pg static
RUN make clean && \
CC="musl-gcc -static" \
CFLAGS="-O2 -Wall -Wextra -std=c99 -g" \
@@ -143,8 +143,8 @@ RUN make clean && \
make
# Strip binary for size
RUN strip build/c_relay_x86
RUN strip build/c_relay_pg_x86
# Multi-stage build to produce minimal output
FROM scratch AS output
COPY --from=builder /build/build/c_relay_x86 /c_relay_static_musl_x86_64
COPY --from=builder /build/build/c_relay_pg_x86 /c_relay_pg_static_musl_x86_64
+8 -8
View File
@@ -19,7 +19,7 @@ RELEASE_MODE=false
VERSION_INCREMENT_TYPE="patch" # "patch", "minor", or "major"
show_usage() {
echo "C-Relay Increment and Push Script"
echo "C-Relay-PG Increment and Push Script"
echo ""
echo "USAGE:"
echo " $0 [OPTIONS] \"commit message\""
@@ -330,7 +330,7 @@ build_release_binary() {
create_source_tarball() {
print_status "Creating source tarball..."
local tarball_name="c-relay-${NEW_VERSION#v}.tar.gz"
local tarball_name="c-relay-pg-${NEW_VERSION#v}.tar.gz"
# Create tarball excluding build artifacts and git files
if tar -czf "$tarball_name" \
@@ -365,7 +365,7 @@ upload_release_assets() {
fi
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
local assets_url="$api_url/releases/$release_id/assets"
print_status "Assets URL: $assets_url"
@@ -427,7 +427,7 @@ create_gitea_release() {
fi
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
# Create release
print_status "Creating release $NEW_VERSION..."
@@ -473,7 +473,7 @@ create_gitea_release() {
# Main execution
main() {
print_status "C-Relay Increment and Push Script"
print_status "C-Relay-PG Increment and Push Script"
# Check prerequisites
check_git_repo
@@ -505,13 +505,13 @@ main() {
# Build release binary
if build_release_binary; then
local binary_path="build/c_relay_static_x86_64"
local binary_path="build/c_relay_pg_static_x86_64"
else
print_warning "Binary build failed, continuing with release creation"
# Check if binary exists from previous build
if [[ -f "build/c_relay_static_x86_64" ]]; then
if [[ -f "build/c_relay_pg_static_x86_64" ]]; then
print_status "Using existing binary from previous build"
binary_path="build/c_relay_static_x86_64"
binary_path="build/c_relay_pg_static_x86_64"
else
binary_path=""
fi
+232 -14
View File
@@ -1,6 +1,6 @@
#!/bin/bash
# C-Relay Build and Restart Script
# C-Relay-PG Build and Restart Script
# Builds the project first, then stops any running relay and starts a new one in the background
echo "=== C Nostr Relay Build and Restart Script ==="
@@ -13,6 +13,13 @@ ADMIN_KEY=""
RELAY_KEY=""
PORT_OVERRIDE=""
DEBUG_LEVEL="5"
DB_BACKEND="postgres"
DB_CONNSTRING=""
DB_HOST=""
DB_PORT=""
DB_NAME=""
DB_USER=""
DB_PASSWORD=""
# Key validation function
validate_hex_key() {
@@ -118,6 +125,84 @@ while [[ $# -gt 0 ]]; do
shift 2
fi
;;
--db-backend)
if [ -z "$2" ]; then
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
HELP=true
shift
else
DB_BACKEND="$2"
shift 2
fi
;;
--db-backend=*)
DB_BACKEND="${1#*=}"
shift
;;
--db-connstring)
if [ -z "$2" ]; then
echo "ERROR: --db-connstring requires a value"
HELP=true
shift
else
DB_CONNSTRING="$2"
shift 2
fi
;;
--db-connstring=*)
DB_CONNSTRING="${1#*=}"
shift
;;
--db-host)
if [ -z "$2" ]; then
echo "ERROR: --db-host requires a value"
HELP=true
shift
else
DB_HOST="$2"
shift 2
fi
;;
--db-port)
if [ -z "$2" ]; then
echo "ERROR: --db-port requires a value"
HELP=true
shift
else
DB_PORT="$2"
shift 2
fi
;;
--db-name)
if [ -z "$2" ]; then
echo "ERROR: --db-name requires a value"
HELP=true
shift
else
DB_NAME="$2"
shift 2
fi
;;
--db-user)
if [ -z "$2" ]; then
echo "ERROR: --db-user requires a value"
HELP=true
shift
else
DB_USER="$2"
shift 2
fi
;;
--db-password)
if [ -z "$2" ]; then
echo "ERROR: --db-password requires a value"
HELP=true
shift
else
DB_PASSWORD="$2"
shift 2
fi
;;
--help|-h)
HELP=true
shift
@@ -164,6 +249,98 @@ if [ -n "$DEBUG_LEVEL" ]; then
fi
fi
# Validate DB backend
if [ "$DB_BACKEND" != "sqlite" ] && [ "$DB_BACKEND" != "postgres" ]; then
echo "ERROR: Invalid --db-backend value '$DB_BACKEND'. Use sqlite or postgres."
exit 1
fi
# Validate DB port if provided
if [ -n "$DB_PORT" ]; then
if ! [[ "$DB_PORT" =~ ^[0-9]+$ ]] || [ "$DB_PORT" -lt 1 ] || [ "$DB_PORT" -gt 65535 ]; then
echo "ERROR: --db-port must be a number between 1 and 65535"
exit 1
fi
fi
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
[ -z "$DB_HOST" ] && DB_HOST="localhost"
[ -z "$DB_PORT" ] && DB_PORT="5432"
[ -z "$DB_NAME" ] && DB_NAME="crelay"
[ -z "$DB_USER" ] && DB_USER="crelay"
[ -z "$DB_PASSWORD" ] && DB_PASSWORD="crelay"
fi
ensure_postgres_database() {
if ! command -v psql >/dev/null 2>&1; then
echo "ERROR: psql not found. Install PostgreSQL client tools to continue."
return 1
fi
echo "Ensuring PostgreSQL database exists: host=$DB_HOST port=$DB_PORT dbname=$DB_NAME user=$DB_USER"
if [ -n "$DB_PASSWORD" ]; then
export PGPASSWORD="$DB_PASSWORD"
fi
# Prefer local postgres superuser for provisioning when available.
if sudo -n -u postgres psql -d postgres -tAc "SELECT 1" >/dev/null 2>&1; then
sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname = '$DB_USER'" | grep -q 1 || \
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "CREATE ROLE \"$DB_USER\" LOGIN PASSWORD '$DB_PASSWORD';" || return 1
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "ALTER ROLE \"$DB_USER\" WITH LOGIN PASSWORD '$DB_PASSWORD';" >/dev/null || return 1
if [ "$PRESERVE_DATABASE" = false ]; then
echo "Resetting PostgreSQL database '$DB_NAME' for fresh start..."
# Ensure no active sessions block DROP DATABASE (common when relay is still running).
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c \
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '$DB_NAME' AND pid <> pg_backend_pid();" >/dev/null || return 1
# Prefer force drop when supported; fallback to regular dropdb.
if ! sudo -u postgres dropdb --if-exists --force "$DB_NAME" >/dev/null 2>&1; then
sudo -u postgres dropdb --if-exists "$DB_NAME" >/dev/null 2>&1 || return 1
fi
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
echo "✓ Recreated PostgreSQL database '$DB_NAME' owned by '$DB_USER'"
else
DB_EXISTS=$(sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
if [ "$DB_EXISTS" != "1" ]; then
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
echo "✓ Created PostgreSQL database '$DB_NAME'"
else
echo "✓ PostgreSQL database '$DB_NAME' already exists"
fi
fi
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "GRANT ALL PRIVILEGES ON DATABASE \"$DB_NAME\" TO \"$DB_USER\";" >/dev/null || return 1
sudo -u postgres psql -d "$DB_NAME" -v ON_ERROR_STOP=1 -c "ALTER SCHEMA public OWNER TO \"$DB_USER\"; GRANT ALL ON SCHEMA public TO \"$DB_USER\";" >/dev/null || return 1
return 0
fi
# Fallback: use configured DB user directly.
if ! psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -v ON_ERROR_STOP=1 -tAc "SELECT 1" >/dev/null 2>&1; then
echo "ERROR: Cannot connect to PostgreSQL server/database 'postgres' with current settings"
echo " host=$DB_HOST port=$DB_PORT user=$DB_USER"
return 1
fi
DB_EXISTS=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
if [ "$DB_EXISTS" != "1" ]; then
echo "Database '$DB_NAME' not found. Creating..."
if ! createdb -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" "$DB_NAME" >/dev/null 2>&1; then
echo "ERROR: Failed to create PostgreSQL database '$DB_NAME'"
return 1
fi
echo "✓ Created PostgreSQL database '$DB_NAME'"
else
echo "✓ PostgreSQL database '$DB_NAME' already exists"
fi
return 0
}
# Show help
if [ "$HELP" = true ]; then
echo "Usage: $0 [OPTIONS]"
@@ -175,6 +352,13 @@ if [ "$HELP" = true ]; then
echo " -d, --debug-level <0-5> Set debug level: 0=none, 1=errors, 2=warnings, 3=info, 4=debug, 5=trace"
echo " --preserve-database Keep existing database files (don't delete for fresh start)"
echo " --test-keys, -t Use deterministic test keys for development (admin: all 'a's, relay: all '1's)"
echo " --db-backend <name> Database backend: postgres (default) or sqlite"
echo " --db-connstring <str> PostgreSQL libpq connection string"
echo " --db-host <host> PostgreSQL host"
echo " --db-port <port> PostgreSQL port"
echo " --db-name <name> PostgreSQL database name"
echo " --db-user <user> PostgreSQL database user"
echo " --db-password <pass> PostgreSQL database password"
echo " --help, -h Show this help message"
echo ""
echo "Event-Based Configuration:"
@@ -194,6 +378,9 @@ if [ "$HELP" = true ]; then
echo " $0 --test-keys # Use test keys for consistent development"
echo " $0 -t --preserve-database # Use test keys and preserve database"
echo ""
echo "Default PostgreSQL connection (when no DB flags provided):"
echo " host=localhost port=5432 dbname=crelay user=crelay password=crelay"
echo ""
echo "Key Format: Keys must be exactly 64 hexadecimal characters (0-9, a-f, A-F)"
echo "Default behavior: Deletes existing database files to start fresh with new keys"
echo " for development purposes"
@@ -217,13 +404,17 @@ fi
rm -rf dev-config/ 2>/dev/null
rm -f db/c_nostr_relay.db* 2>/dev/null
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
ensure_postgres_database || exit 1
fi
# Embed web files into C headers before building
echo "Embedding web files..."
./embed_web_files.sh
# Build the project - ONLY static build
echo "Building project (static binary with SQLite JSON1 extension)..."
./build_static.sh
echo "Building project (static binary, backend: $DB_BACKEND)..."
./build_static.sh --db-backend "$DB_BACKEND"
# Exit if static build fails - no fallback
if [ $? -ne 0 ]; then
@@ -242,13 +433,13 @@ fi
ARCH=$(uname -m)
case "$ARCH" in
x86_64)
BINARY_PATH="./build/c_relay_static_x86_64"
BINARY_PATH="./build/c_relay_pg_static_x86_64"
;;
aarch64|arm64)
BINARY_PATH="./build/c_relay_static_arm64"
BINARY_PATH="./build/c_relay_pg_static_arm64"
;;
*)
BINARY_PATH="./build/c_relay_static_$ARCH"
BINARY_PATH="./build/c_relay_pg_static_$ARCH"
;;
esac
@@ -270,7 +461,7 @@ echo "Build successful. Proceeding with relay restart..."
echo "Stopping any existing relay servers..."
# Get all relay processes and kill them immediately with -9
RELAY_PIDS=$(pgrep -f "c_relay_" || echo "")
RELAY_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$RELAY_PIDS" ]; then
echo "Force killing relay processes immediately: $RELAY_PIDS"
kill -9 $RELAY_PIDS 2>/dev/null
@@ -291,7 +482,7 @@ for attempt in {1..15}; do
fuser -k 8888/tcp 2>/dev/null || true
# Double-check for any remaining relay processes
REMAINING_PIDS=$(pgrep -f "c_relay_" || echo "")
REMAINING_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$REMAINING_PIDS" ]; then
echo "Killing remaining relay processes: $REMAINING_PIDS"
kill -9 $REMAINING_PIDS 2>/dev/null || true
@@ -309,7 +500,7 @@ for attempt in {1..15}; do
done
# Final safety check - ensure no relay processes remain
FINAL_PIDS=$(pgrep -f "c_relay_" || echo "")
FINAL_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$FINAL_PIDS" ]; then
echo "Final cleanup: killing processes $FINAL_PIDS"
kill -9 $FINAL_PIDS 2>/dev/null || true
@@ -325,7 +516,7 @@ echo "Database will be initialized automatically on startup if needed"
# Start relay in background with output redirection
echo "Starting relay server..."
echo "Debug: Current processes: $(ps aux | grep 'c_relay_' | grep -v grep || echo 'None')"
echo "Debug: Current processes: $(ps aux | grep 'c_relay_pg_' | grep -v grep || echo 'None')"
# Build command line arguments for relay binary
RELAY_ARGS=""
@@ -350,15 +541,42 @@ if [ -n "$DEBUG_LEVEL" ]; then
echo "Using debug level: $DEBUG_LEVEL"
fi
if [ -n "$DB_CONNSTRING" ]; then
RELAY_ARGS="$RELAY_ARGS --db-connstring '$DB_CONNSTRING'"
echo "Using PostgreSQL connection string override"
fi
if [ -n "$DB_HOST" ]; then
RELAY_ARGS="$RELAY_ARGS --db-host '$DB_HOST'"
fi
if [ -n "$DB_PORT" ]; then
RELAY_ARGS="$RELAY_ARGS --db-port '$DB_PORT'"
fi
if [ -n "$DB_NAME" ]; then
RELAY_ARGS="$RELAY_ARGS --db-name '$DB_NAME'"
fi
if [ -n "$DB_USER" ]; then
RELAY_ARGS="$RELAY_ARGS --db-user '$DB_USER'"
fi
if [ -n "$DB_PASSWORD" ]; then
RELAY_ARGS="$RELAY_ARGS --db-password '$DB_PASSWORD'"
fi
# Change to build directory before starting relay so database files are created there
cd build
# Start relay in background and capture its PID
if [ "$USE_TEST_KEYS" = true ]; then
echo "Using test keys from .test_keys file..."
./$(basename $BINARY_PATH) -a "$ADMIN_KEY" -r "$RELAY_KEY" --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
# shellcheck disable=SC2086
eval ./$(basename $BINARY_PATH) -a "$ADMIN_KEY" -r "$RELAY_KEY" $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
elif [ -n "$RELAY_ARGS" ]; then
echo "Starting relay with custom configuration..."
./$(basename $BINARY_PATH) $RELAY_ARGS --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
# shellcheck disable=SC2086
eval ./$(basename $BINARY_PATH) $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
else
# No command line arguments needed for random key generation
echo "Starting relay with random key generation..."
@@ -428,8 +646,8 @@ if ps -p "$RELAY_PID" >/dev/null 2>&1; then
echo "=== Event-Based Relay Server Running ==="
echo "Configuration: Event-based (kind 33334 Nostr events)"
echo "Database: Automatically created with relay pubkey naming"
echo "To kill relay: pkill -f 'c_relay_'"
echo "To check status: ps aux | grep c_relay_"
echo "To kill relay: pkill -f 'c_relay_pg_'"
echo "To check status: ps aux | grep c_relay_pg_"
echo "To view logs: tail -f relay.log"
echo "Binary: $BINARY_PATH (zero configuration needed)"
echo "Ready for Nostr client connections!"
+1 -1
View File
@@ -1,3 +1,3 @@
#!/bin/bash
curl -H "Accept: application/nostr+json" http://localhost:8888/
curl -H "Accept: application/nostr+json" https://relay.laantungir.net
+12 -12
View File
@@ -12,11 +12,11 @@ After the next crash, analyze it:
# List all core dumps (most recent first)
sudo coredumpctl list
# View info about the most recent c-relay crash
sudo coredumpctl info c-relay
# View info about the most recent c-relay-pg crash
sudo coredumpctl info c-relay-pg
# Load the core dump in gdb for detailed analysis
sudo coredumpctl gdb c-relay
sudo coredumpctl gdb c-relay-pg
Inside gdb, run these commands:
(gdb) bt full # Full backtrace with all variables
@@ -36,8 +36,8 @@ DEBUGGING
Even simpler: Use this one-liner
# Start relay and immediately attach gdb
cd /usr/local/bin/c_relay
sudo -u c-relay ./c_relay --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay)
cd /usr/local/bin/c_relay_pg
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay_pg)
Inside gdb, after attaching:
@@ -48,17 +48,17 @@ Or shorter:
How to View the Logs
Check systemd journal:
# View all c-relay logs
sudo journalctl -u c-relay
# View all c-relay-pg logs
sudo journalctl -u c-relay-pg
# View recent logs (last 50 lines)
sudo journalctl -u c-relay -n 50
sudo journalctl -u c-relay-pg -n 50
# Follow logs in real-time
sudo journalctl -u c-relay -f
sudo journalctl -u c-relay-pg -f
# View logs since last boot
sudo journalctl -u c-relay -b
sudo journalctl -u c-relay-pg -b
Check if service is running:
@@ -78,9 +78,9 @@ sudo systemctl start rsyslog
sudo systemctl status rsyslog
sudo -u c-relay ./c_relay --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
./c_relay_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
./c_relay_pg_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
sudo ufw allow 8888/tcp
+1 -1
View File
@@ -1,5 +1,5 @@
{
"name": "c-relay",
"name": "c-relay-pg",
"lockfileVersion": 3,
"requires": true,
"packages": {
+178
View File
@@ -0,0 +1,178 @@
# API Upgrade Implementation Plan — Phases 1, 2, and 5
Goal: finish the remaining work from `plans/api_upgrade_plan.md`:
- **Phase 1 (design fix):** convert the `api-worker` thread from job-queue-driven to **timer + subscriber-check** driven.
- **Phase 2:** remove the per-event and per-subscription monitoring triggers so monitoring no longer runs on the event-processing path.
- **Phase 5:** add **PostgreSQL `LISTEN`/`NOTIFY`** so the worker wakes on real data changes instead of pure polling.
Target test relay: **port 7777** (local).
---
## Current State (verified in code)
- `api_worker_main()` ([`src/api.c:172`](src/api.c:172)) blocks on `api_worker_pop_job_blocking()` and reacts to three job types: `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `API_WORK_JOB_STATUS_POST`.
- Monitoring hooks `monitoring_on_event_stored()` ([`src/api.c:752`](src/api.c:752)) and `monitoring_on_subscription_change()` ([`src/api.c:759`](src/api.c:759)) are called from [`src/main.c:1169`](src/main.c:1169) and [`src/subscriptions.c:483`](src/subscriptions.c:483), [`src/subscriptions.c:536`](src/subscriptions.c:536). They enqueue jobs; sync fallback runs if the worker is down.
- Throttling lives in `monitoring_on_event_stored_sync()` / `monitoring_on_subscription_change_sync()` ([`src/api.c:711`](src/api.c:711), [`src/api.c:731`](src/api.c:731)).
- Round-robin selection in `generate_round_robin_monitoring_event()` ([`src/api.c:571`](src/api.c:571)); per-type build/sign/broadcast in `generate_monitoring_event_for_type()` ([`src/api.c:621`](src/api.c:621)).
- `has_subscriptions_for_kind(int)` ([`src/subscriptions.c:980`](src/subscriptions.c:980)) checks only subscriptions with an explicit `kinds` filter — it does **not** consider the `no_kind_filter_subs` list, so it alone is insufficient for a "is anyone listening to kind 24567" check.
- Worker DB connection is a `PGconn*` opened via `db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)). Schema is embedded in [`src/pg_schema.h`](src/pg_schema.h) (version `"4"`) and mirrored in [`src/pg_schema.sql`](src/pg_schema.sql).
- No `LISTEN`/`NOTIFY`/`pg_notify`/`PQnotifies` usage exists anywhere in `src/`.
---
## Design
### Phase 1 + 2 + 5 combined worker loop
The three phases are coupled, so they are implemented together in one rewrite of `api_worker_main()` and its helpers.
### CRITICAL BUG FIX (why the web API isn't getting updates today)
`postgres_db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)) calls
`postgres_db_set_thread_connection(conn)` at line 243 — but `g_thread_pg_conn` is declared `__thread`
([`src/db_ops_postgres.c:45`](src/db_ops_postgres.c:45)), so the binding is set in the **caller's thread**
(lws-main, which calls `start_api_worker`), NOT in the api-worker thread. When `api_worker_main` runs and
the monitoring query functions call `postgres_db_active_connection()`, `g_thread_pg_conn` is NULL in that
thread, so they fall back to the shared global `g_pg_conn` — concurrent access with the main thread. This
causes monitoring queries to fail or interleave, which is why the dashboard stops receiving kind 24567
events.
**Fix:** inside `api_worker_main()`, after opening `worker_db`, call
`db_set_thread_connection(worker_db)` so the `__thread` pointer is bound in the api-worker thread itself.
Call `db_clear_thread_connection()` before closing the connection on shutdown. This must be done in the
new rewrite regardless of the loop design.
```
api-worker thread:
1. Open own PG connection.
2. db_set_thread_connection(worker_db) // bind __thread pointer in THIS thread
3. If PG backend: LISTEN event_stored on this connection.
4. Loop while running:
a. Determine if anyone is listening to kind 24567
(has_subscriptions_for_kind(24567) OR no_kind_filter_subs non-empty).
b. If NO subscribers:
- condvar_timedwait(throttle_sec) // wakeable for shutdown / STATUS_POST job
- on wake: process any pending STATUS_POST job, then continue.
// Zero DB work, zero notify polling.
c. If subscribers exist:
- wait_for_notify_or_timeout(throttle_sec):
select() on { PQsocket, self-pipe read fd } with timeout = throttle_sec.
- If PQsocket readable: PQconsumeInput + drain PQnotifies.
- If self-pipe readable: drain (shutdown or STATUS_POST signal).
- If timeout: fall through.
- generate_round_robin_monitoring_event() // one d-tag per tick
- process any pending STATUS_POST job.
4. On shutdown: UNLISTEN, close PG connection.
```
Key properties:
- **Zero overhead when no subscribers** — condvar sleep, no `select`, no `LISTEN` polling.
- **Reactive when subscribers exist** — wakes within `throttle_sec` of an `event_stored` NOTIFY, but never more than once per `throttle_sec` (rate-limited by the select timeout + the round-robin cadence already in `generate_round_robin_monitoring_event`).
- **STATUS_POST preserved** — still enqueued from [`src/websockets.c:3470`](src/websockets.c:3470); the self-pipe wakes the worker's `select` so status posts are not delayed by a pending notify wait.
- **SQLite fallback**`db_worker_poll_notify()` returns "not supported" on SQLite, so the worker falls back to pure timer behavior (condvar timed wait) — identical to the no-PG path.
### Self-pipe for job/shutdown signalling
A self-pipe (or `eventfd` on Linux) is added to `api.c`:
- `api_worker_enqueue_job(STATUS_POST)` writes one byte to the pipe → wakes `select`.
- `stop_api_worker()` sets `g_api_worker_running = 0`, writes a byte, and broadcasts the condvar (covers both the subscriber and no-subscriber wait paths).
### Subscriber-check helper
Add `int has_any_subscription_for_kind(int event_kind)` in [`src/subscriptions.c`](src/subscriptions.c) that returns true if `has_subscriptions_for_kind(event_kind)` OR the `no_kind_filter_subs` list is non-empty. Export it in [`src/subscriptions.h`](src/subscriptions.h). The worker calls `has_any_subscription_for_kind(24567)`.
### DB abstraction for LISTEN/NOTIFY
Add to [`src/db_ops.h`](src/db_ops.h) / [`src/db_ops_postgres.c`](src/db_ops_postgres.c) / [`src/db_ops_sqlite.c`](src/db_ops_sqlite.c):
- `int db_worker_listen(void* conn, const char* channel);` — issues `LISTEN <channel>`.
- `int db_worker_poll_notify(void* conn, int timeout_ms);` — returns `1` if a notification was consumed, `0` on timeout, `-1` on error/unsupported. Uses `PQsocket()`, `select()`, `PQconsumeInput()`, `PQnotifies()`.
- SQLite stubs return `-1` (unsupported) so the worker falls back to timer mode.
### Schema change (Phase 5)
Add to both [`src/pg_schema.sql`](src/pg_schema.sql) and [`src/pg_schema.h`](src/pg_schema.h):
```sql
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('event_stored', json_build_object(
'kind', NEW.kind,
'pubkey', substring(NEW.pubkey, 1, 8)
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_notify_event_stored ON events;
CREATE TRIGGER trg_notify_event_stored
AFTER INSERT ON events
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
```
Bump `EMBEDDED_PG_SCHEMA_VERSION` from `"4"` to `"5"` in [`src/pg_schema.h`](src/pg_schema.h) and the matching `schema_info` version write in [`src/pg_schema.sql`](src/pg_schema.sql). The trigger is idempotent (`DROP TRIGGER IF EXISTS` + `CREATE`), so existing databases upgrade automatically on next startup via `postgres_db_apply_schema()` ([`src/db_ops_postgres.c:134`](src/db_ops_postgres.c:134)).
### Phase 2 removals
- Delete the `monitoring_on_event_stored()` call at [`src/main.c:1169`](src/main.c:1169) (and its forward decl at [`src/main.c:459`](src/main.c:459)).
- Delete the `monitoring_on_subscription_change()` calls at [`src/subscriptions.c:483`](src/subscriptions.c:483) and [`src/subscriptions.c:536`](src/subscriptions.c:536) (and its forward decl at [`src/subscriptions.c:58`](src/subscriptions.c:58)).
- Remove from [`src/api.h`](src/api.h): `monitoring_on_event_stored`, `monitoring_on_subscription_change`.
- Remove from [`src/api.c`](src/api.c): `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `monitoring_on_event_stored`, `monitoring_on_subscription_change`, `monitoring_on_event_stored_sync`, `monitoring_on_subscription_change_sync`. Keep `API_WORK_JOB_STATUS_POST`, `generate_round_robin_monitoring_event`, `generate_monitoring_event_for_type`, `generate_event_driven_monitoring`/`generate_subscription_driven_monitoring` (still used by `generate_monitoring_event` legacy path and tests).
- The `generate_event_driven_monitoring` / `generate_subscription_driven_monitoring` wrappers become unused by the worker but remain as public helpers for any external/test callers; the worker calls `generate_round_robin_monitoring_event()` directly.
### Thread layout (unchanged from plan target)
```
c_relay_pg process
├── lws-main — WebSocket event loop (no monitoring DB work)
├── db-read-1..N — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion (no monitoring hook)
├── db-write-1..N — Async sub logging, misc writes
└── api-worker — Timer + LISTEN/NOTIFY driven monitoring (own PG conn)
```
---
## Mermaid: api-worker state machine
```mermaid
stateDiagram-v2
[*] --> OpenConn
OpenConn --> BindThread: db_set_thread_connection
BindThread --> Listen: PG backend
BindThread --> Ready: SQLite/no-PG
Listen --> Ready
Ready --> CheckSubs
CheckSubs --> NoSubSleep: no kind-24567 subs
CheckSubs --> WaitNotify: subs present
NoSubSleep --> CheckSubs: throttle_sec or wake
WaitNotify --> Generate: NOTIFY or throttle_sec
Generate --> CheckSubs: after round-robin tick
WaitNotify --> NoSubSleep: subs dropped + wake
Ready --> [*]: stop_api_worker
```
---
## Implementation Todo List
1. Add `has_any_subscription_for_kind()` to `subscriptions.c`/`subscriptions.h` (checks kind index + no-kind-filter list).
2. Add `db_worker_listen()` and `db_worker_poll_notify()` to `db_ops.h`, `db_ops_postgres.c` (libpq), and `db_ops_sqlite.c` (stub).
3. Add `notify_event_stored()` function + `trg_notify_event_stored` trigger to `pg_schema.sql` and `pg_schema.h`; bump schema version to 5.
4. Rewrite `api_worker_main()` in `api.c`: bind the worker PG connection to the thread via `db_set_thread_connection()` (fixes the dashboard-not-updating bug), then run the timer + LISTEN/NOTIFY + subscriber-check loop; add self-pipe for STATUS_POST/shutdown wakeups; keep `start_api_worker`/`stop_api_worker`/`api_worker_process_completions`/`api_worker_enqueue_status_post` signatures.
5. Remove `API_WORK_JOB_EVENT_STORED`/`API_WORK_JOB_SUBSCRIPTION_CHANGE` job types and the `monitoring_on_event_stored`/`monitoring_on_subscription_change` functions + sync wrappers from `api.c` and `api.h`.
6. Remove the `monitoring_on_event_stored()` call from `main.c` (event storage path).
7. Remove the `monitoring_on_subscription_change()` calls from `subscriptions.c` (subscription create/close paths).
8. Build with `./make_and_restart_relay.sh` on port 7777; fix any compile errors.
9. Test: subscribe to kind 24567 on ws://localhost:7777 → confirm monitoring events arrive on throttle cadence (this validates the thread-connection fix).
10. Test: publish a kind-1 event → confirm a NOTIFY-driven monitoring event fires within throttle_sec.
11. Test: with no kind-24567 subscribers → confirm no monitoring events generated (zero overhead) and event storage latency unaffected.
12. Run `tests/quick_error_tests.sh` and `tests/subscribe_all.sh` against port 7777 to confirm no regressions.
---
## Risks & Mitigations
- **Self-pipe + condvar mix complexity.** Mitigation: the no-subscriber path uses only the condvar; the subscriber path uses `select` on the self-pipe + PQ socket. The condvar is only used to wake the no-subscriber sleep, so the two wait mechanisms never overlap.
- **`LISTEN` on a worker connection that also runs monitoring queries.** `LISTEN` is connection-local and persists; running `SELECT`s on the same connection does not cancel it. `PQconsumeInput` must be called before `PQnotifies`. Mitigation: encapsulate all of this in `db_worker_poll_notify()`.
- **Schema trigger on every insert adds per-event overhead.** `pg_notify` is cheap (in-memory queue) and the payload is tiny. The existing `sync_event_tags_from_events` AFTER INSERT trigger already does far more work per row, so this is negligible by comparison.
- **SQLite has no LISTEN/NOTIFY.** Mitigation: `db_worker_poll_notify` returns `-1` on SQLite and the worker falls back to pure timer mode — functionally identical to the no-notify path.
- **Removing the per-event monitoring hook changes dashboard freshness behavior.** Mitigation: with subscribers present, NOTIFY now drives near-real-time updates (better than before); with no subscribers, nothing is generated (matches the plan's "zero overhead" goal). Dashboard already handles event-driven updates via its kind-24567 subscription.
+224
View File
@@ -0,0 +1,224 @@
# API Upgrade Plan: Monitoring Thread + PostgreSQL Push
## Current API Architecture
### How Monitoring Works Today
The relay publishes monitoring data as **kind 24567 ephemeral Nostr events**. The admin dashboard is a standard Nostr client that subscribes to these events.
```
Dashboard connects → subscribes to kind 24567 with d-tags
Relay detects subscription → starts generating monitoring events
Relay queries DB → builds JSON → signs event → broadcasts to subscribers
Dashboard receives events → renders stats/charts
```
### Current Monitoring Event Types
| d-tag | Query Function | What It Contains | Triggered By |
|-------|---------------|-----------------|-------------|
| `event_kinds` | `query_event_kind_distribution()` | Event count per kind, total events | Event storage |
| `time_stats` | `query_time_based_statistics()` | Events in last 24h/7d/30d | Event storage |
| `top_pubkeys` | `query_top_pubkeys()` | Top 10 pubkeys by event count | Event storage |
| `subscription_details` | `query_subscription_details()` | Active subscriptions list | Subscription changes |
| `cpu_metrics` | `query_cpu_metrics()` | PID, memory, CPU, connections | Both triggers |
### Current Admin Command System
Two parallel admin interfaces exist:
**1. Kind 23456 Admin Events (WebSocket)**
- Admin sends NIP-44 encrypted commands as kind 23456 events
- Relay decrypts, processes, responds with kind 23457 events
- Commands: config get/set, auth rules, system commands, SQL queries
**2. NIP-17 DM Admin (Direct Messages)**
- Admin sends gift-wrapped DMs (kind 1059) to relay
- Relay unwraps, processes commands, responds via DM
- Commands: stats, config, help, SQL queries
### Current Problems
1. **All monitoring runs on lws-main thread** — DB queries block the event loop
2. **Monitoring is triggered by event storage**`generate_event_driven_monitoring()` runs synchronously after each event insert
3. **No dedicated monitoring thread** — monitoring piggybacks on the relay's main processing
4. **Throttle is time-based only**`kind_24567_reporting_throttle_sec` (default 5s) but queries still run on main thread
5. **Dashboard polls via subscription** — no way to get data without the relay actively generating events
---
## Upgrade Plan
### Phase 1: Dedicated API Thread (`api-worker`)
**What:** Create a new thread that owns all monitoring event generation. It runs on a timer, queries PostgreSQL with its own connection, builds and signs monitoring events, and pushes them into the broadcast system.
**Thread behavior:**
```
api-worker thread:
1. Open own PG connection
2. Loop:
a. Sleep for throttle_sec interval
b. Check if anyone is subscribed to kind 24567
c. If yes: run all monitoring queries
d. Build kind 24567 events with results
e. Sign events with relay key
f. Push completed events to lws-main for broadcast
g. Repeat
```
**Key design decisions:**
- The api-worker thread does ALL the heavy work (queries + signing)
- It only hands off pre-built, signed events to lws-main for broadcast
- lws-main never blocks on monitoring queries
- The thread sleeps when no one is subscribed (zero overhead)
**Files to change:**
- `src/api.c`: Refactor `generate_monitoring_event_for_type()` to be callable from any thread
- `src/websockets.c`: Add monitor thread lifecycle (start/stop), completion queue for broadcast
- `src/thread_pool.h`: Add api-worker thread config options
**New thread in the process:**
```
c_relay_pg process
├── lws-main — WebSocket event loop
├── db-read-1..N — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion
├── db-write-1..N — Async sub logging, misc writes
└── api-worker — NEW: periodic monitoring event generation
```
### Phase 2: Remove Monitoring from Main Thread Path
**What:** Remove `generate_event_driven_monitoring()` and `generate_subscription_driven_monitoring()` calls from the synchronous event processing path.
**Currently these are called from:**
- After event storage (event-driven monitoring)
- After subscription create/close (subscription-driven monitoring)
**After change:**
- The api-worker thread handles all monitoring on its own timer
- Event storage and subscription changes no longer trigger monitoring directly
- The throttle interval controls how fresh the data is (default 5 seconds)
**Benefit:** Event processing becomes faster — no monitoring overhead per event.
### Phase 3: Dashboard Uses Normal Subscriptions for Real-Time Events
**What:** Instead of the relay generating special monitoring events for "recent events," the dashboard simply subscribes to all events using standard Nostr REQ filters.
**Current approach:** Dashboard subscribes to kind 24567 monitoring events that contain aggregated stats.
**New approach for real-time event feed:**
```javascript
// Dashboard subscribes to ALL events on the relay
relayPool.subscribeMany([url], [
{ kinds: [1, 3, 5, 7, ...], limit: 50 } // Recent events
], {
onevent(event) {
// Render in real-time event feed
addEventToFeed(event);
}
});
```
**What this replaces:** The "Relay Events" page in the dashboard currently shows events from monitoring. Instead, it would show live events as they arrive via normal Nostr subscription.
**What stays as monitoring events:** Aggregated stats (event_kinds, time_stats, top_pubkeys, cpu_metrics, subscription_details) still need to be generated by the relay because they require SQL aggregation queries that a client can't do.
### Phase 4: Evaluate Which Monitoring Events Can Be Replaced
For each current monitoring event type, here's whether it should stay as a relay-generated event or be replaced by dashboard-side logic:
| d-tag | Current: Relay generates | Could dashboard do it? | Recommendation |
|-------|------------------------|----------------------|----------------|
| `event_kinds` | SQL: `SELECT kind, COUNT(*) FROM events GROUP BY kind` | No — requires DB aggregation | **Keep as monitoring event** |
| `time_stats` | SQL: `SELECT COUNT(*) FROM events WHERE created_at > ?` for 24h/7d/30d | No — requires DB aggregation | **Keep as monitoring event** |
| `top_pubkeys` | SQL: `SELECT pubkey, COUNT(*) FROM events GROUP BY pubkey ORDER BY count DESC LIMIT 10` | No — requires DB aggregation | **Keep as monitoring event** |
| `subscription_details` | SQL: `SELECT * FROM active_subscriptions_log` | No — requires DB access | **Keep as monitoring event** |
| `cpu_metrics` | Reads `/proc/self/stat`, memory info | No — requires server-side access | **Keep as monitoring event** |
| Real-time event feed | Currently via monitoring events | **Yes — normal Nostr subscription** | **Replace with REQ subscription** |
| Auth rules list | Currently via admin command | No — requires DB access | **Keep as admin command** |
| Config values | Currently via admin command | No — requires DB access | **Keep as admin command** |
**Conclusion:** All 5 monitoring event types should stay as relay-generated events (they require SQL aggregation or server-side data). The only thing that changes is the real-time event feed, which becomes a normal subscription.
### Phase 5: PostgreSQL LISTEN/NOTIFY for Monitoring Triggers (Future)
**What:** Instead of the monitor thread polling on a timer, PostgreSQL can push notifications when data changes.
**Example triggers:**
```sql
-- Notify when a new event is stored
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('event_stored', json_build_object(
'kind', NEW.kind,
'pubkey', substring(NEW.pubkey, 1, 8)
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER trg_notify_event_stored
AFTER INSERT ON events
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
```
**How the monitor thread would use it:**
```
api-worker thread:
1. LISTEN event_stored
2. Wait for notification (blocks efficiently)
3. On notification: batch up for throttle_sec, then generate monitoring events
4. Repeat
```
**Benefit:** Zero-polling. The monitor thread only wakes up when data actually changes.
**This is optional and can be added later.** The timer-based approach in Phase 1 works well and is simpler.
---
## Updated Thread Layout (After All Phases)
```
c_relay_pg process
├── lws-main — WebSocket event loop (never blocks on DB for monitoring)
├── db-read-1..4 — Async REQ/COUNT queries
├── event-worker — Async EVENT ingestion (validate + store)
├── db-write-1..2 — Async sub logging, IP bans, misc writes
└── api-worker — Periodic monitoring event generation (own PG conn)
```
**Total: 9 threads, 9 PG connections** (with 4 readers + 2 writers)
---
## Implementation Priority
| Phase | Description | Complexity | Impact |
|-------|-------------|-----------|--------|
| 1 | Dedicated api-worker thread | Medium | High — unblocks main thread |
| 2 | Remove monitoring from event processing path | Low | Medium — faster event processing |
| 3 | Dashboard uses normal subscriptions for event feed | Low (frontend only) | Medium — simpler, more reliable |
| 4 | Evaluate monitoring event types | Analysis only | Confirms architecture |
| 5 | PostgreSQL LISTEN/NOTIFY triggers | Medium | Low (optimization) — add later |
**Recommended start:** Phase 1 + 2 together (they're coupled), then Phase 3 (frontend change), then Phase 5 when needed.
---
## Relationship to Thread Pool Plan
This plan complements `thread_pool_pg_optimization_plan.md`:
- Thread pool plan handles **reader/writer scaling** for client-facing operations
- This plan handles **monitoring/API operations** on a dedicated thread
- Both plans share the same PG connection model (one connection per thread)
- The api-worker thread is independent from the thread pool — it has its own lifecycle
The combined target is:
```
lws-main (1) + readers (4) + event-worker (1) + writers (2) + api-worker (1) = 9 threads
```
+240
View File
@@ -0,0 +1,240 @@
# C-Relay-PG Admin Page Migration Plan
## Overview
Migrate the relay admin dashboard from an embedded HTTP-served page (`api/index.html` + `api/index.js`) to a full client-ndk page (`c-relay-pg.html`) built on the client-ndk template. The page will live in the client-ndk project (`~/lt/client-ndk/www/`) alongside other pages and communicate with the relay purely through Nostr protocol subscriptions.
## Why This Migration
The current `/api` page is served directly by the relay's `lws-main` thread, which causes:
- **Slow page delivery** — HTTP file serving competes with WebSocket event processing on a single thread
- **Incomplete page loads** — 10-second `timeout_secs` kills connections when the main thread is busy
- **6+ sequential HTTP round-trips** — each asset requires a two-phase header/body delivery cycle
- **Unnecessary complexity** — embedded file generation, `lws_set_wsi_user()` overwriting, session data type dispatch
The admin page already communicates entirely via Nostr protocol:
- **Kind 23456** → Admin commands (encrypted, sent by admin)
- **Kind 23457** → Admin responses (encrypted, sent by relay)
- **Kind 24567** → Monitoring/stats events (ephemeral, broadcast by relay)
- **NIP-11** → Relay info (fetched via HTTP, independent of file serving)
- **NIP-17** → DM admin commands (gift-wrapped)
Moving to client-ndk means the relay serves zero HTML — it only speaks Nostr protocol.
## Architecture
```mermaid
graph LR
subgraph client-ndk project
A[c-relay-pg.html] --> B[init-ndk.mjs]
B --> C[NDK SharedWorker]
end
subgraph c-relay-pg
D[WebSocket Server]
E[api-worker thread]
end
C -->|kind 23456 admin cmds| D
D -->|kind 23457 responses| C
E -->|kind 24567 monitoring| D
D -->|kind 24567 events| C
C -->|kind 1059 NIP-17 DMs| D
```
## Current Admin Page Sections
| Section | Data Source | Communication |
|---------|-----------|---------------|
| Statistics | kind 24567 monitoring events with d-tags: event_kinds, time_stats, top_pubkeys, cpu_metrics | Subscribe to kind 24567 |
| Subscriptions | kind 24567 monitoring events with d-tag: subscription_details | Subscribe to kind 24567 |
| Configuration | kind 23457 admin responses | Send kind 23456 config_query, receive kind 23457 |
| Authorization | kind 23457 admin responses | Send kind 23456 auth rule commands, receive kind 23457 |
| IP Bans | kind 23457 admin responses | Send kind 23456 ip_ban_* commands, receive kind 23457 |
| Relay Events | Live subscription to all events + kind 0/10050/10002 management | Standard Nostr subscriptions |
| DM | NIP-17 gift-wrapped DMs kind 1059 | NIP-17 encrypt/wrap/publish |
| SQL Query | kind 23457 admin responses | Send kind 23456 sql_query, receive kind 23457 |
## Design Decisions
1. **Full client-ndk page** — Uses `client.css`, `init-ndk.mjs`, `HamburgerMorphing`, `relay-ui.mjs`, `blossom-ui.mjs`, `ai-ui.mjs` — all standard template infrastructure stays intact
2. **Sidenav preserved** — All existing sidenav sections (Relay, Blossom, AI) remain. Admin page navigation links are added to `divSideNavBody` as clickable items
3. **Page-specific CSS** — Follows client-ndk convention: page styles in inline `<style>` block, using only CSS variables from `client.css`
4. **Hardcoded relay URL**`wss://relay.laantungir.net` for now; future enhancement to support relay selection
5. **Auth mode: required** — Admin page requires authentication
## Migration Steps
### Step 1: Create `c-relay-pg.html` from Template
Start from `template.html` in client-ndk. Customize:
- **Title**: `C-Relay-PG`
- **Header text**: `C-Relay-PG Admin`
- **Auth mode**: `required` (default)
- **`divBody` content**: All 8 admin sections, each in its own container div, all `display:none` except Statistics (default)
- **`divSideNavBody`**: Add navigation links for each section:
```html
<div id="divAdminNav">
<div class="adminNavItem active" data-section="statistics">Statistics</div>
<div class="adminNavItem" data-section="subscriptions">Subscriptions</div>
<div class="adminNavItem" data-section="configuration">Configuration</div>
<div class="adminNavItem" data-section="authorization">Authorization</div>
<div class="adminNavItem" data-section="ip-bans">IP Bans</div>
<div class="adminNavItem" data-section="relay-events">Relay Events</div>
<div class="adminNavItem" data-section="dm">DM</div>
<div class="adminNavItem" data-section="database">Database Query</div>
</div>
```
### Step 2: Port Admin Communication Layer
Replace `SimplePool` + `nostr-lite.js` with NDK equivalents:
| Current index.js | New c-relay-pg.html |
|-----------------|---------------------|
| `new SimplePool()` | `subscribe()` from init-ndk.mjs |
| `relayPool.subscribeMany()` | `subscribe(filter, opts)` |
| `relayPool.publish()` | `publishEvent(event)` |
| `window.nostr.signEvent()` | Handled by NDK MessageBasedSigner |
| `window.NostrTools.nip42` | Handled by NDK NIP-42 support |
| `window.NostrTools.nip44` | `window.NostrTools.nip44` from nostr.bundle.js — same as before |
| `nlLite` nostr-login-lite | `initNDKPage()` + `getPubkey()` |
Key function — `sendAdminCommand(commandArray)`:
```javascript
async function sendAdminCommand(commandArray) {
const conversationKey = window.NostrTools.nip44.v2.utils.getConversationKey(
adminPrivateKey, relayPubkey
);
const encrypted = window.NostrTools.nip44.v2.encrypt(
JSON.stringify(commandArray), conversationKey
);
const event = {
created_at: Math.floor(Date.now() / 1000),
kind: 23456,
tags: [['p', relayPubkey]],
content: encrypted
};
return await publishEvent(event);
}
```
Note: NIP-44 encryption requires the admin's private key. The current page uses `window.nostr.nip44.encrypt()` which delegates to the browser extension. NDK's `publishEvent` handles signing, but encryption must be done before publishing. We'll use `window.nostr.nip44.encrypt()` if available (NIP-07 extension), same as the current page does.
### Step 3: Port Subscription Setup
Using NDK's `subscribe()` and `window.addEventListener('ndkEvent', ...)`:
```javascript
// Subscribe to admin responses + monitoring events from relay
const adminSub = subscribe(
{ kinds: [23457, 24567], authors: [RELAY_PUBKEY] },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Live event feed
const liveSub = subscribe(
{ limit: 50 },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Route events to handlers
window.addEventListener('ndkEvent', (e) => {
const evt = e.detail;
if (evt.kind === 23457) processAdminResponse(evt);
else if (evt.kind === 24567) processMonitoringEvent(evt);
else addEventToLiveFeed(evt);
});
```
### Step 4: Port NIP-44 Encryption
Admin commands require NIP-44 encryption. The template already loads `nostr.bundle.js` which includes `window.NostrTools.nip44`. Additionally, `window.nostr.nip44.encrypt()` is available from NIP-07 extensions. Use the same approach as the current page.
### Step 5: Port UI Sections into divBody
Each section's HTML goes into `divBody` as a container div. Page-specific styles go in the inline `<style>` block following client-ndk conventions:
- Use CSS variables only (no hardcoded colors)
- Tables use `var(--primary-color)`, `var(--muted-color)`, `var(--accent-color)`
- Buttons follow the `var(--button-*)` pattern
- Font is always `var(--font-family)`
Sections to port:
1. **Statistics** — DB overview table, event kinds table, time stats table, top pubkeys table, event rate chart area
2. **Subscriptions** — Active subscription details table
3. **Configuration** — Config key/value table with inline edit, refresh button
4. **Authorization** — Auth rules table, whitelist/blacklist inputs, WoT level selector
5. **IP Bans** — Ban stats, manual ban form, IP whitelist, ban list table with filters
6. **Relay Events** — Live event feed table, kind 0/10050/10002 management forms
7. **DM** — NIP-17 message textarea + send button, inbox display
8. **SQL Query** — Query dropdown, SQL textarea, execute button, results table
### Step 6: Page-Specific CSS
Following client-ndk rules from `client.css`:
- Override `#divBody` layout: `flex-direction: column; overflow-y: auto; padding: 20px;`
- Admin section containers: hidden by default, shown when nav item clicked
- Table styles: use `var(--primary-color)` for headers, `var(--muted-color)` for borders
- Button styles: border + transparent background, hover with `var(--accent-color)`
- All in inline `<style media="screen">` block in `<head>`
### Step 7: Relay URL — Hardcoded for Now
```javascript
const RELAY_WS_URL = 'wss://relay.laantungir.net';
const RELAY_HTTP_URL = 'https://relay.laantungir.net';
```
NIP-11 fetch to get relay pubkey:
```javascript
const nip11 = await fetch(RELAY_HTTP_URL, {
headers: { 'Accept': 'application/nostr+json' }
}).then(r => r.json());
const RELAY_PUBKEY = nip11.pubkey;
```
### Step 8: Admin Verification
Same flow as current page:
1. Get relay pubkey from NIP-11
2. Send kind 23456 `system_status` command
3. If relay responds with kind 23457, user is verified as admin
4. Show admin sections; otherwise show access denied
### Step 9 (Optional, Later): Remove Embedded HTTP Serving from Relay
After migration is complete and tested:
- Remove `handle_embedded_file_request()` from `src/api.c`
- Remove `handle_embedded_file_writeable()` from `src/api.c`
- Remove embedded file dispatch from `LWS_CALLBACK_HTTP` in `src/websockets.c`
- Remove `embed_web_files.sh` and `src/embedded_web_content.c`
- Keep NIP-11 serving (lightweight and required by protocol)
## File Changes Summary
### New File (in client-ndk project: ~/lt/client-ndk/www/)
- `c-relay-pg.html` — Complete admin page, self-contained with inline `<style>` and `<script type="module">`
### Files Unchanged
- All relay C code for kind 23456/23457/24567 processing — stays exactly the same
- `src/dm_admin.c` — NIP-17 DM handling stays the same
- `src/nip011.c` — NIP-11 stays the same
- All client-ndk shared files (client.css, init-ndk.mjs, relay-ui.mjs, etc.) — no modifications needed
### Files to Modify Later (optional relay cleanup)
- `src/api.c` — Remove embedded file serving functions
- `src/websockets.c` — Remove embedded file dispatch from HTTP callback
- `Makefile` — Remove embedded web content compilation step
## Risks and Mitigations
| Risk | Mitigation |
|------|-----------|
| NIP-44 encryption via window.nostr.nip44 | Same mechanism as current page — no change in encryption approach |
| NDK SharedWorker relay connection | Use ONLY_RELAY cache usage to ensure direct relay communication |
| Large page size with all sections inline | Sections are display:none by default — no performance impact |
| Event rate chart from text_graph.js | Port chart logic inline or skip initially |
| Relay pubkey needed before subscriptions | Fetch NIP-11 first in initialization sequence |
+632
View File
@@ -0,0 +1,632 @@
# c-relay-pg-pg Plan — PostgreSQL Backend + Multi-Instance + Dashboard
## Overview
**c-relay-pg-pg** is a new project (separate repository) forked from c-relay-pg after the `db_ops` abstraction layer is complete. It replaces the SQLite backend with PostgreSQL, enabling horizontal scaling, external dashboards, and production-grade deployments.
### Prerequisites
- The `db_ops.h` / `db_ops.c` abstraction layer must be complete in c-relay-pg first — see [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1.
- The fork happens after Phase 1 of that plan is done and tested.
### Why a Separate Project?
| | c-relay-pg | c-relay-pg-pg |
|---|---------|-----------|
| **Database** | SQLite (embedded) | PostgreSQL (client-server) |
| **Deployment** | Single binary + DB file | Binary + PostgreSQL server |
| **Scaling** | Single instance | Multiple instances + load balancer |
| **Dashboard** | Embedded web UI | Separate web server + Grafana |
| **Target user** | Personal relay, small community | Medium-large relay, production |
| **Complexity** | Minimal | More infrastructure |
For the full analysis of why PostgreSQL was chosen over MySQL/MariaDB and other databases, see the [database architecture analysis](database_architecture_analysis.md).
## Architecture
```mermaid
flowchart TD
subgraph c-relay-pg-pg - Production Deployment
LB[nginx - TLS + load balancing] -->|WebSocket| R1[c-relay-pg-pg Instance 1]
LB -->|WebSocket| R2[c-relay-pg-pg Instance 2]
LB -->|HTTP| DASH[Dashboard]
R1 -->|db_ops API| PGBACK[PostgreSQL Backend - db_ops_postgres.c]
R2 -->|db_ops API| PGBACK
PGBACK -->|libpq| PG[PostgreSQL Server]
DASH -->|SQL| PG
R1 <-->|LISTEN/NOTIFY| R2
end
```
---
## Phase 1: PostgreSQL Backend
### Goal
Fork c-relay-pg into a new repository called **c-relay-pg-pg**. Replace the SQLite `db_ops` implementation with PostgreSQL using `libpq`. The `db_ops.h` interface stays identical — only the backend changes.
### New Files
- `src/db_ops_sqlite.c` — Renamed from `db_ops.c` (the Phase 1 SQLite implementation from c-relay-pg)
- `src/db_ops_postgres.c` — New PostgreSQL implementation
- `src/db_ops.c` — Thin dispatcher that calls the active backend
### PostgreSQL Schema
```sql
-- PostgreSQL schema for c-relay-pg-pg
-- No event_tags table needed — JSONB + GIN handles everything
CREATE TABLE events (
id TEXT PRIMARY KEY,
pubkey TEXT NOT NULL,
created_at BIGINT NOT NULL,
kind INTEGER NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
content TEXT NOT NULL,
sig TEXT NOT NULL,
tags JSONB NOT NULL DEFAULT '[]',
event_json TEXT NOT NULL,
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Core indexes
CREATE INDEX idx_events_pubkey ON events(pubkey);
CREATE INDEX idx_events_kind ON events(kind);
CREATE INDEX idx_events_created_at ON events(created_at DESC);
CREATE INDEX idx_events_kind_created_at ON events(kind, created_at DESC);
CREATE INDEX idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
CREATE INDEX idx_events_pubkey_kind ON events(pubkey, kind);
-- THE KEY INDEX: GIN on JSONB tags — replaces the entire event_tags table
CREATE INDEX idx_events_tags ON events USING GIN (tags);
-- Partial index: only non-ephemeral events (what REQ queries actually filter)
CREATE INDEX idx_events_non_ephemeral ON events(created_at DESC)
WHERE kind < 20000 OR kind >= 30000;
-- Config table
CREATE TABLE config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
data_type TEXT NOT NULL CHECK (data_type IN ('string', 'integer', 'boolean', 'json')),
description TEXT,
category TEXT DEFAULT 'general',
requires_restart INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Auth rules table
CREATE TABLE auth_rules (
id SERIAL PRIMARY KEY,
rule_type TEXT NOT NULL CHECK (rule_type IN ('whitelist', 'blacklist', 'rate_limit', 'auth_required', 'wot_whitelist')),
pattern_type TEXT NOT NULL CHECK (pattern_type IN ('pubkey', 'kind', 'ip', 'global')),
pattern_value TEXT,
active INTEGER NOT NULL DEFAULT 1,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX idx_auth_rules_lookup ON auth_rules(rule_type, pattern_type, pattern_value) WHERE active = 1;
-- Relay private key storage
CREATE TABLE relay_seckey (
private_key_hex TEXT NOT NULL CHECK (length(private_key_hex) = 64),
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
-- Subscription logging
CREATE TABLE subscriptions (
id SERIAL PRIMARY KEY,
subscription_id TEXT NOT NULL,
wsi_pointer TEXT NOT NULL,
client_ip TEXT NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('created', 'closed', 'expired', 'disconnected')),
filter_json TEXT,
events_sent INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
ended_at BIGINT,
duration INTEGER,
UNIQUE(subscription_id, wsi_pointer)
);
-- IP ban persistence
CREATE TABLE ip_bans (
ip TEXT PRIMARY KEY,
failure_count INTEGER NOT NULL DEFAULT 0,
ban_count INTEGER NOT NULL DEFAULT 0,
banned_until BIGINT NOT NULL DEFAULT 0,
first_failure BIGINT NOT NULL DEFAULT 0,
has_authed_successfully INTEGER NOT NULL DEFAULT 0,
last_success_at BIGINT NOT NULL DEFAULT 0,
total_connections INTEGER NOT NULL DEFAULT 0,
total_failures INTEGER NOT NULL DEFAULT 0,
total_successes INTEGER NOT NULL DEFAULT 0,
first_seen BIGINT NOT NULL DEFAULT 0,
idle_failure_count INTEGER NOT NULL DEFAULT 0,
idle_ban_count INTEGER NOT NULL DEFAULT 0,
idle_banned_until BIGINT NOT NULL DEFAULT 0,
idle_first_failure BIGINT NOT NULL DEFAULT 0
);
-- Materialized views for dashboard (refreshed periodically)
CREATE MATERIALIZED VIEW event_kinds_mv AS
SELECT kind, COUNT(*) as count,
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
FROM events GROUP BY kind;
CREATE MATERIALIZED VIEW time_stats_mv AS
SELECT 'total' as period, COUNT(*) as total_events, COUNT(DISTINCT pubkey) as unique_pubkeys
FROM events
UNION ALL
SELECT '24h', COUNT(*), COUNT(DISTINCT pubkey)
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 86400
UNION ALL
SELECT '7d', COUNT(*), COUNT(DISTINCT pubkey)
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 604800;
CREATE MATERIALIZED VIEW top_pubkeys_mv AS
SELECT pubkey, COUNT(*) as event_count,
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
FROM events GROUP BY pubkey ORDER BY event_count DESC LIMIT 20;
-- Unique indexes required for CONCURRENTLY refresh
CREATE UNIQUE INDEX idx_event_kinds_mv ON event_kinds_mv(kind);
CREATE UNIQUE INDEX idx_time_stats_mv ON time_stats_mv(period);
CREATE UNIQUE INDEX idx_top_pubkeys_mv ON top_pubkeys_mv(pubkey);
```
### Key Implementation Details
#### Tag Queries with JSONB
The biggest win — replacing the `event_tags` subquery with JSONB containment:
```c
// SQLite (current): subquery into event_tags table
// AND id IN (SELECT event_id FROM event_tags WHERE tag_name = ? AND tag_value IN (?))
// PostgreSQL: JSONB containment operator with GIN index
// AND tags @> '[["p", "pubkey_hex"]]'
// For multiple tag values:
// AND (tags @> '[["p", "val1"]]' OR tags @> '[["p", "val2"]]')
```
This eliminates the entire `event_tags` table (4 million rows, ~2 GB indexes in the current SQLite schema). The GIN index on JSONB handles everything in ~100200 MB.
#### LISTEN/NOTIFY Integration
For cross-instance event broadcasting:
```c
// In db_ops_postgres.c:
int db_notify_new_event(const char* event_id) {
char cmd[128];
snprintf(cmd, sizeof(cmd), "NOTIFY new_event, '%s'", event_id);
PGresult* res = PQexec(g_pg_conn, cmd);
PQclear(res);
return 0;
}
// Called from the lws event loop every iteration:
int db_check_notifications(char* event_id_out, size_t max_len) {
PQconsumeInput(g_pg_notify_conn);
PGnotify* notify = PQnotifies(g_pg_notify_conn);
if (notify) {
strncpy(event_id_out, notify->extra, max_len);
PQfreemem(notify);
return 1; // Got a notification
}
return 0; // No notifications
}
```
#### Connection Management
```c
// db_ops_postgres.c connection pool (simple version)
#define DB_POOL_SIZE 4
static PGconn* g_pg_read_pool[DB_POOL_SIZE]; // For REQ queries
static PGconn* g_pg_write_conn; // For EVENT inserts
static PGconn* g_pg_notify_conn; // For LISTEN/NOTIFY
static pthread_mutex_t g_pool_lock;
PGconn* db_get_read_connection(void) {
pthread_mutex_lock(&g_pool_lock);
// Round-robin or find idle connection
// ...
pthread_mutex_unlock(&g_pool_lock);
}
```
### Build System Changes
```makefile
# Makefile additions
DB_BACKEND ?= sqlite # Default to sqlite, override with: make DB_BACKEND=postgres
ifeq ($(DB_BACKEND),postgres)
MAIN_SRC += src/db_ops.c src/db_ops_postgres.c
LIBS += -lpq
CFLAGS += -DDB_BACKEND_POSTGRES
else
MAIN_SRC += src/db_ops.c src/db_ops_sqlite.c
CFLAGS += -DDB_BACKEND_SQLITE
endif
```
### Data Migration Tool
A standalone tool to migrate existing SQLite data to PostgreSQL:
```bash
# migrate_to_postgres.sh
# 1. Export events from SQLite
sqlite3 old_relay.db ".mode csv" "SELECT id,pubkey,created_at,kind,event_type,content,sig,tags,event_json,first_seen FROM events" > events.csv
# 2. Import into PostgreSQL
psql -d crelay -c "\COPY events FROM 'events.csv' WITH CSV"
# 3. Migrate config
sqlite3 old_relay.db ".mode csv" "SELECT key,value,data_type,description,category,requires_restart FROM config" > config.csv
psql -d crelay -c "\COPY config(key,value,data_type,description,category,requires_restart) FROM 'config.csv' WITH CSV"
# 4. Migrate auth rules
sqlite3 old_relay.db ".mode csv" "SELECT rule_type,pattern_type,pattern_value,active FROM auth_rules" > auth_rules.csv
psql -d crelay -c "\COPY auth_rules(rule_type,pattern_type,pattern_value,active) FROM 'auth_rules.csv' WITH CSV"
# 5. Refresh materialized views
psql -d crelay -c "REFRESH MATERIALIZED VIEW event_kinds_mv; REFRESH MATERIALIZED VIEW time_stats_mv; REFRESH MATERIALIZED VIEW top_pubkeys_mv;"
```
---
## Phase 2: Multi-Instance + Dashboard
### Goal
Run multiple c-relay-pg-pg instances behind a load balancer with a separate dashboard web server, all sharing the same PostgreSQL database.
### Components
1. **nginx config** — WebSocket load balancing with `ip_hash` stickiness
2. **systemd template**`c-relay-pg-pg@.service` for multiple instances
3. **LISTEN/NOTIFY integration** — Cross-instance event broadcasting in the `lws_service()` loop
4. **PgBouncer** — Connection pooling between c-relay-pg-pg instances and PostgreSQL
5. **Dashboard** — Separate web server querying PostgreSQL directly
6. **Materialized view refresh** — Background job to refresh analytics views
### Multi-Instance Architecture
```mermaid
flowchart TD
INTERNET[Internet - Nostr Clients] -->|wss://relay.example.com| NGINX[nginx reverse proxy - TLS termination + load balancing]
NGINX -->|ws://localhost:8888| R1[c-relay-pg-pg Instance 1 - port 8888]
NGINX -->|ws://localhost:8889| R2[c-relay-pg-pg Instance 2 - port 8889]
NGINX -->|ws://localhost:8890| R3[c-relay-pg-pg Instance 3 - port 8890]
NGINX -->|http://localhost:3000| DASHWEB[Dashboard Web Server]
R1 -->|libpq connection pool| PGPOOL[PgBouncer - connection pooler]
R2 -->|libpq connection pool| PGPOOL
R3 -->|libpq connection pool| PGPOOL
PGPOOL -->|pooled connections| PG[PostgreSQL Primary]
DASHWEB -->|read queries| PG
PG -->|streaming replication| REPLICA[Read Replica - optional]
DASHWEB -.->|heavy analytics| REPLICA
```
### nginx Load Balancing Config
```nginx
# nginx.conf - WebSocket load balancing for c-relay-pg-pg
upstream relay_backends {
# ip_hash ensures a client always hits the same instance
# (important for WebSocket session stickiness)
ip_hash;
server 127.0.0.1:8888;
server 127.0.0.1:8889;
server 127.0.0.1:8890;
}
server {
listen 443 ssl;
server_name relay.example.com;
# TLS config...
location / {
proxy_pass http://relay_backends;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_read_timeout 86400; # Keep WebSocket alive for 24h
}
# Dashboard on separate path
location /dashboard {
proxy_pass http://127.0.0.1:3000;
}
}
```
**Session stickiness** (`ip_hash`) ensures that once a client connects to Instance 2, all their subsequent WebSocket frames go to Instance 2. This is important because subscriptions are held in-memory per instance.
### Starting Multiple Instances
Each instance is the same binary, just on a different port, all pointing to the same PostgreSQL:
```bash
# Instance 1
./build/c_relay_pg_x86 --port 8888 --db-host localhost --db-name crelay &
# Instance 2
./build/c_relay_pg_x86 --port 8889 --db-host localhost --db-name crelay &
# Instance 3
./build/c_relay_pg_x86 --port 8890 --db-host localhost --db-name crelay &
```
Or with systemd template units:
```ini
# /etc/systemd/system/c-relay-pg-pg@.service
[Unit]
Description=C-Relay-PG-PG Nostr Instance %i
After=postgresql.service
[Service]
ExecStart=/opt/c-relay-pg-pg/c_relay_pg_x86 --port %i --db-host localhost --db-name crelay
Restart=always
User=c-relay-pg
[Install]
WantedBy=multi-user.target
```
```bash
systemctl enable c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
systemctl start c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
```
### Cross-Instance Event Broadcasting
When Instance 1 receives a new EVENT and stores it in PostgreSQL, Instance 2 and Instance 3 need to know about it so they can broadcast to their connected subscribers.
```mermaid
sequenceDiagram
participant Client_A as Client A - connected to Instance 1
participant I1 as Instance 1
participant PG as PostgreSQL
participant I2 as Instance 2
participant I3 as Instance 3
participant Client_B as Client B - connected to Instance 2
participant Client_C as Client C - connected to Instance 3
Client_A->>I1: EVENT - new kind:1 note
I1->>PG: INSERT INTO events...
PG-->>I1: OK
I1->>I1: broadcast to local subscribers
I1->>PG: NOTIFY new_event with event_id
Note over PG: PostgreSQL delivers notification to all listeners
PG-->>I2: NOTIFY: new_event event_id
PG-->>I3: NOTIFY: new_event event_id
I2->>PG: SELECT event_json FROM events WHERE id = event_id
PG-->>I2: event JSON
I2->>I2: match against local subscriptions
I2->>Client_B: EVENT message - if subscription matches
I3->>PG: SELECT event_json FROM events WHERE id = event_id
PG-->>I3: event JSON
I3->>I3: match against local subscriptions
I3->>Client_C: EVENT message - if subscription matches
```
#### PostgreSQL LISTEN/NOTIFY Implementation
Built into PostgreSQL — no additional infrastructure needed:
```c
// === In the relay's event loop (modified lws_service loop) ===
// Setup: create a dedicated connection for LISTEN
PGconn* notify_conn = PQconnectdb("host=localhost dbname=crelay");
PQexec(notify_conn, "LISTEN new_event");
int notify_fd = PQsocket(notify_conn); // Get the socket fd for poll()
// After storing an event:
void on_event_stored(PGconn* write_conn, const char* event_id) {
char notify_cmd[128];
snprintf(notify_cmd, sizeof(notify_cmd),
"NOTIFY new_event, '%s'", event_id);
PQexec(write_conn, notify_cmd);
}
// In the main event loop (runs every lws_service iteration):
void check_cross_instance_events(PGconn* notify_conn) {
// Non-blocking check for notifications
PQconsumeInput(notify_conn);
PGnotify* notify;
while ((notify = PQnotifies(notify_conn)) != NULL) {
// Another instance stored a new event
const char* event_id = notify->extra;
// Fetch the event and check against local subscriptions
cJSON* event = db_get_event_by_id(event_id);
if (event) {
broadcast_event_to_subscriptions(event);
cJSON_Delete(event);
}
PQfreemem(notify);
}
}
```
**Performance**: LISTEN/NOTIFY adds ~15ms latency for cross-instance delivery. For a Nostr relay, this is imperceptible — clients already expect network latency.
**Payload limit**: NOTIFY payloads are limited to 8000 bytes. For event IDs (64 hex chars), this is fine.
#### Alternative: Redis Pub/Sub
If you later need even lower latency or more sophisticated routing:
```c
// Using hiredis (Redis C client)
redisContext* redis = redisConnect("127.0.0.1", 6379);
// After storing event:
redisCommand(redis, "PUBLISH new_event %s", event_json);
// Subscriber (in each instance):
redisCommand(redis, "SUBSCRIBE new_event");
// Then poll for messages in the event loop
```
Redis adds sub-millisecond pub/sub but requires running a Redis server. For most relays, PostgreSQL LISTEN/NOTIFY is sufficient.
### Connection Pooling with PgBouncer
Each relay instance needs multiple database connections. Without pooling, 3 instances × 10 connections = 30 PostgreSQL backend processes. With PgBouncer:
```ini
# /etc/pgbouncer/pgbouncer.ini
[databases]
crelay = host=127.0.0.1 port=5432 dbname=crelay
[pgbouncer]
listen_port = 6432
listen_addr = 127.0.0.1
auth_type = md5
pool_mode = transaction # Return connection to pool after each transaction
max_client_conn = 200 # Total connections from all relay instances
default_pool_size = 20 # Actual PostgreSQL connections
```
Relay instances connect to PgBouncer (port 6432) instead of PostgreSQL directly (port 5432). PgBouncer multiplexes 200 client connections onto 20 actual PostgreSQL connections.
### Zero-Downtime Deployment
```mermaid
sequenceDiagram
participant LB as nginx
participant I1 as Instance 1 - v1.0
participant I2 as Instance 2 - v1.0
participant I3 as Instance 3 - v1.0
participant I1_NEW as Instance 1 - v1.1
Note over LB,I3: Normal operation: 3 instances serving traffic
LB->>I1: Mark upstream as down
Note over I1: Drain: wait for existing connections to close or timeout
I1->>I1: Graceful shutdown
Note over LB: Traffic now goes to I2 and I3 only
I1_NEW->>I1_NEW: Start with new binary
I1_NEW->>LB: Health check passes
LB->>I1_NEW: Mark upstream as up
Note over LB,I1_NEW: Instance 1 now running v1.1, repeat for I2 and I3
```
Rolling deploy script:
```bash
#!/bin/bash
# rolling_deploy.sh - Zero-downtime deployment
for port in 8888 8889 8890; do
echo "Deploying instance on port $port..."
# 1. Tell nginx to stop sending new connections
sed -i "s/server 127.0.0.1:$port;/server 127.0.0.1:$port down;/" /etc/nginx/nginx.conf
nginx -s reload
# 2. Wait for existing connections to drain (30 seconds)
sleep 30
# 3. Stop old instance
systemctl stop c-relay-pg-pg@$port
# 4. Deploy new binary
cp ./build/c_relay_pg_x86 /opt/c-relay-pg-pg/c_relay_pg_x86
# 5. Start new instance
systemctl start c-relay-pg-pg@$port
# 6. Wait for health check
sleep 5
# 7. Re-enable in nginx
sed -i "s/server 127.0.0.1:$port down;/server 127.0.0.1:$port;/" /etc/nginx/nginx.conf
nginx -s reload
echo "Instance on port $port deployed successfully"
done
```
### Dashboard Options
With PostgreSQL, the dashboard can be built with any technology:
- **Grafana** — Point directly at PostgreSQL, zero custom code
- **Custom web app** — React/Vue frontend + any backend (Node, Python, Go) querying PostgreSQL
- **Embedded in c-relay-pg-pg** — Keep current approach but queries go through `db_ops` to PostgreSQL (no longer blocks event loop since PostgreSQL handles concurrency)
### Scaling Scenarios
| Scenario | Instances | Connections | Events/hour | Setup |
|----------|-----------|-------------|-------------|-------|
| **Current** | 1 | ~1,200 | 56 | Single process + SQLite |
| **Small upgrade** | 2 | ~2,500 | 500 | 2 instances + PostgreSQL |
| **Medium relay** | 4 | ~5,000 | 5,000 | 4 instances + PostgreSQL + PgBouncer |
| **Large relay** | 8 | ~10,000 | 50,000 | 8 instances + PostgreSQL + read replica |
| **Multi-region** | 24 per region | ~50,000 | 500,000 | Multiple servers + PostgreSQL replication |
### Cost Perspective
Running multiple relay instances with PostgreSQL on a single VPS:
- **PostgreSQL**: ~200500 MB RAM baseline
- **PgBouncer**: ~10 MB RAM
- **Each relay instance**: ~50100 MB RAM
- **Dashboard web server**: ~50100 MB RAM
- **4 instances + PostgreSQL + PgBouncer + dashboard**: ~12 GB total RAM
- A **$20/month VPS** with 4 cores and 4 GB RAM handles this easily
- A **$40/month VPS** with 8 cores and 8 GB RAM handles 8 instances comfortably
---
## Risk Mitigation
| Risk | Mitigation |
|------|-----------|
| PostgreSQL connection failures | `db_ops` returns error codes. Relay logs errors but doesn't crash. Reconnection logic with exponential backoff. |
| Performance regression | Benchmark before/after. PostgreSQL should be faster for complex queries, similar for simple ones. |
| Data loss during migration | Migration tool is read-only on SQLite. PostgreSQL import is idempotent (INSERT ON CONFLICT). |
| SQLite fallback needed | Keep `db_ops_sqlite.c` working. Compile-time flag switches backends. Can always go back. |
| LISTEN/NOTIFY message loss | NOTIFY is transactional — if the INSERT commits, the NOTIFY is guaranteed. Missed notifications during reconnect handled by periodic full-sync. |
| PgBouncer adds latency | Transaction pooling mode adds <0.1ms. Negligible compared to query time. |
---
## Relationship to c-relay-pg
This project depends on the `db_ops.h` abstraction layer built in [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1. The interface is identical — only the backend implementation changes:
- **c-relay-pg**: `db_ops.c` → SQLite calls via `sqlite3_*`
- **c-relay-pg-pg**: `db_ops_postgres.c` → PostgreSQL calls via `libpq` (`PQexec`, `PQgetvalue`, etc.)
The `db_ops.h` header file is shared between both projects. Changes to the interface should be coordinated.
+332
View File
@@ -0,0 +1,332 @@
# c-relay-pg Thread Pool Plan — db_ops Abstraction + SQLite Thread Pool
## Overview
This plan covers improvements to **c-relay-pg** (this repo) — the lean, single-binary, embedded-SQLite Nostr relay. Two phases:
1. **Phase 1: Database Abstraction Layer** — Consolidate all scattered `sqlite3_*` calls into a single `db_ops.h` / `db_ops.c` module. Pure refactor, no behavior change.
2. **Phase 2: SQLite Thread Pool** — Add worker threads for concurrent reads, unblocking the `lws_service()` event loop.
This is the **final form of c-relay-pg**: an embedded-SQLite relay with clean architecture and non-blocking database access. The abstraction layer also enables a future fork to PostgreSQL — see [c-relay-pg-pg plan](c_relay_pg_pg_plan.md).
### Why This First?
The current architecture has a fundamental bottleneck documented in the [database architecture analysis](database_architecture_analysis.md): the single-threaded event loop blocks on every database call. A 672ms REQ query freezes every connected client. The thread pool fixes this without changing the database engine or deployment model.
| Metric | Current | After Thread Pool |
|--------|---------|-------------------|
| **Event loop blocking** | 0.1672ms per query | Near-zero |
| **Concurrent reads** | 1 | 48 |
| **Deployment** | Single binary + DB file | Same |
| **Database** | SQLite | Same |
| **Code risk** | N/A | Low — additive change |
## Architecture
```mermaid
flowchart TD
subgraph c-relay-pg - Final Form
RELAY[c-relay-pg binary] -->|db_ops API| DBOPS[db_ops.c - abstraction layer]
DBOPS -->|sqlite3 calls| SQLITE[SQLite WAL database]
end
```
```mermaid
flowchart TD
subgraph c-relay-pg with Thread Pool
LWS[lws_service event loop] -->|REQ arrives| Q[Thread-safe job queue]
LWS -->|EVENT arrives| WQ[Write queue - single writer]
Q --> T1[Reader Thread 1 - own sqlite3*]
Q --> T2[Reader Thread 2 - own sqlite3*]
Q --> T3[Reader Thread 3 - own sqlite3*]
Q --> T4[Reader Thread 4 - own sqlite3*]
WQ --> TW[Writer Thread - own sqlite3*]
T1 -->|results| CB[Callback to lws event loop]
T2 -->|results| CB
T3 -->|results| CB
T4 -->|results| CB
TW -->|OK/error| CB
CB -->|queue_message| LWS
end
```
---
## Phase 1: Database Abstraction Layer
### Goal
Consolidate all 258 scattered `sqlite3_*` calls across 8 files into a single `db_ops.h` / `db_ops.c` module with a backend-agnostic interface. SQLite continues to work — this is a pure refactor.
### Files That Currently Touch SQLite Directly
| File | `sqlite3_*` calls | Operations |
|------|-------------------|------------|
| [`src/main.c`](../src/main.c) | ~80 | Event store/retrieve, tag storage, REQ queries, COUNT queries, DB init, schema migration |
| [`src/config.c`](../src/config.c) | ~60 | Config CRUD, auth rules CRUD, WoT sync, relay key storage, startup sequence |
| [`src/api.c`](../src/api.c) | ~40 | Stats queries, monitoring views, admin SQL execution, config management |
| [`src/dm_admin.c`](../src/dm_admin.c) | ~20 | Auth rule management, WoT whitelist operations |
| [`src/websockets.c`](../src/websockets.c) | ~15 | COUNT queries, IP ban stats, connection tracking |
| [`src/subscriptions.c`](../src/subscriptions.c) | ~15 | Subscription logging — create/close/disconnect |
| [`src/nip009.c`](../src/nip009.c) | ~10 | Event deletion — by ID and by address |
| [`src/request_validator.c`](../src/request_validator.c) | ~8 | Blacklist/whitelist checks |
| [`src/ip_ban.c`](../src/ip_ban.c) | ~15 | IP ban table CRUD, persistence |
### Abstraction Layer Design
New files: `src/db_ops.h` and `src/db_ops.c`
```c
// src/db_ops.h — Database operations abstraction layer
#ifndef DB_OPS_H
#define DB_OPS_H
#include "../nostr_core_lib/cjson/cJSON.h"
// ============================================================
// Lifecycle
// ============================================================
int db_init(const char* connection_string); // SQLite: file path, PG: connection string
void db_close(void);
int db_is_available(void);
// ============================================================
// Schema / Migration
// ============================================================
int db_ensure_schema(void);
int db_get_schema_version(void);
int db_execute_raw(const char* sql); // For schema DDL only
// ============================================================
// Event Operations
// ============================================================
int db_store_event(cJSON* event);
int db_event_exists(const char* event_id);
char* db_get_event_json(const char* event_id); // Caller must free
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
int db_delete_events_by_address(const char* pubkey, int kind,
const char* d_tag, long before_timestamp);
// ============================================================
// Event Queries - REQ handling
// ============================================================
typedef struct {
int* kinds; int kind_count;
char** authors; int author_count;
char** ids; int id_count;
char** tag_names; // Parallel arrays: tag_names[i] has tag_values[i][]
char*** tag_values;
int* tag_value_counts;
int tag_filter_count;
long since; // 0 = not set
long until; // 0 = not set
int limit; // 0 = default 500
char* search; // NIP-50 search term, NULL = not set
} db_event_filter_t;
typedef struct db_result db_result_t;
db_result_t* db_query_events(const db_event_filter_t* filter);
const char* db_result_next_json(db_result_t* result); // Returns event_json, NULL when done
int db_result_row_count(db_result_t* result);
void db_result_free(db_result_t* result);
int db_count_events(const db_event_filter_t* filter);
// ============================================================
// Config Operations
// ============================================================
char* db_get_config(const char* key); // Caller must free, NULL if not found
int db_set_config(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart);
int db_update_config(const char* key, const char* value);
int db_config_exists(const char* key);
int db_get_config_count(void);
// ============================================================
// Auth Rules Operations
// ============================================================
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_auth_rule_exists(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_clear_auth_rules(const char* rule_type); // NULL = clear all
int db_is_blacklisted(const char* pubkey);
int db_is_whitelisted(const char* pubkey);
int db_has_any_whitelist(void);
// ============================================================
// Relay Key Storage
// ============================================================
int db_store_relay_private_key(const char* privkey_hex);
char* db_get_relay_private_key(void); // Caller must free
// ============================================================
// Subscription Logging
// ============================================================
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json);
int db_log_subscription_closed(const char* sub_id, const char* client_ip);
int db_log_subscription_disconnected(const char* client_ip);
int db_update_subscription_events_sent(const char* sub_id, int events_sent);
int db_cleanup_orphaned_subscriptions(void);
// ============================================================
// IP Ban Persistence
// ============================================================
int db_ensure_ip_ban_table(void);
int db_load_ip_bans(void* ban_table, int table_size); // Populates in-memory table
int db_save_ip_bans(const void* ban_table, int table_size);
// ============================================================
// Analytics / Stats - for dashboard
// ============================================================
cJSON* db_get_event_kind_distribution(void);
cJSON* db_get_time_based_stats(void);
cJSON* db_get_top_pubkeys(int limit);
cJSON* db_get_subscription_details(void);
int db_get_total_event_count(void);
// ============================================================
// Admin SQL Query
// ============================================================
cJSON* db_execute_admin_query(const char* sql, char* error_msg, size_t error_size);
#endif // DB_OPS_H
```
### Migration Strategy for Phase 1
The key principle: **change the interface, not the behavior**. Each function in `db_ops.c` initially just wraps the existing SQLite calls.
**Step-by-step for each file:**
1. **Create `db_ops.h` and `db_ops.c`** with the interface above
2. **Implement each `db_ops` function** by moving the existing SQLite code from the source files into `db_ops.c`
3. **Replace direct `sqlite3_*` calls** in each source file with `db_ops_*` calls
4. **Remove `extern sqlite3* g_db`** from each file — only `db_ops.c` knows about `g_db`
5. **Remove `#include <sqlite3.h>`** from each file — only `db_ops.c` includes it
6. **Update Makefile** to compile `db_ops.c`
### Order of Migration (by risk, lowest first)
1. **`src/ip_ban.c`** — Self-contained, simple CRUD. Good warmup.
2. **`src/subscriptions.c`** — Logging only, no critical path.
3. **`src/nip009.c`** — Event deletion, small file.
4. **`src/request_validator.c`** — Auth checks, small file.
5. **`src/api.c`** — Stats/monitoring queries. Larger but read-only.
6. **`src/dm_admin.c`** — Auth rules + WoT. Medium complexity.
7. **`src/config.c`** — Config CRUD. Large but well-structured.
8. **`src/websockets.c`** — COUNT queries, minimal DB usage.
9. **`src/main.c`** — Event store/retrieve, REQ queries. The big one — do last.
### Testing Phase 1
After Phase 1, the relay should behave **identically** to before. Run:
- `tests/run_all_tests.sh` — Full test suite
- `tests/performance_benchmarks.sh` — Verify no performance regression
- Manual testing with production-like traffic
---
## Phase 2: SQLite Thread Pool
### Goal
Add a pool of N worker threads, each with its own `sqlite3*` connection to the same database file. SQLite WAL mode (already enabled) supports **concurrent readers**. The event loop dispatches database work to the pool and remains responsive.
### Key Design Points
1. **Read path**: REQ queries dispatched to thread pool. Each worker opens its own `sqlite3*` connection. SQLite WAL allows unlimited concurrent readers.
2. **Write path**: EVENT inserts go through a single dedicated writer thread. SQLite only allows one writer at a time anyway — this serializes writes cleanly.
3. **Result delivery**: Worker threads cannot call `lws_write()` directly (libwebsockets is not thread-safe). Instead, they push results into a per-session message queue and call `lws_cancel_service()` to wake the event loop, which then drains the queue.
4. **Connection lifecycle**: Each thread opens its own connection with `PRAGMA journal_mode=WAL` and `PRAGMA busy_timeout=5000`.
### What Changes in the Codebase
| Component | Current | Thread Pool |
|-----------|---------|-------------|
| [`g_db`](../src/main.c:49) | Single global connection | One per thread + writer connection |
| [`handle_req_message()`](../src/main.c:1101) | Synchronous SQL in callback | Package filter into job, dispatch to pool |
| [`store_event()`](../src/main.c:787) | Synchronous INSERT in callback | Dispatch to writer thread |
| [`handle_count_message()`](../src/websockets.c:2863) | Synchronous COUNT in callback | Dispatch to pool |
| Result delivery | Direct `queue_message()` | Worker pushes to queue + `lws_cancel_service()` |
| Config reads | Direct `sqlite3_prepare` on `g_db` | Can stay synchronous with own connection or cache |
### New Files
- `src/thread_pool.h` — Thread pool interface
- `src/thread_pool.c` — Thread pool implementation (job queue, worker lifecycle, result delivery)
### Thread Pool Interface (sketch)
```c
// src/thread_pool.h
#ifndef THREAD_POOL_H
#define THREAD_POOL_H
typedef enum {
JOB_TYPE_REQ_QUERY,
JOB_TYPE_COUNT_QUERY,
JOB_TYPE_STORE_EVENT,
JOB_TYPE_DELETE_EVENT,
} job_type_t;
typedef struct {
job_type_t type;
void* session; // lws per-session data pointer
db_event_filter_t filter; // For REQ/COUNT jobs
cJSON* event; // For STORE jobs
char event_id[65]; // For DELETE jobs
} db_job_t;
int thread_pool_init(int num_readers, const char* db_path);
void thread_pool_shutdown(void);
int thread_pool_submit_read(db_job_t* job);
int thread_pool_submit_write(db_job_t* job);
#endif // THREAD_POOL_H
```
### Performance Characteristics
| Metric | Value |
|--------|-------|
| **Concurrent reads** | N readers in parallel (N = thread count, typically 48) |
| **Write throughput** | Same as current — SQLite serializes writes regardless |
| **Event loop latency** | Near-zero — REQ no longer blocks the loop |
| **Max theoretical read throughput** | ~48x current (limited by disk I/O, not CPU) |
| **Memory overhead** | ~50100 MB per connection (page cache) |
| **Latency per query** | Same as current per-query, but no head-of-line blocking |
### Limitations
- **Write contention**: SQLite still allows only ONE writer at a time. With WAL, readers don't block writers and writers don't block readers, but two simultaneous writes will serialize. At the current write rate (~56 events/hour), this is a non-issue.
- **Database size**: The 2.7 GB index bloat problem remains. Thread pool doesn't fix the schema — it fixes the concurrency.
- **Scaling ceiling**: Beyond ~8 reader threads, diminishing returns due to disk I/O contention on a single SQLite file.
- **Complexity**: Need a proper job queue, thread lifecycle management, and careful handling of the lws ↔ worker thread boundary.
---
## Risk Mitigation
| Risk | Mitigation |
|------|-----------|
| Phase 1 introduces bugs | Each file migrated independently, tested after each. Full test suite runs after each file. |
| Thread pool race conditions | Worker threads only touch their own `sqlite3*` connection. Result delivery uses a mutex-protected queue. `lws_cancel_service()` is documented as thread-safe. |
| Performance regression from abstraction | Abstraction layer is thin wrappers — no extra allocations or copies. Benchmark before/after. |
| lws thread safety issues | Workers never call `lws_write()` directly. They push to a queue and wake the event loop. This is the documented pattern for libwebsockets multi-threading. |
| Rollback needed | Phase 1 is a pure refactor — easy to revert. Phase 2 thread pool is additive — can be disabled with a compile flag. |
---
## Relationship to c-relay-pg-pg
After Phase 1 (abstraction layer) is complete, the codebase is ready to be forked into a separate **c-relay-pg-pg** project that replaces the SQLite backend with PostgreSQL. See [c-relay-pg-pg plan](c_relay_pg_pg_plan.md) for details.
The `db_ops.h` interface is designed to work with any backend:
- **SQLite** (this plan): `db_result_next_json()` copies from `sqlite3_column_text()`
- **PostgreSQL** (c-relay-pg-pg): `db_result_next_json()` returns from `PQgetvalue()`
- **LMDB** (future possibility): `db_result_next_json()` returns a zero-copy pointer into mmap'd memory
+503
View File
@@ -0,0 +1,503 @@
# Caching Relay Daemon - Architecture Plan
> **Status:** Draft v3 - adds NIP-65 outbox model: bootstrap relays for
> discovery, kind-10002 per-pubkey relay lists, minimum covering set relay
> selection, local-relay-first on subsequent startups.
## Goal
A standalone C99 daemon that acts as a "caching relay feeder". It:
1. Reads a `.jsonc` config file listing one or more **root npubs** (e.g. your own npub), a set of upstream relays, a local relay URL, and configurable event kinds.
2. For each root npub, fetches its kind-3 contact list to discover **followed pubkeys**.
3. Subscribes **live** to events of the configured kinds from the union of followed pubkeys (root npubs + their follows).
4. Performs a **throttled backfill** of historical events per followed pubkey, spread out over time so upstream relays are not hammered.
5. Re-publishes every fetched event to the **local relay** via a plain WebSocket `EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg, or any Nostr relay).
6. Builds as a **statically linked C99 binary** following the c-relay model, linking against `nostr_core_lib` and `c_utils_lib`.
The user's Nostr client then points only at the local relay and gets a fast, pre-populated feed without doing any fan-out itself.
## Design Principles
- **Relay-agnostic on the sink side.** The daemon is just a Nostr client publishing `EVENT` messages over WebSocket. It does not touch the local relay's database directly. This keeps it decoupled and safe.
- **Reuse `nostr_core_lib`.** All Nostr protocol concerns (event validation, kind-3 parsing, relay pool, WebSocket client, NIP-19 npub decoding) come from `nostr_core_lib`. The daemon is orchestration logic only.
- **Single statically linked binary.** Built with the same Makefile pattern as c-relay: link `libnostr_core_x64.a` (or arm64) + `libc_utils.a` + system libs (`-lwebsockets -lsqlite3 -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm`).
- **C99, `-Wall -Wextra -std=c99 -g -O2`** matching c-relay's `CFLAGS`.
- **No config-file framework.** Hand-rolled `.jsonc` parser using cJSON (strip `//` and `/* */` comments before parsing, since cJSON does not natively support JSONC).
## Architecture Overview
```mermaid
flowchart TD
subgraph Config
CFG[caching_relay.jsonc]
end
subgraph Daemon
MAIN[main.c - lifecycle / signals]
CFGP[config.c - parse jsonc]
FOLLOW[follow_graph.c - root npubs + kind-3 resolution]
BACKFILL[backfill.c - throttled historical pull]
LIVE[live_subscriber.c - open subscriptions]
SINK[relay_sink.c - publish EVENT to local relay]
STATE[state.c - in-memory follow set + seen cache]
end
subgraph nostr_core_lib
UPOOL[core_relay_pool.c - upstream pool - query/subscribe]
SPOOL[core_relay_pool.c - sink pool - publish only]
NIP01[nip001.c - validate / parse]
NIP19[nip019.c - npub decode]
WS[nostr_websocket - client WS]
end
subgraph Upstream
R1[wss://relay.damus.io]
R2[wss://nos.lol]
R3[wss://...]
end
subgraph Local
LOCAL[wss://127.0.0.1:8888 - c-relay / c-relay-pg / any]
end
CFG --> CFGP
CFGP --> MAIN
MAIN --> FOLLOW
FOLLOW -->|query kind 3| UPOOL
UPOOL --> R1
UPOOL --> R2
UPOOL --> R3
FOLLOW --> STATE
MAIN --> LIVE
LIVE -->|subscribe authors + kinds| UPOOL
LIVE -->|on_event| SINK
MAIN --> BACKFILL
BACKFILL -->|query per pubkey since T| UPOOL
BACKFILL -->|on_event| SINK
SINK -->|publish_async| SPOOL
SPOOL -->|EVENT json| LOCAL
STATE --> LIVE
STATE --> BACKFILL
CFGP -->|read/write state| CFG
```
## Data Flow
```mermaid
sequenceDiagram
participant D as Daemon
participant UP as upstream_pool
participant SP as sink_pool
participant U as Upstream Relays
participant L as Local Relay
D->>D: parse caching_relay.jsonc + state
D->>UP: add_relay x N upstreams
D->>SP: add_relay local only
D->>UP: query_sync kind=3 authors=root_npubs
U-->>UP: kind-3 events
UP-->>D: followed pubkeys set
D->>D: merge root + follows into author set
D->>UP: subscribe live authors=author_set kinds=configured since=now
loop live loop - pump upstream_pool_run
U-->>UP: EVENT
UP-->>D: on_event callback
D->>SP: publish_async EVENT
SP->>L: EVENT json
D->>SP: pump sink_pool_run to flush callbacks
end
loop backfill - progressive window expansion
D->>UP: query_sync authors=pubkey_i kinds since=now-window limit=K
U-->>UP: historical events
UP-->>D: events
D->>SP: publish_async each EVENT
D->>D: sleep tick_interval_seconds
D->>D: on full round-robin pass - advance window, write state to jsonc
end
```
## Config File Format
`caching_relay.jsonc` (JSONC = JSON with comments). The config file is the
**single source of truth** and is also the daemon's persistent state store:
the daemon rewrites it to disk whenever the backfill window advances, so a
restart resumes exactly where it left off.
```jsonc
{
// Root npubs whose follows list we crawl
"root_npubs": [
"npub1...",
"npub1..."
],
// Upstream relays to pull events from
"upstream_relays": [
"wss://relay.damus.io",
"wss://nos.lol",
"wss://relay.nostr.band"
],
// Local relay to feed events into (publish-only, never queried)
"local_relay": "ws://127.0.0.1:8888",
// Event kinds to cache
"kinds": [1, 3, 6, 10000, 30023],
// ---- Backfill: progressive window expansion ----
"backfill": {
"enabled": true,
// Window schedule in seconds-from-now, applied in order.
// The daemon first pulls everything newer than (now - 24h).
// Once complete, it expands to (now - 7d), then (now - 30d), etc.
"window_schedule_seconds": [86400, 604800, 2592000, 7776000, 31536000],
// Per-pubkey query limit per tick (keeps individual queries light)
"events_per_tick": 50,
// Delay between pubkey backfill ticks (throttle to be polite)
"tick_interval_seconds": 5,
// Delay between completing one window and starting the next
"window_cooldown_seconds": 60
},
// ---- Live subscription ----
"live": {
"enabled": true,
"resubscribe_interval_seconds": 300
},
// Refresh the follow graph periodically
"follow_graph_refresh_seconds": 600,
// ---- Persistent state (managed by the daemon; do not hand-edit) ----
// The daemon writes these back to this file as backfill progresses.
// On restart it reads them to avoid re-pulling already-cached history.
"state": {
// How far back we have fully backfilled, as a unix timestamp.
// Starts at 0 / absent on first run. Advances as each window completes.
"backfilled_until": 0,
// Index into window_schedule_seconds we are currently working on.
"current_window_index": 0,
// Round-robin cursor: which followed pubkey we backfill next.
"backfill_cursor": 0
}
}
```
**State write-back rules:**
- The daemon rewrites the `.jsonc` file (preserving comments is *not* required
on rewrite - it may emit plain JSON once it has been modified at runtime;
the comments are only for the user's initial authoring convenience).
- Write-back happens: (a) when a window completes and `backfilled_until`
advances, (b) on graceful shutdown, (c) periodically (e.g. every 60s) so a
crash loses at most one minute of cursor progress.
- A write-ahead temp file + rename is used so the config is never left
half-written.
## Progressive Window-Expansion Backfill
Instead of a fixed per-pubkey cursor, the daemon uses a **global window** that
expands backwards in time in discrete steps. This is simpler, gives a natural
"recent first" experience, and is trivially resumable from the config file.
### Window schedule
`backfill.window_schedule_seconds` is an ordered list, e.g.
`[86400, 604800, 2592000, 7776000, 31536000]`
= 1 day, 1 week, 1 month, 3 months, 1 year.
### Algorithm
1. On startup, read `state.backfilled_until` (unix ts) and
`state.current_window_index` from the config.
2. If `backfilled_until == 0` (first run), set the current target window to
`window_schedule[0]` (e.g. 24h). The backfill `since` cutoff is
`now - window_schedule[0]`.
3. Round-robin through every followed pubkey. For each pubkey, issue
`nostr_relay_pool_query_sync` with:
- `authors = [pubkey]`
- `kinds = configured kinds`
- `since = now - current_window_seconds` (i.e. the *full* current window,
not an incremental slice - the local relay dedups, so re-overlap is free
and simpler than tracking per-pubkey cursors)
- `limit = events_per_tick`
4. Sleep `tick_interval_seconds` between pubkeys (throttle).
5. When one full round-robin pass over all followed pubkeys completes for the
current window:
- Set `state.backfilled_until = now - current_window_seconds`.
- Advance `state.current_window_index += 1`.
- Persist the config file (temp + rename).
- Sleep `window_cooldown_seconds` before starting the next (wider) window.
6. Repeat with the next (wider) window. The `since` cutoff moves further back
in time, so each window pulls the *additional* older slice. Because the
local relay dedups inserts, events that fall in the overlap with the
previous window are simply ignored on insert - no correctness issue.
7. After the last (widest) window completes, the daemon switches to
**steady-state**: it only keeps the live subscription running and
periodically re-runs the widest window to catch any events that migrated
into scope (e.g. a followed user backfilling their own old notes to a
different relay). `backfilled_until` stays pinned at the oldest window.
### Restart behavior
- On restart, the daemon reads `backfilled_until` and `current_window_index`.
- It resumes at the *current* window (the one that was in progress when it
stopped). Because each window re-pulls `since = now - window_seconds`, a
partial window just re-runs from the start of the round-robin - cheap and
correct.
- `state.backfill_cursor` records which pubkey in the round-robin was next;
this is a minor optimization and may be reset to 0 on restart without harm.
### Throttling summary
- `tick_interval_seconds` paces individual pubkey queries (e.g. 5s).
- `window_cooldown_seconds` paces window-to-window transitions (e.g. 60s).
- `events_per_tick` caps each query's result size (e.g. 50).
- Per-relay query latency from `nostr_relay_pool_get_relay_query_latency`
can be used to prefer fast relays for backfill; slow relays are still used
for the live subscription (where completeness matters more than speed).
## File Layout
```
caching_relay/
plans/plan.md (this file)
Makefile (c-relay-style, static link)
build_static.sh (Alpine MUSL static builder, adapted from c-relay)
Dockerfile.alpine-musl (adapted from c-relay)
src/
main.c (lifecycle, signal handling, main loop)
main.h
config.c / config.h (jsonc parse + validation)
follow_graph.c / .h (resolve root npubs -> kind-3 -> followed set)
live_subscriber.c / .h (open-ended live subscription on the pool)
backfill.c / .h (throttled historical pull worker)
relay_sink.c / .h (publish EVENT to local relay via dedicated sink pool)
state.c / .h (in-memory author set + seen-event ring buffer; config state read/write)
jsonc_strip.c / .h (strip // and /* */ comments before cJSON_Parse)
log.c / log.h (colored stderr logging, c-relay style)
examples/
caching_relay.jsonc (sample config)
README.md
```
## Build Model (mirrors c-relay)
- `Makefile` with:
- `CC = gcc`, `CFLAGS = -Wall -Wextra -std=c99 -g -O2`
- `INCLUDES = -I. -Isrc -I../nostr_core_lib -I../nostr_core_lib/nostr_core -I../nostr_core_lib/cjson -I../nostr_core_lib/nostr_websocket -I../c_utils_lib/src`
- `LIBS = -lwebsockets -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm -L../c_utils_lib -lc_utils`
- Note: no `-lsqlite3` - the daemon itself does not use SQLite. (c-relay uses
SQLite for its event store, but that is the local relay's concern, not the
daemon's.)
- `NOSTR_CORE_LIB = ../nostr_core_lib/libnostr_core_x64.a` (or arm64)
- Build `nostr_core_lib` with `--nips=1,6,19` (1 basic, 6 keys, 19 npub bech32).
NIP-42 is deferred to a later phase (public relays only for now). Kind-3
parsing is plain cJSON tag walking in NIP-01, no special NIP-02 module
needed.
- Single static link line: `$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)`
- `build_static.sh` adapted from c-relay's Alpine MUSL Docker builder to produce a truly static binary.
## Key nostr_core_lib APIs Used
- `nostr_relay_pool_create()` / `nostr_relay_pool_add_relay()` / `nostr_relay_pool_destroy()`
- `nostr_relay_pool_query_sync()` - one-shot historical/backfill queries (kind-3 fetch, per-pubkey backfill)
- `nostr_relay_pool_subscribe()` - long-lived live subscription with `on_event` / `on_eose` callbacks
- `nostr_relay_pool_run()` / `nostr_relay_pool_poll()` - event loop driving both live sub and backfill
- `nostr_relay_pool_publish_async()` - publish fetched events to the local relay
via a **dedicated single-relay sink pool** (separate from the upstream query
pool, so the local relay is never included in `query_sync` fan-out)
- `nostr_validate_event()` - validate before republishing (defensive; upstream events should already be valid)
- NIP-19: `nostr_nip19_decode` (or equivalent) to convert `npub1...` -> hex pubkey for filters
- cJSON for filter construction and kind-3 tag parsing (`["p", "<hex>", "<relay>", "<petname>"]`)
## Concurrency Model
Keep it simple and single-threaded with a cooperative event loop, matching c-relay's spirit:
- **Two `nostr_relay_pool_t` instances**, both driven from the main thread:
1. `upstream_pool` - holds all `upstream_relays`. Used for `query_sync`
(kind-3 fetch, backfill) and the long-lived live `subscribe`.
2. `sink_pool` - holds only `local_relay`. Used exclusively for
`publish_async` of fetched events. Never queried.
- The main loop alternates between:
- `nostr_relay_pool_run(upstream_pool, timeout_ms)` to pump live sub events,
- a time-sliced backfill step (one pubkey `query_sync` per
`tick_interval_seconds`),
- `nostr_relay_pool_run(sink_pool, 0)` to flush pending publish callbacks.
- Live subscription's `on_event` callback calls `relay_sink_publish()` which
enqueues an `publish_async` on the sink pool.
- Backfill `query_sync` is synchronous and blocks the upstream loop briefly -
acceptable since `events_per_tick` is small and `tick_interval_seconds`
provides pacing.
- If profiling later shows backfill blocking the live sub too much, backfill
can be moved to a second thread with its own upstream pool. Start without
that.
## State Persistence
- **No SQLite in the daemon.** The daemon's only persistent state is the
backfill window cursor, and that lives in the `.jsonc` config file itself
under the `state` object (see Config File Format above).
- The followed-pubkey set is re-derived from kind-3 on every
`follow_graph_refresh_seconds` tick and on startup - it is not persisted.
- Event dedup is delegated entirely to the local relay (c-relay's
`INSERT OR IGNORE` on event id). The daemon keeps a small in-memory ring
buffer of recently-published event ids only to avoid redundant publish
*attempts* within a single run; this is not persisted.
- Config write-back uses a temp file + atomic rename so the config is never
left half-written, even on crash.
## Signals & Lifecycle
- `SIGINT` / `SIGTERM` -> graceful shutdown: close subscriptions, destroy pool, close sink WS.
- `SIGHUP` -> reload config (re-read jsonc, refresh follow graph, adjust subscriptions).
- Logs to stderr with c-relay-style color prefixes.
## Decisions Resolved
1. **Sink pool.** Use a *dedicated* single-relay `nostr_relay_pool_t` for the
local sink, separate from the upstream pool. The local relay is never
queried, only published to. **Confirmed.**
- **Update (v3):** On subsequent startups, the local relay IS queried for
kind-3 and kind-10002 events (fast local cache lookup). It is still never
queried for general event backfill -- only for discovery metadata.
2. **NIP-42 auth.** Deferred to a later phase. Phase 1 targets public relays
only. **Confirmed.**
3. **Kind-3 freshness.** Use the most recent kind-3 per root npub
(`nostr_relay_pool_get_event` with `authors=[npub], kinds=[3]`). Re-resolve
on `follow_graph_refresh_seconds` interval. **Confirmed.**
4. **State persistence.** No SQLite in the daemon. The `.jsonc` config file is
the state store; the daemon writes back `state.backfilled_until`,
`state.current_window_index`, and `state.backfill_cursor` as backfill
progresses. **Confirmed.**
5. **Backfill strategy.** Progressive window expansion
(24h -> 7d -> 30d -> 90d -> 365d), round-robin per pubkey within each
window, full-window re-pull (local relay dedups). **Confirmed.**
6. **NIP-65 outbox model.** `upstream_relays` in config are **bootstrap
relays** only. The daemon discovers each followed pubkey's outbox relays
via kind 10002, then computes the **minimum covering set** of relays
(greedy set cover) that covers all followed pubkeys. Bootstrap relays are
always included in the final set. Pubkeys with no kind 10002 fall back to
bootstrap relays. **Confirmed.**
7. **Local-relay-first on subsequent startups.** On startup, if
`state.backfilled_until > 0`, the daemon queries the local relay first for
kind-3 and kind-10002 events (fast, no network). Falls back to bootstrap
relays for any pubkey not found locally. **Confirmed.**
8. **Admin kinds.** Root (admin) npubs get a separate kind list (`admin_kinds`)
with `["*"]` support for all kinds. Two live subscriptions: follows_sub
(non-admin, regular kinds) and admin_sub (admin, admin_kinds). **Confirmed.**
## NIP-65 Outbox Model Design (Phase 2)
### New module: `relay_discovery.c`
Responsibilities:
- For each followed pubkey, fetch their most recent kind 10002 (relay list).
- First-time: query from bootstrap relays.
- Subsequent: query from local relay first, bootstrap fallback.
- Parse `r` tags from kind 10002 events: `["r", "<url>"]` or
`["r", "<url>", "read"]` / `["r", "<url>", "write"]`.
- We care about "read" relays (we are reading events FROM them).
- If no read/write marker, assume both.
- Build a map: `pubkey -> list of outbox relay URLs`.
- Publish all kind-10002 events to the local relay (cache them for next startup).
### Greedy set cover algorithm
```
Input: pubkey_to_relays map {pubkey -> [relay1, relay2, ...]}
Output: minimal set of relay URLs covering all pubkeys
1. uncovered = set of all pubkeys
2. selected = empty set
3. Add all bootstrap relays to selected (always included)
4. For each bootstrap relay, remove its known pubkeys from uncovered
(bootstrap relays cover pubkeys with no 10002)
5. While uncovered is not empty:
a. Find relay R that covers the most pubkeys in uncovered
b. If no relay covers any uncovered pubkey, break (orphaned pubkeys)
c. Add R to selected
d. Remove all pubkeys covered by R from uncovered
6. Return selected
```
### Data structures
```c
/* Per-pubkey outbox relay list */
typedef struct {
char pubkey[CR_HEX_LEN];
char relays[CR_MAX_RELAYS_PER_PUBKEY][CR_URL_LEN];
int relay_count;
} cr_outbox_entry_t;
/* Relay-to-pubkeys coverage map (for set cover) */
typedef struct {
char url[CR_URL_LEN];
char pubkeys[CR_MAX_PUBKEYS_PER_RELAY][CR_HEX_LEN];
int pubkey_count;
} cr_relay_coverage_t;
/* Result of relay discovery */
typedef struct {
cr_outbox_entry_t *outboxes; /* per-pubkey relay lists */
int outbox_count;
char selected_relays[CR_MAX_UPSTREAM][CR_URL_LEN];
int selected_count;
} cr_relay_map_t;
```
### Startup sequence change
```
OLD:
load config -> create pools -> resolve follow graph -> open live sub -> backfill
NEW:
load config -> create pools ->
resolve follow graph (local-first on subsequent) ->
discover outbox relays (kind 10002, local-first on subsequent) ->
compute minimum covering set ->
add selected relays to upstream_pool ->
log selected relays + coverage ->
open live sub -> backfill (per-pubkey from their outbox relays)
```
### Backfill change
Instead of fanning out each backfill query to ALL upstream relays, backfill
queries each pubkey from **that pubkey's specific outbox relays** (or bootstrap
relays as fallback). This is more efficient and more polite to relays that
don't have that pubkey's events.
### New config fields
None required. `upstream_relays` is reinterpreted as bootstrap relays.
Optionally a `max_outbox_relays` cap could be added later if the covering set
grows too large.
## Implementation Todo List
### Phase 0 - Local relay sanity check (DONE)
0. **Start a local c-relay and verify read/write with `nak`.** DONE.
### Phase 1 - Daemon implementation (DONE)
1-12. All implemented, built, and tested. See git history. DONE.
### Phase 2 - NIP-65 outbox model
1. Implement `relay_discovery.c/.h`: fetch kind-10002 per pubkey, parse `r`
tags, build `pubkey -> relays` map, publish 10002 events to local relay.
2. Implement greedy set cover: compute minimum covering relay set from the
outbox map, always include bootstrap relays.
3. Update `follow_graph.c`: query local relay first for kind-3 on subsequent
startups, fall back to bootstrap relays.
4. Update `backfill.c`: query each pubkey from their specific outbox relays
instead of all upstream relays.
5. Update `main.c`: insert relay discovery phase between follow graph
resolution and live subscription. Add discovered relays to upstream_pool.
Log selected relays and coverage.
6. Update `caching_relay_config.jsonc`: update comments to say "bootstrap
relays" instead of "upstream relays".
7. Build and test NIP-65 outbox model end-to-end with a real npub.
8. Update `README.md` if any flowchart details change during implementation.
+543
View File
@@ -0,0 +1,543 @@
# Caching Relay PostgreSQL Inbox Integration Plan
> **Status:** Revised and simplified architecture.
>
> **Decision:** Keep caching as a separate application. The caching application
> focuses on polite upstream collection and writes raw event JSON to a minimal
> PostgreSQL inbox. c-relay-pg removes small priority-ordered batches and handles
> them through relay-owned event processing. The design deliberately accepts a
> small crash-loss window between dequeue and canonical storage; live overlap and
> backfill retries recover those events naturally.
## 1. Goal
Connect the standalone caching application to c-relay-pg without publishing
cached events through a loopback WebSocket and without allowing the caching
application to write directly to the canonical [`events`](../src/pg_schema.sql:12)
table.
The responsibilities are intentionally narrow:
- **Caching application:** sensibly and politely acquire the desired events.
- **PostgreSQL inbox:** temporarily hold structurally plausible event JSON.
- **c-relay-pg:** remain the sole authority for accepting, storing, and
broadcasting events.
This integration is PostgreSQL-only. The relay's normal SQLite operation remains
unchanged, but the external caching service is unavailable with that backend.
## 2. Simplified Architecture
```mermaid
flowchart LR
ROOT[Configured root npubs] --> CACHE[Caching application]
UP[Upstream relays] --> CACHE
CACHE --> INBOX[PostgreSQL caching event inbox]
INBOX --> POLLER[c-relay-pg bounded poller]
POLLER --> INGEST[Relay-owned event ingestion]
INGEST --> EVENTS[Canonical events table]
INGEST --> CLIENTS[Active subscriptions]
UI[Caching admin page] --> CFG[Shared caching configuration]
CFG --> CACHE
CACHE --> STATUS[Service heartbeat and progress]
STATUS --> UI
```
No loopback sink relay pool is used. No direct insert into the canonical event
store is allowed from the caching application.
## 3. Deliberate Simplicity Rules
The first implementation will not include:
- multiple caching workers;
- multiple inbox consumers;
- row leases or claim expiration;
- a queue state machine;
- retry rows;
- a dead-letter queue;
- PostgreSQL `LISTEN`/`NOTIFY`;
- persistent relay-health history;
- exactly-once delivery;
- direct process start/stop through the web page;
- upstream NIP-42 authentication;
- support for the SQLite backend.
The design prefers harmless duplicate acquisition over complicated delivery
coordination. Event IDs already provide natural deduplication in both the inbox
and canonical store.
## 4. Responsibility Boundaries
### 4.1 Caching application
The caching application is responsible for:
1. Reading caching configuration from PostgreSQL.
2. Decoding configured root npubs.
3. Resolving the roots' newest kind-3 follow lists.
4. Discovering useful read relays from kind 10002.
5. Computing a reasonable covering relay set.
6. Maintaining live subscriptions for roots and follows.
7. Performing polite, resumable historical backfill.
8. Inserting received event JSON into the inbox.
9. Persisting only the progress needed to resume discovery and backfill.
10. Publishing a lightweight heartbeat and status snapshot.
It is not responsible for:
- deciding whether an event belongs in the canonical relay database;
- applying c-relay-pg publication authorization;
- processing NIP-09 deletions;
- implementing replacement semantics;
- executing relay administrator commands;
- broadcasting to connected relay clients.
The caching callback may perform minimal application-side checks before insert:
- the callback supplied a JSON object;
- an event ID string exists;
- the serialized event is below a configured maximum size.
Cryptographic verification is optional in the fetcher and is not trusted by
c-relay-pg. The initial version should omit it unless the existing standalone
code already provides it at negligible integration cost.
### 4.2 PostgreSQL inbox
The inbox is deliberately dumb. It rejects obvious malformed data, deduplicates
events currently waiting in the inbox, and establishes live-over-backfill
priority. It does not attempt to verify a Nostr hash or Schnorr signature.
### 4.3 c-relay-pg
c-relay-pg is responsible for:
1. Removing a bounded batch from the inbox.
2. Parsing each event JSON object.
3. Applying authoritative structural, ID, and signature validation.
4. Applying relay-wide event rules such as expiration and PoW where applicable.
5. Bypassing client-session NIP-42 requirements because the source is internal.
6. Preventing imported kind 23456 events from executing administrator commands.
7. Applying duplicate, ephemeral, replaceable, addressable, and NIP-09 behavior.
8. Storing accepted events through its normal database abstraction.
9. Running post-store work and broadcasting newly accepted events from the main
libwebsockets thread.
10. Recording simple in-memory and cumulative import counters.
## 5. Minimal PostgreSQL Schema
Add the following objects to [`src/pg_schema.sql`](../src/pg_schema.sql) and the
embedded PostgreSQL schema header.
### 5.1 `caching_event_inbox`
Suggested logical schema:
```sql
CREATE TABLE IF NOT EXISTS caching_event_inbox (
queue_id BIGSERIAL PRIMARY KEY,
event_id TEXT NOT NULL UNIQUE,
event_json JSONB NOT NULL,
source_relay TEXT,
source_class TEXT NOT NULL DEFAULT 'backfill',
priority SMALLINT NOT NULL DEFAULT 1,
received_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
CHECK (jsonb_typeof(event_json) = 'object'),
CHECK (jsonb_typeof(event_json->'id') = 'string'),
CHECK (length(event_json->>'id') = 64),
CHECK (event_id = event_json->>'id'),
CHECK (jsonb_typeof(event_json->'pubkey') = 'string'),
CHECK (length(event_json->>'pubkey') = 64),
CHECK (jsonb_typeof(event_json->'sig') = 'string'),
CHECK (length(event_json->>'sig') = 128),
CHECK (jsonb_typeof(event_json->'created_at') = 'number'),
CHECK (jsonb_typeof(event_json->'kind') = 'number'),
CHECK (jsonb_typeof(event_json->'tags') = 'array'),
CHECK (jsonb_typeof(event_json->'content') = 'string'),
CHECK (source_class IN ('live', 'discovery', 'backfill')),
CHECK (priority IN (0, 1))
);
CREATE INDEX IF NOT EXISTS idx_caching_inbox_dequeue
ON caching_event_inbox(priority, received_at, queue_id);
```
Priority meanings:
- `0`: live and discovery metadata;
- `1`: historical backfill.
The caching application uses a parameterized insert with
`ON CONFLICT (event_id) DO NOTHING`. It must impose a serialized event-size
limit before sending the row. PostgreSQL schema constraints are structural
protection, not authoritative Nostr validation.
### 5.2 `caching_service_state`
Use one singleton row for inexpensive UI status:
- service version;
- service state: `starting`, `running`, `degraded`, or `stopped`;
- applied config generation;
- heartbeat timestamp;
- followed author count;
- selected and connected relay counts;
- current backfill window and author cursor;
- events fetched and inbox inserts;
- last error text and timestamp.
This is a status snapshot, not an audit log. The caching process overwrites the
same row periodically.
### 5.3 `caching_backfill_progress`
Persist only enough state for the caching application to resume politely:
- author pubkey;
- window index;
- immutable window anchor;
- current inclusive `until` cursor;
- completion flag;
- updated timestamp.
Use a composite primary key on author and window. Do not store individual fetched
event IDs here; inbox and canonical event IDs handle deduplication.
Timestamp-only Nostr pagination can saturate when many events share one second.
For the first version, increase the query limit up to a fixed safety ceiling when
a full page ends at one timestamp. If that ceiling remains saturated, leave the
author incomplete, log it, and retry later rather than falsely marking it done.
## 6. Simple Destructive Dequeue
c-relay-pg uses one PostgreSQL transaction to remove and return a small batch:
```sql
WITH selected AS (
SELECT queue_id
FROM caching_event_inbox
ORDER BY priority ASC, received_at ASC, queue_id ASC
LIMIT $1
FOR UPDATE
)
DELETE FROM caching_event_inbox AS inbox
USING selected
WHERE inbox.queue_id = selected.queue_id
RETURNING inbox.event_id,
inbox.event_json,
inbox.source_relay,
inbox.source_class,
inbox.received_at;
```
Only one c-relay-pg inbox consumer is supported. `SKIP LOCKED`, leases, and claim
owners are therefore unnecessary.
### 6.1 Accepted trade-off
There is a small loss window if c-relay-pg commits the delete and crashes before
storing the returned events. This is accepted deliberately:
- live subscriptions reconnect with overlap;
- backfill revisits incomplete and steady-state windows;
- duplicate reacquisition is safe;
- the reduced implementation complexity is worth the rare temporary loss.
The poller should keep batches small so the loss window and memory footprint are
small.
## 7. c-relay-pg Inbox Consumer
### 7.1 Polling
- Compile the consumer only for `DB_BACKEND_POSTGRES`.
- Poll a small batch at a configurable interval.
- Poll quickly while rows are found and back off to a slower interval when empty.
- Always order live/discovery rows before backfill rows.
- Allow only one batch in memory at a time.
- Stop polling immediately during relay shutdown.
Initial conservative defaults should be small, for example a few dozen rows per
batch and subsecond-to-multisecond active polling with a longer idle interval.
Exact defaults should be selected during integration testing rather than encoded
as architectural requirements.
### 7.2 Ingestion behavior
Do not send dequeued events through a loopback WebSocket. Refactor the existing
inbound event handling into a reusable internal ingestion entry point with an
explicit source mode:
- `CLIENT_EVENT`: normal client authentication and response behavior;
- `CACHING_INBOX_EVENT`: no client session or NIP-42 requirement, no OK response,
and no administrator command execution.
The shared path should preserve:
- event ID and signature verification;
- event limits;
- expiration and PoW behavior;
- NIP-09 authorization and deletion behavior;
- ephemeral handling;
- PostgreSQL replacement semantics in
[`postgres_db_insert_event_with_json()`](../src/db_ops_postgres.c:1596);
- duplicate detection;
- post-store monitoring;
- main-thread broadcast.
Avoid creating a second implementation of these rules solely for inbox events.
### 7.3 Main-thread broadcast
Database and cryptographic work may run off the libwebsockets thread, but
[`store_event_post_actions()`](../src/main.c:1161) and active subscription
broadcast must be queued to the main thread, following the existing asynchronous
completion pattern in
[`process_async_event_completions()`](../src/websockets.c:725).
Newly stored events are broadcast. Duplicates and stale replacements are not.
Validated ephemeral events are broadcast but not stored.
### 7.4 Failure behavior
Because rows have already been removed, malformed or invalid events are simply
counted and logged at a rate-limited level. They are not requeued.
A temporary canonical database failure should stop further inbox polling until
the relay database is healthy. The current in-memory batch may be lost under the
accepted simple-delivery model.
## 8. Caching Application Acquisition Strategy
The external application should spend most of its design effort here.
### 8.1 Follow graph
- Decode configured root npubs.
- Fetch the newest kind 3 for each root from bootstrap relays.
- Include roots themselves in the author set.
- Parse valid `p` tags, deduplicate authors, and enforce a configured author cap.
- Refresh periodically and replace live subscriptions when the set changes.
Local-first querying is optional in this architecture. PostgreSQL already holds
canonical kind-3 events, so the caching application may query the canonical event
store read-only for discovery metadata before contacting upstream relays. It must
not write canonical rows.
### 8.2 NIP-65 relay discovery
- Fetch newest kind 10002 events in bounded author batches.
- Use unmarked or `read` relay tags; ignore `write`-only tags.
- Normalize and deduplicate `wss://` URLs.
- Enforce per-author and global relay caps.
- Use the greedy covering-set logic from the standalone implementation.
- Retain configured bootstrap relays as fallback.
### 8.3 Friendly live subscriptions
- Separate root and followed-author subscriptions when their kind lists differ.
- Batch author lists into reasonable filter sizes.
- Reconnect with bounded exponential backoff and jitter.
- Use a small timestamp overlap when resubscribing.
- Insert callback event JSON into the inbox and continue; do not wait for
c-relay-pg processing.
### 8.4 Friendly backfill
- Use recent-first progressive windows.
- Persist an immutable anchor for each active window.
- Query one author at a time from that author's declared outbox relays, with
bootstrap fallback.
- Use configurable page size and delay between queries.
- Apply relay-specific backoff after errors or timeouts.
- Keep global and per-relay request rates low.
- Advance progress only after the returned page has been inserted into the
inbox or identified as already queued.
- Periodically repeat a bounded widest-window sweep after initial completion.
## 9. Shared Configuration
Keep administrator intent in the existing [`config`](../src/pg_schema.sql:192)
table with category `caching` and `requires_restart = 0`.
Suggested keys:
- `caching_enabled`;
- `caching_config_generation`;
- `caching_root_npubs`;
- `caching_bootstrap_relays`;
- `caching_kinds`;
- `caching_admin_kinds`;
- `caching_live_enabled`;
- `caching_live_resubscribe_seconds`;
- `caching_backfill_enabled`;
- `caching_backfill_windows`;
- `caching_backfill_page_size`;
- `caching_backfill_tick_interval_ms`;
- `caching_backfill_window_cooldown_seconds`;
- `caching_follow_graph_refresh_seconds`;
- `caching_relay_discovery_refresh_seconds`;
- `caching_max_followed_pubkeys`;
- `caching_max_upstream_relays`;
- `caching_max_relays_per_pubkey`;
- `caching_query_timeout_ms`;
- `caching_inbox_batch_size`;
- `caching_inbox_active_poll_ms`;
- `caching_inbox_idle_poll_ms`;
- `caching_max_event_json_bytes`.
The web UI updates these through the existing encrypted administrator API. After
a successful update, increment `caching_config_generation`. The caching
application polls the generation and reloads valid changes. No cross-process
command queue is needed.
`caching_enabled = false` tells the external application to close upstream
activity. systemd or the container runtime, not the web page, owns the process
lifecycle.
## 10. Admin UI
### 10.1 Sidenav placement
In [`api/index.html`](../api/index.html:14), insert **Caching** immediately after
**Relay Events** and before **DM**:
1. Statistics
2. Subscriptions
3. Configuration
4. Authorization
5. IP BAN
6. Relay Events
7. **Caching**
8. DM
9. Database Query
Register `cachingSection` in [`switchPage()`](../api/index.js:5349).
### 10.2 Page layout
Reuse the existing admin UI classes and divide the page into three blocks.
**Service status**
- enabled configuration;
- service heartbeat and state;
- desired and applied config generation;
- followed author count;
- selected and connected relay counts;
- current backfill window and cursor;
- fetched and inbox-inserted counters;
- last service error.
**Relay inbox status**
- pending live/discovery count;
- pending backfill count;
- oldest inbox row age;
- relay-consumed, accepted, duplicate, invalid, and failed counters;
- last successful inbox poll.
**Configuration**
- root npubs and bootstrap relays;
- normal and root kind lists;
- live and backfill toggles;
- backfill windows and pacing;
- discovery refresh intervals;
- author and relay safety caps;
- inbox batch and polling settings;
- **Apply Configuration** button;
- **Reset Backfill Progress** button with confirmation.
The UI reads service state from the singleton status row and lightweight inbox
aggregates. Poll only while the Caching page is visible and use a modest refresh
interval.
## 11. Process and Database Security
Use separate PostgreSQL roles:
- **Caching service role:** read caching configuration and permitted canonical
discovery metadata; insert/select its own status and progress; insert into the
inbox; no insert/update/delete permission on canonical events.
- **c-relay-pg role:** normal relay permissions plus dequeue permission on the
inbox and read access to caching status.
Use parameterized SQL throughout. Restrict public upstreams to normalized
`wss://` URLs. Apply a database statement timeout to caching queries so the
fetcher cannot hold resources indefinitely.
## 12. Lifecycle
### 12.1 Caching service
- Starts and stops independently under systemd or a container runtime.
- On startup, loads configuration and progress, updates heartbeat state, then
begins discovery/live/backfill.
- On shutdown, closes subscriptions, saves current progress, marks status
stopped, and disconnects from PostgreSQL.
- Its failure does not stop c-relay-pg; cached acquisition merely pauses.
### 12.2 c-relay-pg
- Starts the inbox poller only for PostgreSQL builds after database, writer pool,
and WebSocket systems are ready.
- Stops new polls before shutting down the writer pool or WebSocket context.
- Drains already-created main-thread completions before destroying those
dependencies.
- A missing inbox table should be logged as caching unavailable, not terminate
the relay, unless schema migration policy requires otherwise.
## 13. Implementation Sequence
1. Add `caching_event_inbox`, `caching_service_state`, and
`caching_backfill_progress` to the PostgreSQL schema and embedded schema.
2. Add PostgreSQL database abstraction functions for bounded destructive dequeue
and lightweight inbox counts.
3. Refactor c-relay-pg inbound event processing into a shared source-aware
ingestion entry point without changing normal client behavior.
4. Add the PostgreSQL-only inbox poller, bounded in-memory batch, and main-thread
post-action/broadcast completion path.
5. Add caching configuration defaults and validation to c-relay-pg.
6. Adapt the standalone caching application to read PostgreSQL configuration,
insert raw event JSON into the inbox, and update heartbeat/status.
7. Correct the standalone backfill implementation so limited queries paginate
and do not falsely complete an author/window.
8. Persist simple per-author/window progress and implement polite retry/backoff.
9. Add the Caching page after Relay Events and before DM, including service,
inbox, configuration, and reset-progress controls.
10. Add PostgreSQL role/grant documentation and systemd/container deployment
examples for the separate caching process.
11. Test live priority, duplicate acquisition, invalid inbox rows, replacement,
deletion, ephemeral broadcast, process restarts, destructive-dequeue crash
behavior, upstream outages, and graceful shutdown.
12. Build and validate c-relay-pg only through
[`make_and_restart_relay.sh`](../make_and_restart_relay.sh), using
`--preserve-database` where test state must survive.
## 14. Acceptance Criteria
- The caching application cannot write canonical event rows.
- Received upstream events require only minimal fetcher checks before inbox
insertion.
- Inbox constraints reject structurally implausible rows and deduplicate pending
event IDs.
- c-relay-pg removes small batches ordered live/discovery before backfill.
- c-relay-pg remains the authoritative ID/signature validator and storage owner.
- Imported events never execute relay administrator commands or require a client
NIP-42 session.
- Newly accepted events are broadcast to active subscriptions from the main
libwebsockets thread.
- Duplicate and stale replacement events are not rebroadcast.
- The caching service behaves politely through bounded filters, paced backfill,
targeted outbox queries, and retry backoff.
- Backfill progress survives caching-service restarts and does not falsely mark
saturated timestamps complete.
- A caching-service failure does not stop the relay.
- The accepted destructive-dequeue crash window is documented and recoverable
through live overlap and repeated backfill.
- The Caching UI appears after Relay Events and before DM and distinguishes
external service health from relay inbox consumption.
File diff suppressed because it is too large Load Diff
+186
View File
@@ -0,0 +1,186 @@
# Plan: Eliminate g_db from the Main Thread
## Problem Statement
All SQLite calls go through `db_ops.c`, but `db_active_connection()` falls back to `g_db` when `g_thread_db` is NULL. Since the main thread never sets `g_thread_db`, every `db_*` call from lws-main executes synchronous SQLite on the main thread. The thread pool workers have their own connections and work correctly — the problem is the **fallback path**.
Current perf data shows lws-main at **67.6% avg CPU**, dominated by SQLite symbols (`sqlite3BtreeTableMoveto`, `sqlite3VdbeExec`, `pcache1Fetch`).
## Goal
Remove `g_db` as a runtime connection used by the main thread. After this change:
- The main thread has **no SQLite connection** and cannot execute synchronous queries
- All DB work routes through thread pool workers (which have their own connections)
- `g_db` is only used during startup/shutdown (before/after the event loop)
- Any accidental `db_*` call from lws-main returns an error instead of silently blocking
## Architecture After Change
```
lws-main thread:
- WebSocket protocol handling
- Message parsing/routing
- Completion queue draining
- NO SQLite access during event loop
db-read-0 thread:
- REQ queries
- COUNT queries
- Config cache miss reads
- Auth rule checks
- Monitoring/stats queries
db-write thread:
- EVENT inserts
- Subscription logging
- Config updates
- Auth rule modifications
event-worker thread:
- Signature validation
- Duplicate check
- Store via db-write
```
## Categorized g_db Call Sites
### Category A: Startup-only — keep using g_db
These run before the event loop starts. They are fine.
| Function | File | Purpose |
|----------|------|---------|
| `db_init()` | db_ops.c:27 | Open database |
| `db_exec_sql()` for PRAGMAs | main.c:854-882 | WAL, mmap, cache setup |
| `db_table_exists()` | main.c:750 | Schema check |
| `db_get_schema_version_dup()` | main.c:754 | Migration check |
| `db_exec_sql()` for schema | main.c:842 | Create tables |
| `populate_all_config_values_atomic()` | config.c:4290 | First-time config |
| `populate_default_config_values()` | config.c:1657 | Default config |
| `add_pubkeys_to_config_table()` | config.c:1778 | Pubkey storage |
| `apply_cli_overrides_atomic()` | config.c:4211 | CLI overrides |
| `db_store_relay_private_key_hex()` | config.c:495 | Key storage |
| `db_populate_event_tags_from_existing()` | main.c:1156 | Tag migration |
| `cleanup_all_subscriptions_on_startup()` | subscriptions.c:1092 | Orphan cleanup |
### Category B: Shutdown-only — keep using g_db
| Function | File | Purpose |
|----------|------|---------|
| `db_exec_sql()` WAL checkpoint | main.c:900 | Clean shutdown |
| `db_close()` | main.c:904 | Close connection |
### Category C: Runtime hot path — must move off main thread
These are called during the event loop from lws-main context.
| Function | File | Called from | Fix strategy |
|----------|------|------------|--------------|
| `db_get_config_value_dup()` | db_ops.c:777 | config cache miss | Already cached with 5s TTL; pre-warm cache at startup so misses are rare |
| `store_event()` for kind 14/1059 | main.c:983 | websockets.c sync path | Route through async event worker |
| `store_event_post_actions()` | main.c | completion handler | Already runs on main; its DB calls need routing |
| `db_log_subscription_created()` | db_ops.c:145 | subscriptions.c | Fire-and-forget via write queue |
| `db_log_subscription_closed()` | db_ops.c:165 | subscriptions.c | Fire-and-forget via write queue |
| `db_log_subscription_disconnected()` | db_ops.c:193 | subscriptions.c | Fire-and-forget via write queue |
| `db_update_subscription_events_sent()` | db_ops.c:225 | subscriptions.c | Fire-and-forget via write queue |
| `db_cleanup_orphaned_subscriptions()` | db_ops.c:244 | subscriptions.c | Move to startup only |
| `generate_and_post_status_event()` | websockets.c:3408 | periodic timer | Submit to write worker |
| `ip_ban_cleanup()` / `ip_ban_log_stats()` | websockets.c:3196 | periodic timer | Submit to write worker |
| `db_get_total_event_count_ll()` | db_ops.c:589 | api.c monitoring | Submit to read worker |
| `db_get_event_count_since()` | db_ops.c:606 | api.c monitoring | Submit to read worker |
| `db_get_event_kind_distribution_rows()` | db_ops.c:625 | api.c monitoring | Submit to read worker |
| `db_get_top_pubkeys_rows()` | db_ops.c:663 | api.c monitoring | Submit to read worker |
| `db_get_subscription_details_rows()` | db_ops.c:697 | api.c monitoring | Submit to read worker |
| `db_get_all_config_rows()` | db_ops.c:745 | api.c config query | Submit to read worker |
| `db_execute_readonly_query_json()` | db_ops.c:442 | api.c SQL query | Submit to read worker |
| `db_event_id_exists()` | db_ops.c:1041 | main.c event check | Already moved to event worker |
| `db_retrieve_event_by_id()` | db_ops.c:1056 | main.c | Submit to read worker |
| `db_get_event_pubkey()` | db_ops.c:262 | NIP-09 deletion | Submit to read worker |
| `db_delete_event_by_id()` | db_ops.c:285 | NIP-09 deletion | Submit to write worker |
| `db_delete_older_replaceable_events()` | db_ops.c:305 | store_event_core | Already on write worker |
| `db_store_config_event()` | db_ops.c:888 | config.c | Submit to write worker |
| `db_add_auth_rule()` | db_ops.c:1226 | config.c admin | Submit to write worker |
| `db_remove_auth_rule()` | db_ops.c:1242 | config.c admin | Submit to write worker |
| `db_delete_wot_whitelist_rules()` | db_ops.c:1258 | config.c WoT | Submit to write worker |
| `db_count_wot_whitelist_rules()` | db_ops.c:1266 | config.c | Cache or read worker |
| `db_store_event_tags_cjson()` | db_ops.c:1140 | main.c post-actions | Already on write worker path |
| `db_get_config_row_count()` | db_ops.c:1121 | config.c diagnostics | Cache or read worker |
| `db_set_config_value_full()` | db_ops.c:796 | config.c | Submit to write worker |
| `db_update_config_value_only()` | db_ops.c:821 | config.c | Submit to write worker |
| `db_upsert_config_value()` | db_ops.c:838 | api.c | Submit to write worker |
| `db_exec_sql()` for transactions | config.c | admin events | Submit to write worker |
| `db_count_with_sql()` | db_ops.c:415 | config.c admin | Submit to read worker |
| `is_config_table_ready()` | config.c:4595 | config.c hybrid | Cache result at startup |
| `generate_config_event_from_table()` | config.c:4933 | config.c | Cache or read worker |
### Category D: Auth rule checks — already use separate connections
These functions in `db_ops.c` open their own temporary read-only connections:
| Function | Line | Notes |
|----------|------|-------|
| `db_is_pubkey_blacklisted()` | 347 | Opens own connection |
| `db_is_hash_blacklisted()` | 362 | Opens own connection |
| `db_is_pubkey_whitelisted()` | 377 | Opens own connection |
| `db_count_active_whitelist_rules()` | 392 | Opens own connection |
These are already safe — they don't use `g_db`. No change needed.
## Implementation Strategy
### Phase 1: Pre-warm caches and eliminate cache-miss DB reads
1. Pre-warm the config cache at startup by loading all config values into the in-memory cache before the event loop starts
2. Pre-warm the NIP-11 cache at startup
3. Pre-warm the auth rules fast cache at startup
4. This eliminates the most frequent cache-miss `db_get_config_value_dup()` calls
### Phase 2: Route subscription logging through write worker
1. Add a `THREAD_POOL_JOB_FIRE_AND_FORGET` job type to thread_pool
2. Create async wrappers: `db_log_subscription_created_async()`, etc.
3. These submit SQL to the write worker queue and return immediately
4. No completion callback needed — fire and forget
### Phase 3: Route special-kind EVENT store through async worker
1. Remove the `event_is_async_eligible()` exclusion for kinds 14, 1059, 23456
2. For kind 23456: process admin command in completion handler on main thread after store
3. For kind 14/1059: process NIP-17 DM in completion handler after store
### Phase 4: Route periodic timer DB work through workers
1. `generate_and_post_status_event()` — submit event creation to write worker
2. `ip_ban_cleanup()` / `ip_ban_log_stats()` — submit to write worker
3. Monitoring queries — submit to read worker with completion callback
### Phase 5: Null out g_db before event loop
1. After startup is complete and thread pool is initialized, set `g_db = NULL`
2. Change `db_active_connection()` to assert/warn if both `g_thread_db` and `g_db` are NULL
3. Any accidental main-thread DB call will now fail loudly instead of silently blocking
4. Restore `g_db` briefly for shutdown checkpoint
## Implementation Order
1. Phase 1 — lowest risk, immediate benefit from eliminating cache misses
2. Phase 2 — straightforward fire-and-forget pattern
3. Phase 3 — moderate complexity, needs careful completion handler design
4. Phase 4 — moderate complexity, periodic tasks need async patterns
5. Phase 5 — the final enforcement step, only safe after phases 1-4
## Risk Assessment
- **Phase 1**: Very low risk — just pre-warming existing caches
- **Phase 2**: Low risk — subscription logging is non-critical, fire-and-forget is safe
- **Phase 3**: Medium risk — admin/DM event processing has complex state; needs careful testing
- **Phase 4**: Medium risk — periodic tasks have side effects that need to complete
- **Phase 5**: High risk if done prematurely — must verify ALL runtime paths are covered first
## Expected Impact
After all phases:
- lws-main CPU should drop from ~67% to near 0% SQLite overhead
- All SQLite work happens on db-read and db-write threads
- Main thread only does WebSocket I/O, JSON parsing, and completion queue draining
- Thread model becomes: lws-main = pure I/O, workers = all DB
+194
View File
@@ -0,0 +1,194 @@
# Main Thread CPU Offload Plan
## Problem Statement
The main `c_relay_pg` thread consumes ~56% CPU while the DB worker threads (`db-read-1..4`, `db-write`) sit near 0%. All protocol handling, JSON parsing, event validation, subscription matching, and message queueing happens synchronously inside `nostr_relay_callback()` on the main libwebsockets event-loop thread.
## Current Architecture
```mermaid
flowchart TD
LWS[lws_service - main thread] --> CB[nostr_relay_callback]
CB --> PARSE[JSON parse - cJSON_Parse]
CB --> VALIDATE[Signature verify - nostr_validate_unified_request]
CB --> STORE[store_event - builds payload]
STORE --> SYNC_WRITE[thread_pool_execute_store_event_sync]
SYNC_WRITE --> |blocks main thread| DB_WRITE[db-write thread]
DB_WRITE --> |signal| SYNC_WRITE
CB --> REQ[handle_req_message - builds SQL]
REQ --> SYNC_READ[thread_pool_execute_req_sync]
SYNC_READ --> |blocks main thread| DB_READ[db-read-N thread]
DB_READ --> |signal| SYNC_READ
CB --> BROADCAST[broadcast_event_to_subscriptions]
BROADCAST --> QUEUE[queue_message per subscriber]
CB --> WRITEABLE[LWS_CALLBACK_SERVER_WRITEABLE]
WRITEABLE --> DRAIN[process_message_queue - lws_write]
```
### Why the main thread is hot
| Work item | Where | Cost |
|-----------|-------|------|
| JSON parsing | `cJSON_Parse` in `LWS_CALLBACK_RECEIVE` | Medium - per message |
| Signature verification | `nostr_validate_unified_request` - ed25519 crypto | **High** - per EVENT |
| SQL query building | `handle_req_message` filter-to-SQL loop | Low-Medium |
| Sync DB wait | `thread_pool_execute_*_sync` - pthread_cond_wait | Blocks but yields CPU |
| Result iteration + expiration check | Row loop in `handle_req_message` with `cJSON_Parse` per row | Medium-High for large result sets |
| Subscription matching | `broadcast_event_to_subscriptions` - filter matching | Medium - scales with subscriber count |
| Message serialization + queueing | `snprintf` + `queue_message_take_ownership` per subscriber | Medium |
| Config lookups | `get_config_int/bool` called repeatedly in hot paths | Low but frequent |
### Key constraint: libwebsockets is single-threaded
libwebsockets requires that **all `lws_write`, `lws_callback_on_writable`, and `lws_close_reason` calls happen from the service thread** (the thread running `lws_service`). This means we cannot directly write to WebSocket connections from worker threads. However, we **can** do computation on worker threads and post results back to the main thread for I/O.
## Offload Strategy
### Phase 1: Async EVENT Processing (Highest Impact)
Convert EVENT handling from synchronous to async. Currently the main thread does: parse → validate → store → broadcast, all blocking. Instead:
1. **Main thread**: Parse JSON (fast), extract event ID for dedup check, then submit a job to a new "event processing" worker thread
2. **Worker thread**: Signature verification (expensive crypto), store_event (already goes to DB thread), prepare broadcast payload
3. **Main thread callback**: Receive result via `wake_loop_cb` + `lws_cancel_service`, send OK response and broadcast to subscribers
```mermaid
flowchart TD
LWS[lws_service - main thread] --> RECEIVE[LWS_CALLBACK_RECEIVE]
RECEIVE --> PARSE[JSON parse + dedup check]
PARSE --> SUBMIT[Submit to event-worker queue]
SUBMIT --> LWS
WORKER[event-worker thread] --> VERIFY[Signature verification]
VERIFY --> STORE_DB[store_event via DB pool]
STORE_DB --> PREP[Prepare broadcast payload]
PREP --> RESULT_Q[Push result to completion queue]
RESULT_Q --> WAKE[lws_cancel_service]
LWS2[lws_service wakes] --> POLL[Poll completion queue]
POLL --> OK[Send OK response via queue_message]
POLL --> BCAST[broadcast_event_to_subscriptions]
```
**What this offloads**: ed25519 signature verification (~the most expensive per-event operation), event classification, tag serialization, and the synchronous DB store wait.
**What stays on main thread**: JSON parse (needed to extract event ID for dedup), OK response writing, broadcast fan-out (requires lws access).
### Phase 2: Async REQ Query Execution (Medium Impact)
Convert REQ handling from sync to async:
1. **Main thread**: Parse filters, build SQL, create subscription, submit query job
2. **DB reader thread**: Execute query (already happens, but currently blocks main thread via `_sync`)
3. **Main thread callback**: Iterate results, queue EVENT messages, send EOSE
This is simpler than Phase 1 because the thread pool already supports async submission via `thread_pool_submit_read` with a `result_cb`. The `_sync` wrappers just add a condvar wait on top. We need to:
- Use `thread_pool_submit_read` directly instead of `thread_pool_execute_req_sync`
- Store pending REQ context (sub_id, wsi, pss) so the callback can complete the work
- In the result callback, push results to a completion queue and call `lws_cancel_service`
- On the main thread, drain the completion queue and send EVENT + EOSE messages
### Phase 3: Reduce Per-Row Overhead in REQ Results (Low-Medium Impact)
Currently each row from a REQ query gets `cJSON_Parse` just to check NIP-40 expiration. This is wasteful:
- Option A: Add an `expiration` column to the events table so expiration filtering can be done in SQL
- Option B: Store expiration timestamp in a fast-parse format (extract during INSERT, store as integer column)
- Option C: Use string search on the raw JSON for the expiration tag instead of full parse
### Phase 4: Config Value Caching (Low Impact, Easy Win)
`get_config_int` and `get_config_bool` are called on every message in hot paths. These do SQLite queries. Cache config values in memory with a TTL or invalidation signal, so the main loop only refreshes them periodically (already done for `debug_level` every 60s — extend to all hot-path config values).
## Implementation Priority
| Phase | Impact | Risk | Complexity |
|-------|--------|------|------------|
| Phase 1: Async EVENT | **High** - removes crypto from main thread | Medium - async state management | Medium-High |
| Phase 2: Async REQ | **Medium** - unblocks main thread during queries | Low - infrastructure exists | Medium |
| Phase 3: Expiration optimization | **Low-Medium** - reduces per-row parse cost | Low | Low |
| Phase 4: Config caching | **Low** - reduces DB round-trips | Very Low | Low |
## Detailed Design: Phase 1 (Async EVENT Processing)
### New Components
#### Completion Queue (`src/completion_queue.h/.c`)
A thread-safe FIFO queue for posting results from worker threads back to the main thread:
```c
typedef struct {
int type; // COMPLETION_TYPE_EVENT_RESULT, COMPLETION_TYPE_REQ_RESULT, etc.
void* data; // Type-specific result data
struct lws* wsi; // Target WebSocket connection
void* pss; // Per-session data
} completion_item_t;
int completion_queue_init(void);
int completion_queue_push(completion_item_t* item);
completion_item_t* completion_queue_pop(void); // Non-blocking
void completion_queue_shutdown(void);
```
#### Event Worker Thread
A dedicated pthread that processes EVENT validation/storage:
```c
typedef struct {
cJSON* event; // Parsed event JSON - ownership transferred
cJSON* full_message; // Full message JSON for context
struct lws* wsi;
void* pss;
char sub_id[64]; // For response routing
} event_work_item_t;
```
#### Main Loop Integration
Add a completion queue drain step to the main event loop. After `lws_service` returns (either from timeout or `lws_cancel_service` wake), check the completion queue:
```c
while (g_server_running && !g_shutdown_flag) {
int result = lws_service(ws_context, 1000);
// NEW: Drain completion queue
completion_item_t* item;
while ((item = completion_queue_pop()) != NULL) {
process_completion(item); // Send OK, broadcast, etc.
free(item);
}
// ... existing periodic checks ...
}
```
### Changes to Existing Code
1. **`nostr_relay_callback` EVENT path**: After JSON parse and dedup check, instead of calling `nostr_validate_unified_request` + `store_event` + `broadcast_event_to_subscriptions` synchronously, submit an `event_work_item_t` to the event worker queue and return 0 immediately.
2. **`store_event`**: No changes needed — it already uses `thread_pool_execute_store_event_sync` which will run on the DB writer thread. The event worker thread will call it.
3. **`broadcast_event_to_subscriptions`**: No changes needed — it will be called from the main thread when processing the completion item, which is correct since it calls `queue_message_take_ownership` (requires lws thread).
### Thread Safety Considerations
- The `cJSON* event` object must be fully owned by the worker thread during processing. The main thread must not access it after submission.
- The `wsi` and `pss` pointers could become invalid if the client disconnects while the event is being processed. The completion handler must validate that the connection is still alive before sending the OK response.
- A generation counter or epoch on `pss` can detect stale references.
## What Cannot Be Offloaded
- **`lws_write` / `queue_message`**: Must happen on the lws service thread
- **`lws_callback_on_writable`**: Must happen on the lws service thread
- **`lws_close_reason`**: Must happen on the lws service thread
- **Subscription list iteration for broadcast**: Accesses `lws_wsi_user` which is lws-internal
These are fundamental libwebsockets constraints. The pattern is always: do computation off-thread, post result to completion queue, wake main thread, do I/O on main thread.
## Expected Impact
With Phase 1 alone, the main thread would no longer perform:
- ed25519 signature verification (~100-500μs per event depending on CPU)
- Synchronous DB store wait (~50-200μs per event)
- Event classification, tag serialization, JSON serialization for storage
This should reduce main-thread CPU by roughly 30-50% for EVENT-heavy workloads, shifting that work to the event worker thread and DB threads.
+3 -3
View File
@@ -1,8 +1,8 @@
# C-Relay Memory Leak Investigation & Fix Plan
# C-Relay-PG Memory Leak Investigation & Fix Plan
## Problem Statement
c-relay reached **1.56 GB** of its 1.5 GB `MemoryHigh` cgroup limit after only **1 hour 37 minutes** of runtime. The kernel is actively throttling the process with swap pressure (1.8M swap used). This strongly indicates one or more memory leaks causing unbounded growth.
c-relay-pg reached **1.56 GB** of its 1.5 GB `MemoryHigh` cgroup limit after only **1 hour 37 minutes** of runtime. The kernel is actively throttling the process with swap pressure (1.8M swap used). This strongly indicates one or more memory leaks causing unbounded growth.
## Root Cause Analysis
@@ -187,7 +187,7 @@ Add `free((char*)result)` after every `get_config_value()` call that doesn't alr
While fixes are being implemented:
1. **Raise `MemoryHigh`** to 2 GB in the systemd unit to prevent throttling
2. **Add a cron job** to restart the service every 12-24 hours
3. **Monitor with**: `watch -n 5 'cat /proc/$(pgrep c_relay)/status | grep -E "VmRSS|VmSwap"'`
3. **Monitor with**: `watch -n 5 'cat /proc/$(pgrep c_relay_pg)/status | grep -E "VmRSS|VmSwap"'`
---
+201
View File
@@ -0,0 +1,201 @@
# Multi-Admin Support Plan
## Overview
Enable multiple Nostr pubkeys to act as relay administrators. Currently the relay stores a single `admin_pubkey` string in the `config` table and uses `strcmp()` equality checks throughout the codebase. This plan converts that to a comma-separated list of pubkeys stored in the same config key, with a centralized `is_admin_pubkey()` helper that all authorization gates call.
## Design Principles
1. **Minimal schema change** — keep using the existing `config` table with key `admin_pubkey`. The value changes from a single 64-char hex string to a comma-separated list of 64-char hex strings.
2. **Primary admin concept** — the first pubkey in the list is the "primary admin" (the original one). Only the primary admin can add/remove other admins. This prevents privilege escalation.
3. **Centralized check** — introduce one function `is_admin_pubkey(const char* pubkey)` that all 8 authorization gates call instead of inline `strcmp()`.
4. **In-memory cache** — parse the comma-separated list once on startup/config-change into a static array for O(1)-ish lookups without repeated string parsing.
5. **Backward compatible** — a single pubkey with no commas works identically to today.
## Data Model
```
config table:
key = "admin_pubkey"
value = "hex1,hex2,hex3" (comma-separated, no spaces)
data_type = "string"
```
Maximum admins: 10 (compile-time constant `MAX_ADMIN_PUBKEYS`).
## Architecture
```mermaid
flowchart TD
A[Config Table: admin_pubkey = hex1,hex2,hex3] --> B[reload_admin_pubkeys_cache]
B --> C[Static array: g_admin_pubkeys with count]
D[Any authorization check] --> E[is_admin_pubkey - pubkey]
E --> C
F[CLI: -a flag] --> G[first_time_startup_sequence]
G --> A
H[Admin command: add_admin / remove_admin] --> I[modify_admin_list]
I --> A
I --> B
```
## Implementation Steps
### Step 1: Add centralized admin check infrastructure in config.c / config.h
**New constants and types:**
```c
#define MAX_ADMIN_PUBKEYS 10
```
**New functions in config.h:**
```c
int is_admin_pubkey(const char* pubkey); // Returns 1 if pubkey is in admin list
int get_admin_pubkey_count(void); // Returns number of admins
const char* get_primary_admin_pubkey(void); // Returns first admin pubkey
int reload_admin_pubkeys_cache(void); // Parse config value into cache
int add_admin_pubkey(const char* pubkey); // Add pubkey to admin list (primary admin only)
int remove_admin_pubkey(const char* pubkey); // Remove pubkey from admin list (primary admin only)
```
**Implementation in config.c:**
- Static array `g_admin_pubkeys[MAX_ADMIN_PUBKEYS][65]` and `g_admin_pubkey_count`
- `reload_admin_pubkeys_cache()` reads `admin_pubkey` from config table, splits on commas, validates each is 64 hex chars, populates array
- `is_admin_pubkey()` iterates the cached array with `strcmp()`
- `get_primary_admin_pubkey()` returns `g_admin_pubkeys[0]` — used where only the primary admin matters (WoT sync, etc.)
- `add_admin_pubkey()` / `remove_admin_pubkey()` modify the comma-separated string in the config table and call `reload_admin_pubkeys_cache()`
### Step 2: Replace all 8 authorization strcmp gates
Each of these currently does:
```c
const char* admin_pubkey = get_config_value("admin_pubkey");
if (strcmp(sender_pubkey, admin_pubkey) == 0) { /* authorized */ }
```
Replace with:
```c
if (is_admin_pubkey(sender_pubkey)) { /* authorized */ }
```
**Files and locations:**
| File | Function | Line | Change |
|------|----------|------|--------|
| src/config.c | `process_admin_config_event()` | ~1441 | Replace strcmp with `is_admin_pubkey()` |
| src/config.c | `handle_kind_23456_unified()` | ~2727 | Replace strcmp with `is_admin_pubkey()` |
| src/main.c | `verify_admin_event()` | ~1905 | Replace strcmp with `is_admin_pubkey()` |
| src/main.c | Event storage / WoT trigger | ~1174 | Replace strcmp with `is_admin_pubkey()` |
| src/websockets.c | NIP-42 auth bypass | ~2324 | Replace strcmp with `is_admin_pubkey()` |
| src/websockets.c | NIP-17 DM admin check | ~3560 | Replace strcmp with `is_admin_pubkey()` |
| src/dm_admin.c | `process_admin_dm()` | ~503 | Replace strcmp with `is_admin_pubkey()` |
| src/request_validator.c | Event validation bypass | ~307 | Replace strcmp with `is_admin_pubkey()` |
Each replacement also eliminates the `get_config_value("admin_pubkey")` call and its associated `free()`, simplifying the code.
### Step 3: Update startup / initialization pipeline
**config.c `first_time_startup_sequence()`:**
- No change needed — it already stores a single pubkey. The comma-separated format with one entry is identical to the current format.
**config.c `populate_all_config_values_atomic()`:**
- No change needed — inserts the initial admin_pubkey as a single value.
**config.c `add_pubkeys_to_config_table()`:**
- No change needed — stores single pubkey on migration.
**config.c `apply_cli_overrides_atomic()`:**
- No change needed — CLI override replaces the entire admin_pubkey value.
**main.c startup validation:**
- Update the pre-warm validation at ~line 2438 to call `reload_admin_pubkeys_cache()` and validate that at least one admin pubkey exists, rather than checking for exactly 64 chars.
**Call `reload_admin_pubkeys_cache()`** after:
- `populate_all_config_values_atomic()` completes
- `reload_config_from_table()` completes
- Any admin add/remove operation
### Step 4: Add admin management commands to kind 23456 API
Add two new admin commands that can be sent as encrypted kind 23456 events:
```json
["add_admin", "pubkey_hex"]
["remove_admin", "pubkey_hex"]
["list_admins"]
```
**Authorization rule:** Only the primary admin (first in list) can add/remove other admins. This is enforced in the command handler, not in the general `is_admin_pubkey()` check.
**Implementation location:** `handle_system_command_unified()` in config.c, alongside existing commands like `system_status`, `graceful_shutdown`, etc.
### Step 5: Update WoT sync to handle multiple admins
In `wot_sync_from_admin_kind3()`:
- Currently queries kind 3 events from the single admin pubkey
- Change to use `get_primary_admin_pubkey()` — WoT trust anchor should remain the primary admin
- This is a deliberate design choice: secondary admins can manage the relay but the WoT trust graph is rooted in the primary admin's social graph
### Step 6: Update CLI to accept multiple admin pubkeys
**Option A - Comma-separated single flag (recommended):**
```bash
./c_relay_pg -a npub1...,npub2...,npub3...
```
**Changes:**
- `cli_options_t.admin_pubkey_override` becomes `char admin_pubkeys_override[MAX_ADMIN_PUBKEYS * 65]` (larger buffer for comma-separated list)
- CLI parsing in main.c splits on commas, decodes each npub/hex, reassembles as comma-separated hex
- `make_and_restart_relay.sh` `-a` flag documentation updated
### Step 7: Update config validation in api.c
In the config validation table at line ~912:
```c
{"admin_pubkey", "string", 0, 65},
```
Change max length to accommodate multiple pubkeys:
```c
{"admin_pubkey", "string", 0, MAX_ADMIN_PUBKEYS * 65},
```
### Step 8: Update get_admin_pubkey_cached
`get_admin_pubkey_cached()` currently returns the raw config value. Update to return only the primary admin pubkey for backward compatibility with any callers that expect a single 64-char string.
### Step 9: Update test scripts
Update test scripts that hardcode admin keys to work with the multi-admin system:
- `tests/sql_test.sh`
- `tests/white_black_test.sh`
- `tests/17_nip_test.sh`
- `tests/run_all_tests.sh`
- `tests/.test_keys.txt`
Add new test cases for:
- Adding a second admin via kind 23456 command
- Verifying second admin can send admin commands
- Verifying non-admin cannot send admin commands
- Removing a secondary admin
- Verifying primary admin cannot be removed
- Verifying secondary admin cannot add/remove admins
### Step 10: Update documentation
- `AGENTS.md` — update admin API event structure section
- `docs/configuration_guide.md` — document multi-admin setup
- `docs/user_guide.md` — document admin management commands
- `API.md` — document new add_admin/remove_admin/list_admins commands
- `README.md` — mention multi-admin capability
## Migration / Backward Compatibility
- Existing databases with a single `admin_pubkey` value work without any migration — a string with no commas is treated as a single-admin list
- The `is_admin_pubkey()` function handles both formats transparently
- No schema changes needed — the config table already stores strings of arbitrary length
## Security Considerations
1. **Primary admin privilege** — only the first pubkey in the list can add/remove admins, preventing secondary admins from locking out the primary
2. **No self-removal** — the primary admin cannot remove themselves (prevents accidental lockout)
3. **Max admin limit** — capped at 10 to prevent abuse and keep the linear scan fast
4. **Validation** — each pubkey in the list is validated as 64 hex characters before storage
+410
View File
@@ -0,0 +1,410 @@
# C-Relay-PG: PostgreSQL Implementation Plan
## Current State Assessment
### What We Have
- **PostgreSQL 18.3** installed and running on localhost:5432
- **libpq development headers** available via pkg-config (`/usr/include/postgresql`, `/usr/lib/x86_64-linux-gnu`)
- **`db_ops.h`** abstraction layer with 40+ functions already defined
- **`db_ops.c`** — single 1,398-line file implementing all functions against SQLite
- **`ip_ban.c`** — already uses `db_ops` API (not raw SQLite) for persistence
- **`config.c`** — uses `db_ops` API for all database access
### SQLite Coupling Points (Must Fix)
| Location | Issue |
|----------|-------|
| `src/main.c:11` | `#include <sqlite3.h>` |
| `src/main.c:51` | `sqlite3* g_db = NULL;` — global handle owned by main.c |
| `src/main.c:897` | `if (g_db)` — direct handle check |
| `src/main.c:2228` | `if (!g_db)` — direct handle check |
| `src/main.c:2358` | `g_db = NULL;` — strict mode nullification |
| `src/main.c:743` | Log message references `sqlite3_open()` |
| `src/main.c:854-885` | SQLite PRAGMAs (WAL, mmap_size, cache_size) |
| `src/main.c:795-824` | SQLite-specific DDL in migration code |
| `src/main.c:900` | `PRAGMA wal_checkpoint(TRUNCATE)` |
| `src/config.c:4727` | `sqlite_master` query |
| `src/config.c:4388` | `INSERT OR REPLACE` SQL |
| `src/db_ops.c:13` | `extern sqlite3* g_db;` |
| `src/db_ops.c:871` | `strftime('%s', 'now')` — SQLite-specific |
| `src/db_ops.c:889,909,956` | `INSERT OR REPLACE` — SQLite-specific |
| `src/db_ops.c:1341` | `sqlite_master` — SQLite-specific |
| `src/db_ops.c:1387-1395` | `sqlite3_wal_checkpoint_v2()` — SQLite-only |
| `src/ip_ban.c:122,205` | `strftime('%s', 'now')` and `INSERT OR REPLACE` in SQL strings |
| `src/sql_schema.h` | Entire embedded schema is SQLite DDL |
| `src/subscriptions.c:1187-1197` | Commented-out raw SQLite code |
### What Already Works Through `db_ops`
- `ip_ban.c` — uses `db_exec_sql()`, `db_prepare()`, `db_step_stmt()`, etc.
- `config.c` — uses `db_get_config_value_dup()`, `db_set_config_value_full()`, etc.
- `subscriptions.c` — uses `db_log_subscription_*()` functions
- `api.c` — uses `db_execute_readonly_query_json()`, stat helpers
- `nip009.c` — uses `db_delete_event_by_id()`, `db_get_event_pubkey()`
---
## Implementation Phases
### Phase 0: PostgreSQL Setup & Smoke Test
Create the PostgreSQL role, database, and verify connectivity from C.
```
Tasks:
0.1 Create PostgreSQL role "crelay" with password
0.2 Create database "crelay" owned by role "crelay"
0.3 Write a minimal C test program that connects via libpq and runs SELECT 1
0.4 Verify the test compiles and links with: gcc test_pg.c -lpq -o test_pg
```
### Phase 1: Internalize g_db — Move Database Ownership Into db_ops
**Goal**: Eliminate `extern sqlite3* g_db` from main.c. The `db_ops` module must own its connection handle internally. This is the critical prerequisite — without it, swapping backends is impossible.
```mermaid
flowchart LR
subgraph Before
MAIN[main.c owns sqlite3* g_db] --> DBOPS[db_ops.c uses extern g_db]
end
subgraph After
MAIN2[main.c calls db_init/db_close] --> DBOPS2[db_ops.c owns static g_db internally]
end
```
```
Tasks:
1.1 Move `sqlite3* g_db` from main.c into db_ops.c as a static variable
1.2 Move `char g_database_path[512]` from config.c into db_ops.c as static
1.3 Remove `#include <sqlite3.h>` from main.c
1.4 Replace all `g_db` references in main.c with db_ops API calls:
- `if (g_db)``if (db_is_available())`
- `g_db = NULL``db_close()` (or new `db_detach_main()`)
1.5 Replace SQLite PRAGMA calls in main.c init_database() with new db_ops functions:
- `db_set_journal_mode_wal()`
- `db_set_mmap_size(long size)`
- `db_set_cache_size(int kb)`
1.6 Move schema migration logic from main.c into db_ops (new `db_apply_schema()`)
1.7 Replace `sqlite_master` query in config.c:4727 with `db_table_exists()`
1.8 Replace `INSERT OR REPLACE` in config.c:4388 with `db_prepare()`/`db_step_stmt()`
1.9 Build and run existing test suite to confirm no regressions
```
### Phase 2: Rename db_ops.c → db_ops_sqlite.c, Create Dispatcher
**Goal**: Split the single `db_ops.c` into a backend-specific file and a thin dispatcher, preparing the architecture for a second backend.
```
New file layout:
src/db_ops.h — unchanged public API
src/db_ops_sqlite.c — renamed from db_ops.c (all SQLite implementation)
src/db_ops_postgres.c — new file (stub initially)
src/db_ops.c — thin dispatcher that forwards to active backend
```
```
Tasks:
2.1 Rename src/db_ops.c → src/db_ops_sqlite.c
2.2 Prefix all function names in db_ops_sqlite.c with `sqlite_` (e.g., `sqlite_db_init()`)
2.3 Create src/db_ops.c dispatcher with compile-time backend selection:
- #ifdef DB_BACKEND_POSTGRES → call postgres_* functions
- #else → call sqlite_* functions (default)
2.4 Create src/db_ops_postgres.c with stub implementations that return DB_ERROR
2.5 Update Makefile with DB_BACKEND variable:
- `DB_BACKEND ?= sqlite`
- Conditional source file inclusion and -lpq linking
2.6 Build with DB_BACKEND=sqlite — verify identical behavior
2.7 Build with DB_BACKEND=postgres — verify it compiles (stubs return errors)
```
### Phase 3: PostgreSQL Schema
**Goal**: Create the PostgreSQL schema file and apply it to the database.
```
Tasks:
3.1 Create src/pg_schema.sql with PostgreSQL DDL from the plan:
- events table with JSONB tags column
- GIN index on tags
- config, auth_rules, relay_seckey, subscriptions, ip_bans tables
- Materialized views for dashboard
3.2 Create src/pg_schema.h — embedded schema as C string (like sql_schema.h)
3.3 Add `postgres_db_apply_schema()` function in db_ops_postgres.c
3.4 Apply schema to the crelay database and verify with psql
```
### Phase 4: Implement db_ops_postgres.c — Core Functions
**Goal**: Implement the PostgreSQL backend for all `db_ops.h` functions using libpq.
This is the largest phase. Functions are grouped by dependency order.
#### 4A: Connection Management
```
Tasks:
4A.1 Implement postgres_db_init() — PQconnectdb() with connection string
4A.2 Implement postgres_db_close() — PQfinish()
4A.3 Implement postgres_db_is_available() — PQstatus() check
4A.4 Implement postgres_db_last_error() — PQerrorMessage()
4A.5 Implement postgres_db_get_database_path() — return connection string
4A.6 Implement thread-local connection pool:
- postgres_db_set_thread_connection()
- postgres_db_clear_thread_connection()
- postgres_db_open_worker_connection() — new PQconnectdb per worker
- postgres_db_close_worker_connection() — PQfinish per worker
```
#### 4B: Statement API (db_stmt_t wrapper for PGresult)
```
Tasks:
4B.1 Define postgres db_stmt_t struct wrapping PGresult + cursor state
4B.2 Implement postgres_db_prepare() — PQprepare() with auto-generated name
4B.3 Implement postgres_db_bind_text_param/int_param/int64_param
Note: libpq uses $1,$2 params, not ?. Need parameter array approach.
4B.4 Implement postgres_db_step_stmt() — PQexecPrepared() + row iteration
4B.5 Implement postgres_db_column_text_value/int_value/int64_value/double_value
4B.6 Implement postgres_db_reset_stmt() and postgres_db_finalize_stmt()
4B.7 Implement postgres_db_exec_sql() — PQexec() for DDL/DML
```
**Key design decision**: libpq doesn't have a step-based cursor like SQLite. Two approaches:
1. **PQexecPrepared** returns all rows at once — iterate with row index
2. **DECLARE CURSOR / FETCH** for streaming large result sets
For this relay's workload (most queries return <1000 rows), approach 1 is simpler and sufficient.
#### 4C: Config & Key Storage
```
Tasks:
4C.1 Implement postgres_db_get_all_config_rows()
4C.2 Implement postgres_db_get_config_value_dup()
4C.3 Implement postgres_db_set_config_value_full()
— Use INSERT ... ON CONFLICT (key) DO UPDATE instead of INSERT OR REPLACE
4C.4 Implement postgres_db_update_config_value_only()
— Use EXTRACT(EPOCH FROM NOW()) instead of strftime('%s','now')
4C.5 Implement postgres_db_upsert_config_value()
4C.6 Implement postgres_db_store_relay_private_key_hex()
4C.7 Implement postgres_db_get_relay_private_key_hex_dup()
4C.8 Implement postgres_db_store_config_event()
4C.9 Implement postgres_db_get_config_row_count()
```
#### 4D: Event Storage & Retrieval
```
Tasks:
4D.1 Implement postgres_db_insert_event_with_json()
— INSERT ... ON CONFLICT (id) DO NOTHING
— Map extended_errcode to DB_CONSTRAINT for duplicates
4D.2 Implement postgres_db_event_id_exists()
4D.3 Implement postgres_db_retrieve_event_by_id()
4D.4 Implement postgres_db_get_event_pubkey()
4D.5 Implement postgres_db_get_event_time_bounds()
4D.6 Implement postgres_db_get_latest_event_pubkey_for_kind_dup()
```
#### 4E: Event Deletion (NIP-09)
```
Tasks:
4E.1 Implement postgres_db_delete_event_by_id()
4E.2 Implement postgres_db_delete_events_by_address()
```
#### 4F: Tag Operations
```
Tasks:
4F.1 Implement postgres_db_store_event_tags_cjson()
— Note: With JSONB+GIN, this may become a no-op for PostgreSQL
— Tags are already stored in the events.tags JSONB column
4F.2 Implement postgres_db_populate_event_tags_from_existing()
— No-op for PostgreSQL (no separate event_tags table needed)
```
#### 4G: Auth Rules
```
Tasks:
4G.1 Implement postgres_db_is_pubkey_blacklisted()
4G.2 Implement postgres_db_is_hash_blacklisted()
4G.3 Implement postgres_db_is_pubkey_whitelisted()
4G.4 Implement postgres_db_count_active_whitelist_rules()
4G.5 Implement postgres_db_add_auth_rule()
4G.6 Implement postgres_db_remove_auth_rule()
4G.7 Implement postgres_db_delete_wot_whitelist_rules()
4G.8 Implement postgres_db_count_wot_whitelist_rules()
```
#### 4H: Subscription Logging
```
Tasks:
4H.1 Implement postgres_db_log_subscription_created()
— Use INSERT ... ON CONFLICT DO UPDATE
4H.2 Implement postgres_db_log_subscription_closed()
— Use EXTRACT(EPOCH FROM NOW()) for timestamps
4H.3 Implement postgres_db_log_subscription_disconnected()
4H.4 Implement postgres_db_update_subscription_events_sent()
4H.5 Implement postgres_db_cleanup_orphaned_subscriptions()
```
#### 4I: Monitoring & Stats
```
Tasks:
4I.1 Implement postgres_db_get_total_event_count_ll()
4I.2 Implement postgres_db_get_event_count_since()
4I.3 Implement postgres_db_get_event_kind_distribution_rows()
4I.4 Implement postgres_db_get_top_pubkeys_rows()
4I.5 Implement postgres_db_get_subscription_details_rows()
4I.6 Implement postgres_db_count_with_sql()
4I.7 Implement postgres_db_execute_readonly_query_json()
— Map PG column types instead of SQLITE_INTEGER/SQLITE_TEXT/etc.
```
#### 4J: Schema & DDL Helpers
```
Tasks:
4J.1 Implement postgres_db_table_exists()
— Use information_schema.tables instead of sqlite_master
4J.2 Implement postgres_db_get_schema_version_dup()
4J.3 Implement postgres_db_wal_checkpoint_passive() — no-op for PostgreSQL
4J.4 Implement postgres_db_wal_checkpoint_truncate() — no-op for PostgreSQL
```
### Phase 5: Fix SQLite-Specific SQL in Callers
**Goal**: Ensure SQL strings passed through `db_prepare()` / `db_exec_sql()` from outside `db_ops` are portable.
```
Tasks:
5.1 Audit ip_ban.c SQL strings:
— Replace `strftime('%s', 'now')` with `EXTRACT(EPOCH FROM NOW())::BIGINT`
or pass timestamp as bind parameter from C (time(NULL))
— Replace `INSERT OR REPLACE` with `INSERT ... ON CONFLICT DO UPDATE`
5.2 Audit config.c SQL strings:
— Replace `sqlite_master` query with db_table_exists()
— Replace `INSERT OR REPLACE` with ON CONFLICT syntax
5.3 Audit main.c init_database():
— Move all PRAGMA calls behind db_ops backend-specific init
— Move schema migration behind db_apply_schema()
5.4 Strategy decision: use C-side timestamps (time(NULL)) as bind params
instead of database-side timestamp functions for portability
```
### Phase 6: CLI Connection String Support
**Goal**: Accept PostgreSQL connection parameters from the command line.
```
Tasks:
6.1 Add --db-host, --db-port, --db-name, --db-user, --db-password CLI options
6.2 Add --db-connstring option for full libpq connection string
6.3 Construct connection string from individual params if --db-connstring not given
6.4 Pass connection string to db_init() (works for both backends)
6.5 Update make_and_restart_relay.sh to support PostgreSQL connection params
6.6 Update AGENTS.md with new CLI options
```
### Phase 7: Integration Testing
**Goal**: Verify the PostgreSQL backend passes all existing tests.
```
Tasks:
7.1 Build with DB_BACKEND=postgres
7.2 Start relay with PostgreSQL connection string
7.3 Run tests/run_nip_tests.sh — verify NIP compliance
7.4 Run tests/run_all_tests.sh — verify full test suite
7.5 Run tests/performance_benchmarks.sh — compare with SQLite baseline
7.6 Test first-time startup flow (key generation, schema creation)
7.7 Test config event storage and retrieval
7.8 Test auth rules (whitelist/blacklist)
7.9 Test IP ban persistence across restarts
7.10 Test thread pool worker connections with PostgreSQL
```
### Phase 8: Query Builder — JSONB Tag Queries
**Goal**: Replace the `event_tags` JOIN queries with PostgreSQL JSONB containment queries for tag filtering in REQ handling.
```
Tasks:
8.1 Identify where REQ filter → SQL translation happens (websockets.c / subscriptions.c)
8.2 Create backend-aware query builder or use #ifdef for tag filter SQL:
— SQLite: `id IN (SELECT event_id FROM event_tags WHERE tag_name=? AND tag_value IN (?))`
— PostgreSQL: `tags @> '[["p", "value"]]'::jsonb`
8.3 Test tag-filtered REQ queries against PostgreSQL
8.4 Verify GIN index is being used (EXPLAIN ANALYZE)
```
---
## SQL Dialect Mapping Reference
| SQLite | PostgreSQL |
|--------|-----------|
| `INSERT OR REPLACE INTO t ...` | `INSERT INTO t ... ON CONFLICT (pk) DO UPDATE SET ...` |
| `strftime('%s', 'now')` | `EXTRACT(EPOCH FROM NOW())::BIGINT` |
| `SELECT 1 FROM sqlite_master WHERE type='table' AND name=?` | `SELECT 1 FROM information_schema.tables WHERE table_name=?` |
| `PRAGMA journal_mode=WAL` | N/A (PostgreSQL handles WAL internally) |
| `PRAGMA mmap_size=N` | N/A |
| `PRAGMA cache_size=-N` | `SET shared_buffers` (server-level, not per-connection) |
| `sqlite3_wal_checkpoint_v2()` | N/A (no-op) |
| `sqlite3_changes()` | `PQcmdTuples(result)` |
| `sqlite3_extended_errcode()` | `PQresultErrorField(result, PG_DIAG_SQLSTATE)` |
| `?` bind parameter | `$1`, `$2`, ... positional parameters |
| `JSON` column type | `JSONB` column type |
| `json_each()` for tag queries | `@>` containment operator with GIN index |
| `INTEGER PRIMARY KEY` auto-increment | `SERIAL` or `BIGSERIAL` |
---
## Architecture After Implementation
```mermaid
flowchart TD
subgraph Application Layer
MAIN[main.c] --> DBOPS_H[db_ops.h - public API]
CONFIG[config.c] --> DBOPS_H
IPBAN[ip_ban.c] --> DBOPS_H
SUBS[subscriptions.c] --> DBOPS_H
API[api.c] --> DBOPS_H
NIP09[nip009.c] --> DBOPS_H
WS[websockets.c] --> DBOPS_H
end
subgraph Database Abstraction
DBOPS_H --> DISPATCH[db_ops.c - dispatcher]
DISPATCH -->|DB_BACKEND_SQLITE| SQLITE[db_ops_sqlite.c]
DISPATCH -->|DB_BACKEND_POSTGRES| PG[db_ops_postgres.c]
end
subgraph Backends
SQLITE --> SQLITE3[libsqlite3]
PG --> LIBPQ[libpq]
LIBPQ --> PGSERVER[PostgreSQL Server]
end
```
---
## Risk Mitigation
| Risk | Mitigation |
|------|-----------|
| Breaking SQLite backend during refactor | Phase 1-2 are pure refactors — test suite must pass after each |
| libpq statement API mismatch | Use PQexecParams with row-index iteration, not cursor-based |
| SQL dialect differences in caller code | Phase 5 audits all SQL outside db_ops; use C timestamps for portability |
| Connection failures | Implement reconnection with exponential backoff in postgres_db_init |
| Thread safety | Each worker thread gets its own PGconn via db_open_worker_connection |
| Performance regression | Phase 7 benchmarks compare SQLite vs PostgreSQL on same workload |
---
## Build Commands
```bash
# SQLite backend (default, unchanged behavior)
make
# PostgreSQL backend
make DB_BACKEND=postgres
# Run with PostgreSQL
./build/c_relay_pg_x86 --db-connstring "host=localhost dbname=crelay user=crelay password=secret"
```
+257
View File
@@ -0,0 +1,257 @@
# PostgreSQL Nostr Relay Architecture Analysis
This document summarizes common approaches and best practices for building Nostr relays backed by PostgreSQL, based on industry-standard designs and existing open-source relay implementations.
## Identified Projects
* **Nostrss**: High-performance Rust-based relay.
* **Nostrgres**: Focused on PostgreSQL integration with complex event filtering.
* **Relay-rs (Postgres branch)**: General purpose high-throughput relay.
* **Nostream** ([github.com/Cameri/nostream](https://github.com/Cameri/nostream)): Production-ready TypeScript relay backed by PostgreSQL and Redis.
* **Nostr-Relay-NestJS** ([github.com/CodyTseng/nostr-relay-nestjs](https://github.com/CodyTseng/nostr-relay-nestjs)): High-performance relay built with NestJS and PostgreSQL, utilizing Kysely for type-safe database interactions.
## Detailed Analysis
### 1. Common Schema Patterns
* **Event Storage (JSONB)**: Most PostgreSQL-backed relays store the raw event as a JSONB object in a central `events` table. This provides flexibility for the evolving Nostr spec while allowing efficient extraction of fields.
* **Tag Denormalization**: While the raw event is in JSONB, performant relays extract tags (like `p`, `e`, `t`, `d`) into a separate `tags` table with foreign key relationships to the `events` table. This avoids slow JSONB traversal during complex filter queries.
* **Author/Publisher Tracking**: A dedicated `authors` or `pubkeys` table is typically used to maintain indexing on the `pubkey` field, enabling quick lookups of user activity.
### 2. Performance Optimization
* **GIN Indexes on JSONB**: Crucial for filtering on specific event tags or custom properties stored within the event object. GIN indexes with `jsonb_path_ops` are generally preferred for equality checks.
* **Time-Series Partitioning**: Given the append-only nature of Nostr, partitioning the `events` table by time (e.g., daily or weekly chunks) is highly recommended. This significantly improves query performance for recent data and simplifies data expiration/deletion.
* **Clustering**: Clustering the `events` table by the timestamp index can reduce disk I/O, as it keeps temporally related events physically adjacent on the disk.
### 3. Schema Management Approaches
* **Migrations**: Most mature relays utilize migration tools (like `Flyway`, `Diesel migrations`, or `Golang-migrate`) to version control database schema changes. This is critical for production stability.
* **Auto-generation**: Some lightweight prototypes auto-generate schemas at startup. This is generally discouraged for production due to the risk of destructive changes, data loss, or blocking DDL operations on large tables.
## Recommendations for Building a New Relay
1. **Prioritize Denormalization**: Do not rely solely on JSONB queries for high-traffic filters. Extract indexed tags into dedicated columns or tables.
2. **Use Partitioning from Day One**: Implementing native PostgreSQL partitioning (e.g., using `pg_partman`) is much easier before the dataset grows to millions of rows.
3. **Connection Pooling**: PostgreSQL requires aggressive connection management. Use a high-performance pooler like `PgBouncer` to handle the large number of concurrent, short-lived connections common in WebSocket-based relay traffic.
4. **Asynchronous Writes**: Decouple the WebSocket ingestion thread from the database writer thread to ensure that slow database writes do not impact the relay's responsiveness.
## Nostream ([github.com/Cameri/nostream](https://github.com/Cameri/nostream))
A production-ready Nostr relay written in TypeScript (Node.js), backed by PostgreSQL 14+ and Redis. Uses Knex.js for versioned migrations.
### PostgreSQL Schema
The schema is fully normalized — no JSONB blob for the whole event. Tags are stored separately and populated by a trigger.
**`events` table** — one row per event, tags kept as a JSONB column for the trigger to process:
```sql
CREATE TABLE events (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
event_id BYTEA UNIQUE NOT NULL, -- 32-byte hash
event_pubkey BYTEA NOT NULL,
event_kind INTEGER UNSIGNED NOT NULL,
event_created_at INTEGER UNSIGNED NOT NULL,
event_content TEXT NOT NULL,
event_tags JSONB, -- raw tags array, source of truth for trigger
event_signature BYTEA NOT NULL,
remote_address TEXT,
expires_at INTEGER,
deleted_at TIMESTAMP,
event_deduplication JSONB, -- used for replaceable event conflict key
first_seen TIMESTAMP DEFAULT now()
);
-- Indexes
CREATE INDEX ON events (event_pubkey);
CREATE INDEX ON events (event_kind);
CREATE INDEX ON events (event_created_at);
CREATE INDEX ON events (expires_at);
CREATE INDEX event_tags_idx ON events USING GIN (event_tags); -- GIN on raw JSONB
-- Unique index enforcing NIP-16 replaceable event semantics
CREATE UNIQUE INDEX replaceable_events_idx ON events (event_pubkey, event_kind)
WHERE event_kind = 0 OR event_kind = 3
OR (event_kind >= 10000 AND event_kind < 20000);
```
**`event_tags` table** — denormalized single-letter tags, populated by a PostgreSQL trigger on `events`:
```sql
CREATE TABLE event_tags (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
event_id BYTEA NOT NULL,
tag_name TEXT NOT NULL, -- single-letter only (e.g. 'e', 'p', 't')
tag_value TEXT NOT NULL
);
CREATE INDEX ON event_tags (event_id);
CREATE INDEX ON event_tags (tag_name, tag_value);
-- Trigger: on INSERT/UPDATE/DELETE to events, repopulate event_tags
CREATE OR REPLACE FUNCTION process_event_tags() RETURNS TRIGGER AS $$
DECLARE
tag_element jsonb;
BEGIN
DELETE FROM event_tags WHERE event_id = OLD.event_id;
IF TG_OP = 'INSERT' OR TG_OP = 'UPDATE' THEN
FOR tag_element IN SELECT jsonb_array_elements(NEW.event_tags) LOOP
IF length(trim('"' FROM (tag_element->0)::text)) = 1
AND (tag_element->1)::text IS NOT NULL THEN
INSERT INTO event_tags (event_id, tag_name, tag_value)
VALUES (NEW.event_id,
trim('"' FROM (tag_element->0)::text),
trim('"' FROM (tag_element->1)::text));
END IF;
END LOOP;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER insert_event_tags
AFTER INSERT OR UPDATE OR DELETE ON events
FOR EACH ROW EXECUTE FUNCTION process_event_tags();
```
Tag queries (`#e`, `#p`, etc.) join `event_tags` rather than traversing the JSONB array.
**`users` and `invoices` tables** — for paid relay admission (Lightning payments):
```sql
CREATE TABLE users (
pubkey BYTEA PRIMARY KEY,
is_admitted BOOLEAN DEFAULT FALSE,
balance BIGINT DEFAULT 0, -- in msats
tos_accepted_at DATETIME
);
CREATE TABLE invoices (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
pubkey BYTEA NOT NULL,
bolt11 TEXT NOT NULL,
amount_requested BIGINT UNSIGNED NOT NULL,
amount_paid BIGINT UNSIGNED,
unit ENUM('msats','sats','btc'),
status ENUM('pending','completed','expired'),
description TEXT,
confirmed_at DATETIME,
expires_at DATETIME
);
```
### How Redis Is Used
Redis is **not** used for subscription fan-out. Subscription delivery is handled entirely in-process: each `WebSocketAdapter` holds a `Map<subscriptionId, filters[]>`, and a broadcast event walks all connected clients and filters locally.
Redis has two actual roles:
1. **Sliding-window rate limiting** — The `SlidingWindowRateLimiter` uses Redis sorted sets to track request timestamps per key (pubkey, IP, etc.) within a rolling time window:
- `ZREMRANGEBYSCORE key 0 (now - period)` — evict old entries
- `ZADD key timestamp member` — record this hit
- `ZRANGE key 0 -1` — count hits in window
- `EXPIRE key period` — auto-cleanup
- Keys follow the pattern `<pubkey>:events:<period>` or `<ip>:message:<period>`
2. **General-purpose cache** — A `RedisAdapter` wraps the client with `get`/`set`/`hasKey` helpers, used for caching admission checks (e.g. `<pubkey>:is-admitted`). The event handler has a `TODO` comment noting that the `userRepository.findByPubkey()` call should be replaced with a cache lookup — meaning this is partially implemented.
---
## Schema Definitions
### Relay-rs (Postgres)
```sql
-- Events table
CREATE TABLE "event" (
id bytea NOT NULL,
pub_key bytea NOT NULL,
created_at timestamp with time zone NOT NULL,
kind integer NOT NULL,
"content" bytea NOT NULL,
hidden bit(1) NOT NULL DEFAULT 0::bit(1),
delegated_by bytea NULL,
first_seen timestamp with time zone NOT NULL DEFAULT now(),
expires_at timestamp(0) with time zone,
CONSTRAINT event_pkey PRIMARY KEY (id)
);
CREATE INDEX event_created_at_idx ON "event" (created_at,kind);
CREATE INDEX event_pub_key_idx ON "event" (pub_key);
CREATE INDEX event_delegated_by_idx ON "event" (delegated_by);
CREATE INDEX event_expires_at_idx ON "event" (expires_at);
-- Tags table
CREATE TABLE "tag" (
id int8 NOT NULL GENERATED BY DEFAULT AS IDENTITY,
event_id bytea NOT NULL,
"name" varchar NOT NULL,
value bytea NULL,
value_hex bytea NULL,
CONSTRAINT tag_fk FOREIGN KEY (event_id) REFERENCES "event"(id) ON DELETE CASCADE,
CONSTRAINT unique_constraint_name UNIQUE (event_id, "name", value, value_hex)
);
CREATE INDEX tag_event_id_idx ON tag USING btree (event_id, name);
CREATE INDEX tag_value_idx ON tag USING btree (value);
CREATE INDEX tag_value_hex_idx ON tag USING btree (value_hex);
-- Account table
CREATE TABLE "account" (
pubkey varchar NOT NULL,
is_admitted BOOLEAN NOT NULL DEFAULT FALSE,
balance BIGINT NOT NULL DEFAULT 0,
tos_accepted_at TIMESTAMP,
CONSTRAINT account_pkey PRIMARY KEY (pubkey)
);
-- Invoice table
CREATE TYPE status AS ENUM ('Paid', 'Unpaid', 'Expired');
CREATE TABLE "invoice" (
payment_hash varchar NOT NULL,
pubkey varchar NOT NULL,
invoice varchar NOT NULL,
amount BIGINT NOT NULL,
status status NOT NULL DEFAULT 'Unpaid',
description varchar,
created_at timestamp,
confirmed_at timestamp,
CONSTRAINT invoice_payment_hash PRIMARY KEY (payment_hash),
CONSTRAINT invoice_pubkey_fkey FOREIGN KEY (pubkey) REFERENCES account (pubkey) ON DELETE CASCADE
);
```
### Nostr-Relay-NestJS (PostgreSQL via Kysely)
```sql
-- Events table
CREATE TABLE events (
id CHAR(64) PRIMARY KEY,
pubkey CHAR(64) NOT NULL,
created_at BIGINT NOT NULL,
kind INTEGER NOT NULL,
tags JSONB NOT NULL DEFAULT '[]',
generic_tags TEXT[] NOT NULL DEFAULT '{}',
content TEXT NOT NULL DEFAULT '',
sig CHAR(128) NOT NULL,
expired_at BIGINT,
d_tag_value TEXT,
delegator CHAR(64),
create_date TIMESTAMP NOT NULL DEFAULT now(),
update_date TIMESTAMP NOT NULL DEFAULT now(),
delete_date TIMESTAMP
);
-- Indices
CREATE EXTENSION IF NOT EXISTS btree_gin;
CREATE INDEX generic_tags_kind_idx ON events USING gin (generic_tags, kind);
CREATE INDEX pubkey_kind_created_at_idx ON events (pubkey, kind, created_at);
CREATE INDEX delegator_kind_created_at_idx ON events (delegator, kind, created_at);
CREATE INDEX created_at_kind_idx ON events (created_at, kind);
-- Generic tags table (for optimized tag queries)
CREATE TABLE generic_tags (
id SERIAL PRIMARY KEY,
tag TEXT NOT NULL,
author CHAR(64) NOT NULL,
kind INTEGER NOT NULL,
event_id CHAR(64) NOT NULL REFERENCES events(id),
created_at BIGINT NOT NULL
);
-- Indices for tag filtering
CREATE UNIQUE INDEX g_tag_event_id_idx ON generic_tags (tag, event_id);
CREATE INDEX g_tag_kind_created_at_idx ON generic_tags (tag, kind, created_at);
CREATE INDEX g_tag_created_at_idx ON generic_tags (tag, created_at);
```
Nostr-Relay-NestJS features a hybrid storage model where tags are stored both in a JSONB field (for general lookup) and a normalized `generic_tags` table (for optimized tag query performance). The schema uses Kysely for type-safe interaction.
+297
View File
@@ -0,0 +1,297 @@
# Relay Admin Page Migration Plan
## Goal
Migrate the relay admin functionality from the internally-served `api/index.html` + `api/index.js` (served by the C relay binary) to an externally-hosted `relay-admin.html` page in the `client-ndk` project. This page uses the NDK SharedWorker architecture (template.html pattern) and communicates with the relay via kind 23456/23457/24567 Nostr events.
## Current State
### Working Reference: [`c-relay-pg.html`](/home/user/lt/client-ndk/www/c-relay-pg.html)
- Successfully receives kind 24567 monitoring events
- Has `SUBSCRIPTION_ONLY_MODE = true` — admin UI actions are disabled
- Contains all admin section HTML (Statistics, Subscriptions, Configuration, Authorization, IP Bans, Relay Events, DM, Database Query)
- Has sidenav with admin page links
- Has `sendAdminCommand()`, `encryptForRelay()`, `decryptFromRelay()`, `fetchRelayInfo()` functions
- Has `handleMonitoringEvent()` and `handleAdminResponse()` event handlers
### Target File: [`relay-admin.html`](/home/user/lt/client-ndk/www/relay-admin.html)
- Currently an exact copy of `template.html`
- Has full template infrastructure: hamburger menu, sidenav, footer, auth system, relay-ui, blossom-ui, ai-ui
- Has the newer `injectHeaderLoginButton` import (slightly newer than c-relay-pg.html)
- Empty `divBody` — ready for admin content
### Legacy Internal Pages: [`api/index.html`](api/index.html) + [`api/index.js`](api/index.js)
- 6,779 lines of JavaScript — the full admin UI
- Uses `nostr-tools SimplePool` directly (not NDK)
- Uses `NOSTR_LOGIN_LITE` for auth (not NDK SharedWorker)
- Has rich UI: config tables with inline editing, IP ban management with filters, SQL query console with history, WoT management, relay event forms
- All admin commands go through `sendAdminCommand()` → kind 23456 with NIP-44 encryption
- Responses come back as kind 23457 events, decrypted and routed to handlers
## Architecture Comparison
```mermaid
graph TD
subgraph Legacy - api/index.html
A1[NOSTR_LOGIN_LITE auth] --> A2[SimplePool direct WS]
A2 --> A3[Kind 23456 admin commands]
A2 --> A4[Kind 24567 monitoring sub]
A3 --> A5[Kind 23457 responses]
end
subgraph New - relay-admin.html
B1[NDK SharedWorker auth] --> B2[NDK subscribe/publishEvent]
B2 --> B3[Kind 23456 admin commands]
B2 --> B4[Kind 24567 monitoring sub]
B3 --> B5[Kind 23457 responses]
end
```
### Key Difference: NDK vs SimplePool
| Aspect | Legacy api/index.js | New relay-admin.html |
|--------|-------------------|---------------------|
| Auth | `NOSTR_LOGIN_LITE` direct | `initNDKPage()` via SharedWorker |
| Subscriptions | `SimplePool.subscribeMany()` | `subscribe()` from init-ndk.mjs |
| Publishing | `SimplePool.publish()` + manual signing | `publishEvent()` — auto-signed by worker |
| Encryption | `window.nostr.nip44.encrypt/decrypt` | Same — `window.nostr.nip44.encrypt/decrypt` |
| Event listening | SimplePool callbacks | `window.addEventListener('ndkEvent', ...)` |
| Relay URL | Auto-derived from page URL | Configurable constant `ADMIN_RELAY_WS_URL` |
## Migration Strategy
The approach is to merge the admin functionality from `c-relay-pg.html` into `relay-admin.html`, removing the `SUBSCRIPTION_ONLY_MODE` restriction and enabling full admin UI. We use the NDK `publishEvent()` and `subscribe()` APIs instead of SimplePool.
### What stays from template (relay-admin.html already has)
- Full hamburger menu + sidenav infrastructure
- Footer with relay status animations
- Auth system with required/optional/none modes
- Relay-ui, blossom-ui, ai-ui sidenav sections
- Theme toggle, logout, version display
### What gets added from c-relay-pg.html
- Admin nav items in sidenav
- Admin section HTML panels in divBody
- Admin-specific CSS styles
- Relay pubkey fetch via NIP-11
- NIP-44 encrypt/decrypt helpers
- `sendAdminCommand()` for kind 23456
- Kind 24567 monitoring subscription
- Kind 23457 response handling with per-section routing
- All admin UI button handlers
- Boot diagnostics panel
- System info panel
## Detailed Implementation Steps
### Step 1: Add Admin Nav Links to Sidenav
The admin links go directly inside `divSideNavBody` as simple styled items — **not** inside a collapsible `.sidenavSection` container. They should be always-visible, non-collapsible links that match the font/styling of the sidenav sections but remain in the open body area. Place them before `divFiles`:
```html
<div id="divSideNavBody">
<!-- Admin nav links - always visible, not collapsible -->
<div id="divAdminNav">
<div class="adminNavItem active" data-section="statistics">Statistics</div>
<div class="adminNavItem" data-section="subscriptions">Subscriptions</div>
<div class="adminNavItem" data-section="configuration">Configuration</div>
<div class="adminNavItem" data-section="authorization">Authorization</div>
<div class="adminNavItem" data-section="ip-bans">IP Bans</div>
<div class="adminNavItem" data-section="relay-events">Relay Events</div>
<div class="adminNavItem" data-section="dm">DM</div>
<div class="adminNavItem" data-section="database">Database Query</div>
</div>
<div id="divFiles"></div>
</div>
```
The `.adminNavItem` styling uses the same font-family and sizing as sidenav section content (`font-size: 70%`, `font-family: var(--font-family)`) with a simple border-bottom separator, hover accent color, and bold+background for the active state. No section title header, no collapse toggle icon.
### Step 2: Add Admin Section HTML Panels
Port the section HTML from `c-relay-pg.html` into `divBody`. Each section follows the pattern:
```html
<div id="section-{name}" class="adminSection">
<div class="adminSectionHeader">{Title}</div>
<!-- Section-specific content -->
</div>
```
Sections to port:
1. **Statistics** — CPU metrics, time stats, top pubkeys, event kinds cards
2. **Subscriptions** — Subscription details log
3. **Configuration** — Config key/value inputs, fetch/set buttons, log
4. **Authorization** — Pubkey input, whitelist/blacklist/query buttons, log
5. **IP Bans** — IP input, duration, ban/query buttons, log
6. **Relay Events** — Kind 0 metadata form, live event feed
7. **DM** — Message textarea, send button, log
8. **Database Query** — SQL textarea, run button, log
9. **System** (always visible) — Relay WS/HTTP URLs, relay/user pubkeys
10. **Boot Diagnostics** (always visible) — Boot log
### Step 3: Add Admin CSS
Port the admin-specific styles from `c-relay-pg.html` `<style>` block. These use CSS variables from `client.css` so they integrate with the theme system:
- `#divBody` flex layout overrides
- `.adminNavItem` styles with hover/active states
- `.adminSection` show/hide with `.active` class
- `.adminGrid`, `.adminCard`, `.adminTable` layout
- `.adminInput`, `.adminTextArea`, `.adminBtn` form elements
- `.adminLog` scrollable log panels
### Step 4: Add Admin State Variables
```javascript
const ADMIN_RELAY_WS_URL = 'wss://relay.laantungir.net';
const ADMIN_RELAY_HTTP_URL = 'https://relay.laantungir.net';
let relayPubkey = '';
let adminInitialized = false;
let adminUiBound = false;
const adminSubscriptions = [];
const MAX_LOG_LINES = 200;
```
### Step 5: Add Core Admin Functions
Port from `c-relay-pg.html`:
1. **`appendLog(id, msg)`** — Prepend timestamped messages to log panels
2. **`setSystemInfo()`** — Populate system info table
3. **`fetchRelayInfo()`** — NIP-11 fetch to get relay pubkey
4. **`encryptForRelay(content)`** — NIP-44 encrypt using `window.nostr.nip44.encrypt`
5. **`decryptFromRelay(content)`** — NIP-44 decrypt using `window.nostr.nip44.decrypt`
6. **`sendAdminCommand(commandArray, sink)`** — Encrypt + publish kind 23456 event via `publishEvent()`
### Step 6: Add Subscription Setup
```javascript
async function setupAdminSubscriptions() {
// Subscribe to kind 24567 monitoring events
const monitoringSub = subscribe(
{ kinds: [24567], since: Math.floor(Date.now() / 1000) - 60 },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Subscribe to kind 23457 admin responses
const responseSub = subscribe(
{ kinds: [23457], '#p': [currentPubkey], since: Math.floor(Date.now() / 1000) - 60 },
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
);
// Listen for events via ndkEvent window event
window.addEventListener('ndkEvent', async (evt) => {
const event = evt.detail;
if (event.kind === 24567) handleMonitoringEvent(event);
else if (event.kind === 23457) await handleAdminResponse(event);
else handleLiveEvent(event);
});
}
```
### Step 7: Add Event Handlers
Port from `c-relay-pg.html`:
1. **`handleMonitoringEvent(event)`** — Route d-tag values to stat cards
2. **`handleAdminResponse(event)`** — Decrypt kind 23457 and route to logs
3. **`handleLiveEvent(event)`** — Sample non-admin events to live feed
4. **`activateSection(sectionKey)`** — Toggle active section visibility
5. **`bindAdminUi()`** — Wire up all button click handlers
### Step 8: Add Button Handlers
All handlers follow the same pattern: gather input → `sendAdminCommand(array, logId)`:
- **btnRefreshStats**`['system_command', 'system_status']`
- **btnConfigRefresh**`['config_query', 'all']`
- **btnConfigSet**`[key, value]`
- **btnWhitelist**`['whitelist', 'pubkey', pk]`
- **btnBlacklist**`['blacklist', 'pubkey', pk]`
- **btnAuthQuery**`['auth_query', 'all']`
- **btnBanIp**`['ip_ban', 'add', ip, duration]`
- **btnIpQuery**`['ip_ban', 'query', 'all']`
- **btnPublishKind0** → Direct `publishEvent()` with kind 0
- **btnSendDm**`['dm', message]`
- **btnRunSql**`['sql_query', query]`
### Step 9: Integration into main() Flow
```javascript
(async function main() {
// 1. Template init (hamburger, theme, auth)
initHamburgerMenu();
// ... existing template setup ...
// 2. Auth
await initializeAuthentication(authMode);
await initializeAuthenticatedPageFeatures();
// 3. Admin init (NEW)
await initializeAdminFeatures();
// 4. Template finalization
await updateVersionDisplay();
})();
```
Where `initializeAdminFeatures()`:
```javascript
async function initializeAdminFeatures() {
if (!isAuthenticated || adminInitialized) return;
bindAdminUi();
await fetchRelayInfo();
await setupAdminSubscriptions();
setSystemInfo();
adminInitialized = true;
}
```
## Response Routing Enhancement
The current `c-relay-pg.html` broadcasts all responses to all logs. For a better UX, we should parse the decrypted response and route to the appropriate log:
```javascript
async function handleAdminResponse(event) {
const plain = await decryptFromRelay(event.content);
const parsed = JSON.parse(plain);
// Route based on query_type
if (parsed.query_type === 'config_all' || parsed.query_type === 'config_update') {
appendLog('configLog', plain);
} else if (parsed.query_type === 'auth_query' || parsed.query_type === 'auth_rule') {
appendLog('authLog', plain);
} else if (parsed.query_type === 'sql_query') {
appendLog('sqlLog', plain);
} else if (parsed.query_type === 'ip_ban') {
appendLog('ipBanLog', plain);
} else {
// Fallback: broadcast to all
appendLog('configLog', `RESP <- ${plain}`);
}
}
```
## Files Modified
| File | Location | Change |
|------|----------|--------|
| `relay-admin.html` | `~/lt/client-ndk/www/` | Add admin HTML, CSS, and JavaScript |
## Files NOT Modified
- `template.html` — Stays as the clean template
- `c-relay-pg.html` — Stays as the debug/test reference
- `api/index.html` / `api/index.js` — Legacy, stays for now (still served by relay binary)
- No C code changes needed — the relay already handles kind 23456/23457/24567
## Future Enhancements (Out of Scope)
- Rich config table with inline editing (like api/index.html has)
- IP ban table with filter tabs
- SQL query history dropdown
- WoT management section
- Event rate chart visualization
- Admin verification handshake
+204
View File
@@ -0,0 +1,204 @@
# Remaining Implementation Plan — Zero SQLite on Main Thread + PG Fork Readiness
## Overview
This plan covers all remaining work to achieve two goals:
1. **Zero direct SQLite calls on lws-main** during the event loop
2. **Clean `db_ops` abstraction boundary** so the PG fork has no raw `sqlite3_*` calls outside `db_ops.c`
## Current State
### Completed (Fixes 1-5, 7 + Phases 1-3, 5)
- Global config cache with 5s TTL
- Dedup check moved to event worker
- Async COUNT queries via thread pool
- WoT auth rules cached per-session
- Special-kind EVENT store through async worker
- Subscription logging via fire-and-forget write queue
- IP ban periodic save via write queue
- g_db nulled before event loop
### Remaining
- Fix 6: Reduce reader threads (trivial)
- Fix 8: Move periodic tasks off main thread (partial — IP ban save done, monitoring/status not)
- Fix 9: Cache NIP-11 response
- Sync `store_event()` calls from admin/DM/NIP-09 paths
- Raw `sqlite3_*` calls in `thread_pool.c` and `websockets.c`
- COUNT query path in `websockets.c` still uses `json_each()` instead of `event_tags`
---
## Phase A: Move Remaining Periodic Tasks Off Main Thread
**Goal:** Eliminate the last regular SQLite calls from the lws-main event loop timer.
### A1: Move `generate_and_post_status_event()` to write worker
Currently called synchronously from the main event loop at `websockets.c:3455`.
**Changes:**
- Add `THREAD_POOL_JOB_STATUS_POST` to `thread_pool.h` job type enum
- Add executor in `thread_pool.c` that calls `generate_and_post_status_event()`
- In `websockets.c` main loop, replace direct call with `thread_pool_submit_write()` job
- The write worker has a DB connection, so `generate_stats_json()``store_event()``broadcast_event_to_subscriptions()` all work naturally
- Note: `broadcast_event_to_subscriptions()` is thread-safe (uses its own mutex)
**Files:** `thread_pool.h`, `thread_pool.c`, `websockets.c`
### A2: Move monitoring queries to read/write worker
`monitoring_on_event_stored()` and `monitoring_on_subscription_change()` in `api.c` call `generate_event_driven_monitoring()` / `generate_subscription_driven_monitoring()` which do DB queries and then broadcast ephemeral events.
**Changes:**
- Add `THREAD_POOL_JOB_MONITORING` job type
- Submit monitoring generation as a read job (it mostly does SELECTs)
- The completion broadcasts the ephemeral event via `lws_cancel_service()` + completion queue
- Alternative simpler approach: since these are throttled (default 5s) and only fire when kind 24567 subscribers exist, they could run on the write worker as fire-and-forget
**Files:** `thread_pool.h`, `thread_pool.c`, `api.c`, `websockets.c`
### A3: Move `ip_ban_cleanup()` and `ip_ban_log_stats()` to write worker
Currently called from the 60-second periodic timer at `websockets.c:3479-3480`.
**Changes:**
- Add `THREAD_POOL_JOB_IP_BAN_CLEANUP` job type (or reuse a generic callback job)
- Submit as fire-and-forget write job from the periodic timer
- `ip_ban_cleanup()` does in-memory cleanup + optional DB writes
- `ip_ban_log_stats()` is purely logging, no DB
**Files:** `thread_pool.h`, `thread_pool.c`, `ip_ban.c`, `websockets.c`
---
## Phase B: Route Remaining Sync `store_event()` Calls Through Async Worker
**Goal:** Eliminate all synchronous `store_event()` calls from lws-main context.
### Current sync `store_event()` call sites on main thread
| Call site | File | Context | Frequency |
|-----------|------|---------|-----------|
| Kind 1059 gift wrap store | `websockets.c:1795,1803,1819` | NIP-17 DM processing | Low |
| Kind 14 DM store | `websockets.c:1843` | NIP-17 DM processing | Low |
| Regular event fallback | `websockets.c:1862,1876` | Non-async-eligible events | Low |
| Kind 1059 (auth path) | `websockets.c:2594,2602,2618` | NIP-42 auth + DM | Low |
| Kind 14 (auth path) | `websockets.c:2642` | NIP-42 auth + DM | Low |
| Regular (auth path) | `websockets.c:2661,2675` | NIP-42 auth fallback | Low |
| Admin response store | `config.c:2585` | Admin API response | Very low |
| DM admin gift wrap | `dm_admin.c:389` | Admin DM response | Very low |
| NIP-09 deletion store | `nip009.c:135` | Deletion events | Low |
| Status event store | `api.c:595` | Periodic status post | Very low (handled by Phase A1) |
| Admin response event | `api.c:908` | Admin API | Very low |
| DM chat response | `api.c:1377` | Admin DM chat | Very low |
| Relay-generated event | `api.c:2350` | Relay metadata | Very low |
| DM stats response | `websockets.c:3648` | DM stats command | Very low |
### Strategy
Rather than converting each call site individually, create a **synchronous-to-async bridge**:
**Changes:**
- Create `store_event_async(cJSON* event, store_event_callback_t cb, void* ctx)` that submits the event to the write worker
- For call sites that need the result (e.g., to broadcast after store), use a completion callback
- For fire-and-forget sites (admin responses, DM responses), use NULL callback
- The existing `store_event()` wrapper becomes: submit to write worker + block on completion (for backward compat during transition)
- Eventually, all callers migrate to the async version
**Alternative simpler approach:** Since these are all low-frequency paths, and the write worker already has a DB connection, we can submit them as generic write jobs with a callback that does the store + broadcast. The main thread just needs to hand off the event JSON.
**Files:** `main.c`, `websockets.c`, `api.c`, `config.c`, `dm_admin.c`, `nip009.c`, `thread_pool.h`, `thread_pool.c`
---
## Phase C: `db_ops` Abstraction Cleanup for PG Fork
**Goal:** Ensure all raw `sqlite3_*` calls are inside `db_ops.c` only. External code uses only `db_ops.h` functions.
### C1: Add `db_open_worker_connection()` / `db_close_worker_connection()` to `db_ops.h`
Currently, `thread_pool.c:208` and `websockets.c:537` call `sqlite3_open_v2()` directly.
**Changes:**
- Add to `db_ops.h`:
```c
void* db_open_worker_connection(void); // Returns opaque connection handle
void db_close_worker_connection(void* conn);
```
- Implementation in `db_ops.c`: opens a new SQLite connection with WAL + busy timeout
- `thread_pool.c` worker init calls `db_open_worker_connection()` instead of raw `sqlite3_open_v2()`
- `websockets.c` event-worker calls `db_open_worker_connection()` instead of raw `sqlite3_open_v2()`
- Remove `#include <sqlite3.h>` from `thread_pool.c` and `websockets.c`
**Files:** `db_ops.h`, `db_ops.c`, `thread_pool.c`, `websockets.c`
### C2: Audit and remove remaining `sqlite3` includes outside `db_ops.c`
After C1, grep for any remaining `sqlite3` references outside `db_ops.c` and eliminate them.
**Files:** All `.c` files
### C3: Unify COUNT query tag path
The COUNT query builder in `websockets.c:3849` still uses `json_each(json(tags))` while the REQ query builder in `main.c:1604` uses the indexed `event_tags` table. Unify to use `event_tags` for consistency.
**Files:** `websockets.c`
---
## Phase D: Quick Wins
### D1: Fix 6 — Reduce reader threads from 4 to 1
Change default in `thread_pool_init()` or make configurable. With all reads going through the pool, a single reader is sufficient unless profiling shows otherwise.
**Files:** `thread_pool.c` or config default
### D2: Fix 9 — Cache NIP-11 response
Cache the serialized NIP-11 JSON string with a 60-second TTL. Invalidate on admin config change.
**Files:** `nip011.c`
---
## Implementation Order
| # | Task | Risk | Dependencies |
|---|------|------|-------------|
| 1 | A3: ip_ban_cleanup to write worker | Very Low | None |
| 2 | A1: status post to write worker | Low | None |
| 3 | A2: monitoring to worker | Low | None |
| 4 | D1: reduce reader threads to 1 | Very Low | None |
| 5 | D2: cache NIP-11 response | Very Low | None |
| 6 | C1: db_open/close_worker_connection | Low | None |
| 7 | C3: unify COUNT tag query path | Low | None |
| 8 | C2: audit/remove sqlite3 includes | Very Low | C1 |
| 9 | B: route sync store_event through async | Medium | A1 (status post already handled) |
## Expected Outcome
After all phases:
- **lws-main has zero SQLite calls** during the event loop
- **All `sqlite3_*` calls are inside `db_ops.c`** — clean abstraction boundary
- **PG fork can replace `db_ops.c`** without touching any other file
- Thread model is fully realized:
- `lws-main` = WebSocket I/O + JSON parse + subscription match + completion drain
- `event-worker` = signature verify + dedup + store submission
- `db-read` = REQ queries + COUNT queries + monitoring reads
- `db-write` = event INSERTs + subscription logging + periodic tasks + admin writes
```mermaid
flowchart LR
CLIENT[Nostr Clients] --> LWS[lws-main<br/>Zero SQLite<br/>WS I/O + JSON + Sub Match]
LWS -->|EVENT| EW[event-worker<br/>Verify + Dedup]
EW -->|store job| DW[db-write<br/>INSERT events<br/>Sub logging<br/>Periodic tasks<br/>Admin writes]
EW -->|completion| LWS
DW -->|completion| LWS
LWS -->|REQ/COUNT| DR[db-read<br/>SELECT queries<br/>Monitoring reads]
DR -->|completion| LWS
LWS -->|OK/EVENT/EOSE/COUNT| CLIENT
```
+379
View File
@@ -0,0 +1,379 @@
# Plan: Remove All SQLite from lws-main Thread
## Problem Statement
After implementing the 4 offload fixes (v2.1.4), `lws-main` still consumes **53.87% avg CPU** (down from 68.71%). Perf profiling shows **100% of the top symbols are SQLite** — B-tree traversal, page cache, VFS reads. The worker threads (`db-read`, `db-write`, `event-worker`) are nearly idle at 0.00.1%.
The goal is to eliminate all synchronous SQLite access from `lws-main` so it only does: JSON parsing, WebSocket I/O, subscription matching (in-memory), and message queuing.
## Thread Architecture Change
Reduce from 7 threads to 4:
```
Current: lws-main, db-read-1, db-read-2, db-read-3, db-read-4, db-write, event-worker
Proposed: lws-main, db-read, db-write, event-worker
```
The 4 reader threads are overkill — they average 0.070.10% CPU each. A single `db-read` thread is sufficient for the current workload. This can be made configurable later if needed.
## Remaining SQLite Calls on lws-main
Every synchronous SQLite call that currently runs on `lws-main`:
### Hot Path — Per-Event (every incoming EVENT message)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `event_id_exists_in_db()` | websockets.c:1324, 2134 | Every EVENT | ~10-50us per B-tree lookup |
| `get_config_value("relay_pubkey")` | websockets.c:2050 | Every kind-14 EVENT | SQLite SELECT |
| `get_config_value("admin_pubkey")` | websockets.c:2073 | Every kind-23456 EVENT | SQLite SELECT |
### Hot Path — Per-Connection (every new WebSocket connection)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("trust_proxy_headers")` | websockets.c:1071 | Every connection | SQLite SELECT |
| `get_config_bool("nip42_auth_required_events")` | websockets.c:1134 | Every connection | SQLite SELECT |
| `get_config_bool("nip42_auth_required_subscriptions")` | websockets.c:1135 | Every connection | SQLite SELECT |
### Hot Path — Per-REQ (every subscription request)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("expiration_enabled")` | main.c:1424 | Every REQ | SQLite SELECT |
| `get_config_bool("expiration_filter")` | main.c:1425 | Every REQ | SQLite SELECT |
| `check_database_auth_rules()` | websockets.c:1706 | Every REQ when WoT enabled | SQLite SELECT |
### Medium Path — Per-COUNT (NIP-45)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `thread_pool_execute_count_sync()` | websockets.c:3689 | Every COUNT message | Blocks main thread waiting for db-read |
### Low Path — Periodic (every 60s or on timer)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `refresh_hot_config_if_needed()` | websockets.c:228-247 | Every 5s | 8 SQLite SELECTs |
| `generate_and_post_status_event()` | websockets.c:3213 | Every N hours | store_event + broadcast |
| `ip_ban_cleanup/log_stats` | websockets.c:3004-3005 | Every 60s | SQLite reads/writes |
### Per-EVENT — Validator Path (called from event-worker AND sync path)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("admin_pubkey")` | request_validator.c:308 | Every kind-23456 EVENT | SQLite SELECT |
| `get_config_value("nip42_auth_enabled")` | request_validator.c:326 | Every EVENT | SQLite SELECT |
| `get_config_bool("pow_enabled")` | request_validator.c:370 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_min_difficulty")` | request_validator.c:371 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_validation_flags")` | request_validator.c:372 | Every EVENT | SQLite SELECT |
| `get_config_int("nip40_expiration_grace_period")` | request_validator.c:432 | Every EVENT with expiration | SQLite SELECT |
| `get_config_value("auth_enabled")` | request_validator.c:212 | Every EVENT (reload_auth_config) | SQLite SELECT |
| `get_config_value("auth_rules_enabled")` | request_validator.c:220 | Every EVENT (reload_auth_config) | SQLite SELECT |
| `db_count_active_whitelist_rules()` | request_validator.c:536 | Every 5s (cached) | Opens separate SQLite connection |
**Note:** These run on the **event-worker** thread for async-eligible events, but on **lws-main** for special kinds (14, 1059, 23456) that go through the sync validation path. They also run on lws-main for the `nostr_validate_unified_request()` call in the sync fallback.
### Per-EVENT — NIP-13 PoW Validation
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("pow_enabled")` | nip013.c:27 | Every EVENT | SQLite SELECT |
| `get_config_int("pow_min_difficulty")` | nip013.c:28 | Every EVENT | SQLite SELECT |
| `get_config_value("pow_mode")` | nip013.c:29 | Every EVENT | SQLite SELECT |
### Per-Broadcast — NIP-40 Expiration Check
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_bool("expiration_enabled")` | subscriptions.c:764 | Every broadcast | SQLite SELECT |
| `get_config_bool("expiration_filter")` | subscriptions.c:765 | Every broadcast | SQLite SELECT |
### Per-NIP-11 Request (HTTP, not WebSocket)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| 15+ `get_config_value()` calls | nip011.c:96-110 | Every NIP-11 HTTP request | 15 SQLite SELECTs |
| 10+ `get_config_int()`/`get_config_bool()` calls | nip011.c:113-123 | Every NIP-11 HTTP request | 10 SQLite SELECTs |
### Per-NIP-42 Auth Event
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("relay_url")` | nip042.c:99 | Every AUTH event | SQLite SELECT |
| `get_config_int("relay_port")` | nip042.c:103 | Every AUTH event (fallback) | SQLite SELECT |
### Per-NIP-40 Expiration Check (EVENT submission)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| 5 `get_config_bool()`/`get_config_int()` calls | nip040.c:37-41 | Every EVENT with expiration tag | 5 SQLite SELECTs |
### Subscription Lifecycle (per sub create/close/disconnect)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `db_log_subscription_created()` | subscriptions.c:1043 | Every REQ | SQLite INSERT |
| `db_log_subscription_closed()` | subscriptions.c:1053 | Every CLOSE | SQLite INSERT + UPDATE |
| `db_log_subscription_disconnected()` | subscriptions.c:1060 | Every disconnect | SQLite UPDATE + INSERT |
| `db_update_subscription_events_sent()` | subscriptions.c:1088 | Every broadcast match | SQLite UPDATE |
| `db_cleanup_orphaned_subscriptions()` | subscriptions.c:1100 | Startup only | SQLite DELETE |
### IP Ban System (per-connection and periodic)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("idle_ban_whitelist")` | ip_ban.c:254 | Every ban check | SQLite SELECT |
| `get_config_bool("auth_fail_ban_enabled")` | ip_ban.c:301, 343 | Every ban check + failure | SQLite SELECT |
| `get_config_bool("idle_ban_enabled")` | ip_ban.c:307, 394 | Every ban check + idle failure | SQLite SELECT |
| 6+ `get_config_int()` calls | ip_ban.c:345-399, 464-465 | Every failure recording | SQLite SELECTs |
| `db_prepare()`/`db_step_stmt()` for ip_bans | ip_ban.c:141-238 | Load at startup + periodic save | SQLite reads/writes |
### Special Kind Sync Path (kinds 14, 1059, 23456)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `store_event()` | websockets.c:1548-1629 | Every special-kind EVENT | Full sync store |
| `process_admin_event_in_config()` | websockets.c:1468 | Every kind-23456 | Config DB writes |
| `process_nip17_admin_message()` | websockets.c:1523 | Every kind-1059 when enabled | Decrypt + DB |
| `process_dm_stats_command()` | websockets.c:3240 | Admin DM stats | Decrypt + query + encrypt |
| `generate_stats_json()` | api.c:1167 (via dm_admin.c:805) | Admin stats command | Multiple SQLite queries |
### DM Admin Path (called from lws-main for kind 1059/14)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `get_config_value("relay_pubkey")` | dm_admin.c:358, 451 | Every admin DM | SQLite SELECT |
| `get_config_value("admin_pubkey")` | dm_admin.c:504 | Every admin DM | SQLite SELECT |
| `get_config_int("nip59_timestamp_max_delay_sec")` | dm_admin.c:373 | Every admin DM | SQLite SELECT |
| `get_config_int("wot_enabled")` | dm_admin.c:613, 640, 660 | WoT commands | SQLite SELECT |
### API/Monitoring (called from lws-main periodic timer)
| Call | Location | Frequency | Cost |
|------|----------|-----------|------|
| `generate_stats_json()` | api.c:1167 | Status post + admin stats | ~10 SQLite queries |
| `query_time_based_statistics()` | api.c:102 | Monitoring events | SQLite queries |
| `query_subscription_details()` | api.c:197 | Monitoring events | SQLite queries |
| `get_config_value("relay_pubkey")` | api.c:419, 821, 1291 | Every monitoring event | SQLite SELECT |
| `get_config_int("kind_1_status_posts_hours")` | api.c:547 | Status post check | SQLite SELECT |
| `get_config_int("kind_24567_reporting_throttle_sec")` | api.c:60 | Monitoring throttle | SQLite SELECT |
## Implementation Plan
### Fix 1: Create Global Config Cache — Eliminate ALL get_config_* SQLite Calls
**Impact: Very High** — eliminates 50+ distinct `get_config_value/int/bool` call sites across 8 source files
The audit found `get_config_value/int/bool` calls in: `websockets.c`, `main.c`, `subscriptions.c`, `request_validator.c`, `nip011.c`, `nip013.c`, `nip040.c`, `nip042.c`, `ip_ban.c`, `dm_admin.c`, `api.c`, `config.c`. Every single one does a `SELECT value FROM config WHERE key=?` SQLite query.
**Approach:** Replace `get_config_value_from_table()` with an in-memory hash map that is loaded once at startup and refreshed periodically (every 5s) or on admin config change events.
**Changes:**
1. **New: `config_cache` in config.c** — A simple hash map or fixed array of key-value pairs loaded from the config table:
```c
typedef struct {
char key[64];
char value[512];
} config_cache_entry_t;
static config_cache_entry_t g_config_cache[256];
static int g_config_cache_count = 0;
static time_t g_config_cache_last_refresh = 0;
static pthread_mutex_t g_config_cache_mutex;
#define CONFIG_CACHE_TTL_SEC 5
```
2. **Modify `get_config_value_from_table()`** — Instead of querying SQLite, look up in the in-memory cache. If cache is stale, refresh from SQLite (but only once per TTL period, not per call).
3. **Add `config_cache_refresh()`** — Loads all rows from `SELECT key, value FROM config` into the hash map. Called:
- Once at startup after config table is populated
- Every 5 seconds from the existing `refresh_hot_config_if_needed()` timer
- Immediately after `process_admin_event_in_config()` modifies config
4. **Thread safety:** The cache is read from lws-main and event-worker threads. Use a read-write lock or double-buffer pattern. Since config changes are rare, a simple mutex with short hold time is fine.
5. **Remove the existing `hot_config_cache_t`** in websockets.c — it becomes redundant since the global config cache serves the same purpose with broader coverage.
This single change eliminates SQLite calls from:
- Per-connection: `trust_proxy_headers`, `nip42_auth_required_*` (websockets.c)
- Per-REQ: `expiration_enabled`, `expiration_filter` (main.c)
- Per-EVENT: `admin_pubkey`, `relay_pubkey`, `nip42_auth_enabled`, `pow_*` (request_validator.c, nip013.c)
- Per-broadcast: `expiration_enabled`, `expiration_filter` (subscriptions.c)
- Per-NIP-11: 25+ config values (nip011.c)
- Per-NIP-42: `relay_url`, `relay_port` (nip042.c)
- Per-NIP-40: 5 expiration config values (nip040.c)
- Per-IP-ban: `idle_ban_*`, `auth_fail_*` config values (ip_ban.c)
- Per-admin-DM: `relay_pubkey`, `admin_pubkey`, `wot_enabled` (dm_admin.c)
- Per-monitoring: `relay_pubkey`, throttle values (api.c)
### Fix 2: Move Duplicate Check to Event Worker Thread
**Impact: High** — eliminates the most frequent per-EVENT SQLite call from lws-main
Currently `event_id_exists_in_db()` runs on lws-main before async submission. Move it into the `async_event_worker_main()` function, which already has its own SQLite connection.
**Changes:**
- Remove `event_id_exists_in_db()` calls from both EVENT paths in websockets.c (lines 1324 and 2134)
- Add the duplicate check at the start of `async_event_worker_main()` before `nostr_validate_unified_request()`
- If duplicate found, set `completion->success = 1` and `completion->error_message = "duplicate: already have this event"` and skip crypto
- The completion handler already sends OK responses, so this works seamlessly
**Trade-off:** Without the early dedup check on lws-main, duplicate events will be submitted to the event worker queue and consume a queue slot + thread wakeup before being detected. This is acceptable because:
- The worker thread dedup check is still fast (B-tree index lookup)
- It avoids the much larger cost of running SQLite on lws-main for every event
- Queue depth is 4096, so even under heavy duplicate traffic this won't overflow
### Fix 3: Make COUNT Queries Async
**Impact: Medium** — eliminates blocking wait on lws-main for NIP-45 COUNT
Currently `handle_count_message()` calls `thread_pool_execute_count_sync()` which blocks lws-main waiting for the db-read worker. Convert to the same async pattern used for REQ queries.
**Changes:**
- Create `count_async_state_t` similar to `req_async_state_t`
- Create `count_async_completion_t` with count result
- Add `submit_count_query_async()` that uses `thread_pool_submit_read()` with a callback
- The callback pushes to a count completion queue
- Add `process_count_async_completions()` to drain the queue on lws-main and send COUNT responses
- Call it from the main event loop alongside `process_req_async_completions()`
### Fix 4: Cache Auth Rules Check Result
**Impact: Low-Medium** — eliminates per-REQ SQLite when WoT is enabled
`check_database_auth_rules()` at websockets.c:1706 runs a SQLite query for every REQ when `wot_enabled == 2`. Cache the result per-session.
**Changes:**
- Add `wot_checked` and `wot_allowed` fields to `per_session_data`
- On first REQ, call `check_database_auth_rules()` and cache the result
- On subsequent REQs from the same session, use the cached value
- Reset cache when auth state changes (e.g., after NIP-42 AUTH)
### Fix 5: Move Special-Kind EVENT Store to Event Worker
**Impact: Low** — special kinds are rare but currently do full sync store on lws-main
Kinds 14, 1059, and 23456 are currently excluded from async processing because they have special handling. However, the `store_event()` call within those paths still blocks lws-main.
**Changes:**
- For kind 1059 and kind 14: after special processing completes, submit the store to the event worker instead of calling `store_event()` synchronously
- For kind 23456: admin events are not stored (processed by admin API), so no change needed
- This requires the special processing (decryption, admin check) to still happen on lws-main (it needs `pss` context), but the final DB write goes async
### Fix 6: Reduce Reader Threads from 4 to 1
**Impact: Resource savings** — 3 fewer threads, 3 fewer SQLite connections
**Changes:**
- Change default `reader_threads` in `thread_pool_init()` from 4 to 1
- Or make it configurable via config with default 1
- The single `db-read` thread handles all REQ and COUNT queries sequentially
- If future profiling shows the read thread becoming a bottleneck, increase back to 2+
### Fix 7: Move Subscription Logging Off Main Thread
**Impact: Medium** — eliminates per-REQ INSERT, per-CLOSE UPDATE, per-broadcast UPDATE
The subscription lifecycle functions in subscriptions.c do synchronous SQLite writes on lws-main:
- `db_log_subscription_created()` — INSERT on every REQ
- `db_log_subscription_closed()` — INSERT + UPDATE on every CLOSE
- `db_log_subscription_disconnected()` — UPDATE + INSERT on every disconnect
- `db_update_subscription_events_sent()` — UPDATE on every broadcast match
**Changes:**
- Submit these as fire-and-forget write jobs to the db-write thread via `thread_pool_submit_write()`
- Create a new job type `THREAD_POOL_JOB_SUBSCRIPTION_LOG` or simply use a generic callback job
- The main thread doesn't need to wait for these to complete — they're purely observational logging
### Fix 8: Move Periodic Tasks Off Main Thread
**Impact: Low** — these run every 60s but can still cause latency spikes
- `generate_and_post_status_event()`: Submit as an async job to the event worker
- `ip_ban_cleanup()`/`ip_ban_log_stats()`/`ip_ban_save()`: These do SQLite reads/writes. Move to a periodic job on the db-write thread
- Config cache refresh (Fix 1) replaces the old `refresh_hot_config_if_needed()` SQLite queries
### Fix 9: Cache NIP-11 Response
**Impact: Low-Medium** — eliminates 25+ SQLite SELECTs per NIP-11 HTTP request
`generate_relay_info_json()` in nip011.c calls `get_config_value()` 15 times and `get_config_int()`/`get_config_bool()` 10 times. After Fix 1 (global config cache), these become in-memory lookups. But we can go further:
**Changes:**
- Cache the serialized NIP-11 JSON response string with a TTL (e.g., 60 seconds)
- On NIP-11 request, return the cached string directly without rebuilding
- Invalidate on admin config change events
## Implementation Order
| # | Fix | Risk | Complexity |
|---|-----|------|------------|
| 1 | Global config cache (replaces all get_config_* SQLite) | Low | Medium |
| 2 | Move dedup check to event worker | Low | Low |
| 3 | Async COUNT queries | Low | Medium |
| 4 | Cache WoT auth rules per-session | Very Low | Low |
| 5 | Async store for special kinds | Medium | Medium |
| 6 | Reduce reader threads to 1 | Very Low | Trivial |
| 7 | Move subscription logging off main thread | Low | Low-Medium |
| 8 | Move periodic tasks off main thread | Low | Medium |
| 9 | Cache NIP-11 response | Very Low | Low |
## Expected Impact
After all 9 fixes, `lws-main` should have **zero direct SQLite calls**. Its work becomes:
- JSON parse incoming messages (~fast)
- In-memory subscription matching (~fast)
- In-memory config lookups (~fast, hash map)
- Message formatting and queuing (~fast)
- `lws_service()` I/O (~fast)
- Draining completion queues (~fast)
### Impact by fix
| Fix | SQLite calls eliminated | Frequency |
|-----|------------------------|-----------|
| Fix 1: Global config cache | ~50+ `get_config_*` call sites across 8 files | Per-event, per-connection, per-REQ, per-broadcast, per-NIP-11 |
| Fix 2: Dedup to event worker | 2 `event_id_exists_in_db()` calls | Per-EVENT |
| Fix 3: Async COUNT | 1 `thread_pool_execute_count_sync()` | Per-COUNT message |
| Fix 4: WoT cache per-session | 1 `check_database_auth_rules()` | Per-REQ when WoT enabled |
| Fix 5: Async special-kind store | ~6 `store_event()` calls | Per special-kind EVENT |
| Fix 6: Reduce readers | N/A (resource savings) | N/A |
| Fix 7: Async subscription logging | 4 `db_log_*`/`db_update_*` calls | Per-REQ, per-CLOSE, per-broadcast |
| Fix 8: Async periodic tasks | `generate_stats_json()`, `ip_ban_*` | Every 60s |
| Fix 9: Cache NIP-11 | 25+ config lookups per request | Per NIP-11 HTTP request |
Estimated main-thread CPU reduction: from **54% to ~5-10%** (mostly WebSocket I/O, subscription matching, and JSON parsing).
## Thread Model After Changes
```
lws-main — WebSocket I/O, JSON parse, subscription match, message queue
NO SQLite. Pure compute + I/O.
db-read — All SELECT queries: REQ results, COUNT results, config refresh,
auth rule checks, event dedup checks via completion queue
db-write — All INSERT/UPDATE: event storage, tag inserts, IP ban persistence,
status event generation
event-worker — EVENT validation: signature verification, then submits store to
db-write via completion queue, results flow back to lws-main
```
```mermaid
flowchart LR
CLIENT[Nostr Clients] --> LWS[lws-main<br/>WebSocket I/O<br/>JSON parse<br/>Sub matching<br/>NO SQLite]
LWS -->|EVENT submit| EW[event-worker<br/>Sig verify<br/>Dedup check]
EW -->|store job| DW[db-write<br/>INSERT events<br/>INSERT tags<br/>IP ban save]
EW -->|completion| LWS
DW -->|completion| LWS
LWS -->|REQ/COUNT submit| DR[db-read<br/>SELECT queries<br/>Config refresh<br/>Auth checks]
DR -->|completion| LWS
LWS -->|OK/EVENT/EOSE| CLIENT
```
+215
View File
@@ -0,0 +1,215 @@
# Thread Pool PostgreSQL Optimization Plan
## Current State
### Thread Inventory
| Thread | Name | Role | PG Connection | Config Key |
|--------|------|------|--------------|------------|
| lws-main | `lws-main` | WebSocket event loop, HTTP, NIP-11, admin events, subscription matching, broadcast, monitoring event generation | `g_pg_conn` (main) | N/A — always runs |
| reader-1 | `db-read-1` | Async REQ and COUNT query execution | `g_thread_pg_conn` (TLS) | `thread_pool_readers` (default: 1) |
| writer | `db-write` | Async EVENT store, delete, sub logging, IP ban save, WAL checkpoint | `g_thread_pg_conn` (TLS) | N/A — always 1 |
| event-worker | `event-worker` | Async EVENT ingestion — duplicate check, signature validation, `store_event_core()`, ephemeral event handling | Own `worker_db` via `db_open_worker_connection()` | N/A — always 1 |
**Total: 4 threads, 4 PG connections**
### event-worker Thread (websockets.c — separate from thread pool)
The `event-worker` is a dedicated async pipeline for incoming EVENT messages, defined in [`src/websockets.c`](src/websockets.c:543). It is **not** part of the thread pool — it has its own job queue, mutex, and condvar.
**What it does for each incoming EVENT:**
1. Checks if event ID already exists in DB (`event_id_exists_in_db()`)
2. Validates the event with `nostr_validate_unified_request()` (signature check)
3. For ephemeral events (kind 20000-29999): marks for broadcast only, no storage
4. For regular events: calls `store_event_core()` to insert into PostgreSQL
5. Pushes a completion struct and wakes `lws-main` via `lws_cancel_service()`
6. `lws-main` picks up the completion, sends OK response, and broadcasts to subscribers
**Why it exists separately from the thread pool writer:**
- The thread pool writer handles generic DB operations (sub logging, IP bans, etc.)
- The event-worker handles the full EVENT ingestion pipeline including validation and duplicate detection — not just the DB insert
- It was added to move CPU-intensive signature verification off the main thread
**Relationship to thread pool writer:**
There is overlap — both can write to the events table. The event-worker calls `store_event_core()` directly, while the thread pool writer uses `db_insert_event_with_json()`. Some EVENT paths still go through the thread pool writer (sync path for admin/DM/protected events).
### What Runs on lws-main Today (Shouldn't Block)
- WebSocket accept/read/write for all Nostr clients
- JSON parsing of REQ/EVENT/CLOSE/COUNT messages
- Event signature verification (CPU-bound)
- Subscription matching and broadcast (iterates all subscriptions)
- NIP-11 HTTP responses
- NIP-42 auth challenge generation
- Admin event processing (kind 23456) — includes NIP-44 decrypt + config changes
- **Monitoring event generation** — queries DB, signs events, broadcasts (BLOCKS)
- **Sync config reads**`get_config_value()` hits PG on main thread (BLOCKS)
### SQLite-Era Constraints (No Longer Apply)
- Single writer thread was required (SQLite serializes writes)
- WAL checkpoint job was required (SQLite-specific)
- Reader/writer separation was critical for avoiding SQLITE_BUSY
- `db_wal_checkpoint_truncate()` on writer shutdown — SQLite-only
---
## Changes for PostgreSQL
### Phase 1: Increase Reader Thread Count
**What:** Change default reader count from 1 to 4.
**Why:** With SQLite, multiple readers had diminishing returns due to file-level locking. PostgreSQL handles concurrent reads perfectly — each reader has its own connection and queries execute in parallel.
**Files:**
- `src/main.c` line ~2524: Change default from 1 to 4
- Config key `thread_pool_readers` already exists — just change the default
**Risk:** Low. Each reader opens its own PG connection via `db_open_worker_connection()`. PostgreSQL default `max_connections = 100` easily handles 4-8 readers.
### Phase 2: Allow Multiple Writer Threads
**What:** Make writer thread count configurable (default: 2). Convert the single `pthread_t writer` to an array like readers.
**Why:** PostgreSQL supports concurrent writes with row-level locking. The single-writer bottleneck was a SQLite constraint. With multiple writers, EVENT inserts from different clients can execute in parallel.
**Files:**
- `src/thread_pool.h`: Add `int writer_threads` to `thread_pool_config_t`
- `src/thread_pool.c`:
- Change `pthread_t writer` to `pthread_t* writers` + `int writer_count`
- `thread_pool_init()`: create N writer threads
- `thread_pool_shutdown()`: join all writer threads
- `writer_worker_main()`: set thread name to `db-write-N`
- `src/main.c`: Add config key `thread_pool_writers` (default: 2)
**Risk:** Medium. Need to verify that no write job assumes serialized execution. Current write jobs:
- `STORE_EVENT` — PG handles concurrent inserts with ON CONFLICT
- `LOG_SUB_CREATED/CLOSED/DISCONNECTED` — independent rows, safe
- `UPDATE_SUB_EVENTS_SENT` — updates by sub_id, safe
- `IP_BAN_SAVE` — writes to ip_bans table, safe
- `WAL_CHECKPOINT` — SQLite-only, skip for PG (see Phase 3)
All write jobs operate on independent rows or use conflict resolution. Multiple writers are safe.
### Phase 3: Remove SQLite-Specific WAL Checkpoint Logic
**What:** Make WAL checkpoint a no-op when running PostgreSQL backend.
**Why:** PostgreSQL manages its own WAL internally. The `db_wal_checkpoint_passive()` and `db_wal_checkpoint_truncate()` calls are SQLite-specific.
**Files:**
- `src/thread_pool.c`:
- `execute_wal_checkpoint_job()`: Return OK immediately if PG backend
- `writer_worker_main()` shutdown path: Skip `db_wal_checkpoint_truncate()` if PG backend
- `src/db_ops.c` or `src/db_ops_postgres.c`: Ensure WAL checkpoint functions are no-ops for PG
**Risk:** Low. These are already effectively no-ops since the PG backend doesn't implement them, but making it explicit avoids log noise.
### Phase 4: Move Monitoring Event Generation Off Main Thread
**What:** Run monitoring queries and event generation on a reader thread instead of lws-main.
**Why:** The monitoring system (`generate_monitoring_event_for_type()` in `src/api.c`) currently:
1. Queries PostgreSQL for stats (blocks main thread 5-50ms)
2. Creates and signs a Nostr event (CPU work on main thread)
3. Broadcasts to subscriptions (main thread)
Step 1 and 2 should happen on a worker thread. Only step 3 (broadcast) needs to happen on lws-main.
**Approach:** Add a new job type `THREAD_POOL_JOB_MONITORING_QUERY` to the read queue. The reader thread executes the query and prepares the event JSON. The completion callback on lws-main just broadcasts the pre-built event.
**Files:**
- `src/thread_pool.h`: Add `THREAD_POOL_JOB_MONITORING_QUERY` job type and payload struct
- `src/thread_pool.c`: Add `execute_monitoring_query_job()` handler
- `src/api.c`: Refactor `generate_monitoring_event_for_type()` to submit async job
- `src/websockets.c`: Handle monitoring completion in the event loop callback
**Risk:** Medium. The monitoring event signing requires the relay private key. Need to pass it to the worker thread or pre-compute the unsigned event and sign on completion.
### Phase 5: Add Configurable Thread Pool Sizing via Config Events
**What:** Allow thread pool reader/writer counts to be set via the admin config event system.
**Config keys:**
- `thread_pool_readers` — number of reader threads (default: 4)
- `thread_pool_writers` — number of writer threads (default: 2)
- `thread_pool_max_queue_depth` — max pending jobs per queue (default: 4096)
**Note:** Thread count changes require relay restart (cannot dynamically resize thread pool).
**Files:**
- `src/default_config_event.h`: Add defaults for new config keys
- `src/main.c`: Read config values when initializing thread pool
**Risk:** Low. Config keys already work this way for `thread_pool_readers`.
### Phase 6: Fix Error Messages (Cosmetic)
**What:** Update SQLite-specific error messages in thread pool to be backend-neutral.
**Files:**
- `src/thread_pool.c` line 508: "Reader worker failed to open SQLite connection" → "Reader worker failed to open database connection"
- `src/thread_pool.c` line 530: "Writer worker failed to open SQLite connection" → "Writer worker failed to open database connection"
**Risk:** None.
---
## Target Thread Layout After Implementation
```
c_relay_pg process
├── lws-main — WebSocket event loop (never blocks on DB)
├── db-read-1 — Async REQ/COUNT/monitoring queries
├── db-read-2 — Async REQ/COUNT/monitoring queries
├── db-read-3 — Async REQ/COUNT/monitoring queries
├── db-read-4 — Async REQ/COUNT/monitoring queries
├── event-worker — Async EVENT ingestion (validate + store)
├── db-write-1 — Async sub logging, IP bans, misc writes
└── db-write-2 — Async sub logging, IP bans, misc writes
```
**Total: 8 threads, 8 PG connections** (configurable)
---
## PostgreSQL Connection Budget
| Component | Connections | Notes |
|-----------|------------|-------|
| lws-main | 1 | Sync config reads, admin event processing |
| Reader threads (4) | 4 | One per reader |
| event-worker | 1 | EVENT ingestion pipeline |
| Writer threads (2) | 2 | Sub logging, IP bans, misc |
| **Total per instance** | **8** | Well within PG default max_connections=100 |
| Future: admin-ws | +1 | When/if added later |
| Future: pg-listener | +1 | When/if added later |
| **Future total** | **10** | Still very comfortable |
For load balancing with N relay instances: N × 8 connections. With 10 instances = 80 connections. PgBouncer recommended at that scale.
---
## Implementation Order
| Phase | Description | Complexity | Impact |
|-------|-------------|-----------|--------|
| 1 | Increase reader count to 4 | Trivial — one line change | High — parallel query execution |
| 2 | Multiple writer threads | Medium — refactor writer array | Medium — parallel event inserts |
| 3 | Remove WAL checkpoint logic | Low — conditional no-op | Low — cleaner code |
| 4 | Move monitoring off main thread | Medium — new job type + async flow | High — unblocks event loop |
| 5 | Config event integration | Low — add config keys | Low — operational convenience |
| 6 | Fix error messages | Trivial | None — cosmetic |
**Recommended start:** Phase 1 + 6 (trivial, immediate benefit), then Phase 3 (cleanup), then Phase 4 (biggest architectural improvement), then Phase 2 (nice-to-have), then Phase 5 (polish).
---
## Future: Load Balancing Considerations
The thread pool design is already compatible with horizontal scaling because:
1. **No shared in-process state between instances** — all state is in PostgreSQL
2. **Subscription matching is per-instance** — each instance tracks its own connected clients
3. **Cross-instance event notification** via PostgreSQL LISTEN/NOTIFY (future phase)
4. **Connection pooling** via PgBouncer when connection count becomes a concern
The only addition needed for multi-instance is a LISTEN/NOTIFY subscriber thread that receives "new event stored" notifications from PostgreSQL and triggers subscription matching on the local instance. This is the `pg-listener` thread discussed earlier — it can be added independently of the thread pool changes in this plan.
+107
View File
@@ -0,0 +1,107 @@
# WAL Checkpoint Fix — db-write Thread CPU Plan
## Problem
Fresh profiling of the production server shows the `db-write` thread consuming **77% average CPU** (peaking at 100%) while `lws-main` sits at 0.3%.
The `perf` callgraph confirms 100% of the hot symbols are SQLite **B-tree read operations** running on the `db-write` thread:
| % CPU | Symbol | Role |
|-------|--------|------|
| 13.88 | `sqlite3BtreeTableMoveto` | B-tree seek |
| 10.05 | `sqlite3VdbeExec` | VM execution |
| 7.77 | `rep_movs_alternative` | kernel memcpy in pread |
| 7.76 | `sqlite3GetVarint` | varint decode |
| 6.66 | `pcache1Fetch` | page cache lookup |
| 6.02 | `memcpy` | data copy |
These are **read-path** functions, not write-path. The `event-worker` thread accumulated only 3 CPU ticks over 5 minutes, meaning very few events are actually being stored.
## Root Cause
SQLite WAL auto-checkpoint. By default SQLite triggers a checkpoint every 1000 WAL pages. The checkpoint is executed by the **next writer** — which is always the `db-write` thread since all writes are funnelled through it.
A WAL checkpoint reads every dirty page from the WAL file and writes it back to the main database file. This involves B-tree traversals to locate pages, which explains the `BtreeTableMoveto` dominance.
The current `db_open_worker_connection` in `src/db_ops.c` sets `PRAGMA journal_mode=WAL` and `busy_timeout=5000` but does **not** configure `wal_autocheckpoint`. SQLite's default of 1000 pages applies.
In earlier profiling runs the same CPU burn appeared on `lws-main` at 55-67% — because that thread was doing the writes directly before the thread-pool refactoring. The work simply moved threads; the total cost is unchanged.
## Implementation Plan
### Step 1 — Disable auto-checkpoint on the write connection
In `db_open_worker_connection` add:
```c
sqlite3_exec(db, "PRAGMA wal_autocheckpoint=0;", NULL, NULL, NULL);
```
This prevents the write thread from ever running a checkpoint inline with normal writes.
### Step 2 — Add a periodic checkpoint job type to the thread pool
Add a new job type `THREAD_POOL_JOB_WAL_CHECKPOINT` to `thread_pool.h`.
The handler in `thread_pool.c` calls:
```c
sqlite3_wal_checkpoint_v2(db, NULL, SQLITE_CHECKPOINT_PASSIVE, NULL, NULL);
```
`PASSIVE` mode checkpoints only pages that are not currently being read, so it never blocks readers.
### Step 3 — Submit the checkpoint job on a timer
In the existing 60-second periodic timer in `websockets.c`, add a call to submit a WAL checkpoint job to the write queue:
```c
thread_pool_submit_wal_checkpoint();
```
This runs the checkpoint once per minute on the write thread, but as a **bounded, predictable** operation rather than being triggered unpredictably by every INSERT.
### Step 4 — Also disable auto-checkpoint on reader connections
In `db_open_worker_connection`, the same `wal_autocheckpoint=0` applies to reader connections too. Readers can also trigger checkpoints in SQLite; disabling it on all connections ensures only the explicit periodic job does checkpointing.
### Step 5 — Run a TRUNCATE checkpoint at shutdown
In `thread_pool_shutdown`, before closing the write connection, run:
```c
sqlite3_wal_checkpoint_v2(db, NULL, SQLITE_CHECKPOINT_TRUNCATE, NULL, NULL);
```
This ensures the WAL is fully flushed and truncated on clean shutdown, keeping the database file compact.
## Files Changed
| File | Change |
|------|--------|
| `src/db_ops.c` | Add `PRAGMA wal_autocheckpoint=0` in `db_open_worker_connection` |
| `src/thread_pool.h` | Add `THREAD_POOL_JOB_WAL_CHECKPOINT` enum value |
| `src/thread_pool.c` | Add `execute_wal_checkpoint_job` handler; add checkpoint-at-shutdown in `thread_pool_shutdown`; add `thread_pool_submit_wal_checkpoint` helper |
| `src/websockets.c` | Call `thread_pool_submit_wal_checkpoint()` in the 60-second periodic timer |
## Expected Impact
- `db-write` CPU should drop from ~77% to near 0% when idle (no events to store)
- Periodic checkpoint bursts of a few seconds every 60s instead of continuous burn
- No change to data durability — WAL still protects against crashes; checkpoint just moves data from WAL to main DB file
- Readers are unaffected — PASSIVE checkpoint never blocks them
## Risk
- If the relay crashes between checkpoints, the WAL file may be larger than before (up to 60s of accumulated writes). This is safe — SQLite replays the WAL on next open. The WAL file size is bounded by the write rate, which is low.
- PASSIVE checkpoint may not checkpoint all pages if readers hold them. This is fine — the next checkpoint will catch up.
## Verification
After deploying, re-run the profiler:
```bash
./deploy_lt_debug.sh && DURATION=300 INTERVAL=5 ./tests/thread_cpu_profile.sh
```
Expected: `db-write` avg CPU drops below 5%.
+1 -1
View File
@@ -1 +1 @@
3134018
257278
+618 -443
View File
File diff suppressed because it is too large Load Diff
+6 -2
View File
@@ -60,10 +60,14 @@ char* execute_sql_query(const char* query, const char* request_id, char* error_m
int handle_sql_query_unified(cJSON* event, const char* query, char* error_message, size_t error_size, struct lws* wsi);
// Monitoring system functions
void monitoring_on_event_stored(void);
void monitoring_on_subscription_change(void);
int get_monitoring_throttle_seconds(void);
// Dedicated API worker thread lifecycle
int start_api_worker(void);
void stop_api_worker(void);
void api_worker_process_completions(void);
int api_worker_enqueue_status_post(void);
// Kind 1 status posts
int generate_and_post_status_event(void);
+253
View File
@@ -0,0 +1,253 @@
/*
* Caching Inbox Poller
*
* Periodically dequeues events from the PostgreSQL `caching_event_inbox` table
* and feeds them through the shared ingest_event() pipeline with
* EVENT_SOURCE_CACHING_INBOX. The poller runs on the main libwebsockets
* service thread (no separate thread) and is compiled only for the PostgreSQL
* backend. For SQLite builds every entry point is a no-op.
*
* The poller maintains a simple two-state machine:
* POLLER_IDLE - slow poll interval (no rows were found last cycle)
* POLLER_ACTIVE - fast poll interval (a full batch was dequeued last cycle)
*/
#include "caching_inbox_poller.h"
#include <stddef.h>
#include <string.h>
#include <time.h>
#include "debug.h"
#include "config.h"
#include "db_ops.h"
#include "main.h"
#ifdef DB_BACKEND_POSTGRES
#include <cjson/cJSON.h>
#endif
// ---- Configuration keys (read each tick) -------------------------------------
#define CFG_KEY_ENABLED "caching_inbox_enabled"
#define CFG_KEY_BATCH_SIZE "caching_inbox_batch_size"
#define CFG_KEY_ACTIVE_POLL_MS "caching_inbox_active_poll_ms"
#define CFG_KEY_IDLE_POLL_MS "caching_inbox_idle_poll_ms"
// ---- Defaults ----------------------------------------------------------------
#define DEFAULT_BATCH_SIZE 25
#define DEFAULT_ACTIVE_POLL_MS 200
#define DEFAULT_IDLE_POLL_MS 5000
// ---- State machine -----------------------------------------------------------
typedef enum {
POLLER_IDLE = 0,
POLLER_ACTIVE = 1
} poller_state_t;
// ---- Module state (PostgreSQL only; trivial for SQLite) ----------------------
#ifdef DB_BACKEND_POSTGRES
static poller_state_t g_state = POLLER_IDLE;
static long long g_last_poll_ms = 0; // monotonic-ish ms timestamp of last poll
static int g_initialized = 0;
// Statistics counters
static int g_total_dequeued = 0;
static int g_total_accepted = 0;
static int g_total_rejected = 0;
static int g_total_duplicates = 0;
static int g_last_batch_size = 0;
#endif
// ---- Helpers -----------------------------------------------------------------
#ifdef DB_BACKEND_POSTGRES
// Return a coarse millisecond timestamp using time(NULL) scaled to ms.
// We avoid clock_gettime(CLOCK_MONOTONIC) portability concerns and keep the
// resolution sufficient for the configured poll intervals (>= 200ms).
static long long now_ms(void) {
return (long long)time(NULL) * 1000LL;
}
#endif
// ---- Public API --------------------------------------------------------------
int caching_inbox_poller_init(void) {
#ifdef DB_BACKEND_POSTGRES
g_state = POLLER_IDLE;
g_last_poll_ms = 0;
g_total_dequeued = 0;
g_total_accepted = 0;
g_total_rejected = 0;
g_total_duplicates = 0;
g_last_batch_size = 0;
g_initialized = 1;
DEBUG_LOG("caching_inbox_poller: initialized (PostgreSQL backend)");
#else
// SQLite build: no-op. The inbox feature is PostgreSQL-only.
#endif
return 0;
}
int caching_inbox_poller_tick(void) {
#ifdef DB_BACKEND_POSTGRES
if (!g_initialized) {
return 0;
}
// Read configuration each tick (kept simple; values are cheap to read).
int enabled = get_config_bool(CFG_KEY_ENABLED, 0);
if (!enabled) {
return 0;
}
int batch_size = get_config_int(CFG_KEY_BATCH_SIZE, DEFAULT_BATCH_SIZE);
if (batch_size <= 0) {
batch_size = DEFAULT_BATCH_SIZE;
}
int active_poll_ms = get_config_int(CFG_KEY_ACTIVE_POLL_MS, DEFAULT_ACTIVE_POLL_MS);
if (active_poll_ms < 0) {
active_poll_ms = DEFAULT_ACTIVE_POLL_MS;
}
int idle_poll_ms = get_config_int(CFG_KEY_IDLE_POLL_MS, DEFAULT_IDLE_POLL_MS);
if (idle_poll_ms < 0) {
idle_poll_ms = DEFAULT_IDLE_POLL_MS;
}
// Throttle: only poll when the configured interval has elapsed.
long long now = now_ms();
long long interval_ms = (g_state == POLLER_ACTIVE) ? active_poll_ms : idle_poll_ms;
if (g_last_poll_ms != 0 && (now - g_last_poll_ms) < interval_ms) {
return 0;
}
g_last_poll_ms = now;
// Dequeue a bounded batch in a single SQL transaction.
int count = 0;
cJSON* rows = db_caching_inbox_dequeue_batch(batch_size, &count);
if (count <= 0 || rows == NULL) {
// Empty queue (or dequeue failure). Back off to idle mode.
if (rows != NULL) {
// Defensive: count==0 but a non-null array was returned.
cJSON_Delete(rows);
}
if (g_state == POLLER_ACTIVE) {
DEBUG_TRACE("caching_inbox_poller: queue empty, switching to IDLE");
}
g_state = POLLER_IDLE;
g_last_batch_size = 0;
// Note: db_caching_inbox_dequeue_batch logs its own errors on failure;
// a NULL return with count==0 is the normal "empty queue" case.
return 0;
}
DEBUG_LOG("caching_inbox_poller: dequeued %d event(s) (batch_size=%d, state=%s)",
count, batch_size, g_state == POLLER_ACTIVE ? "ACTIVE" : "IDLE");
g_last_batch_size = count;
g_total_dequeued += count;
// Feed each dequeued event through the shared ingestion pipeline.
cJSON* row = NULL;
cJSON_ArrayForEach(row, rows) {
cJSON* event_json_obj = cJSON_GetObjectItemCaseSensitive(row, "event_json");
if (!event_json_obj || !cJSON_IsString(event_json_obj)) {
DEBUG_WARN("caching_inbox_poller: row missing event_json string; skipping");
g_total_rejected++;
continue;
}
const char* event_json = cJSON_GetStringValue(event_json_obj);
size_t event_json_len = strlen(event_json);
int rc = ingest_event(event_json, event_json_len,
EVENT_SOURCE_CACHING_INBOX, NULL, NULL);
if (rc == 0) {
g_total_accepted++;
} else {
g_total_rejected++;
DEBUG_WARN("caching_inbox_poller: ingest_event rejected an event (rc=%d)", rc);
}
}
cJSON_Delete(rows);
// State transition: a full batch suggests more work is waiting.
if (count >= batch_size) {
if (g_state != POLLER_ACTIVE) {
DEBUG_TRACE("caching_inbox_poller: full batch, switching to ACTIVE");
}
g_state = POLLER_ACTIVE;
} else {
if (g_state != POLLER_IDLE) {
DEBUG_TRACE("caching_inbox_poller: partial batch, switching to IDLE");
}
g_state = POLLER_IDLE;
}
return 0;
#else
// SQLite build: no-op.
return 0;
#endif
}
void caching_inbox_poller_shutdown(void) {
#ifdef DB_BACKEND_POSTGRES
DEBUG_LOG("caching_inbox_poller: shutdown (dequeued=%d accepted=%d rejected=%d)",
g_total_dequeued, g_total_accepted, g_total_rejected);
g_initialized = 0;
g_state = POLLER_IDLE;
g_last_poll_ms = 0;
g_last_batch_size = 0;
#else
// SQLite build: no-op.
#endif
}
void caching_inbox_poller_get_stats(int* out_total_dequeued,
int* out_total_accepted,
int* out_total_rejected,
int* out_total_duplicates,
int* out_last_batch_size,
int* out_pending_live,
int* out_pending_backfill,
int* out_oldest_age_seconds) {
#ifdef DB_BACKEND_POSTGRES
if (out_total_dequeued) *out_total_dequeued = g_total_dequeued;
if (out_total_accepted) *out_total_accepted = g_total_accepted;
if (out_total_rejected) *out_total_rejected = g_total_rejected;
if (out_total_duplicates) *out_total_duplicates = g_total_duplicates;
if (out_last_batch_size) *out_last_batch_size = g_last_batch_size;
// Pending counts and oldest age come from the database; query on demand.
int live = 0, backfill = 0, oldest = 0;
if (out_pending_live || out_pending_backfill) {
if (db_caching_inbox_pending_counts(&live, &backfill) != 0) {
live = 0;
backfill = 0;
}
}
if (out_pending_live) *out_pending_live = live;
if (out_pending_backfill) *out_pending_backfill = backfill;
if (out_oldest_age_seconds) {
if (db_caching_inbox_oldest_age(&oldest) != 0) {
oldest = 0;
}
*out_oldest_age_seconds = oldest;
}
#else
if (out_total_dequeued) *out_total_dequeued = 0;
if (out_total_accepted) *out_total_accepted = 0;
if (out_total_rejected) *out_total_rejected = 0;
if (out_total_duplicates) *out_total_duplicates = 0;
if (out_last_batch_size) *out_last_batch_size = 0;
if (out_pending_live) *out_pending_live = 0;
if (out_pending_backfill) *out_pending_backfill = 0;
if (out_oldest_age_seconds) *out_oldest_age_seconds = 0;
#endif
}
+27
View File
@@ -0,0 +1,27 @@
#ifndef CACHING_INBOX_POLLER_H
#define CACHING_INBOX_POLLER_H
// Initialize the inbox poller. Called once at startup (PostgreSQL only).
// Returns 0 on success, -1 on error.
int caching_inbox_poller_init(void);
// Perform one polling cycle. Called from the main lws service loop.
// Dequeues a bounded batch and feeds events through ingest_event().
// Returns 0 on success, -1 on error.
int caching_inbox_poller_tick(void);
// Shut down the poller and free resources.
void caching_inbox_poller_shutdown(void);
// Get current poller statistics for status display.
// All out-parameters are optional (may be NULL).
void caching_inbox_poller_get_stats(int* out_total_dequeued,
int* out_total_accepted,
int* out_total_rejected,
int* out_total_duplicates,
int* out_last_batch_size,
int* out_pending_live,
int* out_pending_backfill,
int* out_oldest_age_seconds);
#endif // CACHING_INBOX_POLLER_H
+265
View File
@@ -0,0 +1,265 @@
#define _GNU_SOURCE
#include "caching_service_launcher.h"
#include "config.h"
#include "debug.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <signal.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <errno.h>
#include <fcntl.h>
// Tracked child PID for fork mode.
// -1 means not running (or not launched by us).
static pid_t g_caching_child_pid = -1;
// ---------------------------------------------------------------------------
// Fork/exec implementation
// ---------------------------------------------------------------------------
static int caching_service_start_fork(const char* binary_path, const char* pg_conn) {
if (!binary_path || binary_path[0] == '\0') {
DEBUG_ERROR("caching_service_start: caching_service_binary_path is not set");
return -1;
}
// Check that the binary exists and is executable
if (access(binary_path, X_OK) != 0) {
DEBUG_ERROR("caching_service_start: binary '%s' not found or not executable: %s",
binary_path, strerror(errno));
return -1;
}
// If already running, don't start again
if (g_caching_child_pid > 0) {
// Check if the process is still alive
int status;
pid_t result = waitpid(g_caching_child_pid, &status, WNOHANG);
if (result == 0) {
// Still running
DEBUG_WARN("caching_service_start: caching service already running (PID %d)",
g_caching_child_pid);
return 0;
}
// Process exited — reset PID
g_caching_child_pid = -1;
}
DEBUG_INFO("caching_service_start: forking '%s' with --pg-conn", binary_path);
pid_t pid = fork();
if (pid < 0) {
DEBUG_ERROR("caching_service_start: fork failed: %s", strerror(errno));
return -1;
}
if (pid == 0) {
// Child process
// Detach from parent's process group so signals to the relay
// don't automatically propagate to the caching service.
setsid();
// Redirect stdin from /dev/null
int devnull = open("/dev/null", O_RDONLY);
if (devnull >= 0) {
dup2(devnull, STDIN_FILENO);
close(devnull);
}
// Redirect stdout/stderr to a log file or /dev/null
// For now, send to /dev/null. A future improvement could
// redirect to a configurable log file.
devnull = open("/dev/null", O_WRONLY);
if (devnull >= 0) {
dup2(devnull, STDOUT_FILENO);
dup2(devnull, STDERR_FILENO);
close(devnull);
}
// Build argv
// argv[0] = binary path
// argv[1] = "--pg-conn"
// argv[2] = connection string (if provided)
// argv[3] = NULL
char* argv[4];
int argc = 0;
argv[argc++] = (char*)binary_path;
if (pg_conn && pg_conn[0] != '\0') {
argv[argc++] = (char*)"--pg-conn";
argv[argc++] = (char*)pg_conn;
}
argv[argc] = NULL;
execv(binary_path, argv);
// If we get here, exec failed
// Use _exit to avoid flushing parent's stdio buffers
_exit(127);
}
// Parent process
g_caching_child_pid = pid;
DEBUG_INFO("caching_service_start: caching service started (PID %d)", pid);
return 0;
}
static int caching_service_stop_fork(void) {
if (g_caching_child_pid <= 0) {
DEBUG_LOG("caching_service_stop: caching service not running");
return 0;
}
DEBUG_INFO("caching_service_stop: sending SIGTERM to PID %d", g_caching_child_pid);
if (kill(g_caching_child_pid, SIGTERM) != 0) {
if (errno == ESRCH) {
// Process already exited
DEBUG_LOG("caching_service_stop: process already exited");
g_caching_child_pid = -1;
return 0;
}
DEBUG_ERROR("caching_service_stop: kill failed: %s", strerror(errno));
return -1;
}
// Wait briefly for graceful shutdown
int status;
int wait_ms = 0;
while (wait_ms < 5000) {
pid_t result = waitpid(g_caching_child_pid, &status, WNOHANG);
if (result == g_caching_child_pid) {
// Child exited
if (WIFEXITED(status)) {
DEBUG_INFO("caching_service_stop: process exited with status %d",
WEXITSTATUS(status));
} else if (WIFSIGNALED(status)) {
DEBUG_INFO("caching_service_stop: process killed by signal %d",
WTERMSIG(status));
}
g_caching_child_pid = -1;
return 0;
}
usleep(100000); // 100ms
wait_ms += 100;
}
// Force kill if still running after 5 seconds
DEBUG_WARN("caching_service_stop: process did not exit gracefully, sending SIGKILL");
kill(g_caching_child_pid, SIGKILL);
waitpid(g_caching_child_pid, &status, 0);
g_caching_child_pid = -1;
return 0;
}
static int caching_service_is_running_fork(void) {
if (g_caching_child_pid <= 0) {
return 0;
}
int status;
pid_t result = waitpid(g_caching_child_pid, &status, WNOHANG);
if (result == 0) {
return 1; // Still running
}
// Process exited
g_caching_child_pid = -1;
return 0;
}
// ---------------------------------------------------------------------------
// Systemd implementation (stub — not yet implemented)
// ---------------------------------------------------------------------------
static int caching_service_start_systemd(void) {
DEBUG_ERROR("caching_service_start: systemd launch mode not yet implemented");
return -1;
}
static int caching_service_stop_systemd(void) {
DEBUG_ERROR("caching_service_stop: systemd launch mode not yet implemented");
return -1;
}
static int caching_service_is_running_systemd(void) {
// Could check systemctl is-active caching-relay
return 0;
}
// ---------------------------------------------------------------------------
// Public API
// ---------------------------------------------------------------------------
int caching_service_start(void) {
const char* launch_mode = get_config_value("caching_service_launch_mode");
if (!launch_mode || strcmp(launch_mode, "fork") == 0) {
const char* binary_path = get_config_value("caching_service_binary_path");
const char* pg_conn = get_config_value("caching_service_pg_conn");
int rc = caching_service_start_fork(binary_path, pg_conn);
if (binary_path) free((char*)binary_path);
if (pg_conn) free((char*)pg_conn);
if (launch_mode) free((char*)launch_mode);
return rc;
}
if (strcmp(launch_mode, "systemd") == 0) {
free((char*)launch_mode);
return caching_service_start_systemd();
}
DEBUG_ERROR("caching_service_start: unknown launch mode '%s'", launch_mode);
free((char*)launch_mode);
return -1;
}
int caching_service_stop(void) {
const char* launch_mode = get_config_value("caching_service_launch_mode");
if (!launch_mode || strcmp(launch_mode, "fork") == 0) {
if (launch_mode) free((char*)launch_mode);
return caching_service_stop_fork();
}
if (strcmp(launch_mode, "systemd") == 0) {
free((char*)launch_mode);
return caching_service_stop_systemd();
}
free((char*)launch_mode);
return -1;
}
int caching_service_is_running(void) {
const char* launch_mode = get_config_value("caching_service_launch_mode");
int rc;
if (!launch_mode || strcmp(launch_mode, "fork") == 0) {
rc = caching_service_is_running_fork();
} else if (strcmp(launch_mode, "systemd") == 0) {
rc = caching_service_is_running_systemd();
} else {
rc = 0;
}
if (launch_mode) free((char*)launch_mode);
return rc;
}
int caching_service_get_pid(void) {
if (caching_service_is_running_fork()) {
return (int)g_caching_child_pid;
}
return -1;
}
int caching_service_shutdown(void) {
// Only stop if we launched it (fork mode with a tracked PID)
if (g_caching_child_pid > 0) {
DEBUG_INFO("caching_service_shutdown: stopping caching service (PID %d)",
g_caching_child_pid);
return caching_service_stop_fork();
}
return 0;
}
+37
View File
@@ -0,0 +1,37 @@
#ifndef CACHING_SERVICE_LAUNCHER_H
#define CACHING_SERVICE_LAUNCHER_H
// Caching service process launcher.
//
// Supports two launch modes controlled by the "caching_service_launch_mode" config key:
// "fork" — c-relay-pg forks/execs the caching_relay binary directly (default)
// "systemd" — delegates to systemctl start/stop (not yet implemented, returns error)
//
// The binary path is read from "caching_service_binary_path".
// The PostgreSQL connection string is read from "caching_service_pg_conn"
// and passed to the binary via the --pg-conn argument.
// Start the caching service.
// Returns 0 on success, -1 on error.
// On success in fork mode, the child PID is tracked internally.
int caching_service_start(void);
// Stop the caching service.
// Returns 0 on success (or if not running), -1 on error.
// In fork mode, sends SIGTERM to the tracked child process.
int caching_service_stop(void);
// Check if the caching service is currently running.
// Returns 1 if running, 0 if not running, -1 on error.
int caching_service_is_running(void);
// Get the current child PID (fork mode only).
// Returns the PID, or -1 if not running or not in fork mode.
int caching_service_get_pid(void);
// Shut down the caching service if it was launched by us.
// Called during c-relay-pg graceful shutdown.
// Returns 0 on success, -1 on error.
int caching_service_shutdown(void);
#endif // CACHING_SERVICE_LAUNCHER_H
+1625 -754
View File
File diff suppressed because it is too large Load Diff
+20 -3
View File
@@ -1,7 +1,6 @@
#ifndef CONFIG_H
#define CONFIG_H
#include <sqlite3.h>
#include <cjson/cJSON.h>
#include <time.h>
#include <pthread.h>
@@ -16,6 +15,9 @@ struct lws;
#define RELAY_URL_MAX_LENGTH 512
#define RELAY_PUBKEY_MAX_LENGTH 65
#define RELAY_CONTACT_MAX_LENGTH 256
#define MAX_ADMIN_PUBKEYS 10
#define ADMIN_PUBKEY_HEX_LEN 64
#define ADMIN_PUBKEY_LIST_MAX_LENGTH ((MAX_ADMIN_PUBKEYS * ADMIN_PUBKEY_HEX_LEN) + (MAX_ADMIN_PUBKEYS - 1) + 1)
#define SUBSCRIPTION_ID_MAX_LENGTH 64
#define CLIENT_IP_MAX_LENGTH 46
#define MAX_SUBSCRIPTIONS_PER_CLIENT 25
@@ -30,10 +32,16 @@ extern char g_database_path[512];
// Command line options structure for first-time startup
typedef struct {
int port_override; // -1 = not set, >0 = port value
char admin_pubkey_override[65]; // Empty string = not set, 64-char hex = override
char admin_pubkey_override[ADMIN_PUBKEY_LIST_MAX_LENGTH]; // Empty string = not set, comma-separated 64-char hex pubkeys
char relay_privkey_override[65]; // Empty string = not set, 64-char hex = override
int strict_port; // 0 = allow port increment, 1 = fail if exact port unavailable
int debug_level; // 0-5, default 0 (no debug output)
char db_connstring_override[1024];
char db_host[128];
int db_port; // 0 = not set
char db_name[128];
char db_user[128];
char db_password[128];
} cli_options_t;
// Core configuration functions (temporary compatibility)
@@ -62,7 +70,7 @@ int get_config_int(const char* key, int default_value);
int get_config_bool(const char* key, int default_value);
// First-time startup functions
int is_first_time_startup(void);
int is_first_time_startup(const char* db_connection_target);
int first_time_startup_sequence(const cli_options_t* cli_options, char* admin_pubkey_out, char* relay_pubkey_out, char* relay_privkey_out);
int startup_existing_relay(const char* relay_pubkey, const cli_options_t* cli_options);
@@ -78,6 +86,7 @@ char* extract_pubkey_from_filename(const char* filename);
int store_relay_private_key(const char* relay_privkey_hex);
char* get_relay_private_key(void);
const char* get_temp_relay_private_key(void); // For first-time startup only
int preload_relay_private_key_cache(void); // Preload relay private key before strict DB mode
// NIP-42 authentication configuration functions
int parse_auth_required_kinds(const char* kinds_str, int* kinds_array, int max_kinds);
@@ -128,6 +137,14 @@ const char* get_relay_pubkey_cached(void);
void invalidate_config_cache(void);
int reload_config_from_table(void);
// Multi-admin helpers
int reload_admin_pubkeys_cache(void);
int is_admin_pubkey(const char* pubkey);
int get_admin_pubkey_count(void);
const char* get_primary_admin_pubkey(void);
int add_admin_pubkey(const char* pubkey);
int remove_admin_pubkey(const char* pubkey);
// Hybrid config access functions
const char* get_config_value_hybrid(const char* key);
int is_config_table_ready(void);
+375
View File
@@ -0,0 +1,375 @@
#define _GNU_SOURCE
#include "db_ops.h"
#include "sqlite_db_ops.h"
#include "db_ops_postgres.h"
#ifdef DB_BACKEND_POSTGRES
int db_init(const char* connection_string) { return postgres_db_init(connection_string); }
void db_close(void) { postgres_db_close(); }
int db_is_available(void) { return postgres_db_is_available(); }
const char* db_last_error(void) { return postgres_db_last_error(); }
const char* db_get_database_path(void) { return postgres_db_get_database_path(); }
int db_set_thread_connection(void* connection) { return postgres_db_set_thread_connection(connection); }
void db_clear_thread_connection(void) { postgres_db_clear_thread_connection(); }
int db_open_worker_connection(const char* db_path, void** out_connection) {
return postgres_db_open_worker_connection(db_path, out_connection);
}
void db_close_worker_connection(void* connection) { postgres_db_close_worker_connection(connection); }
int db_worker_listen(void* connection, const char* channel) {
return postgres_db_worker_listen(connection, channel);
}
int db_worker_poll_notify(void* connection, int timeout_ms) {
return postgres_db_worker_poll_notify(connection, timeout_ms);
}
int db_prepare(const char* sql, db_stmt_t** out_stmt) {
return postgres_db_prepare(sql, (postgres_db_stmt_t**)out_stmt);
}
int db_bind_text_param(db_stmt_t* stmt, int index, const char* value) {
return postgres_db_bind_text_param((postgres_db_stmt_t*)stmt, index, value);
}
int db_bind_int_param(db_stmt_t* stmt, int index, int value) {
return postgres_db_bind_int_param((postgres_db_stmt_t*)stmt, index, value);
}
int db_bind_int64_param(db_stmt_t* stmt, int index, long long value) {
return postgres_db_bind_int64_param((postgres_db_stmt_t*)stmt, index, value);
}
int db_step_stmt(db_stmt_t* stmt) { return postgres_db_step_stmt((postgres_db_stmt_t*)stmt); }
int db_reset_stmt(db_stmt_t* stmt) { return postgres_db_reset_stmt((postgres_db_stmt_t*)stmt); }
const char* db_column_text_value(db_stmt_t* stmt, int col) {
return postgres_db_column_text_value((postgres_db_stmt_t*)stmt, col);
}
int db_column_int_value(db_stmt_t* stmt, int col) {
return postgres_db_column_int_value((postgres_db_stmt_t*)stmt, col);
}
long long db_column_int64_value(db_stmt_t* stmt, int col) {
return postgres_db_column_int64_value((postgres_db_stmt_t*)stmt, col);
}
double db_column_double_value(db_stmt_t* stmt, int col) {
return postgres_db_column_double_value((postgres_db_stmt_t*)stmt, col);
}
void db_finalize_stmt(db_stmt_t* stmt) { postgres_db_finalize_stmt((postgres_db_stmt_t*)stmt); }
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json) {
return postgres_db_log_subscription_created(sub_id, wsi_ptr, client_ip, filter_json);
}
int db_log_subscription_closed(const char* sub_id, const char* client_ip) {
return postgres_db_log_subscription_closed(sub_id, client_ip);
}
int db_log_subscription_disconnected(const char* client_ip) {
return postgres_db_log_subscription_disconnected(client_ip);
}
int db_update_subscription_events_sent(const char* sub_id, int events_sent) {
return postgres_db_update_subscription_events_sent(sub_id, events_sent);
}
int db_cleanup_orphaned_subscriptions(void) { return postgres_db_cleanup_orphaned_subscriptions(); }
int db_get_event_pubkey(const char* event_id, char* pubkey_out, size_t pubkey_out_size) {
return postgres_db_get_event_pubkey(event_id, pubkey_out, pubkey_out_size);
}
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey) {
return postgres_db_delete_event_by_id(event_id, requester_pubkey);
}
int db_delete_events_by_address(const char* pubkey, int kind, const char* d_tag, long before_timestamp) {
return postgres_db_delete_events_by_address(pubkey, kind, d_tag, before_timestamp);
}
int db_is_pubkey_blacklisted(const char* pubkey) { return postgres_db_is_pubkey_blacklisted(pubkey); }
int db_is_hash_blacklisted(const char* resource_hash) { return postgres_db_is_hash_blacklisted(resource_hash); }
int db_is_pubkey_whitelisted(const char* pubkey) { return postgres_db_is_pubkey_whitelisted(pubkey); }
int db_count_active_whitelist_rules(void) { return postgres_db_count_active_whitelist_rules(); }
int db_count_with_sql(const char* sql, const char** bind_params, int bind_param_count, int* out_count) {
return postgres_db_count_with_sql(sql, bind_params, bind_param_count, out_count);
}
char* db_execute_readonly_query_json(const char* query, const char* request_id,
char* error_message, size_t error_size,
int max_rows, int timeout_ms) {
return postgres_db_execute_readonly_query_json(query, request_id, error_message, error_size,
max_rows, timeout_ms);
}
int db_get_total_event_count_ll(long long* out_count) { return postgres_db_get_total_event_count_ll(out_count); }
int db_get_event_count_since(time_t cutoff, long long* out_count) {
return postgres_db_get_event_count_since(cutoff, out_count);
}
int db_get_storage_size_bytes(long long* out_size) {
return postgres_db_get_storage_size_bytes(out_size);
}
cJSON* db_get_event_kind_distribution_rows(long long* out_total_events) {
return postgres_db_get_event_kind_distribution_rows(out_total_events);
}
cJSON* db_get_top_pubkeys_rows(int limit) { return postgres_db_get_top_pubkeys_rows(limit); }
cJSON* db_get_subscription_details_rows(void) { return postgres_db_get_subscription_details_rows(); }
cJSON* db_get_all_config_rows(void) { return postgres_db_get_all_config_rows(); }
char* db_get_config_value_dup(const char* key) { return postgres_db_get_config_value_dup(key); }
int db_set_config_value_full(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart) {
return postgres_db_set_config_value_full(key, value, data_type, description, category, requires_restart);
}
int db_update_config_value_only(const char* key, const char* value) {
return postgres_db_update_config_value_only(key, value);
}
int db_upsert_config_value(const char* key, const char* value, const char* data_type) {
return postgres_db_upsert_config_value(key, value, data_type);
}
int db_store_relay_private_key_hex(const char* relay_privkey_hex) {
return postgres_db_store_relay_private_key_hex(relay_privkey_hex);
}
char* db_get_relay_private_key_hex_dup(void) { return postgres_db_get_relay_private_key_hex_dup(); }
int db_store_config_event(const cJSON* event) { return postgres_db_store_config_event(event); }
int db_insert_event_with_json(const char* id, const char* pubkey, long long created_at,
int kind, const char* event_type, const char* content,
const char* sig, const char* tags_json, const char* event_json,
int* out_step_rc, int* out_extended_errcode) {
return postgres_db_insert_event_with_json(id, pubkey, created_at, kind, event_type, content,
sig, tags_json, event_json, out_step_rc, out_extended_errcode);
}
int db_get_event_time_bounds(long long* out_min_created_at, long long* out_max_created_at) {
return postgres_db_get_event_time_bounds(out_min_created_at, out_max_created_at);
}
int db_event_id_exists(const char* event_id, int* out_exists) {
return postgres_db_event_id_exists(event_id, out_exists);
}
cJSON* db_retrieve_event_by_id(const char* event_id) { return postgres_db_retrieve_event_by_id(event_id); }
char* db_get_latest_event_pubkey_for_kind_dup(int kind) {
return postgres_db_get_latest_event_pubkey_for_kind_dup(kind);
}
int db_get_config_row_count(int* out_count) { return postgres_db_get_config_row_count(out_count); }
int db_store_event_tags_cjson(const char* event_id, const cJSON* tags) {
return postgres_db_store_event_tags_cjson(event_id, tags);
}
int db_populate_event_tags_from_existing(void) { return postgres_db_populate_event_tags_from_existing(); }
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value) {
return postgres_db_add_auth_rule(rule_type, pattern_type, pattern_value);
}
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value) {
return postgres_db_remove_auth_rule(rule_type, pattern_type, pattern_value);
}
int db_delete_wot_whitelist_rules(void) { return postgres_db_delete_wot_whitelist_rules(); }
int db_count_wot_whitelist_rules(void) { return postgres_db_count_wot_whitelist_rules(); }
int db_table_exists(const char* table_name, int* out_exists) {
return postgres_db_table_exists(table_name, out_exists);
}
char* db_get_schema_version_dup(void) { return postgres_db_get_schema_version_dup(); }
int db_exec_sql(const char* sql) { return postgres_db_exec_sql(sql); }
int db_wal_checkpoint_passive(void) { return postgres_db_wal_checkpoint_passive(); }
int db_wal_checkpoint_truncate(void) { return postgres_db_wal_checkpoint_truncate(); }
// Caching relay inbox helpers (PostgreSQL-only).
cJSON* db_caching_inbox_dequeue_batch(int batch_size, int* out_count) {
return postgres_db_caching_inbox_dequeue_batch(batch_size, out_count);
}
int db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count) {
return postgres_db_caching_inbox_pending_counts(out_live_count, out_backfill_count);
}
int db_caching_inbox_oldest_age(int* out_oldest_age_seconds) {
return postgres_db_caching_inbox_oldest_age(out_oldest_age_seconds);
}
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority) {
return postgres_db_caching_inbox_insert(event_id, event_json, source_relay, source_class, priority);
}
#else
int db_init(const char* connection_string) { return sqlite_db_init(connection_string); }
void db_close(void) { sqlite_db_close(); }
int db_is_available(void) { return sqlite_db_is_available(); }
const char* db_last_error(void) { return sqlite_db_last_error(); }
const char* db_get_database_path(void) { return sqlite_db_get_database_path(); }
int db_set_thread_connection(void* connection) { return sqlite_db_set_thread_connection(connection); }
void db_clear_thread_connection(void) { sqlite_db_clear_thread_connection(); }
int db_open_worker_connection(const char* db_path, void** out_connection) {
return sqlite_db_open_worker_connection(db_path, out_connection);
}
void db_close_worker_connection(void* connection) { sqlite_db_close_worker_connection(connection); }
int db_worker_listen(void* connection, const char* channel) {
return sqlite_db_worker_listen(connection, channel);
}
int db_worker_poll_notify(void* connection, int timeout_ms) {
return sqlite_db_worker_poll_notify(connection, timeout_ms);
}
int db_prepare(const char* sql, db_stmt_t** out_stmt) {
return sqlite_db_prepare(sql, (sqlite_db_stmt_t**)out_stmt);
}
int db_bind_text_param(db_stmt_t* stmt, int index, const char* value) {
return sqlite_db_bind_text_param((sqlite_db_stmt_t*)stmt, index, value);
}
int db_bind_int_param(db_stmt_t* stmt, int index, int value) {
return sqlite_db_bind_int_param((sqlite_db_stmt_t*)stmt, index, value);
}
int db_bind_int64_param(db_stmt_t* stmt, int index, long long value) {
return sqlite_db_bind_int64_param((sqlite_db_stmt_t*)stmt, index, value);
}
int db_step_stmt(db_stmt_t* stmt) { return sqlite_db_step_stmt((sqlite_db_stmt_t*)stmt); }
int db_reset_stmt(db_stmt_t* stmt) { return sqlite_db_reset_stmt((sqlite_db_stmt_t*)stmt); }
const char* db_column_text_value(db_stmt_t* stmt, int col) {
return sqlite_db_column_text_value((sqlite_db_stmt_t*)stmt, col);
}
int db_column_int_value(db_stmt_t* stmt, int col) {
return sqlite_db_column_int_value((sqlite_db_stmt_t*)stmt, col);
}
long long db_column_int64_value(db_stmt_t* stmt, int col) {
return sqlite_db_column_int64_value((sqlite_db_stmt_t*)stmt, col);
}
double db_column_double_value(db_stmt_t* stmt, int col) {
return sqlite_db_column_double_value((sqlite_db_stmt_t*)stmt, col);
}
void db_finalize_stmt(db_stmt_t* stmt) { sqlite_db_finalize_stmt((sqlite_db_stmt_t*)stmt); }
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json) {
return sqlite_db_log_subscription_created(sub_id, wsi_ptr, client_ip, filter_json);
}
int db_log_subscription_closed(const char* sub_id, const char* client_ip) {
return sqlite_db_log_subscription_closed(sub_id, client_ip);
}
int db_log_subscription_disconnected(const char* client_ip) {
return sqlite_db_log_subscription_disconnected(client_ip);
}
int db_update_subscription_events_sent(const char* sub_id, int events_sent) {
return sqlite_db_update_subscription_events_sent(sub_id, events_sent);
}
int db_cleanup_orphaned_subscriptions(void) { return sqlite_db_cleanup_orphaned_subscriptions(); }
int db_get_event_pubkey(const char* event_id, char* pubkey_out, size_t pubkey_out_size) {
return sqlite_db_get_event_pubkey(event_id, pubkey_out, pubkey_out_size);
}
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey) {
return sqlite_db_delete_event_by_id(event_id, requester_pubkey);
}
int db_delete_events_by_address(const char* pubkey, int kind, const char* d_tag, long before_timestamp) {
return sqlite_db_delete_events_by_address(pubkey, kind, d_tag, before_timestamp);
}
int db_is_pubkey_blacklisted(const char* pubkey) { return sqlite_db_is_pubkey_blacklisted(pubkey); }
int db_is_hash_blacklisted(const char* resource_hash) { return sqlite_db_is_hash_blacklisted(resource_hash); }
int db_is_pubkey_whitelisted(const char* pubkey) { return sqlite_db_is_pubkey_whitelisted(pubkey); }
int db_count_active_whitelist_rules(void) { return sqlite_db_count_active_whitelist_rules(); }
int db_count_with_sql(const char* sql, const char** bind_params, int bind_param_count, int* out_count) {
return sqlite_db_count_with_sql(sql, bind_params, bind_param_count, out_count);
}
char* db_execute_readonly_query_json(const char* query, const char* request_id,
char* error_message, size_t error_size,
int max_rows, int timeout_ms) {
return sqlite_db_execute_readonly_query_json(query, request_id, error_message, error_size,
max_rows, timeout_ms);
}
int db_get_total_event_count_ll(long long* out_count) { return sqlite_db_get_total_event_count_ll(out_count); }
int db_get_event_count_since(time_t cutoff, long long* out_count) {
return sqlite_db_get_event_count_since(cutoff, out_count);
}
int db_get_storage_size_bytes(long long* out_size) {
return sqlite_db_get_storage_size_bytes(out_size);
}
cJSON* db_get_event_kind_distribution_rows(long long* out_total_events) {
return sqlite_db_get_event_kind_distribution_rows(out_total_events);
}
cJSON* db_get_top_pubkeys_rows(int limit) { return sqlite_db_get_top_pubkeys_rows(limit); }
cJSON* db_get_subscription_details_rows(void) { return sqlite_db_get_subscription_details_rows(); }
cJSON* db_get_all_config_rows(void) { return sqlite_db_get_all_config_rows(); }
char* db_get_config_value_dup(const char* key) { return sqlite_db_get_config_value_dup(key); }
int db_set_config_value_full(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart) {
return sqlite_db_set_config_value_full(key, value, data_type, description, category, requires_restart);
}
int db_update_config_value_only(const char* key, const char* value) {
return sqlite_db_update_config_value_only(key, value);
}
int db_upsert_config_value(const char* key, const char* value, const char* data_type) {
return sqlite_db_upsert_config_value(key, value, data_type);
}
int db_store_relay_private_key_hex(const char* relay_privkey_hex) {
return sqlite_db_store_relay_private_key_hex(relay_privkey_hex);
}
char* db_get_relay_private_key_hex_dup(void) { return sqlite_db_get_relay_private_key_hex_dup(); }
int db_store_config_event(const cJSON* event) { return sqlite_db_store_config_event(event); }
int db_insert_event_with_json(const char* id, const char* pubkey, long long created_at,
int kind, const char* event_type, const char* content,
const char* sig, const char* tags_json, const char* event_json,
int* out_step_rc, int* out_extended_errcode) {
return sqlite_db_insert_event_with_json(id, pubkey, created_at, kind, event_type, content,
sig, tags_json, event_json, out_step_rc, out_extended_errcode);
}
int db_get_event_time_bounds(long long* out_min_created_at, long long* out_max_created_at) {
return sqlite_db_get_event_time_bounds(out_min_created_at, out_max_created_at);
}
int db_event_id_exists(const char* event_id, int* out_exists) {
return sqlite_db_event_id_exists(event_id, out_exists);
}
cJSON* db_retrieve_event_by_id(const char* event_id) { return sqlite_db_retrieve_event_by_id(event_id); }
char* db_get_latest_event_pubkey_for_kind_dup(int kind) {
return sqlite_db_get_latest_event_pubkey_for_kind_dup(kind);
}
int db_get_config_row_count(int* out_count) { return sqlite_db_get_config_row_count(out_count); }
int db_store_event_tags_cjson(const char* event_id, const cJSON* tags) {
return sqlite_db_store_event_tags_cjson(event_id, tags);
}
int db_populate_event_tags_from_existing(void) { return sqlite_db_populate_event_tags_from_existing(); }
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value) {
return sqlite_db_add_auth_rule(rule_type, pattern_type, pattern_value);
}
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value) {
return sqlite_db_remove_auth_rule(rule_type, pattern_type, pattern_value);
}
int db_delete_wot_whitelist_rules(void) { return sqlite_db_delete_wot_whitelist_rules(); }
int db_count_wot_whitelist_rules(void) { return sqlite_db_count_wot_whitelist_rules(); }
int db_table_exists(const char* table_name, int* out_exists) {
return sqlite_db_table_exists(table_name, out_exists);
}
char* db_get_schema_version_dup(void) { return sqlite_db_get_schema_version_dup(); }
int db_exec_sql(const char* sql) { return sqlite_db_exec_sql(sql); }
int db_wal_checkpoint_passive(void) { return sqlite_db_wal_checkpoint_passive(); }
int db_wal_checkpoint_truncate(void) { return sqlite_db_wal_checkpoint_truncate(); }
// Caching relay inbox helpers are PostgreSQL-only. The SQLite backend has no
// caching_event_inbox table, so the dispatch path returns errors/no-ops here
// rather than delegating to sqlite_db_ops stubs.
cJSON* db_caching_inbox_dequeue_batch(int batch_size, int* out_count) {
if (out_count) *out_count = 0;
(void)batch_size;
return NULL;
}
int db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count) {
if (out_live_count) *out_live_count = 0;
if (out_backfill_count) *out_backfill_count = 0;
return DB_ERROR;
}
int db_caching_inbox_oldest_age(int* out_oldest_age_seconds) {
if (out_oldest_age_seconds) *out_oldest_age_seconds = 0;
return DB_ERROR;
}
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority) {
(void)event_id; (void)event_json; (void)source_relay; (void)source_class; (void)priority;
return DB_ERROR;
}
#endif
+140
View File
@@ -0,0 +1,140 @@
#ifndef DB_OPS_H
#define DB_OPS_H
#include <stddef.h>
#include <time.h>
#include <cjson/cJSON.h>
// Generic helpers
int db_init(const char* connection_string);
void db_close(void);
int db_is_available(void);
const char* db_last_error(void);
const char* db_get_database_path(void);
// Per-thread connection override (used by thread pool workers)
int db_set_thread_connection(void* connection);
void db_clear_thread_connection(void);
// Worker/runtime SQLite connection lifecycle (kept internal to db_ops.c)
int db_open_worker_connection(const char* db_path, void** out_connection);
void db_close_worker_connection(void* connection);
// LISTEN/NOTIFY support for worker connections (PostgreSQL-only; SQLite stubs
// return -1 so callers can fall back to timer-based polling).
// db_worker_listen: issue LISTEN <channel> on the given worker connection.
// db_worker_poll_notify: block up to timeout_ms waiting for a notification on
// the given worker connection. Returns 1 if a notification was consumed,
// 0 on timeout, -1 on error/unsupported backend.
int db_worker_listen(void* connection, const char* channel);
int db_worker_poll_notify(void* connection, int timeout_ms);
// DB result codes (backend-agnostic)
#define DB_OK 0
#define DB_ERROR 1
#define DB_CONSTRAINT 19
#define DB_MISUSE 21
#define DB_ROW 100
#define DB_DONE 101
typedef struct db_stmt db_stmt_t;
// Generic statement helpers (Phase 1 migration)
int db_prepare(const char* sql, db_stmt_t** out_stmt);
int db_bind_text_param(db_stmt_t* stmt, int index, const char* value);
int db_bind_int_param(db_stmt_t* stmt, int index, int value);
int db_bind_int64_param(db_stmt_t* stmt, int index, long long value);
int db_step_stmt(db_stmt_t* stmt);
int db_reset_stmt(db_stmt_t* stmt);
const char* db_column_text_value(db_stmt_t* stmt, int col);
int db_column_int_value(db_stmt_t* stmt, int col);
long long db_column_int64_value(db_stmt_t* stmt, int col);
double db_column_double_value(db_stmt_t* stmt, int col);
void db_finalize_stmt(db_stmt_t* stmt);
// Subscription logging
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json);
int db_log_subscription_closed(const char* sub_id, const char* client_ip);
int db_log_subscription_disconnected(const char* client_ip);
int db_update_subscription_events_sent(const char* sub_id, int events_sent);
int db_cleanup_orphaned_subscriptions(void);
// NIP-09 event deletion helpers
int db_get_event_pubkey(const char* event_id, char* pubkey_out, size_t pubkey_out_size);
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
int db_delete_events_by_address(const char* pubkey, int kind,
const char* d_tag, long before_timestamp);
// Auth rule checks for request validator
int db_is_pubkey_blacklisted(const char* pubkey);
int db_is_hash_blacklisted(const char* resource_hash);
int db_is_pubkey_whitelisted(const char* pubkey);
int db_count_active_whitelist_rules(void);
// Generic prepared COUNT helper
int db_count_with_sql(const char* sql, const char** bind_params, int bind_param_count, int* out_count);
char* db_execute_readonly_query_json(const char* query, const char* request_id,
char* error_message, size_t error_size,
int max_rows, int timeout_ms);
// Monitoring/stat helpers (Phase 1 api.c migration)
int db_get_total_event_count_ll(long long* out_count);
int db_get_event_count_since(time_t cutoff, long long* out_count);
int db_get_storage_size_bytes(long long* out_size);
cJSON* db_get_event_kind_distribution_rows(long long* out_total_events);
cJSON* db_get_top_pubkeys_rows(int limit);
cJSON* db_get_subscription_details_rows(void);
// Config helpers
cJSON* db_get_all_config_rows(void);
char* db_get_config_value_dup(const char* key);
int db_set_config_value_full(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart);
int db_update_config_value_only(const char* key, const char* value);
int db_upsert_config_value(const char* key, const char* value, const char* data_type);
int db_store_relay_private_key_hex(const char* relay_privkey_hex);
char* db_get_relay_private_key_hex_dup(void);
int db_store_config_event(const cJSON* event);
// Event storage/timestamp/retrieval helpers
int db_insert_event_with_json(const char* id, const char* pubkey, long long created_at,
int kind, const char* event_type, const char* content,
const char* sig, const char* tags_json, const char* event_json,
int* out_step_rc, int* out_extended_errcode);
int db_get_event_time_bounds(long long* out_min_created_at, long long* out_max_created_at);
int db_event_id_exists(const char* event_id, int* out_exists);
cJSON* db_retrieve_event_by_id(const char* event_id);
char* db_get_latest_event_pubkey_for_kind_dup(int kind);
// Config table helpers
int db_get_config_row_count(int* out_count);
// Event tag helpers
int db_store_event_tags_cjson(const char* event_id, const cJSON* tags);
int db_populate_event_tags_from_existing(void);
// Auth/WoT rule helpers
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int db_delete_wot_whitelist_rules(void);
int db_count_wot_whitelist_rules(void);
// Schema/DDL helpers
int db_table_exists(const char* table_name, int* out_exists);
char* db_get_schema_version_dup(void);
int db_exec_sql(const char* sql);
int db_wal_checkpoint_passive(void);
int db_wal_checkpoint_truncate(void);
// Caching relay inbox helpers (PostgreSQL-only; SQLite path returns error/no-op).
// These back the c-relay-pg inbox poller that destructively dequeues events
// written into the caching_event_inbox table by the external caching app.
cJSON* db_caching_inbox_dequeue_batch(int batch_size, int* out_count);
int db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count);
int db_caching_inbox_oldest_age(int* out_oldest_age_seconds);
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority);
#endif // DB_OPS_H
File diff suppressed because it is too large Load Diff
+109
View File
@@ -0,0 +1,109 @@
#ifndef DB_OPS_POSTGRES_H
#define DB_OPS_POSTGRES_H
#include "db_ops.h"
typedef struct postgres_db_stmt postgres_db_stmt_t;
int postgres_db_init(const char* connection_string);
void postgres_db_close(void);
int postgres_db_is_available(void);
const char* postgres_db_last_error(void);
const char* postgres_db_get_database_path(void);
int postgres_db_apply_schema(void);
int postgres_db_set_thread_connection(void* connection);
void postgres_db_clear_thread_connection(void);
int postgres_db_open_worker_connection(const char* db_path, void** out_connection);
void postgres_db_close_worker_connection(void* connection);
int postgres_db_worker_listen(void* connection, const char* channel);
int postgres_db_worker_poll_notify(void* connection, int timeout_ms);
int postgres_db_prepare(const char* sql, postgres_db_stmt_t** out_stmt);
int postgres_db_bind_text_param(postgres_db_stmt_t* stmt, int index, const char* value);
int postgres_db_bind_int_param(postgres_db_stmt_t* stmt, int index, int value);
int postgres_db_bind_int64_param(postgres_db_stmt_t* stmt, int index, long long value);
int postgres_db_step_stmt(postgres_db_stmt_t* stmt);
int postgres_db_reset_stmt(postgres_db_stmt_t* stmt);
const char* postgres_db_column_text_value(postgres_db_stmt_t* stmt, int col);
int postgres_db_column_int_value(postgres_db_stmt_t* stmt, int col);
long long postgres_db_column_int64_value(postgres_db_stmt_t* stmt, int col);
double postgres_db_column_double_value(postgres_db_stmt_t* stmt, int col);
void postgres_db_finalize_stmt(postgres_db_stmt_t* stmt);
int postgres_db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json);
int postgres_db_log_subscription_closed(const char* sub_id, const char* client_ip);
int postgres_db_log_subscription_disconnected(const char* client_ip);
int postgres_db_update_subscription_events_sent(const char* sub_id, int events_sent);
int postgres_db_cleanup_orphaned_subscriptions(void);
int postgres_db_get_event_pubkey(const char* event_id, char* pubkey_out, size_t pubkey_out_size);
int postgres_db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
int postgres_db_delete_events_by_address(const char* pubkey, int kind,
const char* d_tag, long before_timestamp);
int postgres_db_is_pubkey_blacklisted(const char* pubkey);
int postgres_db_is_hash_blacklisted(const char* resource_hash);
int postgres_db_is_pubkey_whitelisted(const char* pubkey);
int postgres_db_count_active_whitelist_rules(void);
int postgres_db_count_with_sql(const char* sql, const char** bind_params, int bind_param_count, int* out_count);
char* postgres_db_execute_readonly_query_json(const char* query, const char* request_id,
char* error_message, size_t error_size,
int max_rows, int timeout_ms);
int postgres_db_get_total_event_count_ll(long long* out_count);
int postgres_db_get_event_count_since(time_t cutoff, long long* out_count);
int postgres_db_get_storage_size_bytes(long long* out_size);
cJSON* postgres_db_get_event_kind_distribution_rows(long long* out_total_events);
cJSON* postgres_db_get_top_pubkeys_rows(int limit);
cJSON* postgres_db_get_subscription_details_rows(void);
cJSON* postgres_db_get_all_config_rows(void);
char* postgres_db_get_config_value_dup(const char* key);
int postgres_db_set_config_value_full(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart);
int postgres_db_update_config_value_only(const char* key, const char* value);
int postgres_db_upsert_config_value(const char* key, const char* value, const char* data_type);
int postgres_db_store_relay_private_key_hex(const char* relay_privkey_hex);
char* postgres_db_get_relay_private_key_hex_dup(void);
int postgres_db_store_config_event(const cJSON* event);
int postgres_db_insert_event_with_json(const char* id, const char* pubkey, long long created_at,
int kind, const char* event_type, const char* content,
const char* sig, const char* tags_json, const char* event_json,
int* out_step_rc, int* out_extended_errcode);
int postgres_db_get_event_time_bounds(long long* out_min_created_at, long long* out_max_created_at);
int postgres_db_event_id_exists(const char* event_id, int* out_exists);
cJSON* postgres_db_retrieve_event_by_id(const char* event_id);
char* postgres_db_get_latest_event_pubkey_for_kind_dup(int kind);
int postgres_db_get_config_row_count(int* out_count);
int postgres_db_store_event_tags_cjson(const char* event_id, const cJSON* tags);
int postgres_db_populate_event_tags_from_existing(void);
int postgres_db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int postgres_db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int postgres_db_delete_wot_whitelist_rules(void);
int postgres_db_count_wot_whitelist_rules(void);
int postgres_db_table_exists(const char* table_name, int* out_exists);
char* postgres_db_get_schema_version_dup(void);
int postgres_db_exec_sql(const char* sql);
int postgres_db_wal_checkpoint_passive(void);
int postgres_db_wal_checkpoint_truncate(void);
// Caching relay inbox helpers (PostgreSQL-only).
cJSON* postgres_db_caching_inbox_dequeue_batch(int batch_size, int* out_count);
int postgres_db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count);
int postgres_db_caching_inbox_oldest_age(int* out_oldest_age_seconds);
int postgres_db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority);
#endif // DB_OPS_POSTGRES_H
+1428
View File
File diff suppressed because it is too large Load Diff
+37 -1
View File
@@ -128,7 +128,43 @@ static const struct {
// NIP-17 Admin DM Settings
// When false (default), Kind 1059 gift wrap events are stored normally without expensive decryption attempts.
// Enable only if you intend to use NIP-17 DMs to send admin commands to the relay.
{"nip17_admin_enabled", "false"}
{"nip17_admin_enabled", "false"},
// Thread Pool Settings
{"thread_pool_enabled", "true"},
{"thread_pool_readers", "4"},
{"thread_pool_writers", "2"},
{"thread_pool_max_queue_depth", "4096"},
// Caching Relay Settings
// External caching service configuration. All caching_ keys are dynamic (no restart required).
{"caching_enabled", "false"}, // Enable caching inbox consumer
{"caching_config_generation", "0"}, // Configuration generation counter for external service
{"caching_root_npubs", ""}, // Comma-separated root npubs to follow
{"caching_bootstrap_relays", ""}, // Comma-separated bootstrap relay URLs
{"caching_kinds", "1,3,6,10000,30023"}, // Event kinds to cache for non-root follows
{"caching_admin_kinds", "*"}, // Event kinds for root npubs (* = all)
{"caching_live_enabled", "true"}, // Enable live subscriptions
{"caching_live_resubscribe_seconds", "300"}, // Live subscription resubscribe interval
{"caching_backfill_enabled", "true"}, // Enable historical backfill
{"caching_backfill_windows", "86400,604800,2592000,7776000,31536000"}, // Backfill window schedule in seconds
{"caching_backfill_page_size", "50"}, // Initial backfill query page size
{"caching_backfill_tick_interval_ms", "5000"}, // Minimum delay between backfill queries
{"caching_backfill_window_cooldown_seconds", "60"}, // Delay between completed windows
{"caching_follow_graph_refresh_seconds", "600"}, // Kind-3 follow graph refresh interval
{"caching_relay_discovery_refresh_seconds", "3600"}, // Kind-10002 relay discovery refresh interval
{"caching_max_followed_pubkeys", "5000"}, // Maximum followed author count
{"caching_max_upstream_relays", "32"}, // Maximum upstream relay count
{"caching_max_relays_per_pubkey", "5"}, // Maximum outbox relays per author
{"caching_query_timeout_ms", "15000"}, // Upstream query timeout
{"caching_inbox_enabled", "false"}, // Enable inbox poller in c-relay-pg
{"caching_inbox_batch_size", "25"}, // Inbox dequeue batch size
{"caching_inbox_active_poll_ms", "200"}, // Poll interval when inbox has events
{"caching_inbox_idle_poll_ms", "5000"}, // Poll interval when inbox is empty
{"caching_max_event_json_bytes", "65536"}, // Maximum event JSON size for inbox insert
{"caching_service_binary_path", ""}, // Path to caching_relay binary (for fork/exec launch)
{"caching_service_pg_conn", ""}, // PostgreSQL connection string for caching service
{"caching_service_launch_mode", "fork"} // Launch mode: "fork" or "systemd" (systemd not yet implemented)
};
// Number of default configuration values
+4 -17
View File
@@ -5,6 +5,7 @@
#include "config.h"
#include "debug.h"
#include "api.h"
#include "db_ops.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
#include "../nostr_core_lib/nostr_core/nip017.h"
#include "../nostr_core_lib/nostr_core/nip044.h"
@@ -17,9 +18,6 @@
#include <cjson/cJSON.h>
#include <libwebsockets.h>
// External database connection (from main.c)
extern sqlite3* g_db;
// Forward declarations for unified handlers
extern int handle_auth_query_unified(cJSON* event, const char* query_type, char* error_message, size_t error_size, struct lws* wsi);
extern int handle_config_query_unified(cJSON* event, const char* query_type, char* error_message, size_t error_size, struct lws* wsi);
@@ -503,8 +501,7 @@ int process_nip17_admin_command(cJSON* dm_event, char* error_message, size_t err
const char* sender_pubkey = cJSON_GetStringValue(sender_pubkey_obj);
// Check if sender is admin
const char* admin_pubkey = get_config_value("admin_pubkey");
int is_admin = admin_pubkey && strlen(admin_pubkey) > 0 && strcmp(sender_pubkey, admin_pubkey) == 0;
int is_admin = is_admin_pubkey(sender_pubkey);
DEBUG_INFO("DM_ADMIN: Sender pubkey: %.16s... (admin: %s)", sender_pubkey, is_admin ? "YES" : "NO");
@@ -622,8 +619,7 @@ int process_nip17_admin_command(cJSON* dm_event, char* error_message, size_t err
update_config_in_table("nip42_auth_required_subscriptions", "false");
// Clear wot_whitelist rules
const char* delete_sql = "DELETE FROM auth_rules WHERE rule_type = 'wot_whitelist'";
sqlite3_exec(g_db, delete_sql, NULL, NULL, NULL);
db_delete_wot_whitelist_rules();
snprintf(response_msg, sizeof(response_msg),
"🔓 Web of Trust Disabled\n"
@@ -695,16 +691,7 @@ int process_nip17_admin_command(cJSON* dm_event, char* error_message, size_t err
else if (strcmp(wot_cmd, "status") == 0 || strlen(wot_cmd) == 0) {
// Show status
// Count whitelisted pubkeys
sqlite3_stmt* stmt;
const char* count_sql = "SELECT COUNT(*) FROM auth_rules WHERE rule_type = 'wot_whitelist' AND active = 1";
int whitelist_count = 0;
if (sqlite3_prepare_v2(g_db, count_sql, -1, &stmt, NULL) == SQLITE_OK) {
if (sqlite3_step(stmt) == SQLITE_ROW) {
whitelist_count = sqlite3_column_int(stmt, 0);
}
sqlite3_finalize(stmt);
}
int whitelist_count = db_count_wot_whitelist_rules();
const char* level_str;
if (wot_level == 0) level_str = "Off (open relay)";
File diff suppressed because one or more lines are too long
+95 -59
View File
@@ -2,6 +2,8 @@
#include "ip_ban.h"
#include "debug.h"
#include "config.h"
#include "db_ops.h"
#include "thread_pool.h"
#include <string.h>
#include <stdlib.h>
#include <pthread.h>
@@ -100,8 +102,8 @@ void ip_ban_init(void) {
DEBUG_LOG("IP ban table initialized (%d slots)", IP_BAN_TABLE_SIZE);
}
void ip_ban_load_from_db(sqlite3* db) {
if (!db || !g_initialized) return;
void ip_ban_load_from_db(void) {
if (!db_is_available() || !g_initialized) return;
// Create table if it doesn't exist (handles existing databases)
const char* create_sql =
@@ -117,20 +119,18 @@ void ip_ban_load_from_db(sqlite3* db) {
" total_failures INTEGER NOT NULL DEFAULT 0,"
" total_successes INTEGER NOT NULL DEFAULT 0,"
" first_seen INTEGER NOT NULL DEFAULT 0,"
" updated_at INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))"
" updated_at INTEGER NOT NULL DEFAULT 0"
");";
char* err = NULL;
if (sqlite3_exec(db, create_sql, NULL, NULL, &err) != SQLITE_OK) {
DEBUG_ERROR("Failed to create ip_bans table: %s", err ? err : "unknown");
if (err) sqlite3_free(err);
if (db_exec_sql(create_sql) != 0) {
DEBUG_ERROR("Failed to create ip_bans table: %s", db_last_error());
return;
}
// Migration: Add idle_* columns if they don't exist (ignore errors if already exists)
sqlite3_exec(db, "ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0", NULL, NULL, NULL);
sqlite3_exec(db, "ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0", NULL, NULL, NULL);
sqlite3_exec(db, "ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0", NULL, NULL, NULL);
sqlite3_exec(db, "ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0", NULL, NULL, NULL);
(void)db_exec_sql("ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0");
(void)db_exec_sql("ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0");
(void)db_exec_sql("ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0");
(void)db_exec_sql("ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0");
const char* sql =
"SELECT ip, failure_count, ban_count, banned_until, first_failure,"
@@ -139,16 +139,16 @@ void ip_ban_load_from_db(sqlite3* db) {
" idle_failure_count, idle_ban_count, idle_banned_until, idle_first_failure"
" FROM ip_bans";
sqlite3_stmt* stmt;
if (sqlite3_prepare_v2(db, sql, -1, &stmt, NULL) != SQLITE_OK) {
db_stmt_t* stmt;
if (db_prepare(sql, &stmt) != DB_OK) {
DEBUG_ERROR("Failed to prepare ip_bans load query");
return;
}
int loaded = 0;
pthread_mutex_lock(&g_ban_mutex);
while (sqlite3_step(stmt) == SQLITE_ROW) {
const char* ip = (const char*)sqlite3_column_text(stmt, 0);
while (db_step_stmt(stmt) == DB_ROW) {
const char* ip = db_column_text_value(stmt, 0);
if (!ip) continue;
int idx = find_slot(ip);
@@ -158,25 +158,25 @@ void ip_ban_load_from_db(sqlite3* db) {
entry->state = IP_BAN_ACTIVE;
strncpy(entry->ip, ip, sizeof(entry->ip) - 1);
entry->ip[sizeof(entry->ip) - 1] = '\0';
entry->failure_count = sqlite3_column_int(stmt, 1);
entry->ban_count = sqlite3_column_int(stmt, 2);
entry->banned_until = (time_t)sqlite3_column_int64(stmt, 3);
entry->first_failure = (time_t)sqlite3_column_int64(stmt, 4);
entry->has_authed_successfully = sqlite3_column_int(stmt, 5);
entry->last_success_at = (time_t)sqlite3_column_int64(stmt, 6);
entry->total_connections = sqlite3_column_int(stmt, 7);
entry->total_failures = sqlite3_column_int(stmt, 8);
entry->total_successes = sqlite3_column_int(stmt, 9);
entry->first_seen = (time_t)sqlite3_column_int64(stmt, 10);
entry->failure_count = db_column_int_value(stmt, 1);
entry->ban_count = db_column_int_value(stmt, 2);
entry->banned_until = (time_t)db_column_int64_value(stmt, 3);
entry->first_failure = (time_t)db_column_int64_value(stmt, 4);
entry->has_authed_successfully = db_column_int_value(stmt, 5);
entry->last_success_at = (time_t)db_column_int64_value(stmt, 6);
entry->total_connections = db_column_int_value(stmt, 7);
entry->total_failures = db_column_int_value(stmt, 8);
entry->total_successes = db_column_int_value(stmt, 9);
entry->first_seen = (time_t)db_column_int64_value(stmt, 10);
// Load idle tracking fields (default to 0 if columns don't exist yet)
entry->idle_failure_count = sqlite3_column_int(stmt, 11);
entry->idle_ban_count = sqlite3_column_int(stmt, 12);
entry->idle_banned_until = (time_t)sqlite3_column_int64(stmt, 13);
entry->idle_first_failure = (time_t)sqlite3_column_int64(stmt, 14);
entry->idle_failure_count = db_column_int_value(stmt, 11);
entry->idle_ban_count = db_column_int_value(stmt, 12);
entry->idle_banned_until = (time_t)db_column_int64_value(stmt, 13);
entry->idle_first_failure = (time_t)db_column_int64_value(stmt, 14);
loaded++;
}
pthread_mutex_unlock(&g_ban_mutex);
sqlite3_finalize(stmt);
db_finalize_stmt(stmt);
// Count how many are still actively banned
time_t now = time(NULL);
@@ -193,24 +193,40 @@ void ip_ban_load_from_db(sqlite3* db) {
DEBUG_WARN("IP ban table loaded: %d IPs restored (%d still banned)", loaded, still_banned);
}
void ip_ban_save_to_db(sqlite3* db) {
if (!db || !g_initialized) return;
void ip_ban_save_to_db(void) {
if (!db_is_available() || !g_initialized) return;
const char* upsert_sql =
"INSERT OR REPLACE INTO ip_bans"
"INSERT INTO ip_bans"
" (ip, failure_count, ban_count, banned_until, first_failure,"
" has_authed_successfully, last_success_at, total_connections,"
" total_failures, total_successes, first_seen, updated_at,"
" idle_failure_count, idle_ban_count, idle_banned_until, idle_first_failure)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, strftime('%s', 'now'), ?, ?, ?, ?)";
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
" ON CONFLICT(ip) DO UPDATE SET"
" failure_count=excluded.failure_count,"
" ban_count=excluded.ban_count,"
" banned_until=excluded.banned_until,"
" first_failure=excluded.first_failure,"
" has_authed_successfully=excluded.has_authed_successfully,"
" last_success_at=excluded.last_success_at,"
" total_connections=excluded.total_connections,"
" total_failures=excluded.total_failures,"
" total_successes=excluded.total_successes,"
" first_seen=excluded.first_seen,"
" updated_at=excluded.updated_at,"
" idle_failure_count=excluded.idle_failure_count,"
" idle_ban_count=excluded.idle_ban_count,"
" idle_banned_until=excluded.idle_banned_until,"
" idle_first_failure=excluded.idle_first_failure";
sqlite3_stmt* stmt;
if (sqlite3_prepare_v2(db, upsert_sql, -1, &stmt, NULL) != SQLITE_OK) {
db_stmt_t* stmt;
if (db_prepare(upsert_sql, &stmt) != DB_OK) {
DEBUG_ERROR("Failed to prepare ip_bans save query");
return;
}
sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, NULL);
(void)db_exec_sql("BEGIN TRANSACTION");
int saved = 0;
pthread_mutex_lock(&g_ban_mutex);
@@ -218,25 +234,26 @@ void ip_ban_save_to_db(sqlite3* db) {
ip_ban_entry_t* entry = &g_ban_table[i];
if (entry->state != IP_BAN_ACTIVE) continue;
sqlite3_reset(stmt);
sqlite3_bind_text(stmt, 1, entry->ip, -1, SQLITE_STATIC);
sqlite3_bind_int(stmt, 2, entry->failure_count);
sqlite3_bind_int(stmt, 3, entry->ban_count);
sqlite3_bind_int64(stmt, 4, (sqlite3_int64)entry->banned_until);
sqlite3_bind_int64(stmt, 5, (sqlite3_int64)entry->first_failure);
sqlite3_bind_int(stmt, 6, entry->has_authed_successfully);
sqlite3_bind_int64(stmt, 7, (sqlite3_int64)entry->last_success_at);
sqlite3_bind_int(stmt, 8, entry->total_connections);
sqlite3_bind_int(stmt, 9, entry->total_failures);
sqlite3_bind_int(stmt, 10, entry->total_successes);
sqlite3_bind_int64(stmt, 11, (sqlite3_int64)entry->first_seen);
(void)db_reset_stmt(stmt);
db_bind_text_param(stmt, 1, entry->ip);
db_bind_int_param(stmt, 2, entry->failure_count);
db_bind_int_param(stmt, 3, entry->ban_count);
db_bind_int64_param(stmt, 4, (long long)entry->banned_until);
db_bind_int64_param(stmt, 5, (long long)entry->first_failure);
db_bind_int_param(stmt, 6, entry->has_authed_successfully);
db_bind_int64_param(stmt, 7, (long long)entry->last_success_at);
db_bind_int_param(stmt, 8, entry->total_connections);
db_bind_int_param(stmt, 9, entry->total_failures);
db_bind_int_param(stmt, 10, entry->total_successes);
db_bind_int64_param(stmt, 11, (long long)entry->first_seen);
db_bind_int64_param(stmt, 12, (long long)time(NULL));
// Idle tracking fields
sqlite3_bind_int(stmt, 12, entry->idle_failure_count);
sqlite3_bind_int(stmt, 13, entry->idle_ban_count);
sqlite3_bind_int64(stmt, 14, (sqlite3_int64)entry->idle_banned_until);
sqlite3_bind_int64(stmt, 15, (sqlite3_int64)entry->idle_first_failure);
db_bind_int_param(stmt, 13, entry->idle_failure_count);
db_bind_int_param(stmt, 14, entry->idle_ban_count);
db_bind_int64_param(stmt, 15, (long long)entry->idle_banned_until);
db_bind_int64_param(stmt, 16, (long long)entry->idle_first_failure);
if (sqlite3_step(stmt) != SQLITE_DONE) {
if (db_step_stmt(stmt) != DB_DONE) {
DEBUG_WARN("Failed to save ip_ban entry for %s", entry->ip);
} else {
saved++;
@@ -244,14 +261,23 @@ void ip_ban_save_to_db(sqlite3* db) {
}
pthread_mutex_unlock(&g_ban_mutex);
sqlite3_finalize(stmt);
sqlite3_exec(db, "COMMIT", NULL, NULL, NULL);
db_finalize_stmt(stmt);
(void)db_exec_sql("COMMIT");
DEBUG_TRACE("IP ban table saved: %d entries written to DB", saved);
}
// Check if an IP is in the idle_ban_whitelist config (comma-separated list)
static int ip_is_whitelisted(const char* ip) {
if (!ip) {
return 0;
}
// Always trust loopback to avoid local test/dev self-bans.
if (strcmp(ip, "127.0.0.1") == 0 || strcmp(ip, "::1") == 0) {
return 1;
}
const char* whitelist = get_config_value("idle_ban_whitelist");
if (!whitelist || whitelist[0] == '\0') {
if (whitelist) free((char*)whitelist);
@@ -547,7 +573,7 @@ int ip_ban_get_tracked_count(void) {
return count;
}
void ip_ban_log_stats(sqlite3* db) {
void ip_ban_log_stats(void) {
if (!g_initialized) return;
static time_t last_log = 0;
@@ -557,8 +583,18 @@ void ip_ban_log_stats(sqlite3* db) {
last_log = now;
// Save to DB every 5 minutes
if (db) {
ip_ban_save_to_db(db);
if (thread_pool_is_running()) {
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_IP_BAN_SAVE;
thread_pool_status_t rc = thread_pool_submit_write(&job, NULL);
if (rc != THREAD_POOL_STATUS_OK) {
DEBUG_WARN("Failed to queue IP_BAN_SAVE job (%d); saving synchronously", rc);
ip_ban_save_to_db();
}
} else {
ip_ban_save_to_db();
}
pthread_mutex_lock(&g_ban_mutex);
+3 -4
View File
@@ -15,7 +15,6 @@
// auth_fail_ban_duration_sec int default: 300 (initial ban duration, doubles each time)
#include <time.h>
#include <sqlite3.h>
// Maximum number of IPs tracked simultaneously (fixed-size, no malloc)
#define IP_BAN_TABLE_SIZE 4096
@@ -25,11 +24,11 @@ void ip_ban_init(void);
// Load ban state from the ip_bans database table.
// Call after ip_ban_init() and after the database is open.
void ip_ban_load_from_db(sqlite3* db);
void ip_ban_load_from_db(void);
// Save current ban state to the ip_bans database table.
// Called every 5 minutes from the maintenance timer.
void ip_ban_save_to_db(sqlite3* db);
void ip_ban_save_to_db(void);
// Check if an IP is currently banned.
// Returns 1 if banned (connection should be rejected), 0 if allowed.
@@ -57,7 +56,7 @@ void ip_ban_cleanup(void);
// Emit a periodic WARN-level log summary of banned IPs (every 5 minutes).
// Also saves state to DB.
void ip_ban_log_stats(sqlite3* db);
void ip_ban_log_stats(void);
// Get stats for logging/monitoring
int ip_ban_get_banned_count(void);
+1259 -542
View File
File diff suppressed because it is too large Load Diff
+68 -8
View File
@@ -1,5 +1,5 @@
/*
* C-Relay Main Header - Version and Metadata Information
* C-Relay-PG Main Header - Version and Metadata Information
*
* This header contains version information and relay metadata.
* Version macros are auto-updated by the build system.
@@ -11,16 +11,16 @@
// Version information (auto-updated by build system)
// Using CRELAY_ prefix to avoid conflicts with nostr_core_lib VERSION macros
#define CRELAY_VERSION_MAJOR 1
#define CRELAY_VERSION_MINOR 2
#define CRELAY_VERSION_PATCH 55
#define CRELAY_VERSION "v1.2.55"
#define CRELAY_VERSION_MAJOR 2
#define CRELAY_VERSION_MINOR 1
#define CRELAY_VERSION_PATCH 17
#define CRELAY_VERSION "v2.1.17"
// Relay metadata (authoritative source for NIP-11 information)
#define RELAY_NAME "C-Relay"
#define RELAY_DESCRIPTION "High-performance C Nostr relay with SQLite storage"
#define RELAY_NAME "C-Relay-PG"
#define RELAY_DESCRIPTION "High-performance C Nostr relay with PostgreSQL storage"
#define RELAY_CONTACT ""
#define RELAY_SOFTWARE "https://git.laantungir.net/laantungir/c-relay.git"
#define RELAY_SOFTWARE "https://git.laantungir.net/laantungir/c-relay-pg.git"
#define RELAY_VERSION CRELAY_VERSION // Use the same version as the build
#define SUPPORTED_NIPS "1,2,4,9,11,12,13,15,16,20,22,33,40,42,50,70"
#define LANGUAGE_TAGS ""
@@ -28,4 +28,64 @@
#define POSTING_POLICY ""
#define PAYMENTS_URL ""
// Forward declaration to avoid pulling cJSON headers into all includers.
typedef struct cJSON cJSON;
// Forward declaration for libwebsockets struct (used by ingest_event).
struct lws;
// Async REQ handler status used by websocket callback to defer EOSE until worker completion
#define HANDLE_REQ_ASYNC_PENDING (-2)
// Async EVENT storage split:
// - store_event_core(): worker-safe core DB write path
// returns 0 when inserted, 1 when handled without insert (duplicate/ephemeral), -1 on error
// - store_event_post_actions(): main-thread-only follow-up actions (monitoring/tags/WoT sync)
// - store_event(): legacy wrapper preserving original behavior for synchronous call sites
int store_event_core(cJSON* event);
void store_event_post_actions(cJSON* event);
int store_event(cJSON* event);
// Drains completed async REQ jobs and queues EVENT/EOSE on the lws service thread.
void process_req_async_completions(void);
// Source mode for the shared event ingestion entry point.
typedef enum {
EVENT_SOURCE_CLIENT, // Normal WebSocket client event
EVENT_SOURCE_CACHING_INBOX // Event from the caching inbox (internal import)
} event_source_t;
// Process an event through the authoritative validation and storage pipeline.
// This is the shared entry point for both client events and caching inbox events.
//
// For EVENT_SOURCE_CLIENT: behaves exactly as the current client path does.
// For EVENT_SOURCE_CACHING_INBOX:
// - No NIP-42 client authentication requirement
// - No OK response sent (no wsi)
// - No administrator command execution (kind 23456 events are stored as data, not processed)
// - Still validates signature, structure, expiration, PoW
// - Still handles replaceable/addressable/ephemeral/duplicate/NIP-09 semantics
// - Still stores through the writer thread pool
// - Still queues post-actions and broadcast to the main thread
//
// Parameters:
// event_json - raw JSON string of the Nostr event
// event_json_len - length of the JSON string
// source - EVENT_SOURCE_CLIENT or EVENT_SOURCE_CACHING_INBOX
// wsi - WebSocket instance for client responses (NULL for caching inbox)
// pss - per-session data for client responses (NULL for caching inbox)
//
// Returns:
// 0 = accepted and stored (or duplicate, or ephemeral broadcast)
// -1 = rejected (validation failure, storage failure)
int ingest_event(const char* event_json, size_t event_json_len,
event_source_t source, struct lws* wsi, void* pss);
// Drains completed caching-inbox ingestion jobs on the lws service thread.
// Runs store_event_post_actions() and broadcast_event_to_subscriptions() for
// events that were stored off the main thread by ingest_event() with
// EVENT_SOURCE_CACHING_INBOX. Safe to call unconditionally; no-op when empty
// or when the PostgreSQL backend is not in use.
void process_inbox_event_completions(void);
#endif /* MAIN_H */
+33 -89
View File
@@ -7,7 +7,7 @@
/////////////////////////////////////////////////////////////////////////////////////////
#include <cjson/cJSON.h>
#include "debug.h"
#include <sqlite3.h>
#include "db_ops.h"
#include <string.h>
#include <stdlib.h>
#include <time.h>
@@ -21,10 +21,6 @@ int store_event(cJSON* event);
int delete_events_by_id(const char* requester_pubkey, cJSON* event_ids);
int delete_events_by_address(const char* requester_pubkey, cJSON* addresses, long deletion_timestamp);
// Global database variable
extern sqlite3* g_db;
// Handle NIP-09 deletion request event (kind 5)
int handle_deletion_request(cJSON* event, char* error_message, size_t error_size) {
if (!event) {
@@ -146,97 +142,72 @@ int handle_deletion_request(cJSON* event, char* error_message, size_t error_size
// Delete events by ID (with pubkey authorization)
int delete_events_by_id(const char* requester_pubkey, cJSON* event_ids) {
if (!g_db || !requester_pubkey || !event_ids || !cJSON_IsArray(event_ids)) {
if (!db_is_available() || !requester_pubkey || !event_ids || !cJSON_IsArray(event_ids)) {
return 0;
}
int deleted_count = 0;
cJSON* event_id = NULL;
cJSON_ArrayForEach(event_id, event_ids) {
if (!cJSON_IsString(event_id)) {
continue;
}
const char* id = cJSON_GetStringValue(event_id);
// First check if event exists and if requester is authorized
const char* check_sql = "SELECT pubkey FROM events WHERE id = ?";
sqlite3_stmt* check_stmt;
int rc = sqlite3_prepare_v2(g_db, check_sql, -1, &check_stmt, NULL);
if (rc != SQLITE_OK) {
char event_pubkey[65] = {0};
int exists = db_get_event_pubkey(id, event_pubkey, sizeof(event_pubkey));
if (exists <= 0) {
continue;
}
sqlite3_bind_text(check_stmt, 1, id, -1, SQLITE_STATIC);
if (sqlite3_step(check_stmt) == SQLITE_ROW) {
const char* event_pubkey = (char*)sqlite3_column_text(check_stmt, 0);
// Only delete if the requester is the author
if (event_pubkey && strcmp(event_pubkey, requester_pubkey) == 0) {
sqlite3_finalize(check_stmt);
// Delete the event
const char* delete_sql = "DELETE FROM events WHERE id = ? AND pubkey = ?";
sqlite3_stmt* delete_stmt;
rc = sqlite3_prepare_v2(g_db, delete_sql, -1, &delete_stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(delete_stmt, 1, id, -1, SQLITE_STATIC);
sqlite3_bind_text(delete_stmt, 2, requester_pubkey, -1, SQLITE_STATIC);
if (sqlite3_step(delete_stmt) == SQLITE_DONE && sqlite3_changes(g_db) > 0) {
deleted_count++;
}
sqlite3_finalize(delete_stmt);
}
} else {
sqlite3_finalize(check_stmt);
char warning_msg[128];
snprintf(warning_msg, sizeof(warning_msg), "Unauthorized deletion attempt for event: %.16s...", id);
DEBUG_WARN(warning_msg);
// Only delete if the requester is the author
if (strcmp(event_pubkey, requester_pubkey) == 0) {
int changes = db_delete_event_by_id(id, requester_pubkey);
if (changes > 0) {
deleted_count += changes;
}
} else {
sqlite3_finalize(check_stmt);
char warning_msg[128];
snprintf(warning_msg, sizeof(warning_msg), "Unauthorized deletion attempt for event: %.16s...", id);
DEBUG_WARN(warning_msg);
}
}
return deleted_count;
}
// Delete events by addressable reference (kind:pubkey:d-identifier)
int delete_events_by_address(const char* requester_pubkey, cJSON* addresses, long deletion_timestamp) {
if (!g_db || !requester_pubkey || !addresses || !cJSON_IsArray(addresses)) {
if (!db_is_available() || !requester_pubkey || !addresses || !cJSON_IsArray(addresses)) {
return 0;
}
int deleted_count = 0;
cJSON* address = NULL;
cJSON_ArrayForEach(address, addresses) {
if (!cJSON_IsString(address)) {
continue;
}
const char* addr = cJSON_GetStringValue(address);
// Parse address format: kind:pubkey:d-identifier
char* addr_copy = strdup(addr);
if (!addr_copy) continue;
char* kind_str = strtok(addr_copy, ":");
char* pubkey_str = strtok(NULL, ":");
char* d_identifier = strtok(NULL, ":");
if (!kind_str || !pubkey_str) {
free(addr_copy);
continue;
}
int kind = atoi(kind_str);
// Only delete if the requester is the author
if (strcmp(pubkey_str, requester_pubkey) != 0) {
free(addr_copy);
@@ -245,42 +216,15 @@ int delete_events_by_address(const char* requester_pubkey, cJSON* addresses, lon
DEBUG_WARN(warning_msg);
continue;
}
// Build deletion query based on whether we have d-identifier
const char* delete_sql;
sqlite3_stmt* delete_stmt;
if (d_identifier && strlen(d_identifier) > 0) {
// Delete specific addressable event with d-tag
delete_sql = "DELETE FROM events WHERE kind = ? AND pubkey = ? AND created_at <= ? "
"AND json_extract(tags, '$[*]') LIKE '%[\"d\",\"' || ? || '\"]%'";
} else {
// Delete all events of this kind by this author up to deletion timestamp
delete_sql = "DELETE FROM events WHERE kind = ? AND pubkey = ? AND created_at <= ?";
int changes = db_delete_events_by_address(requester_pubkey, kind, d_identifier, deletion_timestamp);
if (changes > 0) {
deleted_count += changes;
}
int rc = sqlite3_prepare_v2(g_db, delete_sql, -1, &delete_stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_int(delete_stmt, 1, kind);
sqlite3_bind_text(delete_stmt, 2, requester_pubkey, -1, SQLITE_STATIC);
sqlite3_bind_int64(delete_stmt, 3, deletion_timestamp);
if (d_identifier && strlen(d_identifier) > 0) {
sqlite3_bind_text(delete_stmt, 4, d_identifier, -1, SQLITE_STATIC);
}
if (sqlite3_step(delete_stmt) == SQLITE_DONE) {
int changes = sqlite3_changes(g_db);
if (changes > 0) {
deleted_count += changes;
}
}
sqlite3_finalize(delete_stmt);
}
free(addr_copy);
}
return deleted_count;
}
+71 -36
View File
@@ -8,7 +8,7 @@
#include <libwebsockets.h>
#include "../nostr_core_lib/cjson/cJSON.h"
#include "config.h"
#include "main.h"
// Forward declarations for configuration functions
const char* get_config_value(const char* key);
@@ -23,6 +23,67 @@ int get_config_bool(const char* key, int default_value);
#define HTTP_STATUS_INTERNAL_SERVER_ERROR 500
#define NIP11_CACHE_TTL_SEC 5
static pthread_mutex_t g_nip11_cache_mutex = PTHREAD_MUTEX_INITIALIZER;
static char* g_nip11_cached_json = NULL;
static size_t g_nip11_cached_json_len = 0;
static time_t g_nip11_cache_expires_at = 0;
cJSON* generate_relay_info_json();
static void invalidate_nip11_cache_locked(void) {
if (g_nip11_cached_json) {
free(g_nip11_cached_json);
g_nip11_cached_json = NULL;
}
g_nip11_cached_json_len = 0;
g_nip11_cache_expires_at = 0;
}
static char* get_nip11_json_cached_dup(size_t* out_len) {
time_t now = time(NULL);
pthread_mutex_lock(&g_nip11_cache_mutex);
if (g_nip11_cached_json && now < g_nip11_cache_expires_at) {
char* copy = strdup(g_nip11_cached_json);
if (copy && out_len) {
*out_len = g_nip11_cached_json_len;
}
pthread_mutex_unlock(&g_nip11_cache_mutex);
return copy;
}
pthread_mutex_unlock(&g_nip11_cache_mutex);
cJSON* info_json = generate_relay_info_json();
if (!info_json) {
return NULL;
}
char* fresh_json = cJSON_Print(info_json);
cJSON_Delete(info_json);
if (!fresh_json) {
return NULL;
}
size_t fresh_len = strlen(fresh_json);
pthread_mutex_lock(&g_nip11_cache_mutex);
invalidate_nip11_cache_locked();
g_nip11_cached_json = strdup(fresh_json);
if (g_nip11_cached_json) {
g_nip11_cached_json_len = fresh_len;
g_nip11_cache_expires_at = now + NIP11_CACHE_TTL_SEC;
}
pthread_mutex_unlock(&g_nip11_cache_mutex);
if (out_len) {
*out_len = fresh_len;
}
return fresh_json;
}
/////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////
// NIP-11 RELAY INFORMATION DOCUMENT
@@ -80,8 +141,9 @@ void init_relay_info() {
// Clean up relay information JSON objects
void cleanup_relay_info() {
// NIP-11 relay information is now generated dynamically from config table
// No cleanup needed - data is fetched directly from database when requested
pthread_mutex_lock(&g_nip11_cache_mutex);
invalidate_nip11_cache_locked();
pthread_mutex_unlock(&g_nip11_cache_mutex);
}
// Generate NIP-11 compliant JSON document
@@ -184,14 +246,14 @@ cJSON* generate_relay_info_json() {
cJSON_AddStringToObject(info, "software", relay_software);
free((char*)relay_software);
} else {
cJSON_AddStringToObject(info, "software", "https://git.laantungir.net/laantungir/c-relay.git");
cJSON_AddStringToObject(info, "software", RELAY_SOFTWARE);
}
if (relay_version && strlen(relay_version) > 0) {
cJSON_AddStringToObject(info, "version", relay_version);
free((char*)relay_version);
} else {
cJSON_AddStringToObject(info, "version", "0.2.0");
cJSON_AddStringToObject(info, "version", RELAY_VERSION);
}
// Add policies
@@ -348,9 +410,10 @@ int handle_nip11_http_request(struct lws* wsi, const char* accept_header) {
return -1; // Close connection
}
// Generate relay information JSON
cJSON* info_json = generate_relay_info_json();
if (!info_json) {
// Generate (or fetch cached) relay information JSON
size_t json_len = 0;
char* json_string = get_nip11_json_cached_dup(&json_len);
if (!json_string) {
DEBUG_ERROR("Failed to generate relay info JSON");
unsigned char buf[LWS_PRE + 256];
unsigned char *p = &buf[LWS_PRE];
@@ -373,34 +436,6 @@ int handle_nip11_http_request(struct lws* wsi, const char* accept_header) {
return -1;
}
char* json_string = cJSON_Print(info_json);
cJSON_Delete(info_json);
if (!json_string) {
DEBUG_ERROR("Failed to serialize relay info JSON");
unsigned char buf[LWS_PRE + 256];
unsigned char *p = &buf[LWS_PRE];
unsigned char *start = p;
unsigned char *end = &buf[sizeof(buf) - 1];
if (lws_add_http_header_status(wsi, HTTP_STATUS_INTERNAL_SERVER_ERROR, &p, end)) {
return -1;
}
if (lws_add_http_header_by_token(wsi, WSI_TOKEN_HTTP_CONTENT_TYPE, (unsigned char*)"text/plain", 10, &p, end)) {
return -1;
}
if (lws_add_http_header_content_length(wsi, 0, &p, end)) {
return -1;
}
if (lws_finalize_http_header(wsi, &p, end)) {
return -1;
}
lws_write(wsi, start, p - start, LWS_WRITE_HTTP_HEADERS);
return -1;
}
size_t json_len = strlen(json_string);
// Allocate session data to manage buffer lifetime across callbacks
struct nip11_session_data* session_data = malloc(sizeof(struct nip11_session_data));
if (!session_data) {
+489
View File
@@ -0,0 +1,489 @@
#ifndef PG_SCHEMA_H
#define PG_SCHEMA_H
#define EMBEDDED_PG_SCHEMA_VERSION "5"
static const char* const EMBEDDED_PG_SCHEMA_SQL =
"-- C-Relay-PG PostgreSQL Schema\n"
"-- Initial PostgreSQL backend schema\n"
"\n"
"BEGIN;\n"
"\n"
"CREATE TABLE IF NOT EXISTS schema_info (\n"
" key TEXT PRIMARY KEY,\n"
" value TEXT NOT NULL,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"CREATE TABLE IF NOT EXISTS events (\n"
" id TEXT PRIMARY KEY,\n"
" pubkey TEXT NOT NULL,\n"
" created_at BIGINT NOT NULL,\n"
" kind INTEGER NOT NULL,\n"
" event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),\n"
" content TEXT NOT NULL,\n"
" sig TEXT NOT NULL,\n"
" tags JSONB NOT NULL DEFAULT '[]'::jsonb,\n"
" d_tag_value TEXT NOT NULL DEFAULT '',\n"
" expires_at BIGINT,\n"
" event_json TEXT NOT NULL,\n"
" first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"ALTER TABLE events\n"
" ADD COLUMN IF NOT EXISTS expires_at BIGINT;\n"
"\n"
"DO $$\n"
"BEGIN\n"
" IF EXISTS (\n"
" SELECT 1\n"
" FROM information_schema.columns\n"
" WHERE table_schema = current_schema()\n"
" AND table_name = 'events'\n"
" AND column_name = 'd_tag_value'\n"
" AND is_generated = 'ALWAYS'\n"
" ) THEN\n"
" DROP INDEX IF EXISTS uq_events_addressable_pubkey_kind_dtag;\n"
" DROP INDEX IF EXISTS idx_events_d_tag_value;\n"
" ALTER TABLE events DROP COLUMN d_tag_value;\n"
" ALTER TABLE events ADD COLUMN d_tag_value TEXT NOT NULL DEFAULT '';\n"
" ELSIF NOT EXISTS (\n"
" SELECT 1\n"
" FROM information_schema.columns\n"
" WHERE table_schema = current_schema()\n"
" AND table_name = 'events'\n"
" AND column_name = 'd_tag_value'\n"
" ) THEN\n"
" ALTER TABLE events ADD COLUMN d_tag_value TEXT NOT NULL DEFAULT '';\n"
" END IF;\n"
"\n"
" -- One-time backfill of d_tag_value from tags JSONB.\n"
" -- Guarded by schema_info version check so it only runs once on upgrade\n"
" -- from v3, NOT on every startup (avoids full-table scan + write on boot).\n"
" IF COALESCE((SELECT value FROM schema_info WHERE key = 'version'), '0') < '4' THEN\n"
" UPDATE events e\n"
" SET d_tag_value = COALESCE((\n"
" SELECT tag->>1\n"
" FROM jsonb_array_elements(e.tags) AS tag\n"
" WHERE jsonb_typeof(tag) = 'array'\n"
" AND jsonb_array_length(tag) >= 2\n"
" AND tag->>0 = 'd'\n"
" LIMIT 1\n"
" ), '')\n"
" WHERE COALESCE(e.d_tag_value, '') = '';\n"
" END IF;\n"
"END\n"
"$$;\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_events_pubkey ON events(pubkey);\n"
"CREATE INDEX IF NOT EXISTS idx_events_kind ON events(kind);\n"
"CREATE INDEX IF NOT EXISTS idx_events_created_at ON events(created_at DESC);\n"
"CREATE INDEX IF NOT EXISTS idx_events_event_type ON events(event_type);\n"
"CREATE INDEX IF NOT EXISTS idx_events_kind_created_at ON events(kind, created_at DESC);\n"
"CREATE INDEX IF NOT EXISTS idx_events_pubkey_created_at ON events(pubkey, created_at DESC);\n"
"CREATE INDEX IF NOT EXISTS idx_events_pubkey_kind ON events(pubkey, kind);\n"
"CREATE INDEX IF NOT EXISTS idx_events_tags_gin ON events USING GIN(tags);\n"
"CREATE INDEX IF NOT EXISTS idx_events_expires_at ON events(expires_at DESC) WHERE expires_at IS NOT NULL;\n"
"CREATE INDEX IF NOT EXISTS idx_events_d_tag_value ON events(d_tag_value) WHERE d_tag_value <> '';\n"
"CREATE UNIQUE INDEX IF NOT EXISTS uq_events_replaceable_pubkey_kind\n"
" ON events(pubkey, kind)\n"
" WHERE event_type = 'replaceable';\n"
"CREATE UNIQUE INDEX IF NOT EXISTS uq_events_addressable_pubkey_kind_dtag\n"
" ON events(pubkey, kind, d_tag_value)\n"
" WHERE event_type = 'addressable';\n"
"\n"
"-- Keep this table for compatibility with existing query paths during migration.\n"
"CREATE TABLE IF NOT EXISTS event_tags (\n"
" event_id TEXT NOT NULL REFERENCES events(id) ON DELETE CASCADE,\n"
" tag_name TEXT NOT NULL,\n"
" tag_value TEXT NOT NULL,\n"
" tag_index INTEGER NOT NULL DEFAULT 0,\n"
" PRIMARY KEY (event_id, tag_name, tag_value, tag_index)\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_event_tags_lookup ON event_tags(tag_name, tag_value);\n"
"CREATE INDEX IF NOT EXISTS idx_event_tags_event ON event_tags(event_id);\n"
"CREATE INDEX IF NOT EXISTS idx_event_tags_value_name ON event_tags(tag_value, tag_name);\n"
"\n"
"CREATE OR REPLACE FUNCTION set_event_derived_fields()\n"
"RETURNS TRIGGER AS $$\n"
"DECLARE\n"
" expiration_value TEXT;\n"
" d_value TEXT;\n"
"BEGIN\n"
" NEW.expires_at := NULL;\n"
" NEW.d_tag_value := '';\n"
"\n"
" SELECT tag->>1\n"
" INTO expiration_value\n"
" FROM jsonb_array_elements(NEW.tags) AS tag\n"
" WHERE jsonb_typeof(tag) = 'array'\n"
" AND jsonb_array_length(tag) >= 2\n"
" AND tag->>0 = 'expiration'\n"
" LIMIT 1;\n"
"\n"
" IF expiration_value IS NOT NULL AND expiration_value ~ '^[0-9]+$' THEN\n"
" NEW.expires_at := expiration_value::BIGINT;\n"
" END IF;\n"
"\n"
"\n"
" SELECT tag->>1\n"
" INTO d_value\n"
" FROM jsonb_array_elements(NEW.tags) AS tag\n"
" WHERE jsonb_typeof(tag) = 'array'\n"
" AND jsonb_array_length(tag) >= 2\n"
" AND tag->>0 = 'd'\n"
" LIMIT 1;\n"
"\n"
" IF d_value IS NOT NULL THEN\n"
" NEW.d_tag_value := d_value;\n"
" END IF;\n"
"\n"
" RETURN NEW;\n"
"END;\n"
"$$ LANGUAGE plpgsql;\n"
"\n"
"CREATE OR REPLACE FUNCTION sync_event_tags_from_events()\n"
"RETURNS TRIGGER AS $$\n"
"BEGIN\n"
" DELETE FROM event_tags WHERE event_id = NEW.id;\n"
"\n"
" INSERT INTO event_tags (event_id, tag_name, tag_value, tag_index)\n"
" SELECT NEW.id,\n"
" tag->>0 AS tag_name,\n"
" tag->>1 AS tag_value,\n"
" (ord - 1)::INTEGER AS tag_index\n"
" FROM jsonb_array_elements(NEW.tags) WITH ORDINALITY AS expanded(tag, ord)\n"
" WHERE jsonb_typeof(tag) = 'array'\n"
" AND jsonb_array_length(tag) >= 2\n"
" AND COALESCE(tag->>0, '') <> ''\n"
" AND COALESCE(tag->>1, '') <> '';\n"
"\n"
" RETURN NEW;\n"
"END;\n"
"$$ LANGUAGE plpgsql;\n"
"\n"
"DROP TRIGGER IF EXISTS trg_events_set_derived_fields ON events;\n"
"CREATE TRIGGER trg_events_set_derived_fields\n"
"BEFORE INSERT OR UPDATE OF tags ON events\n"
"FOR EACH ROW\n"
"EXECUTE FUNCTION set_event_derived_fields();\n"
"\n"
"DROP TRIGGER IF EXISTS trg_events_sync_event_tags ON events;\n"
"CREATE TRIGGER trg_events_sync_event_tags\n"
"AFTER INSERT OR UPDATE OF tags ON events\n"
"FOR EACH ROW\n"
"EXECUTE FUNCTION sync_event_tags_from_events();\n"
"\n"
"CREATE TABLE IF NOT EXISTS relay_seckey (\n"
" id SMALLINT PRIMARY KEY DEFAULT 1,\n"
" private_key_hex TEXT NOT NULL CHECK (char_length(private_key_hex) = 64),\n"
" created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" CONSTRAINT relay_seckey_singleton CHECK (id = 1)\n"
");\n"
"\n"
"CREATE TABLE IF NOT EXISTS auth_rules (\n"
" id BIGSERIAL PRIMARY KEY,\n"
" rule_type TEXT NOT NULL CHECK (rule_type IN ('whitelist', 'blacklist', 'rate_limit', 'auth_required', 'wot_whitelist')),\n"
" pattern_type TEXT NOT NULL CHECK (pattern_type IN ('pubkey', 'kind', 'ip', 'global', 'event_id', 'content', 'hash')),\n"
" pattern_value TEXT,\n"
" active INTEGER NOT NULL DEFAULT 1,\n"
" created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_auth_rules_pattern ON auth_rules(pattern_type, pattern_value);\n"
"CREATE INDEX IF NOT EXISTS idx_auth_rules_type ON auth_rules(rule_type);\n"
"CREATE INDEX IF NOT EXISTS idx_auth_rules_active ON auth_rules(active);\n"
"\n"
"CREATE TABLE IF NOT EXISTS config (\n"
" key TEXT PRIMARY KEY,\n"
" value TEXT NOT NULL,\n"
" data_type TEXT NOT NULL CHECK (data_type IN ('string', 'integer', 'boolean', 'json')),\n"
" description TEXT,\n"
" category TEXT DEFAULT 'general',\n"
" requires_restart INTEGER DEFAULT 0,\n"
" created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_config_category ON config(category);\n"
"CREATE INDEX IF NOT EXISTS idx_config_restart ON config(requires_restart);\n"
"CREATE INDEX IF NOT EXISTS idx_config_updated ON config(updated_at DESC);\n"
"\n"
"CREATE TABLE IF NOT EXISTS subscriptions (\n"
" id BIGSERIAL PRIMARY KEY,\n"
" subscription_id TEXT NOT NULL,\n"
" wsi_pointer TEXT NOT NULL,\n"
" client_ip TEXT NOT NULL,\n"
" event_type TEXT NOT NULL CHECK (event_type IN ('created', 'closed', 'expired', 'disconnected')),\n"
" filter_json TEXT,\n"
" events_sent INTEGER DEFAULT 0,\n"
" created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" ended_at BIGINT,\n"
" duration BIGINT,\n"
" UNIQUE(subscription_id, wsi_pointer)\n"
");\n"
"\n"
"CREATE TABLE IF NOT EXISTS subscription_metrics (\n"
" id BIGSERIAL PRIMARY KEY,\n"
" date TEXT NOT NULL UNIQUE,\n"
" total_created INTEGER DEFAULT 0,\n"
" total_closed INTEGER DEFAULT 0,\n"
" total_events_broadcast INTEGER DEFAULT 0,\n"
" avg_duration DOUBLE PRECISION DEFAULT 0,\n"
" peak_concurrent INTEGER DEFAULT 0,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_id ON subscriptions(subscription_id);\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_type ON subscriptions(event_type);\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_created ON subscriptions(created_at DESC);\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_client ON subscriptions(client_ip);\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_wsi ON subscriptions(wsi_pointer);\n"
"CREATE INDEX IF NOT EXISTS idx_subscriptions_active_log ON subscriptions(event_type, ended_at, created_at DESC);\n"
"CREATE INDEX IF NOT EXISTS idx_subscription_metrics_date ON subscription_metrics(date DESC);\n"
"\n"
"CREATE TABLE IF NOT EXISTS ip_bans (\n"
" ip TEXT PRIMARY KEY,\n"
" failure_count INTEGER NOT NULL DEFAULT 0,\n"
" ban_count INTEGER NOT NULL DEFAULT 0,\n"
" banned_until BIGINT NOT NULL DEFAULT 0,\n"
" first_failure BIGINT NOT NULL DEFAULT 0,\n"
" has_authed_successfully INTEGER NOT NULL DEFAULT 0,\n"
" last_success_at BIGINT NOT NULL DEFAULT 0,\n"
" total_connections INTEGER NOT NULL DEFAULT 0,\n"
" total_failures INTEGER NOT NULL DEFAULT 0,\n"
" total_successes INTEGER NOT NULL DEFAULT 0,\n"
" first_seen BIGINT NOT NULL DEFAULT 0,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" idle_failure_count INTEGER NOT NULL DEFAULT 0,\n"
" idle_ban_count INTEGER NOT NULL DEFAULT 0,\n"
" idle_banned_until BIGINT NOT NULL DEFAULT 0,\n"
" idle_first_failure BIGINT NOT NULL DEFAULT 0\n"
");\n"
"\n"
"-- Helper views for common queries\n"
"CREATE OR REPLACE VIEW recent_events AS\n"
"SELECT id, pubkey, created_at, kind, event_type, content\n"
"FROM events\n"
"WHERE event_type <> 'ephemeral'\n"
"ORDER BY created_at DESC\n"
"LIMIT 1000;\n"
"\n"
"CREATE OR REPLACE VIEW event_stats AS\n"
"SELECT\n"
" event_type,\n"
" COUNT(*) AS count,\n"
" AVG(char_length(content)) AS avg_content_length,\n"
" MIN(created_at) AS earliest,\n"
" MAX(created_at) AS latest\n"
"FROM events\n"
"GROUP BY event_type;\n"
"\n"
"-- Configuration events view (kind 33334)\n"
"CREATE OR REPLACE VIEW configuration_events AS\n"
"SELECT\n"
" id,\n"
" pubkey AS admin_pubkey,\n"
" created_at,\n"
" content,\n"
" tags,\n"
" sig\n"
"FROM events\n"
"WHERE kind = 33334\n"
"ORDER BY created_at DESC;\n"
"\n"
"-- View for subscription analytics\n"
"CREATE OR REPLACE VIEW subscription_analytics AS\n"
"SELECT\n"
" to_timestamp(created_at)::date AS date,\n"
" COUNT(*) AS subscriptions_created,\n"
" COUNT(CASE WHEN ended_at IS NOT NULL THEN 1 END) AS subscriptions_ended,\n"
" AVG(CASE WHEN duration IS NOT NULL THEN duration END) AS avg_duration_seconds,\n"
" MAX(events_sent) AS max_events_sent,\n"
" AVG(events_sent) AS avg_events_sent,\n"
" COUNT(DISTINCT client_ip) AS unique_clients\n"
"FROM subscriptions\n"
"GROUP BY to_timestamp(created_at)::date\n"
"ORDER BY date DESC;\n"
"\n"
"-- View for current active subscriptions (from log perspective)\n"
"CREATE OR REPLACE VIEW active_subscriptions_log AS\n"
"SELECT\n"
" subscription_id,\n"
" client_ip,\n"
" filter_json,\n"
" events_sent,\n"
" created_at,\n"
" (EXTRACT(EPOCH FROM NOW())::BIGINT - created_at) AS duration_seconds,\n"
" wsi_pointer\n"
"FROM subscriptions\n"
"WHERE event_type = 'created'\n"
" AND ended_at IS NULL;\n"
"\n"
"-- Event kinds distribution view\n"
"CREATE OR REPLACE VIEW event_kinds_view AS\n"
"SELECT\n"
" kind,\n"
" COUNT(*) AS count,\n"
" ROUND((COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0))::numeric, 2) AS percentage\n"
"FROM events\n"
"GROUP BY kind\n"
"ORDER BY count DESC;\n"
"\n"
"-- Top pubkeys by event count view\n"
"CREATE OR REPLACE VIEW top_pubkeys_view AS\n"
"SELECT\n"
" pubkey,\n"
" COUNT(*) AS event_count,\n"
" ROUND((COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0))::numeric, 2) AS percentage\n"
"FROM events\n"
"GROUP BY pubkey\n"
"ORDER BY event_count DESC;\n"
"\n"
"-- Time-based statistics view\n"
"CREATE OR REPLACE VIEW time_stats_view AS\n"
"SELECT\n"
" 'total' AS period,\n"
" COUNT(*) AS total_events,\n"
" COUNT(DISTINCT pubkey) AS unique_pubkeys,\n"
" MIN(created_at) AS oldest_event,\n"
" MAX(created_at) AS newest_event\n"
"FROM events\n"
"UNION ALL\n"
"SELECT\n"
" '24h' AS period,\n"
" COUNT(*) AS total_events,\n"
" COUNT(DISTINCT pubkey) AS unique_pubkeys,\n"
" MIN(created_at) AS oldest_event,\n"
" MAX(created_at) AS newest_event\n"
"FROM events\n"
"WHERE created_at >= (EXTRACT(EPOCH FROM NOW())::BIGINT - 86400)\n"
"UNION ALL\n"
"SELECT\n"
" '7d' AS period,\n"
" COUNT(*) AS total_events,\n"
" COUNT(DISTINCT pubkey) AS unique_pubkeys,\n"
" MIN(created_at) AS oldest_event,\n"
" MAX(created_at) AS newest_event\n"
"FROM events\n"
"WHERE created_at >= (EXTRACT(EPOCH FROM NOW())::BIGINT - 604800)\n"
"UNION ALL\n"
"SELECT\n"
" '30d' AS period,\n"
" COUNT(*) AS total_events,\n"
" COUNT(DISTINCT pubkey) AS unique_pubkeys,\n"
" MIN(created_at) AS oldest_event,\n"
" MAX(created_at) AS newest_event\n"
"FROM events\n"
"WHERE created_at >= (EXTRACT(EPOCH FROM NOW())::BIGINT - 2592000);\n"
"\n"
"INSERT INTO schema_info(key, value, updated_at)\n"
"VALUES ('version', '5', EXTRACT(EPOCH FROM NOW())::BIGINT)\n"
"ON CONFLICT (key) DO UPDATE SET\n"
" value = EXCLUDED.value,\n"
" updated_at = EXCLUDED.updated_at;\n"
"\n"
"-- =====================================================================\n"
"-- Caching relay integration tables\n"
"-- These tables are written by an external caching application and\n"
"-- consumed by c-relay-pg. They are schema-only here; the database\n"
"-- abstraction functions are added in a later phase.\n"
"-- =====================================================================\n"
"\n"
"-- Queue table where the external caching application inserts raw event\n"
"-- JSON. c-relay-pg destructively dequeues from it.\n"
"CREATE TABLE IF NOT EXISTS caching_event_inbox (\n"
" queue_id BIGSERIAL PRIMARY KEY,\n"
" event_id TEXT NOT NULL UNIQUE,\n"
" event_json JSONB NOT NULL,\n"
" source_relay TEXT,\n"
" source_class TEXT NOT NULL DEFAULT 'backfill',\n"
" priority SMALLINT NOT NULL DEFAULT 1,\n"
" received_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
"\n"
" CHECK (jsonb_typeof(event_json) = 'object'),\n"
" CHECK (jsonb_typeof(event_json->'id') = 'string'),\n"
" CHECK (length(event_json->>'id') = 64),\n"
" CHECK (event_id = event_json->>'id'),\n"
" CHECK (jsonb_typeof(event_json->'pubkey') = 'string'),\n"
" CHECK (length(event_json->>'pubkey') = 64),\n"
" CHECK (jsonb_typeof(event_json->'sig') = 'string'),\n"
" CHECK (length(event_json->>'sig') = 128),\n"
" CHECK (jsonb_typeof(event_json->'created_at') = 'number'),\n"
" CHECK (jsonb_typeof(event_json->'kind') = 'number'),\n"
" CHECK (jsonb_typeof(event_json->'tags') = 'array'),\n"
" CHECK (jsonb_typeof(event_json->'content') = 'string'),\n"
" CHECK (source_class IN ('live', 'discovery', 'backfill')),\n"
" CHECK (priority IN (0, 1))\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_caching_inbox_dequeue\n"
" ON caching_event_inbox(priority, received_at, queue_id);\n"
"\n"
"-- Singleton status row for the external caching application. The caching\n"
"-- process overwrites this periodically.\n"
"CREATE TABLE IF NOT EXISTS caching_service_state (\n"
" id SMALLINT PRIMARY KEY DEFAULT 1,\n"
" service_version TEXT,\n"
" service_state TEXT NOT NULL DEFAULT 'stopped',\n"
" config_generation BIGINT NOT NULL DEFAULT 0,\n"
" heartbeat_at BIGINT NOT NULL DEFAULT 0,\n"
" followed_author_count INTEGER NOT NULL DEFAULT 0,\n"
" selected_relay_count INTEGER NOT NULL DEFAULT 0,\n"
" connected_relay_count INTEGER NOT NULL DEFAULT 0,\n"
" current_window_index INTEGER NOT NULL DEFAULT 0,\n"
" backfill_cursor INTEGER NOT NULL DEFAULT 0,\n"
" events_fetched BIGINT NOT NULL DEFAULT 0,\n"
" inbox_inserts BIGINT NOT NULL DEFAULT 0,\n"
" last_error TEXT,\n"
" last_error_at BIGINT,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
"\n"
" CHECK (id = 1),\n"
" CHECK (service_state IN ('stopped', 'starting', 'running', 'degraded'))\n"
");\n"
"\n"
"-- Per-author backfill progress for the external caching application.\n"
"CREATE TABLE IF NOT EXISTS caching_backfill_progress (\n"
" author_pubkey TEXT NOT NULL,\n"
" window_index INTEGER NOT NULL,\n"
" window_anchor BIGINT NOT NULL,\n"
" until_cursor BIGINT NOT NULL DEFAULT 0,\n"
" complete BOOLEAN NOT NULL DEFAULT FALSE,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
"\n"
" PRIMARY KEY (author_pubkey, window_index)\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_caching_backfill_progress_window\n"
" ON caching_backfill_progress(window_index)\n"
" WHERE complete = FALSE;\n"
"\n"
"-- =====================================================================\n"
"-- LISTEN/NOTIFY support for api-worker monitoring (Phase 5)\n"
"-- Fires a notification on the 'event_stored' channel whenever a new event\n"
"-- is inserted, so the api-worker thread can wake reactively instead of\n"
"-- polling on a timer. The payload is a small JSON object with kind and a\n"
"-- truncated pubkey prefix (kept tiny to minimize per-insert overhead).\n"
"-- =====================================================================\n"
"CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$\n"
"BEGIN\n"
" PERFORM pg_notify('event_stored', json_build_object(\n"
" 'kind', NEW.kind,\n"
" 'pubkey', substring(NEW.pubkey, 1, 8)\n"
" )::text);\n"
" RETURN NEW;\n"
"END;\n"
"$$ LANGUAGE plpgsql;\n"
"\n"
"DROP TRIGGER IF EXISTS trg_notify_event_stored ON events;\n"
"CREATE TRIGGER trg_notify_event_stored\n"
" AFTER INSERT ON events\n"
" FOR EACH ROW EXECUTE FUNCTION notify_event_stored();\n"
"\n"
"COMMIT;\n"
;
#endif // PG_SCHEMA_H
+363
View File
@@ -0,0 +1,363 @@
-- C-Relay-PG PostgreSQL Schema
-- Initial PostgreSQL backend schema
BEGIN;
CREATE TABLE IF NOT EXISTS schema_info (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE TABLE IF NOT EXISTS events (
id TEXT PRIMARY KEY,
pubkey TEXT NOT NULL,
created_at BIGINT NOT NULL,
kind INTEGER NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
content TEXT NOT NULL,
sig TEXT NOT NULL,
tags JSONB NOT NULL DEFAULT '[]'::jsonb,
d_tag_value TEXT NOT NULL DEFAULT '',
expires_at BIGINT,
event_json TEXT NOT NULL,
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
ALTER TABLE events
ADD COLUMN IF NOT EXISTS expires_at BIGINT;
DO $$
BEGIN
IF EXISTS (
SELECT 1
FROM information_schema.columns
WHERE table_schema = current_schema()
AND table_name = 'events'
AND column_name = 'd_tag_value'
AND is_generated = 'ALWAYS'
) THEN
DROP INDEX IF EXISTS uq_events_addressable_pubkey_kind_dtag;
DROP INDEX IF EXISTS idx_events_d_tag_value;
ALTER TABLE events DROP COLUMN d_tag_value;
ALTER TABLE events ADD COLUMN d_tag_value TEXT NOT NULL DEFAULT '';
ELSIF NOT EXISTS (
SELECT 1
FROM information_schema.columns
WHERE table_schema = current_schema()
AND table_name = 'events'
AND column_name = 'd_tag_value'
) THEN
ALTER TABLE events ADD COLUMN d_tag_value TEXT NOT NULL DEFAULT '';
END IF;
-- One-time backfill of d_tag_value from tags JSONB.
-- Guarded by schema_info version check so it only runs once on upgrade
-- from v3, NOT on every startup (avoids full-table scan + write on boot).
IF COALESCE((SELECT value FROM schema_info WHERE key = 'version'), '0') < '4' THEN
UPDATE events e
SET d_tag_value = COALESCE((
SELECT tag->>1
FROM jsonb_array_elements(e.tags) AS tag
WHERE jsonb_typeof(tag) = 'array'
AND jsonb_array_length(tag) >= 2
AND tag->>0 = 'd'
LIMIT 1
), '')
WHERE COALESCE(e.d_tag_value, '') = '';
END IF;
END
$$;
CREATE INDEX IF NOT EXISTS idx_events_pubkey ON events(pubkey);
CREATE INDEX IF NOT EXISTS idx_events_kind ON events(kind);
CREATE INDEX IF NOT EXISTS idx_events_created_at ON events(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_event_type ON events(event_type);
CREATE INDEX IF NOT EXISTS idx_events_kind_created_at ON events(kind, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_events_pubkey_kind ON events(pubkey, kind);
CREATE INDEX IF NOT EXISTS idx_events_tags_gin ON events USING GIN(tags);
CREATE INDEX IF NOT EXISTS idx_events_expires_at ON events(expires_at DESC) WHERE expires_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_events_d_tag_value ON events(d_tag_value) WHERE d_tag_value <> '';
CREATE UNIQUE INDEX IF NOT EXISTS uq_events_replaceable_pubkey_kind
ON events(pubkey, kind)
WHERE event_type = 'replaceable';
CREATE UNIQUE INDEX IF NOT EXISTS uq_events_addressable_pubkey_kind_dtag
ON events(pubkey, kind, d_tag_value)
WHERE event_type = 'addressable';
-- Keep this table for compatibility with existing query paths during migration.
CREATE TABLE IF NOT EXISTS event_tags (
event_id TEXT NOT NULL REFERENCES events(id) ON DELETE CASCADE,
tag_name TEXT NOT NULL,
tag_value TEXT NOT NULL,
tag_index INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (event_id, tag_name, tag_value, tag_index)
);
CREATE INDEX IF NOT EXISTS idx_event_tags_lookup ON event_tags(tag_name, tag_value);
CREATE INDEX IF NOT EXISTS idx_event_tags_event ON event_tags(event_id);
CREATE INDEX IF NOT EXISTS idx_event_tags_value_name ON event_tags(tag_value, tag_name);
CREATE OR REPLACE FUNCTION set_event_derived_fields()
RETURNS TRIGGER AS $$
DECLARE
expiration_value TEXT;
d_value TEXT;
BEGIN
NEW.expires_at := NULL;
NEW.d_tag_value := '';
SELECT tag->>1
INTO expiration_value
FROM jsonb_array_elements(NEW.tags) AS tag
WHERE jsonb_typeof(tag) = 'array'
AND jsonb_array_length(tag) >= 2
AND tag->>0 = 'expiration'
LIMIT 1;
IF expiration_value IS NOT NULL AND expiration_value ~ '^[0-9]+$' THEN
NEW.expires_at := expiration_value::BIGINT;
END IF;
SELECT tag->>1
INTO d_value
FROM jsonb_array_elements(NEW.tags) AS tag
WHERE jsonb_typeof(tag) = 'array'
AND jsonb_array_length(tag) >= 2
AND tag->>0 = 'd'
LIMIT 1;
IF d_value IS NOT NULL THEN
NEW.d_tag_value := d_value;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION sync_event_tags_from_events()
RETURNS TRIGGER AS $$
BEGIN
DELETE FROM event_tags WHERE event_id = NEW.id;
INSERT INTO event_tags (event_id, tag_name, tag_value, tag_index)
SELECT NEW.id,
tag->>0 AS tag_name,
tag->>1 AS tag_value,
(ord - 1)::INTEGER AS tag_index
FROM jsonb_array_elements(NEW.tags) WITH ORDINALITY AS expanded(tag, ord)
WHERE jsonb_typeof(tag) = 'array'
AND jsonb_array_length(tag) >= 2
AND COALESCE(tag->>0, '') <> ''
AND COALESCE(tag->>1, '') <> '';
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_events_set_derived_fields ON events;
CREATE TRIGGER trg_events_set_derived_fields
BEFORE INSERT OR UPDATE OF tags ON events
FOR EACH ROW
EXECUTE FUNCTION set_event_derived_fields();
DROP TRIGGER IF EXISTS trg_events_sync_event_tags ON events;
CREATE TRIGGER trg_events_sync_event_tags
AFTER INSERT OR UPDATE OF tags ON events
FOR EACH ROW
EXECUTE FUNCTION sync_event_tags_from_events();
CREATE TABLE IF NOT EXISTS relay_seckey (
id SMALLINT PRIMARY KEY DEFAULT 1,
private_key_hex TEXT NOT NULL CHECK (char_length(private_key_hex) = 64),
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
CONSTRAINT relay_seckey_singleton CHECK (id = 1)
);
CREATE TABLE IF NOT EXISTS auth_rules (
id BIGSERIAL PRIMARY KEY,
rule_type TEXT NOT NULL CHECK (rule_type IN ('whitelist', 'blacklist', 'rate_limit', 'auth_required', 'wot_whitelist')),
pattern_type TEXT NOT NULL CHECK (pattern_type IN ('pubkey', 'kind', 'ip', 'global', 'event_id', 'content', 'hash')),
pattern_value TEXT,
active INTEGER NOT NULL DEFAULT 1,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX IF NOT EXISTS idx_auth_rules_pattern ON auth_rules(pattern_type, pattern_value);
CREATE INDEX IF NOT EXISTS idx_auth_rules_type ON auth_rules(rule_type);
CREATE INDEX IF NOT EXISTS idx_auth_rules_active ON auth_rules(active);
CREATE TABLE IF NOT EXISTS config (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
data_type TEXT NOT NULL CHECK (data_type IN ('string', 'integer', 'boolean', 'json')),
description TEXT,
category TEXT DEFAULT 'general',
requires_restart INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX IF NOT EXISTS idx_config_category ON config(category);
CREATE INDEX IF NOT EXISTS idx_config_restart ON config(requires_restart);
CREATE INDEX IF NOT EXISTS idx_config_updated ON config(updated_at DESC);
CREATE TABLE IF NOT EXISTS subscriptions (
id BIGSERIAL PRIMARY KEY,
subscription_id TEXT NOT NULL,
wsi_pointer TEXT NOT NULL,
client_ip TEXT NOT NULL,
event_type TEXT NOT NULL CHECK (event_type IN ('created', 'closed', 'expired', 'disconnected')),
filter_json TEXT,
events_sent INTEGER DEFAULT 0,
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
ended_at BIGINT,
duration BIGINT,
UNIQUE(subscription_id, wsi_pointer)
);
CREATE TABLE IF NOT EXISTS subscription_metrics (
id BIGSERIAL PRIMARY KEY,
date TEXT NOT NULL UNIQUE,
total_created INTEGER DEFAULT 0,
total_closed INTEGER DEFAULT 0,
total_events_broadcast INTEGER DEFAULT 0,
avg_duration DOUBLE PRECISION DEFAULT 0,
peak_concurrent INTEGER DEFAULT 0,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX IF NOT EXISTS idx_subscriptions_id ON subscriptions(subscription_id);
CREATE INDEX IF NOT EXISTS idx_subscriptions_type ON subscriptions(event_type);
CREATE INDEX IF NOT EXISTS idx_subscriptions_created ON subscriptions(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_subscriptions_client ON subscriptions(client_ip);
CREATE INDEX IF NOT EXISTS idx_subscriptions_wsi ON subscriptions(wsi_pointer);
CREATE INDEX IF NOT EXISTS idx_subscriptions_active_log ON subscriptions(event_type, ended_at, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_subscription_metrics_date ON subscription_metrics(date DESC);
CREATE TABLE IF NOT EXISTS ip_bans (
ip TEXT PRIMARY KEY,
failure_count INTEGER NOT NULL DEFAULT 0,
ban_count INTEGER NOT NULL DEFAULT 0,
banned_until BIGINT NOT NULL DEFAULT 0,
first_failure BIGINT NOT NULL DEFAULT 0,
has_authed_successfully INTEGER NOT NULL DEFAULT 0,
last_success_at BIGINT NOT NULL DEFAULT 0,
total_connections INTEGER NOT NULL DEFAULT 0,
total_failures INTEGER NOT NULL DEFAULT 0,
total_successes INTEGER NOT NULL DEFAULT 0,
first_seen BIGINT NOT NULL DEFAULT 0,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
idle_failure_count INTEGER NOT NULL DEFAULT 0,
idle_ban_count INTEGER NOT NULL DEFAULT 0,
idle_banned_until BIGINT NOT NULL DEFAULT 0,
idle_first_failure BIGINT NOT NULL DEFAULT 0
);
INSERT INTO schema_info(key, value, updated_at)
VALUES ('version', '5', EXTRACT(EPOCH FROM NOW())::BIGINT)
ON CONFLICT (key) DO UPDATE SET
value = EXCLUDED.value,
updated_at = EXCLUDED.updated_at;
-- =====================================================================
-- LISTEN/NOTIFY support for api-worker monitoring (Phase 5)
-- Fires a notification on the 'event_stored' channel whenever a new event
-- is inserted, so the api-worker thread can wake reactively instead of
-- polling on a timer. The payload is a small JSON object with kind and a
-- truncated pubkey prefix (kept tiny to minimize per-insert overhead).
-- =====================================================================
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('event_stored', json_build_object(
'kind', NEW.kind,
'pubkey', substring(NEW.pubkey, 1, 8)
)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS trg_notify_event_stored ON events;
CREATE TRIGGER trg_notify_event_stored
AFTER INSERT ON events
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
-- =====================================================================
-- Caching relay integration tables
-- These tables are written by an external caching application and
-- consumed by c-relay-pg. They are schema-only here; the database
-- abstraction functions are added in a later phase.
-- =====================================================================
-- Queue table where the external caching application inserts raw event
-- JSON. c-relay-pg destructively dequeues from it.
CREATE TABLE IF NOT EXISTS caching_event_inbox (
queue_id BIGSERIAL PRIMARY KEY,
event_id TEXT NOT NULL UNIQUE,
event_json JSONB NOT NULL,
source_relay TEXT,
source_class TEXT NOT NULL DEFAULT 'backfill',
priority SMALLINT NOT NULL DEFAULT 1,
received_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
CHECK (jsonb_typeof(event_json) = 'object'),
CHECK (jsonb_typeof(event_json->'id') = 'string'),
CHECK (length(event_json->>'id') = 64),
CHECK (event_id = event_json->>'id'),
CHECK (jsonb_typeof(event_json->'pubkey') = 'string'),
CHECK (length(event_json->>'pubkey') = 64),
CHECK (jsonb_typeof(event_json->'sig') = 'string'),
CHECK (length(event_json->>'sig') = 128),
CHECK (jsonb_typeof(event_json->'created_at') = 'number'),
CHECK (jsonb_typeof(event_json->'kind') = 'number'),
CHECK (jsonb_typeof(event_json->'tags') = 'array'),
CHECK (jsonb_typeof(event_json->'content') = 'string'),
CHECK (source_class IN ('live', 'discovery', 'backfill')),
CHECK (priority IN (0, 1))
);
CREATE INDEX IF NOT EXISTS idx_caching_inbox_dequeue
ON caching_event_inbox(priority, received_at, queue_id);
-- Singleton status row for the external caching application. The caching
-- process overwrites this periodically.
CREATE TABLE IF NOT EXISTS caching_service_state (
id SMALLINT PRIMARY KEY DEFAULT 1,
service_version TEXT,
service_state TEXT NOT NULL DEFAULT 'stopped',
config_generation BIGINT NOT NULL DEFAULT 0,
heartbeat_at BIGINT NOT NULL DEFAULT 0,
followed_author_count INTEGER NOT NULL DEFAULT 0,
selected_relay_count INTEGER NOT NULL DEFAULT 0,
connected_relay_count INTEGER NOT NULL DEFAULT 0,
current_window_index INTEGER NOT NULL DEFAULT 0,
backfill_cursor INTEGER NOT NULL DEFAULT 0,
events_fetched BIGINT NOT NULL DEFAULT 0,
inbox_inserts BIGINT NOT NULL DEFAULT 0,
last_error TEXT,
last_error_at BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
CHECK (id = 1),
CHECK (service_state IN ('stopped', 'starting', 'running', 'degraded'))
);
-- Per-author backfill progress for the external caching application.
CREATE TABLE IF NOT EXISTS caching_backfill_progress (
author_pubkey TEXT NOT NULL,
window_index INTEGER NOT NULL,
window_anchor BIGINT NOT NULL,
until_cursor BIGINT NOT NULL DEFAULT 0,
complete BOOLEAN NOT NULL DEFAULT FALSE,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
PRIMARY KEY (author_pubkey, window_index)
);
CREATE INDEX IF NOT EXISTS idx_caching_backfill_progress_window
ON caching_backfill_progress(window_index)
WHERE complete = FALSE;
COMMIT;
+41 -101
View File
@@ -1,5 +1,5 @@
/*
* C-Relay Request Validator - Integrated Authentication System
* C-Relay-PG Request Validator - Integrated Authentication System
*
* Provides complete request validation including:
* - Protocol validation via nostr_core_lib (signatures, pubkey extraction,
@@ -17,17 +17,13 @@
#include "../nostr_core_lib/nostr_core/utils.h"
#include "debug.h" // C-relay debug system
#include "config.h" // C-relay configuration system
#include <sqlite3.h>
#include "db_ops.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <time.h>
// External references to C-relay global state
extern sqlite3* g_db;
extern char g_database_path[512];
// Forward declaration for C-relay event storage function
extern int store_event(cJSON* event);
@@ -133,6 +129,14 @@ typedef struct {
static nip42_challenge_manager_t g_challenge_manager = {0};
static int g_validator_initialized = 0;
typedef struct {
int has_active_whitelist_rules;
time_t last_refresh;
} auth_rules_fast_cache_t;
static auth_rules_fast_cache_t g_auth_rules_fast_cache = {0};
#define AUTH_RULES_CACHE_TTL_SEC 5
// Last rule violation details for status code mapping
struct {
char violation_type[100]; // "pubkey_blacklist", "hash_blacklist",
@@ -301,14 +305,11 @@ int nostr_validate_unified_request(const char* json_string, size_t json_length)
// 8. Check if this is a kind 23456 admin event from authorized admin
// This must happen AFTER signature validation but BEFORE auth rules
if (event_kind == 23456) {
const char* admin_pubkey = get_config_value("admin_pubkey");
if (admin_pubkey && strcmp(event_pubkey, admin_pubkey) == 0) {
if (is_admin_pubkey(event_pubkey)) {
// Valid admin event - bypass remaining validation
free((char*)admin_pubkey);
cJSON_Delete(event);
return NOSTR_SUCCESS;
}
if (admin_pubkey) free((char*)admin_pubkey);
// Not from admin - continue with normal validation
}
@@ -506,10 +507,10 @@ void nostr_request_result_free_file_data(nostr_request_result_t *result) {
/**
* Force cache refresh - cache no longer exists, function kept for compatibility
* Force cache refresh for auth rule fast cache.
*/
void nostr_request_validator_force_cache_refresh(void) {
// Cache no longer exists - direct database queries are used
g_auth_rules_fast_cache.last_refresh = 0;
}
/**
@@ -524,118 +525,57 @@ static int reload_auth_config(void) {
// Note: Blossom protocol validation removed - C-relay uses standard Nostr events only
static int has_active_whitelist_rules_cached(void) {
time_t now = time(NULL);
if (g_auth_rules_fast_cache.last_refresh == 0 ||
(now - g_auth_rules_fast_cache.last_refresh) >= AUTH_RULES_CACHE_TTL_SEC) {
g_auth_rules_fast_cache.has_active_whitelist_rules =
(db_count_active_whitelist_rules() > 0) ? 1 : 0;
g_auth_rules_fast_cache.last_refresh = now;
}
return g_auth_rules_fast_cache.has_active_whitelist_rules;
}
/**
* Check database authentication rules for the request
* Implements the 6-step rule evaluation engine from AUTH_API.md
*/
int check_database_auth_rules(const char *pubkey, const char *operation __attribute__((unused)),
const char *resource_hash) {
sqlite3 *db = NULL;
sqlite3_stmt *stmt = NULL;
int rc;
DEBUG_TRACE("Checking auth rules for pubkey: %s", pubkey);
if (!pubkey) {
return NOSTR_ERROR_INVALID_INPUT;
}
// Open database using global database path
if (strlen(g_database_path) == 0) {
return NOSTR_SUCCESS; // Default allow on DB error
}
rc = sqlite3_open_v2(g_database_path, &db, SQLITE_OPEN_READONLY, NULL);
if (rc != SQLITE_OK) {
return NOSTR_SUCCESS; // Default allow on DB error
}
// Step 1: Check pubkey blacklist (highest priority)
const char *blacklist_sql =
"SELECT rule_type FROM auth_rules WHERE rule_type = "
"'blacklist' AND pattern_type = 'pubkey' AND pattern_value = ? AND active = 1 LIMIT 1";
DEBUG_TRACE("Blacklist SQL: %s", blacklist_sql);
rc = sqlite3_prepare_v2(db, blacklist_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, pubkey, -1, SQLITE_STATIC);
int step_result = sqlite3_step(stmt);
DEBUG_TRACE("Blacklist query result: %s", step_result == SQLITE_ROW ? "FOUND" : "NOT_FOUND");
if (step_result == SQLITE_ROW) {
DEBUG_TRACE("BLACKLIST HIT: Denying access for pubkey: %s", pubkey);
// Set specific violation details for status code mapping
strcpy(g_last_rule_violation.violation_type, "pubkey_blacklist");
sprintf(g_last_rule_violation.reason, "Public key blacklisted");
sqlite3_finalize(stmt);
sqlite3_close(db);
return NOSTR_ERROR_AUTH_REQUIRED;
}
sqlite3_finalize(stmt);
if (db_is_pubkey_blacklisted(pubkey)) {
DEBUG_TRACE("BLACKLIST HIT: Denying access for pubkey: %s", pubkey);
strcpy(g_last_rule_violation.violation_type, "pubkey_blacklist");
sprintf(g_last_rule_violation.reason, "Public key blacklisted");
return NOSTR_ERROR_AUTH_REQUIRED;
}
// Step 2: Check hash blacklist
if (resource_hash) {
const char *hash_blacklist_sql =
"SELECT rule_type FROM auth_rules WHERE rule_type = "
"'blacklist' AND pattern_type = 'hash' AND pattern_value = ? AND active = 1 LIMIT 1";
rc = sqlite3_prepare_v2(db, hash_blacklist_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, resource_hash, -1, SQLITE_STATIC);
if (sqlite3_step(stmt) == SQLITE_ROW) {
// Set specific violation details for status code mapping
strcpy(g_last_rule_violation.violation_type, "hash_blacklist");
sprintf(g_last_rule_violation.reason, "File hash blacklisted");
sqlite3_finalize(stmt);
sqlite3_close(db);
return NOSTR_ERROR_AUTH_REQUIRED;
}
sqlite3_finalize(stmt);
}
if (resource_hash && db_is_hash_blacklisted(resource_hash)) {
strcpy(g_last_rule_violation.violation_type, "hash_blacklist");
sprintf(g_last_rule_violation.reason, "File hash blacklisted");
return NOSTR_ERROR_AUTH_REQUIRED;
}
// Step 3: Check pubkey whitelist (includes wot_whitelist)
const char *whitelist_sql =
"SELECT rule_type FROM auth_rules WHERE rule_type IN "
"('whitelist', 'wot_whitelist') AND pattern_type = 'pubkey' AND pattern_value = ? AND active = 1 LIMIT 1";
rc = sqlite3_prepare_v2(db, whitelist_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, pubkey, -1, SQLITE_STATIC);
if (sqlite3_step(stmt) == SQLITE_ROW) {
sqlite3_finalize(stmt);
sqlite3_close(db);
return NOSTR_SUCCESS; // Allow whitelisted pubkey
}
sqlite3_finalize(stmt);
if (db_is_pubkey_whitelisted(pubkey)) {
return NOSTR_SUCCESS; // Allow whitelisted pubkey
}
// Step 4: Check if any whitelist rules exist - if yes, deny by default (includes wot_whitelist)
const char *whitelist_exists_sql =
"SELECT COUNT(*) FROM auth_rules WHERE rule_type IN ('whitelist', 'wot_whitelist') "
"AND pattern_type = 'pubkey' AND active = 1 LIMIT 1";
rc = sqlite3_prepare_v2(db, whitelist_exists_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
if (sqlite3_step(stmt) == SQLITE_ROW) {
int whitelist_count = sqlite3_column_int(stmt, 0);
if (whitelist_count > 0) {
// Set specific violation details for status code mapping
strcpy(g_last_rule_violation.violation_type, "whitelist_violation");
strcpy(g_last_rule_violation.reason,
"Public key not whitelisted for this operation");
sqlite3_finalize(stmt);
sqlite3_close(db);
return NOSTR_ERROR_AUTH_REQUIRED;
}
}
sqlite3_finalize(stmt);
// Step 4: If any whitelist rules exist, deny by default
if (has_active_whitelist_rules_cached()) {
strcpy(g_last_rule_violation.violation_type, "whitelist_violation");
strcpy(g_last_rule_violation.reason,
"Public key not whitelisted for this operation");
return NOSTR_ERROR_AUTH_REQUIRED;
}
sqlite3_close(db);
return NOSTR_SUCCESS; // Default allow if no restrictive rules matched
}
+99
View File
@@ -0,0 +1,99 @@
#ifndef SQLITE_DB_OPS_H
#define SQLITE_DB_OPS_H
#include "db_ops.h"
typedef struct sqlite_db_stmt sqlite_db_stmt_t;
int sqlite_db_init(const char* connection_string);
void sqlite_db_close(void);
int sqlite_db_is_available(void);
const char* sqlite_db_last_error(void);
const char* sqlite_db_get_database_path(void);
int sqlite_db_set_thread_connection(void* connection);
void sqlite_db_clear_thread_connection(void);
int sqlite_db_open_worker_connection(const char* db_path, void** out_connection);
void sqlite_db_close_worker_connection(void* connection);
int sqlite_db_worker_listen(void* connection, const char* channel);
int sqlite_db_worker_poll_notify(void* connection, int timeout_ms);
int sqlite_db_prepare(const char* sql, sqlite_db_stmt_t** out_stmt);
int sqlite_db_bind_text_param(sqlite_db_stmt_t* stmt, int index, const char* value);
int sqlite_db_bind_int_param(sqlite_db_stmt_t* stmt, int index, int value);
int sqlite_db_bind_int64_param(sqlite_db_stmt_t* stmt, int index, long long value);
int sqlite_db_step_stmt(sqlite_db_stmt_t* stmt);
int sqlite_db_reset_stmt(sqlite_db_stmt_t* stmt);
const char* sqlite_db_column_text_value(sqlite_db_stmt_t* stmt, int col);
int sqlite_db_column_int_value(sqlite_db_stmt_t* stmt, int col);
long long sqlite_db_column_int64_value(sqlite_db_stmt_t* stmt, int col);
double sqlite_db_column_double_value(sqlite_db_stmt_t* stmt, int col);
void sqlite_db_finalize_stmt(sqlite_db_stmt_t* stmt);
int sqlite_db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
const char* client_ip, const char* filter_json);
int sqlite_db_log_subscription_closed(const char* sub_id, const char* client_ip);
int sqlite_db_log_subscription_disconnected(const char* client_ip);
int sqlite_db_update_subscription_events_sent(const char* sub_id, int events_sent);
int sqlite_db_cleanup_orphaned_subscriptions(void);
int sqlite_db_get_event_pubkey(const char* event_id, char* pubkey_out, size_t pubkey_out_size);
int sqlite_db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
int sqlite_db_delete_events_by_address(const char* pubkey, int kind,
const char* d_tag, long before_timestamp);
int sqlite_db_is_pubkey_blacklisted(const char* pubkey);
int sqlite_db_is_hash_blacklisted(const char* resource_hash);
int sqlite_db_is_pubkey_whitelisted(const char* pubkey);
int sqlite_db_count_active_whitelist_rules(void);
int sqlite_db_count_with_sql(const char* sql, const char** bind_params, int bind_param_count, int* out_count);
char* sqlite_db_execute_readonly_query_json(const char* query, const char* request_id,
char* error_message, size_t error_size,
int max_rows, int timeout_ms);
int sqlite_db_get_total_event_count_ll(long long* out_count);
int sqlite_db_get_event_count_since(time_t cutoff, long long* out_count);
int sqlite_db_get_storage_size_bytes(long long* out_size);
cJSON* sqlite_db_get_event_kind_distribution_rows(long long* out_total_events);
cJSON* sqlite_db_get_top_pubkeys_rows(int limit);
cJSON* sqlite_db_get_subscription_details_rows(void);
cJSON* sqlite_db_get_all_config_rows(void);
char* sqlite_db_get_config_value_dup(const char* key);
int sqlite_db_set_config_value_full(const char* key, const char* value, const char* data_type,
const char* description, const char* category, int requires_restart);
int sqlite_db_update_config_value_only(const char* key, const char* value);
int sqlite_db_upsert_config_value(const char* key, const char* value, const char* data_type);
int sqlite_db_store_relay_private_key_hex(const char* relay_privkey_hex);
char* sqlite_db_get_relay_private_key_hex_dup(void);
int sqlite_db_store_config_event(const cJSON* event);
int sqlite_db_insert_event_with_json(const char* id, const char* pubkey, long long created_at,
int kind, const char* event_type, const char* content,
const char* sig, const char* tags_json, const char* event_json,
int* out_step_rc, int* out_extended_errcode);
int sqlite_db_get_event_time_bounds(long long* out_min_created_at, long long* out_max_created_at);
int sqlite_db_event_id_exists(const char* event_id, int* out_exists);
cJSON* sqlite_db_retrieve_event_by_id(const char* event_id);
char* sqlite_db_get_latest_event_pubkey_for_kind_dup(int kind);
int sqlite_db_get_config_row_count(int* out_count);
int sqlite_db_store_event_tags_cjson(const char* event_id, const cJSON* tags);
int sqlite_db_populate_event_tags_from_existing(void);
int sqlite_db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int sqlite_db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
int sqlite_db_delete_wot_whitelist_rules(void);
int sqlite_db_count_wot_whitelist_rules(void);
int sqlite_db_table_exists(const char* table_name, int* out_exists);
char* sqlite_db_get_schema_version_dup(void);
int sqlite_db_exec_sql(const char* sql);
int sqlite_db_wal_checkpoint_passive(void);
int sqlite_db_wal_checkpoint_truncate(void);
#endif // SQLITE_DB_OPS_H
+193 -126
View File
@@ -1,7 +1,8 @@
#define _GNU_SOURCE
#include <cjson/cJSON.h>
#include "debug.h"
#include <sqlite3.h>
#include "db_ops.h"
#include "thread_pool.h"
#include <string.h>
#include <stdlib.h>
#include <time.h>
@@ -10,6 +11,34 @@
#include <libwebsockets.h>
#include "subscriptions.h"
static void free_sub_log_created_payload_local(thread_pool_sub_log_created_payload_t* payload) {
if (!payload) return;
free(payload->sub_id);
free(payload->wsi_ptr);
free(payload->client_ip);
free(payload->filter_json);
free(payload);
}
static void free_sub_log_closed_payload_local(thread_pool_sub_log_closed_payload_t* payload) {
if (!payload) return;
free(payload->sub_id);
free(payload->client_ip);
free(payload);
}
static void free_sub_log_disconnected_payload_local(thread_pool_sub_log_disconnected_payload_t* payload) {
if (!payload) return;
free(payload->client_ip);
free(payload);
}
static void free_sub_update_events_payload_local(thread_pool_sub_update_events_payload_t* payload) {
if (!payload) return;
free(payload->sub_id);
free(payload);
}
// Forward declarations for logging functions
// Forward declarations for configuration functions
@@ -25,12 +54,6 @@ int validate_timestamp_range(long since, long until, char* error_message, size_t
int validate_numeric_limits(int limit, char* error_message, size_t error_size);
int validate_search_term(const char* search_term, char* error_message, size_t error_size);
// Forward declaration for monitoring function
void monitoring_on_subscription_change(void);
// Global database variable
extern sqlite3* g_db;
// Configuration functions from config.c
extern int get_config_bool(const char* key, int default_value);
@@ -453,9 +476,6 @@ int add_subscription_to_manager(subscription_t* sub) {
// Log subscription creation to database (INSERT OR REPLACE handles duplicates)
log_subscription_created(sub);
// Trigger monitoring update for subscription changes
monitoring_on_subscription_change();
return 0;
}
@@ -506,9 +526,6 @@ int remove_subscription_from_manager(const char* sub_id, struct lws* wsi) {
// Update events sent counter before freeing
update_subscription_events_sent(sub_id_copy, events_sent_copy);
// Trigger monitoring update for subscription changes
monitoring_on_subscription_change();
free_subscription(sub);
return 0;
}
@@ -982,6 +999,39 @@ int has_subscriptions_for_kind(int event_kind) {
return 0; // No matching subscriptions
}
// Check if any active subscription would receive a specific event kind.
// Unlike has_subscriptions_for_kind(), this also returns 1 when there are
// active subscriptions with NO kind filter (they match every kind), which
// is required for the api-worker's "is anyone listening to kind 24567" gate.
int has_any_subscription_for_kind(int event_kind) {
pthread_mutex_lock(&g_subscription_manager.subscriptions_lock);
// 1) Subscriptions with an explicit kind filter matching event_kind.
if (event_kind >= 0 && event_kind <= 65535) {
kind_subscription_node_t* node = g_subscription_manager.kind_index[event_kind];
while (node) {
if (node->subscription && node->subscription->active) {
pthread_mutex_unlock(&g_subscription_manager.subscriptions_lock);
return 1;
}
node = node->next;
}
}
// 2) Subscriptions with no kind filter (match all kinds).
no_kind_filter_node_t* nk = g_subscription_manager.no_kind_filter_subs;
while (nk) {
if (nk->subscription && nk->subscription->active) {
pthread_mutex_unlock(&g_subscription_manager.subscriptions_lock);
return 1;
}
nk = nk->next;
}
pthread_mutex_unlock(&g_subscription_manager.subscriptions_lock);
return 0;
}
/////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////
@@ -991,21 +1041,21 @@ int has_subscriptions_for_kind(int event_kind) {
// Log subscription creation to database
void log_subscription_created(const subscription_t* sub) {
if (!g_db || !sub) return;
if (!sub) return;
// Convert wsi pointer to string
char wsi_str[32];
snprintf(wsi_str, sizeof(wsi_str), "%p", (void*)sub->wsi);
// Create filter JSON for logging
char* filter_json = NULL;
if (sub->filters) {
cJSON* filters_array = cJSON_CreateArray();
subscription_filter_t* filter = sub->filters;
while (filter) {
cJSON* filter_obj = cJSON_CreateObject();
if (filter->kinds) {
cJSON_AddItemToObject(filter_obj, "kinds", cJSON_Duplicate(filter->kinds, 1));
}
@@ -1034,99 +1084,119 @@ void log_subscription_created(const subscription_t* sub) {
}
cJSON_Delete(tags_obj);
}
cJSON_AddItemToArray(filters_array, filter_obj);
filter = filter->next;
}
filter_json = cJSON_Print(filters_array);
cJSON_Delete(filters_array);
}
// Use INSERT OR REPLACE to handle duplicates automatically
const char* sql =
"INSERT OR REPLACE INTO subscriptions (subscription_id, wsi_pointer, client_ip, event_type, filter_json) "
"VALUES (?, ?, ?, 'created', ?)";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, sub->id, -1, SQLITE_STATIC);
sqlite3_bind_text(stmt, 2, wsi_str, -1, SQLITE_TRANSIENT);
sqlite3_bind_text(stmt, 3, sub->client_ip, -1, SQLITE_STATIC);
sqlite3_bind_text(stmt, 4, filter_json ? filter_json : "[]", -1, SQLITE_TRANSIENT);
sqlite3_step(stmt);
sqlite3_finalize(stmt);
if (!thread_pool_is_running()) {
db_log_subscription_created(sub->id, wsi_str, sub->client_ip, filter_json ? filter_json : "[]");
if (filter_json) free(filter_json);
return;
}
thread_pool_sub_log_created_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) {
if (filter_json) free(filter_json);
return;
}
payload->sub_id = strdup(sub->id);
payload->wsi_ptr = strdup(wsi_str);
payload->client_ip = strdup(sub->client_ip);
payload->filter_json = strdup(filter_json ? filter_json : "[]");
if (!payload->sub_id || !payload->wsi_ptr || !payload->client_ip || !payload->filter_json) {
free(payload->sub_id);
free(payload->wsi_ptr);
free(payload->client_ip);
free(payload->filter_json);
free(payload);
if (filter_json) free(filter_json);
return;
}
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_LOG_SUB_CREATED;
job.payload = payload;
job.payload_free = (thread_pool_payload_free_cb)free_sub_log_created_payload_local;
thread_pool_status_t submit_rc = thread_pool_submit_write(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_sub_log_created_payload_local(payload);
}
if (filter_json) free(filter_json);
}
// Log subscription closure to database
void log_subscription_closed(const char* sub_id, const char* client_ip, const char* reason) {
(void)reason; // Mark as intentionally unused
if (!g_db || !sub_id) return;
const char* sql =
"INSERT INTO subscriptions (subscription_id, wsi_pointer, client_ip, event_type) "
"VALUES (?, '', ?, 'closed')";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, sub_id, -1, SQLITE_STATIC);
sqlite3_bind_text(stmt, 2, client_ip ? client_ip : "unknown", -1, SQLITE_STATIC);
sqlite3_step(stmt);
sqlite3_finalize(stmt);
if (!sub_id) return;
if (!thread_pool_is_running()) {
db_log_subscription_closed(sub_id, client_ip);
return;
}
// Update the corresponding 'created' entry with end time and events sent
const char* update_sql =
"UPDATE subscriptions "
"SET ended_at = strftime('%s', 'now') "
"WHERE subscription_id = ? AND event_type = 'created' AND ended_at IS NULL";
rc = sqlite3_prepare_v2(g_db, update_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, sub_id, -1, SQLITE_STATIC);
sqlite3_step(stmt);
sqlite3_finalize(stmt);
thread_pool_sub_log_closed_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) {
return;
}
payload->sub_id = strdup(sub_id);
payload->client_ip = strdup(client_ip ? client_ip : "unknown");
if (!payload->sub_id || !payload->client_ip) {
free_sub_log_closed_payload_local(payload);
return;
}
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_LOG_SUB_CLOSED;
job.payload = payload;
job.payload_free = (thread_pool_payload_free_cb)free_sub_log_closed_payload_local;
thread_pool_status_t submit_rc = thread_pool_submit_write(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_sub_log_closed_payload_local(payload);
}
}
// Log subscription disconnection to database
void log_subscription_disconnected(const char* client_ip) {
if (!g_db || !client_ip) return;
// Mark all active subscriptions for this client as disconnected
const char* sql =
"UPDATE subscriptions "
"SET ended_at = strftime('%s', 'now') "
"WHERE client_ip = ? AND event_type = 'created' AND ended_at IS NULL";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, client_ip, -1, SQLITE_STATIC);
int changes = sqlite3_changes(g_db);
sqlite3_step(stmt);
sqlite3_finalize(stmt);
if (changes > 0) {
// Log a disconnection event
const char* insert_sql =
"INSERT INTO subscriptions (subscription_id, wsi_pointer, client_ip, event_type) "
"VALUES ('disconnect', '', ?, 'disconnected')";
rc = sqlite3_prepare_v2(g_db, insert_sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_text(stmt, 1, client_ip, -1, SQLITE_STATIC);
sqlite3_step(stmt);
sqlite3_finalize(stmt);
}
}
if (!client_ip) return;
if (!thread_pool_is_running()) {
db_log_subscription_disconnected(client_ip);
return;
}
thread_pool_sub_log_disconnected_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) {
return;
}
payload->client_ip = strdup(client_ip);
if (!payload->client_ip) {
free_sub_log_disconnected_payload_local(payload);
return;
}
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_LOG_SUB_DISCONNECTED;
job.payload = payload;
job.payload_free = (thread_pool_payload_free_cb)free_sub_log_disconnected_payload_local;
thread_pool_status_t submit_rc = thread_pool_submit_write(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_sub_log_disconnected_payload_local(payload);
}
}
@@ -1153,55 +1223,52 @@ void log_subscription_disconnected(const char* client_ip) {
// Update events sent counter for a subscription
void update_subscription_events_sent(const char* sub_id, int events_sent) {
if (!g_db || !sub_id) return;
if (!sub_id) return;
const char* sql =
"UPDATE subscriptions "
"SET events_sent = ? "
"WHERE subscription_id = ? AND event_type = 'created'";
if (!thread_pool_is_running()) {
db_update_subscription_events_sent(sub_id, events_sent);
return;
}
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc == SQLITE_OK) {
sqlite3_bind_int(stmt, 1, events_sent);
sqlite3_bind_text(stmt, 2, sub_id, -1, SQLITE_STATIC);
thread_pool_sub_update_events_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) {
return;
}
sqlite3_step(stmt);
sqlite3_finalize(stmt);
payload->sub_id = strdup(sub_id);
payload->events_sent = events_sent;
if (!payload->sub_id) {
free_sub_update_events_payload_local(payload);
return;
}
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_UPDATE_SUB_EVENTS_SENT;
job.payload = payload;
job.payload_free = (thread_pool_payload_free_cb)free_sub_update_events_payload_local;
thread_pool_status_t submit_rc = thread_pool_submit_write(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_sub_update_events_payload_local(payload);
}
}
// Cleanup all subscriptions on startup
void cleanup_all_subscriptions_on_startup(void) {
if (!g_db) {
if (!db_is_available()) {
DEBUG_ERROR("Database not available for startup cleanup");
return;
}
DEBUG_LOG("Performing startup subscription cleanup");
// Mark all active subscriptions as disconnected
const char* sql =
"UPDATE subscriptions "
"SET ended_at = strftime('%s', 'now') "
"WHERE event_type = 'created' AND ended_at IS NULL";
sqlite3_stmt* stmt;
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
if (rc != SQLITE_OK) {
DEBUG_ERROR("Failed to prepare startup cleanup query");
return;
}
rc = sqlite3_step(stmt);
int changes = sqlite3_changes(g_db);
sqlite3_finalize(stmt);
if (rc != SQLITE_DONE) {
int changes = db_cleanup_orphaned_subscriptions();
if (changes < 0) {
DEBUG_ERROR("Failed to execute startup cleanup");
return;
}
if (changes > 0) {
DEBUG_LOG("Startup cleanup: marked %d orphaned subscriptions as disconnected", changes);
} else {
+2 -1
View File
@@ -1,4 +1,4 @@
// Subscription system structures and functions for C-Relay
// Subscription system structures and functions for C-Relay-PG
// This header defines subscription management functionality
#ifndef SUBSCRIPTIONS_H
@@ -140,6 +140,7 @@ void update_subscription_events_sent(const char* sub_id, int events_sent);
// Subscription query functions
int has_subscriptions_for_kind(int event_kind);
int has_any_subscription_for_kind(int event_kind);
// Startup cleanup function
void cleanup_all_subscriptions_on_startup(void);
+986
View File
@@ -0,0 +1,986 @@
#define _GNU_SOURCE
#include "thread_pool.h"
#include "debug.h"
#include "db_ops.h"
#include "ip_ban.h"
#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct thread_pool_job_node {
uint64_t job_id;
thread_pool_job_t job;
struct thread_pool_job_node* next;
} thread_pool_job_node_t;
typedef struct {
thread_pool_job_node_t* head;
thread_pool_job_node_t* tail;
int size;
int max_size;
pthread_mutex_t mutex;
pthread_cond_t cond;
} thread_pool_queue_t;
typedef struct {
int running;
uint64_t next_job_id;
int reader_count;
pthread_t* readers;
int writer_count;
pthread_t* writers;
thread_pool_queue_t read_q;
thread_pool_queue_t write_q;
thread_pool_wake_loop_cb wake_loop_cb;
void* wake_loop_ctx;
char db_path[512];
pthread_mutex_t state_mutex;
} thread_pool_state_t;
static thread_pool_state_t g_pool = {0};
typedef struct {
pthread_mutex_t mutex;
pthread_cond_t cond;
int done;
thread_pool_status_t status;
void* result_data;
size_t result_size;
} thread_pool_wait_ctx_t;
static void wait_ctx_init(thread_pool_wait_ctx_t* ctx) {
memset(ctx, 0, sizeof(*ctx));
pthread_mutex_init(&ctx->mutex, NULL);
pthread_cond_init(&ctx->cond, NULL);
}
static void wait_ctx_destroy(thread_pool_wait_ctx_t* ctx) {
pthread_mutex_destroy(&ctx->mutex);
pthread_cond_destroy(&ctx->cond);
}
static void wait_ctx_result_cb(const thread_pool_result_t* result, void* user_ctx) {
thread_pool_wait_ctx_t* ctx = (thread_pool_wait_ctx_t*)user_ctx;
if (!ctx || !result) return;
pthread_mutex_lock(&ctx->mutex);
ctx->status = result->status;
ctx->result_data = result->result_data;
ctx->result_size = result->result_size;
ctx->done = 1;
pthread_cond_signal(&ctx->cond);
pthread_mutex_unlock(&ctx->mutex);
}
static void wait_ctx_wait(thread_pool_wait_ctx_t* ctx) {
pthread_mutex_lock(&ctx->mutex);
while (!ctx->done) {
pthread_cond_wait(&ctx->cond, &ctx->mutex);
}
pthread_mutex_unlock(&ctx->mutex);
}
static void free_count_payload(void* p) {
thread_pool_count_payload_t* payload = (thread_pool_count_payload_t*)p;
if (!payload) return;
free(payload->sql);
if (payload->bind_params) {
for (int i = 0; i < payload->bind_param_count; i++) {
free(payload->bind_params[i]);
}
free(payload->bind_params);
}
free(payload);
}
static void free_req_payload(void* p) {
thread_pool_req_payload_t* payload = (thread_pool_req_payload_t*)p;
if (!payload) return;
free(payload->sql);
if (payload->bind_params) {
for (int i = 0; i < payload->bind_param_count; i++) {
free(payload->bind_params[i]);
}
free(payload->bind_params);
}
free(payload);
}
static void free_store_event_payload(void* p) {
thread_pool_store_event_payload_t* payload = (thread_pool_store_event_payload_t*)p;
if (!payload) return;
free(payload->id);
free(payload->pubkey);
free(payload->event_type);
free(payload->content);
free(payload->sig);
free(payload->tags_json);
free(payload->event_json);
free(payload);
}
static void queue_init(thread_pool_queue_t* q, int max_size) {
memset(q, 0, sizeof(*q));
q->max_size = (max_size > 0) ? max_size : 4096;
pthread_mutex_init(&q->mutex, NULL);
pthread_cond_init(&q->cond, NULL);
}
static void queue_destroy(thread_pool_queue_t* q) {
pthread_mutex_lock(&q->mutex);
thread_pool_job_node_t* cur = q->head;
while (cur) {
thread_pool_job_node_t* next = cur->next;
if (cur->job.payload_free && cur->job.payload) {
cur->job.payload_free(cur->job.payload);
}
free(cur);
cur = next;
}
q->head = q->tail = NULL;
q->size = 0;
pthread_mutex_unlock(&q->mutex);
pthread_mutex_destroy(&q->mutex);
pthread_cond_destroy(&q->cond);
}
static int queue_push(thread_pool_queue_t* q, thread_pool_job_node_t* node) {
pthread_mutex_lock(&q->mutex);
if (q->size >= q->max_size) {
pthread_mutex_unlock(&q->mutex);
return 0;
}
node->next = NULL;
if (!q->tail) {
q->head = q->tail = node;
} else {
q->tail->next = node;
q->tail = node;
}
q->size++;
pthread_cond_signal(&q->cond);
pthread_mutex_unlock(&q->mutex);
return 1;
}
static thread_pool_job_node_t* queue_pop(thread_pool_queue_t* q) {
pthread_mutex_lock(&q->mutex);
while (g_pool.running && q->size == 0) {
pthread_cond_wait(&q->cond, &q->mutex);
}
if (!g_pool.running && q->size == 0) {
pthread_mutex_unlock(&q->mutex);
return NULL;
}
thread_pool_job_node_t* node = q->head;
q->head = node->next;
if (!q->head) q->tail = NULL;
q->size--;
pthread_mutex_unlock(&q->mutex);
return node;
}
static void wake_event_loop(void) {
if (g_pool.wake_loop_cb) {
g_pool.wake_loop_cb(g_pool.wake_loop_ctx);
}
}
static void* open_worker_connection(void) {
void* db_conn = NULL;
if (db_open_worker_connection(g_pool.db_path, &db_conn) != 0) {
return NULL;
}
return db_conn;
}
static void complete_job_with_result(thread_pool_job_node_t* node,
thread_pool_status_t status,
const char* message,
void* result_data,
size_t result_size) {
thread_pool_result_t result;
memset(&result, 0, sizeof(result));
result.job_id = node->job_id;
result.type = node->job.type;
result.status = status;
result.session = node->job.session;
result.result_data = result_data;
result.result_size = result_size;
result.message = message;
if (node->job.result_cb) {
node->job.result_cb(&result, node->job.result_cb_ctx);
}
wake_event_loop();
if (node->job.payload_free && node->job.payload) {
node->job.payload_free(node->job.payload);
}
free(node);
}
static void execute_count_job(thread_pool_job_node_t* node) {
thread_pool_count_payload_t* payload = (thread_pool_count_payload_t*)node->job.payload;
if (!payload || !payload->sql) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"COUNT payload missing", NULL, 0);
return;
}
int count = 0;
if (db_count_with_sql(payload->sql, (const char**)payload->bind_params,
payload->bind_param_count, &count) != 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"COUNT query failed", NULL, 0);
return;
}
thread_pool_count_result_t* out = calloc(1, sizeof(*out));
if (!out) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"COUNT result allocation failed", NULL, 0);
return;
}
out->count = count;
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"COUNT query completed", out, sizeof(*out));
}
static void execute_req_job(thread_pool_job_node_t* node) {
thread_pool_req_payload_t* payload = (thread_pool_req_payload_t*)node->job.payload;
if (!payload || !payload->sql) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"REQ payload missing", NULL, 0);
return;
}
db_stmt_t* stmt = NULL;
if (db_prepare(payload->sql, &stmt) != DB_OK) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"REQ prepare failed", NULL, 0);
return;
}
for (int i = 0; i < payload->bind_param_count; i++) {
const char* v = (payload->bind_params && payload->bind_params[i]) ? payload->bind_params[i] : "";
(void)db_bind_text_param(stmt, i + 1, v);
}
int cap = (payload->row_limit > 0) ? payload->row_limit : 500;
if (cap < 1) cap = 1;
thread_pool_req_result_t* out = calloc(1, sizeof(*out));
if (!out) {
db_finalize_stmt(stmt);
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"REQ result allocation failed", NULL, 0);
return;
}
out->event_json_rows = calloc((size_t)cap, sizeof(char*));
if (!out->event_json_rows) {
db_finalize_stmt(stmt);
free(out);
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"REQ row array allocation failed", NULL, 0);
return;
}
int rc = DB_OK;
while ((rc = db_step_stmt(stmt)) == DB_ROW) {
if (out->row_count >= cap) break;
const char* row = db_column_text_value(stmt, 0);
if (!row) continue;
out->event_json_rows[out->row_count] = strdup(row);
if (!out->event_json_rows[out->row_count]) {
break;
}
out->row_count++;
}
db_finalize_stmt(stmt);
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"REQ query completed", out, sizeof(*out));
}
static void execute_store_event_job(thread_pool_job_node_t* node) {
thread_pool_store_event_payload_t* payload = (thread_pool_store_event_payload_t*)node->job.payload;
if (!payload || !payload->id || !payload->pubkey || !payload->content || !payload->sig) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"STORE_EVENT payload missing required fields", NULL, 0);
return;
}
thread_pool_store_event_result_t* out = calloc(1, sizeof(*out));
if (!out) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"STORE_EVENT result allocation failed", NULL, 0);
return;
}
int step_rc = DB_ERROR;
int extended_errcode = 0;
if (db_insert_event_with_json(payload->id,
payload->pubkey,
payload->created_at,
payload->kind,
payload->event_type ? payload->event_type : "regular",
payload->content,
payload->sig,
payload->tags_json ? payload->tags_json : "[]",
payload->event_json ? payload->event_json : "{}",
&step_rc,
&extended_errcode) != 0) {
free(out);
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"STORE_EVENT execution failed", NULL, 0);
return;
}
out->step_rc = step_rc;
out->extended_errcode = extended_errcode;
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"STORE_EVENT completed", out, sizeof(*out));
}
static void execute_sub_log_created_job(thread_pool_job_node_t* node) {
thread_pool_sub_log_created_payload_t* payload = (thread_pool_sub_log_created_payload_t*)node->job.payload;
if (!payload || !payload->sub_id || !payload->wsi_ptr || !payload->client_ip) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_CREATED payload missing", NULL, 0);
return;
}
if (db_log_subscription_created(payload->sub_id,
payload->wsi_ptr,
payload->client_ip,
payload->filter_json ? payload->filter_json : "[]") != 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_CREATED failed", NULL, 0);
return;
}
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"SUB_LOG_CREATED completed", NULL, 0);
}
static void execute_sub_log_closed_job(thread_pool_job_node_t* node) {
thread_pool_sub_log_closed_payload_t* payload = (thread_pool_sub_log_closed_payload_t*)node->job.payload;
if (!payload || !payload->sub_id) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_CLOSED payload missing", NULL, 0);
return;
}
if (db_log_subscription_closed(payload->sub_id, payload->client_ip) != 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_CLOSED failed", NULL, 0);
return;
}
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"SUB_LOG_CLOSED completed", NULL, 0);
}
static void execute_sub_log_disconnected_job(thread_pool_job_node_t* node) {
thread_pool_sub_log_disconnected_payload_t* payload = (thread_pool_sub_log_disconnected_payload_t*)node->job.payload;
if (!payload || !payload->client_ip) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_DISCONNECTED payload missing", NULL, 0);
return;
}
if (db_log_subscription_disconnected(payload->client_ip) < 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_LOG_DISCONNECTED failed", NULL, 0);
return;
}
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"SUB_LOG_DISCONNECTED completed", NULL, 0);
}
static void execute_sub_update_events_job(thread_pool_job_node_t* node) {
thread_pool_sub_update_events_payload_t* payload = (thread_pool_sub_update_events_payload_t*)node->job.payload;
if (!payload || !payload->sub_id) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_UPDATE_EVENTS payload missing", NULL, 0);
return;
}
if (db_update_subscription_events_sent(payload->sub_id, payload->events_sent) != 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"SUB_UPDATE_EVENTS failed", NULL, 0);
return;
}
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"SUB_UPDATE_EVENTS completed", NULL, 0);
}
static void execute_ip_ban_save_job(thread_pool_job_node_t* node) {
ip_ban_save_to_db();
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"IP_BAN_SAVE completed", NULL, 0);
}
static void execute_wal_checkpoint_job(thread_pool_job_node_t* node) {
#ifdef DB_BACKEND_POSTGRES
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"WAL_CHECKPOINT skipped for PostgreSQL", NULL, 0);
#else
if (db_wal_checkpoint_passive() != 0) {
complete_job_with_result(node, THREAD_POOL_STATUS_INTERNAL_ERROR,
"WAL_CHECKPOINT failed", NULL, 0);
return;
}
complete_job_with_result(node, THREAD_POOL_STATUS_OK,
"WAL_CHECKPOINT completed", NULL, 0);
#endif
}
static void execute_read_job(thread_pool_job_node_t* node) {
switch (node->job.type) {
case THREAD_POOL_JOB_REQ_QUERY:
execute_req_job(node);
break;
case THREAD_POOL_JOB_COUNT_QUERY:
execute_count_job(node);
break;
default:
complete_job_with_result(node, THREAD_POOL_STATUS_NOT_IMPLEMENTED,
"Read job type not implemented", NULL, 0);
break;
}
}
static void execute_write_job(thread_pool_job_node_t* node) {
switch (node->job.type) {
case THREAD_POOL_JOB_STORE_EVENT:
execute_store_event_job(node);
break;
case THREAD_POOL_JOB_LOG_SUB_CREATED:
execute_sub_log_created_job(node);
break;
case THREAD_POOL_JOB_LOG_SUB_CLOSED:
execute_sub_log_closed_job(node);
break;
case THREAD_POOL_JOB_LOG_SUB_DISCONNECTED:
execute_sub_log_disconnected_job(node);
break;
case THREAD_POOL_JOB_UPDATE_SUB_EVENTS_SENT:
execute_sub_update_events_job(node);
break;
case THREAD_POOL_JOB_IP_BAN_SAVE:
execute_ip_ban_save_job(node);
break;
case THREAD_POOL_JOB_WAL_CHECKPOINT:
execute_wal_checkpoint_job(node);
break;
default:
complete_job_with_result(node, THREAD_POOL_STATUS_NOT_IMPLEMENTED,
"Write job type not implemented", NULL, 0);
break;
}
}
static void* reader_worker_main(void* arg) {
int reader_index = (int)(intptr_t)arg;
char thread_name[16];
snprintf(thread_name, sizeof(thread_name), "db-read-%d", reader_index);
pthread_setname_np(pthread_self(), thread_name);
void* worker_db = open_worker_connection();
if (!worker_db) {
DEBUG_ERROR("Reader worker failed to open database connection");
return NULL;
}
while (g_pool.running) {
thread_pool_job_node_t* node = queue_pop(&g_pool.read_q);
if (!node) break;
execute_read_job(node);
}
db_close_worker_connection(worker_db);
return NULL;
}
static void* writer_worker_main(void* arg) {
int writer_index = (int)(intptr_t)arg;
char thread_name[16];
snprintf(thread_name, sizeof(thread_name), "db-write-%d", writer_index);
pthread_setname_np(pthread_self(), thread_name);
void* worker_db = open_worker_connection();
if (!worker_db) {
DEBUG_ERROR("Writer worker failed to open database connection");
return NULL;
}
while (g_pool.running) {
thread_pool_job_node_t* node = queue_pop(&g_pool.write_q);
if (!node) break;
execute_write_job(node);
}
#ifndef DB_BACKEND_POSTGRES
if (db_wal_checkpoint_truncate() != 0) {
DEBUG_WARN("Writer shutdown TRUNCATE checkpoint failed");
}
#endif
db_close_worker_connection(worker_db);
return NULL;
}
int thread_pool_init(const thread_pool_config_t* config) {
if (!config) return -1;
pthread_mutex_lock(&g_pool.state_mutex);
if (g_pool.running) {
pthread_mutex_unlock(&g_pool.state_mutex);
return 0;
}
g_pool.reader_count = (config->reader_threads > 0) ? config->reader_threads : 4;
g_pool.writer_count = (config->writer_threads > 0) ? config->writer_threads : 2;
g_pool.readers = calloc((size_t)g_pool.reader_count, sizeof(pthread_t));
if (!g_pool.readers) {
pthread_mutex_unlock(&g_pool.state_mutex);
return -1;
}
g_pool.writers = calloc((size_t)g_pool.writer_count, sizeof(pthread_t));
if (!g_pool.writers) {
free(g_pool.readers);
g_pool.readers = NULL;
pthread_mutex_unlock(&g_pool.state_mutex);
return -1;
}
queue_init(&g_pool.read_q, config->max_queue_depth);
queue_init(&g_pool.write_q, config->max_queue_depth);
g_pool.next_job_id = 1;
g_pool.wake_loop_cb = config->wake_loop_cb;
g_pool.wake_loop_ctx = config->wake_loop_ctx;
const char* db_path = (config->db_path && config->db_path[0] != '\0') ? config->db_path : db_get_database_path();
if (!db_path || db_path[0] == '\0') {
free(g_pool.readers);
free(g_pool.writers);
g_pool.readers = NULL;
g_pool.writers = NULL;
pthread_mutex_unlock(&g_pool.state_mutex);
return -1;
}
strncpy(g_pool.db_path, db_path, sizeof(g_pool.db_path) - 1);
g_pool.db_path[sizeof(g_pool.db_path) - 1] = '\0';
g_pool.running = 1;
for (int i = 0; i < g_pool.reader_count; i++) {
if (pthread_create(&g_pool.readers[i], NULL, reader_worker_main, (void*)(intptr_t)(i + 1)) != 0) {
g_pool.running = 0;
pthread_cond_broadcast(&g_pool.read_q.cond);
pthread_cond_broadcast(&g_pool.write_q.cond);
pthread_mutex_unlock(&g_pool.state_mutex);
return -1;
}
}
for (int i = 0; i < g_pool.writer_count; i++) {
if (pthread_create(&g_pool.writers[i], NULL, writer_worker_main, (void*)(intptr_t)(i + 1)) != 0) {
g_pool.running = 0;
pthread_cond_broadcast(&g_pool.read_q.cond);
pthread_cond_broadcast(&g_pool.write_q.cond);
pthread_mutex_unlock(&g_pool.state_mutex);
return -1;
}
}
pthread_mutex_unlock(&g_pool.state_mutex);
DEBUG_LOG("Thread pool initialized: %d readers + %d writers", g_pool.reader_count, g_pool.writer_count);
return 0;
}
void thread_pool_shutdown(void) {
pthread_mutex_lock(&g_pool.state_mutex);
if (!g_pool.running) {
pthread_mutex_unlock(&g_pool.state_mutex);
return;
}
g_pool.running = 0;
pthread_cond_broadcast(&g_pool.read_q.cond);
pthread_cond_broadcast(&g_pool.write_q.cond);
pthread_mutex_unlock(&g_pool.state_mutex);
for (int i = 0; i < g_pool.reader_count; i++) {
pthread_join(g_pool.readers[i], NULL);
}
for (int i = 0; i < g_pool.writer_count; i++) {
pthread_join(g_pool.writers[i], NULL);
}
free(g_pool.readers);
free(g_pool.writers);
g_pool.readers = NULL;
g_pool.writers = NULL;
g_pool.reader_count = 0;
g_pool.writer_count = 0;
queue_destroy(&g_pool.read_q);
queue_destroy(&g_pool.write_q);
DEBUG_LOG("Thread pool shutdown complete");
}
int thread_pool_is_running(void) {
return g_pool.running;
}
static thread_pool_status_t submit_to_queue(thread_pool_queue_t* q, const thread_pool_job_t* job, uint64_t* out_job_id) {
if (!g_pool.running) return THREAD_POOL_STATUS_SHUTTING_DOWN;
if (!job) return THREAD_POOL_STATUS_INTERNAL_ERROR;
thread_pool_job_node_t* node = calloc(1, sizeof(*node));
if (!node) return THREAD_POOL_STATUS_INTERNAL_ERROR;
node->job = *job;
pthread_mutex_lock(&g_pool.state_mutex);
node->job_id = g_pool.next_job_id++;
pthread_mutex_unlock(&g_pool.state_mutex);
if (out_job_id) *out_job_id = node->job_id;
if (!queue_push(q, node)) {
if (job->payload_free && job->payload) {
job->payload_free(job->payload);
}
free(node);
return THREAD_POOL_STATUS_QUEUE_FULL;
}
return THREAD_POOL_STATUS_OK;
}
thread_pool_status_t thread_pool_submit_read(const thread_pool_job_t* job, uint64_t* out_job_id) {
return submit_to_queue(&g_pool.read_q, job, out_job_id);
}
thread_pool_status_t thread_pool_submit_write(const thread_pool_job_t* job, uint64_t* out_job_id) {
return submit_to_queue(&g_pool.write_q, job, out_job_id);
}
int thread_pool_submit_wal_checkpoint(void) {
#ifdef DB_BACKEND_POSTGRES
return 0;
#else
if (!thread_pool_is_running()) {
return -1;
}
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_WAL_CHECKPOINT;
thread_pool_status_t rc = thread_pool_submit_write(&job, NULL);
return (rc == THREAD_POOL_STATUS_OK) ? 0 : -1;
#endif
}
int thread_pool_execute_count_sync(const char* sql, const char** bind_params, int bind_param_count, int* out_count) {
if (!sql || !out_count) {
return -1;
}
if (!thread_pool_is_running()) {
return db_count_with_sql(sql, bind_params, bind_param_count, out_count);
}
thread_pool_count_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) return -1;
payload->sql = strdup(sql);
payload->bind_param_count = bind_param_count;
if (!payload->sql) {
free_count_payload(payload);
return -1;
}
if (bind_param_count > 0) {
payload->bind_params = calloc((size_t)bind_param_count, sizeof(char*));
if (!payload->bind_params) {
free_count_payload(payload);
return -1;
}
for (int i = 0; i < bind_param_count; i++) {
const char* v = (bind_params && bind_params[i]) ? bind_params[i] : "";
payload->bind_params[i] = strdup(v);
if (!payload->bind_params[i]) {
free_count_payload(payload);
return -1;
}
}
}
thread_pool_wait_ctx_t wait_ctx;
wait_ctx_init(&wait_ctx);
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_COUNT_QUERY;
job.payload = payload;
job.payload_free = free_count_payload;
job.result_cb = wait_ctx_result_cb;
job.result_cb_ctx = &wait_ctx;
thread_pool_status_t submit_rc = thread_pool_submit_read(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_count_payload(payload);
wait_ctx_destroy(&wait_ctx);
return -1;
}
wait_ctx_wait(&wait_ctx);
int rc = -1;
if (wait_ctx.status == THREAD_POOL_STATUS_OK && wait_ctx.result_data) {
thread_pool_count_result_t* result = (thread_pool_count_result_t*)wait_ctx.result_data;
*out_count = result->count;
free(result);
rc = 0;
}
wait_ctx_destroy(&wait_ctx);
return rc;
}
int thread_pool_execute_req_sync(const char* sql, const char** bind_params, int bind_param_count,
int row_limit, thread_pool_req_result_t** out_result) {
if (!sql || !out_result) {
return -1;
}
if (!thread_pool_is_running()) {
db_stmt_t* stmt = NULL;
if (db_prepare(sql, &stmt) != DB_OK) {
return -1;
}
for (int i = 0; i < bind_param_count; i++) {
const char* v = (bind_params && bind_params[i]) ? bind_params[i] : "";
(void)db_bind_text_param(stmt, i + 1, v);
}
int cap = (row_limit > 0) ? row_limit : 500;
if (cap < 1) cap = 1;
thread_pool_req_result_t* result = calloc(1, sizeof(*result));
if (!result) {
db_finalize_stmt(stmt);
return -1;
}
result->event_json_rows = calloc((size_t)cap, sizeof(char*));
if (!result->event_json_rows) {
free(result);
db_finalize_stmt(stmt);
return -1;
}
int step_rc = DB_OK;
while ((step_rc = db_step_stmt(stmt)) == DB_ROW) {
if (result->row_count >= cap) break;
const char* row = db_column_text_value(stmt, 0);
if (!row) continue;
result->event_json_rows[result->row_count] = strdup(row);
if (!result->event_json_rows[result->row_count]) {
break;
}
result->row_count++;
}
db_finalize_stmt(stmt);
*out_result = result;
return 0;
}
thread_pool_req_payload_t* payload = calloc(1, sizeof(*payload));
if (!payload) return -1;
payload->sql = strdup(sql);
payload->bind_param_count = bind_param_count;
payload->row_limit = row_limit;
if (!payload->sql) {
free_req_payload(payload);
return -1;
}
if (bind_param_count > 0) {
payload->bind_params = calloc((size_t)bind_param_count, sizeof(char*));
if (!payload->bind_params) {
free_req_payload(payload);
return -1;
}
for (int i = 0; i < bind_param_count; i++) {
const char* v = (bind_params && bind_params[i]) ? bind_params[i] : "";
payload->bind_params[i] = strdup(v);
if (!payload->bind_params[i]) {
free_req_payload(payload);
return -1;
}
}
}
thread_pool_wait_ctx_t wait_ctx;
wait_ctx_init(&wait_ctx);
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_REQ_QUERY;
job.payload = payload;
job.payload_free = free_req_payload;
job.result_cb = wait_ctx_result_cb;
job.result_cb_ctx = &wait_ctx;
thread_pool_status_t submit_rc = thread_pool_submit_read(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_req_payload(payload);
wait_ctx_destroy(&wait_ctx);
return -1;
}
wait_ctx_wait(&wait_ctx);
int rc = -1;
if (wait_ctx.status == THREAD_POOL_STATUS_OK && wait_ctx.result_data) {
*out_result = (thread_pool_req_result_t*)wait_ctx.result_data;
rc = 0;
}
wait_ctx_destroy(&wait_ctx);
return rc;
}
void thread_pool_free_req_result(thread_pool_req_result_t* result) {
if (!result) return;
if (result->event_json_rows) {
for (int i = 0; i < result->row_count; i++) {
free(result->event_json_rows[i]);
}
free(result->event_json_rows);
}
free(result);
}
int thread_pool_execute_store_event_sync(const thread_pool_store_event_payload_t* payload,
thread_pool_store_event_result_t* out_result) {
if (!payload || !out_result || !payload->id || !payload->pubkey || !payload->content || !payload->sig) {
return -1;
}
if (!thread_pool_is_running()) {
int step_rc = DB_ERROR;
int extended_errcode = 0;
if (db_insert_event_with_json(payload->id,
payload->pubkey,
payload->created_at,
payload->kind,
payload->event_type ? payload->event_type : "regular",
payload->content,
payload->sig,
payload->tags_json ? payload->tags_json : "[]",
payload->event_json ? payload->event_json : "{}",
&step_rc,
&extended_errcode) != 0) {
return -1;
}
out_result->step_rc = step_rc;
out_result->extended_errcode = extended_errcode;
return 0;
}
thread_pool_store_event_payload_t* payload_copy = calloc(1, sizeof(*payload_copy));
if (!payload_copy) return -1;
payload_copy->id = payload->id ? strdup(payload->id) : NULL;
payload_copy->pubkey = payload->pubkey ? strdup(payload->pubkey) : NULL;
payload_copy->created_at = payload->created_at;
payload_copy->kind = payload->kind;
payload_copy->event_type = payload->event_type ? strdup(payload->event_type) : strdup("regular");
payload_copy->content = payload->content ? strdup(payload->content) : NULL;
payload_copy->sig = payload->sig ? strdup(payload->sig) : NULL;
payload_copy->tags_json = payload->tags_json ? strdup(payload->tags_json) : strdup("[]");
payload_copy->event_json = payload->event_json ? strdup(payload->event_json) : strdup("{}");
if (!payload_copy->id || !payload_copy->pubkey || !payload_copy->event_type ||
!payload_copy->content || !payload_copy->sig || !payload_copy->tags_json || !payload_copy->event_json) {
free_store_event_payload(payload_copy);
return -1;
}
thread_pool_wait_ctx_t wait_ctx;
wait_ctx_init(&wait_ctx);
thread_pool_job_t job;
memset(&job, 0, sizeof(job));
job.type = THREAD_POOL_JOB_STORE_EVENT;
job.payload = payload_copy;
job.payload_free = free_store_event_payload;
job.result_cb = wait_ctx_result_cb;
job.result_cb_ctx = &wait_ctx;
thread_pool_status_t submit_rc = thread_pool_submit_write(&job, NULL);
if (submit_rc != THREAD_POOL_STATUS_OK) {
free_store_event_payload(payload_copy);
wait_ctx_destroy(&wait_ctx);
return -1;
}
wait_ctx_wait(&wait_ctx);
int rc = -1;
if (wait_ctx.status == THREAD_POOL_STATUS_OK && wait_ctx.result_data) {
thread_pool_store_event_result_t* result = (thread_pool_store_event_result_t*)wait_ctx.result_data;
out_result->step_rc = result->step_rc;
out_result->extended_errcode = result->extended_errcode;
free(result);
rc = 0;
}
wait_ctx_destroy(&wait_ctx);
return rc;
}
__attribute__((constructor))
static void thread_pool_state_init_once(void) {
pthread_mutex_init(&g_pool.state_mutex, NULL);
}
+146
View File
@@ -0,0 +1,146 @@
#ifndef THREAD_POOL_H
#define THREAD_POOL_H
#include <stddef.h>
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef enum {
THREAD_POOL_JOB_REQ_QUERY = 0,
THREAD_POOL_JOB_COUNT_QUERY,
THREAD_POOL_JOB_STORE_EVENT,
THREAD_POOL_JOB_DELETE_EVENT,
THREAD_POOL_JOB_LOG_SUB_CREATED,
THREAD_POOL_JOB_LOG_SUB_CLOSED,
THREAD_POOL_JOB_LOG_SUB_DISCONNECTED,
THREAD_POOL_JOB_UPDATE_SUB_EVENTS_SENT,
THREAD_POOL_JOB_IP_BAN_SAVE,
THREAD_POOL_JOB_WAL_CHECKPOINT,
THREAD_POOL_JOB_CUSTOM
} thread_pool_job_type_t;
typedef enum {
THREAD_POOL_STATUS_OK = 0,
THREAD_POOL_STATUS_NOT_IMPLEMENTED,
THREAD_POOL_STATUS_QUEUE_FULL,
THREAD_POOL_STATUS_SHUTTING_DOWN,
THREAD_POOL_STATUS_INTERNAL_ERROR
} thread_pool_status_t;
typedef struct {
uint64_t job_id;
thread_pool_job_type_t type;
thread_pool_status_t status;
void* session;
void* result_data;
size_t result_size;
const char* message;
} thread_pool_result_t;
typedef void (*thread_pool_result_cb)(const thread_pool_result_t* result, void* user_ctx);
typedef void (*thread_pool_payload_free_cb)(void* payload);
typedef void (*thread_pool_wake_loop_cb)(void* wake_ctx);
typedef struct {
thread_pool_job_type_t type;
void* session;
void* payload;
size_t payload_size;
thread_pool_payload_free_cb payload_free;
thread_pool_result_cb result_cb;
void* result_cb_ctx;
} thread_pool_job_t;
typedef struct {
int reader_threads;
int writer_threads;
int max_queue_depth;
const char* db_path;
thread_pool_wake_loop_cb wake_loop_cb;
void* wake_loop_ctx;
} thread_pool_config_t;
typedef struct {
char* sql;
char** bind_params;
int bind_param_count;
} thread_pool_count_payload_t;
typedef struct {
char* sql;
char** bind_params;
int bind_param_count;
int row_limit;
} thread_pool_req_payload_t;
typedef struct {
char* id;
char* pubkey;
long long created_at;
int kind;
char* event_type;
char* content;
char* sig;
char* tags_json;
char* event_json;
} thread_pool_store_event_payload_t;
typedef struct {
int count;
} thread_pool_count_result_t;
typedef struct {
char** event_json_rows;
int row_count;
} thread_pool_req_result_t;
typedef struct {
int step_rc;
int extended_errcode;
} thread_pool_store_event_result_t;
typedef struct {
char* sub_id;
char* wsi_ptr;
char* client_ip;
char* filter_json;
} thread_pool_sub_log_created_payload_t;
typedef struct {
char* sub_id;
char* client_ip;
} thread_pool_sub_log_closed_payload_t;
typedef struct {
char* client_ip;
} thread_pool_sub_log_disconnected_payload_t;
typedef struct {
char* sub_id;
int events_sent;
} thread_pool_sub_update_events_payload_t;
int thread_pool_init(const thread_pool_config_t* config);
void thread_pool_shutdown(void);
int thread_pool_is_running(void);
thread_pool_status_t thread_pool_submit_read(const thread_pool_job_t* job, uint64_t* out_job_id);
thread_pool_status_t thread_pool_submit_write(const thread_pool_job_t* job, uint64_t* out_job_id);
int thread_pool_submit_wal_checkpoint(void);
int thread_pool_execute_count_sync(const char* sql, const char** bind_params, int bind_param_count, int* out_count);
int thread_pool_execute_req_sync(const char* sql, const char** bind_params, int bind_param_count,
int row_limit, thread_pool_req_result_t** out_result);
void thread_pool_free_req_result(thread_pool_req_result_t* result);
int thread_pool_execute_store_event_sync(const thread_pool_store_event_payload_t* payload,
thread_pool_store_event_result_t* out_result);
#ifdef __cplusplus
}
#endif
#endif // THREAD_POOL_H

Some files were not shown because too many files have changed in this diff Show More