Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
20902e1c5d | ||
|
|
7233818da9 | ||
|
|
86cbf7ff16 | ||
|
|
3035e71ca5 | ||
|
|
179b160cf3 | ||
|
|
aedd22436e | ||
|
|
8fdc362a8b | ||
|
|
67cf3973c5 | ||
|
|
c02ae2be5b | ||
|
|
fd3efdd01f | ||
|
|
232f93c16f | ||
|
|
9a9365dcfa | ||
|
|
5577b57149 | ||
|
|
22c943d495 | ||
|
|
7e3d9b6825 | ||
|
|
547d22d09f | ||
|
|
ce9ae5a488 | ||
|
|
9b248b740f | ||
|
|
0f77aeb72b | ||
|
|
bb64651a0c | ||
|
|
17be9c2b03 | ||
|
|
f396c622b6 | ||
|
|
cbd260c1ae | ||
|
|
bc9ac290ab | ||
|
|
192eeb248d | ||
|
|
da6c505420 | ||
|
|
c077c209e5 | ||
|
|
9bf02702c2 | ||
|
|
0920cc092d | ||
|
|
a89460be5d | ||
|
|
ff2a3aa335 | ||
|
|
94b61e8a7c | ||
|
|
b0c0754e83 | ||
|
|
3265e3d114 | ||
|
|
927659ece1 | ||
|
|
b6ff4150b4 | ||
|
|
63bc526163 | ||
|
|
a1f712236a | ||
|
|
06e6c17b7b | ||
|
|
0751a7c55c | ||
|
|
f1728932a9 | ||
|
|
ef8bdef2a8 | ||
|
|
c11a8ba292 | ||
|
|
0f124fe575 | ||
|
|
6b20452fab | ||
|
|
10c19bc243 | ||
|
|
3d7aa2196f | ||
|
|
a416c3f275 | ||
|
|
bba9baabc3 | ||
|
|
31187c4c4f | ||
|
|
55f862b879 | ||
|
|
76c9b3fcf0 | ||
|
|
929bd09164 | ||
|
|
d17f1dd8d5 | ||
|
|
207a949835 | ||
|
|
d08f4e4221 | ||
|
|
c96736fa6a | ||
|
|
f8ec4ae924 | ||
|
|
fe7304ac7f | ||
|
|
e5d39c984b | ||
|
|
5e45f21e35 | ||
|
|
5321a238b8 | ||
|
|
083bc14972 | ||
|
|
11aaccba9b | ||
|
|
bd1bbd763d | ||
|
|
2bd7aa5a10 | ||
|
|
361912ec85 | ||
|
|
0de491382e | ||
|
|
3148bbbee7 | ||
|
|
3965ba04d8 | ||
|
|
b96af938bd | ||
|
|
89c8248013 | ||
|
|
d8f477c6cf | ||
|
|
040eeadb13 | ||
|
|
83f8b0ab88 | ||
|
|
b82f7eaaf3 | ||
|
|
0dc5b75d7c | ||
|
|
e94b1a81e3 | ||
|
|
1adabdbc4e | ||
|
|
81d44c3d8c | ||
|
|
7acc0bdd90 | ||
|
|
c4ef71d673 | ||
|
|
086d2af56c | ||
|
|
18a7deec54 | ||
|
|
c794370a3f | ||
|
|
2ed4b96058 | ||
|
|
c0051b22be | ||
|
|
4cc2d2376e |
@@ -11,3 +11,4 @@ copy_executable_local.sh
|
||||
nostr_login_lite/
|
||||
style_guide/
|
||||
nostr-tools
|
||||
.test_keys
|
||||
@@ -0,0 +1,2 @@
|
||||
ADMIN_PUBKEY='6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3'
|
||||
SERVER_PRIVKEY='1111111111111111111111111111111111111111111111111111111111111111'
|
||||
+11
-11
@@ -1,12 +1,12 @@
|
||||
|
||||
# AGENTS.md - AI Agent Integration Guide for Architect Mode
|
||||
|
||||
**Project-Specific Information for AI Agents Working with C-Relay in Architect Mode**
|
||||
**Project-Specific Information for AI Agents Working with C-Relay-PG in Architect Mode**
|
||||
|
||||
## Critical Architecture Understanding
|
||||
|
||||
### System Architecture Overview
|
||||
C-Relay implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
|
||||
C-Relay-PG implements a **unique event-based configuration architecture** that fundamentally differs from traditional Nostr relays:
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||
@@ -51,9 +51,9 @@ C-Relay implements a **unique event-based configuration architecture** that fund
|
||||
## Architectural Decision Analysis
|
||||
|
||||
### Configuration System Design
|
||||
**Traditional Approach vs C-Relay:**
|
||||
**Traditional Approach vs C-Relay-PG:**
|
||||
```
|
||||
Traditional: C-Relay:
|
||||
Traditional: C-Relay-PG:
|
||||
config.json → kind 33334 events
|
||||
ENV variables → cryptographically signed tags
|
||||
File watching → database polling/restart
|
||||
@@ -119,18 +119,18 @@ File watching → database polling/restart
|
||||
```
|
||||
Developer Machine:
|
||||
├── ./make_and_restart_relay.sh
|
||||
├── build/c_relay_x86
|
||||
├── build/c_relay_pg_x86
|
||||
├── build/<relay_pubkey>.db
|
||||
└── relay.log
|
||||
```
|
||||
|
||||
### Production SystemD Deployment
|
||||
```
|
||||
/opt/c-relay/:
|
||||
├── c_relay_x86
|
||||
/opt/c-relay-pg/:
|
||||
├── c_relay_pg_x86
|
||||
├── <relay_pubkey>.db
|
||||
├── systemd service (c-relay.service)
|
||||
└── c-relay user isolation
|
||||
├── systemd service (c-relay-pg.service)
|
||||
└── c-relay-pg user isolation
|
||||
```
|
||||
|
||||
### Container Deployment Architecture
|
||||
@@ -144,7 +144,7 @@ Container:
|
||||
|
||||
### Reverse Proxy Architecture
|
||||
```
|
||||
Internet → Nginx/HAProxy → C-Relay
|
||||
Internet → Nginx/HAProxy → C-Relay-PG
|
||||
├── WebSocket upgrade handling
|
||||
├── SSL termination
|
||||
└── Rate limiting
|
||||
@@ -292,7 +292,7 @@ Admin Signs Event → WebSocket Submit → Validate → Store → Restart Requir
|
||||
**Decision**: Same port serves both protocols
|
||||
**Consequences**: Simplified deployment, protocol detection overhead, libwebsockets dependency
|
||||
|
||||
These architectural decisions form the foundation of C-Relay's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
|
||||
These architectural decisions form the foundation of C-Relay-PG's unique approach to Nostr relay implementation and should be carefully considered when planning extensions or modifications.
|
||||
**
|
||||
|
||||
[Response interrupted by a tool use result. Only one tool may be used at a time and should be placed at the end of the message.]
|
||||
Executable
BIN
Binary file not shown.
@@ -1,6 +1,6 @@
|
||||
# AGENTS.md - AI Agent Integration Guide
|
||||
|
||||
**Project-Specific Information for AI Agents Working with C-Relay**
|
||||
**Project-Specific Information for AI Agents Working with C-Relay-PG**
|
||||
|
||||
## Critical Build Commands
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
- Starts relay in background with proper logging
|
||||
|
||||
### Architecture-Specific Binary Outputs
|
||||
- **x86_64**: `./build/c_relay_x86`
|
||||
- **ARM64**: `./build/c_relay_arm64`
|
||||
- **Other**: `./build/c_relay_$(ARCH)`
|
||||
- **x86_64**: `./build/c_relay_pg_x86`
|
||||
- **ARM64**: `./build/c_relay_pg_arm64`
|
||||
- **Other**: `./build/c_relay_pg_$(ARCH)`
|
||||
|
||||
### Database File Naming Convention
|
||||
- **Format**: `<relay_pubkey>.db` (NOT `.nrdb` as shown in docs)
|
||||
@@ -75,10 +75,10 @@
|
||||
### Process Management
|
||||
```bash
|
||||
# Kill existing relay processes
|
||||
pkill -f "c_relay_"
|
||||
pkill -f "c_relay_pg_"
|
||||
|
||||
# Check running processes
|
||||
ps aux | grep c_relay_
|
||||
ps aux | grep c_relay_pg_
|
||||
|
||||
# Force kill port binding
|
||||
fuser -k 8888/tcp
|
||||
@@ -95,7 +95,7 @@ fuser -k 8888/tcp
|
||||
- Event configuration tests: `tests/event_config_tests.sh`
|
||||
|
||||
### SystemD Integration Considerations
|
||||
- Service runs as `c-relay` user in `/opt/c-relay`
|
||||
- Service runs as `c-relay-pg` user in `/opt/c-relay-pg`
|
||||
- Database files created in WorkingDirectory automatically
|
||||
- No environment variables needed (event-based config)
|
||||
- Resource limits: 65536 file descriptors, 4096 processes
|
||||
@@ -137,7 +137,7 @@ fuser -k 8888/tcp
|
||||
## Quick Debugging Commands
|
||||
```bash
|
||||
# Check relay status
|
||||
ps aux | grep c_relay_ && netstat -tln | grep 8888
|
||||
ps aux | grep c_relay_pg_ && netstat -tln | grep 8888
|
||||
|
||||
# View logs
|
||||
tail -f relay.log
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# C-Relay API Documentation
|
||||
# C-Relay-PG API Documentation
|
||||
|
||||
Complete API reference for the C-Relay event-based administration system and advanced features.
|
||||
Complete API reference for the C-Relay-PG event-based administration system and advanced features.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -21,7 +21,7 @@ Complete API reference for the C-Relay event-based administration system and adv
|
||||
|
||||
## Overview
|
||||
|
||||
C-Relay uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
|
||||
C-Relay-PG uses an innovative **event-based administration system** where all configuration and management commands are sent as cryptographically signed Nostr events. This provides:
|
||||
|
||||
- **Cryptographic security**: All commands must be signed with the admin private key
|
||||
- **Audit trail**: Complete history of all administrative actions
|
||||
@@ -63,8 +63,8 @@ Store the admin private key securely:
|
||||
export C_RELAY_ADMIN_KEY="nsec1abc123..."
|
||||
|
||||
# Secure file
|
||||
echo "nsec1abc123..." > ~/.c-relay-admin
|
||||
chmod 600 ~/.c-relay-admin
|
||||
echo "nsec1abc123..." > ~/.c-relay-pg-admin
|
||||
chmod 600 ~/.c-relay-pg-admin
|
||||
|
||||
# Password manager (recommended)
|
||||
# Store in 1Password, Bitwarden, etc.
|
||||
@@ -167,7 +167,7 @@ Examples:
|
||||
"data": [
|
||||
{
|
||||
"key": "relay_name",
|
||||
"value": "C-Relay",
|
||||
"value": "C-Relay-PG",
|
||||
"data_type": "string",
|
||||
"category": "relay",
|
||||
"description": "Relay name displayed in NIP-11"
|
||||
@@ -506,10 +506,10 @@ ORDER BY count DESC
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|-----|------|---------|-------------|
|
||||
| `relay_name` | string | "C-Relay" | Relay name (NIP-11) |
|
||||
| `relay_name` | string | "C-Relay-PG" | Relay name (NIP-11) |
|
||||
| `relay_description` | string | "C Nostr Relay" | Relay description |
|
||||
| `relay_contact` | string | "" | Admin contact info |
|
||||
| `relay_software` | string | "c-relay" | Software identifier |
|
||||
| `relay_software` | string | "c-relay-pg" | Software identifier |
|
||||
| `relay_version` | string | auto | Software version |
|
||||
| `supported_nips` | string | "1,9,11,13,15,20,33,40,42,45,50,70" | Supported NIPs |
|
||||
| `language_tags` | string | "*" | Supported languages |
|
||||
@@ -576,7 +576,7 @@ ORDER BY count DESC
|
||||
|
||||
## Real-time Monitoring
|
||||
|
||||
C-Relay provides subscription-based real-time monitoring using ephemeral events (kind 24567).
|
||||
C-Relay-PG provides subscription-based real-time monitoring using ephemeral events (kind 24567).
|
||||
|
||||
### Activation
|
||||
|
||||
|
||||
+43
-33
@@ -1,12 +1,14 @@
|
||||
# Alpine-based MUSL static binary builder for C-Relay
|
||||
# Alpine-based MUSL static binary builder for C-Relay-PG
|
||||
# Produces truly portable binaries with zero runtime dependencies
|
||||
|
||||
ARG DEBUG_BUILD=false
|
||||
ARG DB_BACKEND=sqlite
|
||||
|
||||
FROM alpine:3.19 AS builder
|
||||
|
||||
# Re-declare build argument in this stage
|
||||
# Re-declare build arguments in this stage
|
||||
ARG DEBUG_BUILD=false
|
||||
ARG DB_BACKEND=sqlite
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache \
|
||||
@@ -26,9 +28,11 @@ RUN apk add --no-cache \
|
||||
curl-static \
|
||||
sqlite-dev \
|
||||
sqlite-static \
|
||||
postgresql-dev \
|
||||
linux-headers \
|
||||
wget \
|
||||
bash
|
||||
bash \
|
||||
openssh-client
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /build
|
||||
@@ -68,15 +72,8 @@ RUN cd /tmp && \
|
||||
make install && \
|
||||
rm -rf /tmp/libwebsockets
|
||||
|
||||
# Copy only submodule configuration and git directory
|
||||
COPY .gitmodules /build/.gitmodules
|
||||
COPY .git /build/.git
|
||||
|
||||
# Clean up any stale submodule references (nips directory is not a submodule)
|
||||
RUN git rm --cached nips 2>/dev/null || true
|
||||
|
||||
# Initialize submodules (cached unless .gitmodules changes)
|
||||
RUN git submodule update --init --recursive
|
||||
# Submodules are provided in the local build context and copied explicitly below.
|
||||
# Avoid network/SSH dependency inside Docker image build.
|
||||
|
||||
# Copy nostr_core_lib source files (cached unless nostr_core_lib changes)
|
||||
COPY nostr_core_lib /build/nostr_core_lib/
|
||||
@@ -84,11 +81,13 @@ COPY nostr_core_lib /build/nostr_core_lib/
|
||||
# Copy c_utils_lib source files (cached unless c_utils_lib changes)
|
||||
COPY c_utils_lib /build/c_utils_lib/
|
||||
|
||||
# Build c_utils_lib with MUSL-compatible flags (cached unless c_utils_lib changes)
|
||||
# Build c_utils_lib static library for relay logging utilities
|
||||
# (build only debug.c to avoid broken/missing version header surface in submodule revision)
|
||||
RUN cd c_utils_lib && \
|
||||
sed -i 's/CFLAGS = -Wall -Wextra -std=c99 -O2 -g/CFLAGS = -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g/' Makefile && \
|
||||
make clean && \
|
||||
make
|
||||
mkdir -p build && \
|
||||
gcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -O2 -g \
|
||||
-Isrc -Iinclude -c src/debug.c -o build/debug.o && \
|
||||
ar rcs libc_utils.a build/debug.o
|
||||
|
||||
# Build nostr_core_lib with required NIPs (cached unless nostr_core_lib changes)
|
||||
# Disable fortification in build.sh to prevent __*_chk symbol issues
|
||||
@@ -99,44 +98,55 @@ RUN cd nostr_core_lib && \
|
||||
rm -f *.o *.a 2>/dev/null || true && \
|
||||
./build.sh --nips=1,6,13,17,19,44,59
|
||||
|
||||
# Copy c-relay source files LAST (only this layer rebuilds on source changes)
|
||||
# Copy c-relay-pg source files LAST (only this layer rebuilds on source changes)
|
||||
COPY src/ /build/src/
|
||||
COPY Makefile /build/Makefile
|
||||
|
||||
# Build c-relay with full static linking (only rebuilds when src/ changes)
|
||||
# Build c-relay-pg with full static linking (only rebuilds when src/ changes)
|
||||
# Disable fortification to avoid __*_chk symbols that don't exist in MUSL
|
||||
# Use conditional compilation flags based on DEBUG_BUILD argument
|
||||
# Use conditional compilation flags based on DEBUG_BUILD and DB_BACKEND build args
|
||||
RUN if [ "$DEBUG_BUILD" = "true" ]; then \
|
||||
CFLAGS="-g -O0 -DDEBUG"; \
|
||||
STRIP_CMD=""; \
|
||||
echo "Building with DEBUG symbols enabled"; \
|
||||
CFLAGS="-g -O2 -DDEBUG"; \
|
||||
STRIP_CMD="echo 'Keeping debug symbols'"; \
|
||||
echo "Building with DEBUG symbols enabled (optimized with -O2)"; \
|
||||
else \
|
||||
CFLAGS="-O2"; \
|
||||
STRIP_CMD="strip /build/c_relay_static"; \
|
||||
echo "Building optimized production binary"; \
|
||||
STRIP_CMD="strip /build/c_relay_pg_static"; \
|
||||
echo "Building optimized production binary (symbols stripped)"; \
|
||||
fi && \
|
||||
gcc -static $CFLAGS -Wall -Wextra -std=c99 \
|
||||
if [ "$DB_BACKEND" = "postgres" ]; then \
|
||||
DB_FLAGS="-DDB_BACKEND_POSTGRES -DHAVE_LIBPQ"; \
|
||||
DB_LIBS="-lpq -lpgcommon -lpgport"; \
|
||||
echo "Compiling with PostgreSQL backend"; \
|
||||
else \
|
||||
DB_FLAGS=""; \
|
||||
DB_LIBS=""; \
|
||||
echo "Compiling with SQLite backend"; \
|
||||
fi && \
|
||||
gcc -static $CFLAGS $DB_FLAGS -Wall -Wextra -std=c99 \
|
||||
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
|
||||
-I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core \
|
||||
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket \
|
||||
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql \
|
||||
src/main.c src/config.c src/dm_admin.c src/request_validator.c \
|
||||
src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c \
|
||||
src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c \
|
||||
-o /build/c_relay_static \
|
||||
src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c \
|
||||
src/caching_inbox_poller.c src/caching_service_launcher.c \
|
||||
src/db_ops.c src/db_ops_sqlite.c src/db_ops_postgres.c src/thread_pool.c \
|
||||
-o /build/c_relay_pg_static \
|
||||
c_utils_lib/libc_utils.a \
|
||||
nostr_core_lib/libnostr_core_x64.a \
|
||||
-lwebsockets -lssl -lcrypto -lsqlite3 -lsecp256k1 \
|
||||
-lcurl -lz -lpthread -lm -ldl && \
|
||||
-lcurl -lz -lpthread -lm -ldl $DB_LIBS && \
|
||||
eval "$STRIP_CMD"
|
||||
|
||||
# Verify it's truly static
|
||||
RUN echo "=== Binary Information ===" && \
|
||||
file /build/c_relay_static && \
|
||||
ls -lh /build/c_relay_static && \
|
||||
file /build/c_relay_pg_static && \
|
||||
ls -lh /build/c_relay_pg_static && \
|
||||
echo "=== Checking for dynamic dependencies ===" && \
|
||||
(ldd /build/c_relay_static 2>&1 || echo "Binary is static") && \
|
||||
(ldd /build/c_relay_pg_static 2>&1 || echo "Binary is static") && \
|
||||
echo "=== Build complete ==="
|
||||
|
||||
# Output stage - just the binary
|
||||
FROM scratch AS output
|
||||
COPY --from=builder /build/c_relay_static /c_relay_static
|
||||
COPY --from=builder /build/c_relay_pg_static /c_relay_pg_static
|
||||
@@ -1,28 +1,40 @@
|
||||
# C-Relay Makefile
|
||||
# C-Relay-PG Makefile
|
||||
|
||||
CC = gcc
|
||||
CFLAGS = -Wall -Wextra -std=c99 -g -O2
|
||||
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket
|
||||
INCLUDES = -I. -Ic_utils_lib/src -Inostr_core_lib -Inostr_core_lib/nostr_core -Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket -I/usr/include/postgresql
|
||||
LIBS = -lsqlite3 -lwebsockets -lz -ldl -lpthread -lm -L/usr/local/lib -lsecp256k1 -lssl -lcrypto -L/usr/local/lib -lcurl -Lc_utils_lib -lc_utils
|
||||
|
||||
DB_BACKEND ?= sqlite
|
||||
|
||||
# Build directory
|
||||
BUILD_DIR = build
|
||||
|
||||
# Source files
|
||||
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c
|
||||
MAIN_SRC = src/main.c src/config.c src/dm_admin.c src/request_validator.c src/nip009.c src/nip011.c src/nip013.c src/nip040.c src/nip042.c src/websockets.c src/subscriptions.c src/api.c src/embedded_web_content.c src/ip_ban.c src/thread_pool.c src/caching_inbox_poller.c src/caching_service_launcher.c
|
||||
DB_OPS_SRC = src/db_ops.c
|
||||
|
||||
ifeq ($(DB_BACKEND),postgres)
|
||||
CFLAGS += -DDB_BACKEND_POSTGRES -DHAVE_LIBPQ
|
||||
DB_OPS_SRC += src/db_ops_postgres.c
|
||||
LIBS += -lpq
|
||||
else
|
||||
DB_OPS_SRC += src/db_ops_sqlite.c
|
||||
endif
|
||||
|
||||
NOSTR_CORE_LIB = nostr_core_lib/libnostr_core_x64.a
|
||||
C_UTILS_LIB = c_utils_lib/libc_utils.a
|
||||
|
||||
# Architecture detection
|
||||
ARCH = $(shell uname -m)
|
||||
ifeq ($(ARCH),x86_64)
|
||||
TARGET = $(BUILD_DIR)/c_relay_x86
|
||||
TARGET = $(BUILD_DIR)/c_relay_pg_x86
|
||||
else ifeq ($(ARCH),aarch64)
|
||||
TARGET = $(BUILD_DIR)/c_relay_arm64
|
||||
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
|
||||
else ifeq ($(ARCH),arm64)
|
||||
TARGET = $(BUILD_DIR)/c_relay_arm64
|
||||
TARGET = $(BUILD_DIR)/c_relay_pg_arm64
|
||||
else
|
||||
TARGET = $(BUILD_DIR)/c_relay_$(ARCH)
|
||||
TARGET = $(BUILD_DIR)/c_relay_pg_$(ARCH)
|
||||
endif
|
||||
|
||||
# Default target
|
||||
@@ -81,19 +93,19 @@ force-version:
|
||||
@$(MAKE) src/main.h
|
||||
|
||||
# Build the relay
|
||||
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Compiling C-Relay for architecture: $(ARCH)"
|
||||
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
|
||||
$(TARGET): $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Compiling C-Relay-PG for architecture: $(ARCH) (backend: $(DB_BACKEND))"
|
||||
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
|
||||
@echo "Build complete: $(TARGET)"
|
||||
|
||||
# Build for specific architectures
|
||||
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Building C-Relay for x86_64..."
|
||||
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
|
||||
@echo "Build complete: $(BUILD_DIR)/c_relay_x86"
|
||||
x86: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Building C-Relay-PG for x86_64 (backend: $(DB_BACKEND))..."
|
||||
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_x86 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)
|
||||
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_x86"
|
||||
|
||||
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Cross-compiling C-Relay for ARM64..."
|
||||
arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(DB_OPS_SRC) $(NOSTR_CORE_LIB) $(C_UTILS_LIB)
|
||||
@echo "Cross-compiling C-Relay-PG for ARM64 (backend: $(DB_BACKEND))..."
|
||||
@if ! command -v aarch64-linux-gnu-gcc >/dev/null 2>&1; then \
|
||||
echo "ERROR: ARM64 cross-compiler not found."; \
|
||||
echo "Install with: make install-cross-tools"; \
|
||||
@@ -116,9 +128,9 @@ arm64: $(BUILD_DIR) src/main.h src/sql_schema.h $(MAIN_SRC) $(NOSTR_CORE_LIB) $(
|
||||
fi
|
||||
@echo "Using aarch64-linux-gnu-gcc with ARM64 libraries..."
|
||||
PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig:/usr/share/pkgconfig \
|
||||
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(BUILD_DIR)/c_relay_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
|
||||
aarch64-linux-gnu-gcc $(CFLAGS) $(INCLUDES) $(MAIN_SRC) $(DB_OPS_SRC) -o $(BUILD_DIR)/c_relay_pg_arm64 $(NOSTR_CORE_LIB) $(C_UTILS_LIB) \
|
||||
-L/usr/lib/aarch64-linux-gnu $(LIBS)
|
||||
@echo "Build complete: $(BUILD_DIR)/c_relay_arm64"
|
||||
@echo "Build complete: $(BUILD_DIR)/c_relay_pg_arm64"
|
||||
|
||||
# Install ARM64 cross-compilation dependencies
|
||||
install-arm64-deps:
|
||||
@@ -160,7 +172,7 @@ test: $(TARGET)
|
||||
init-db:
|
||||
@echo "Database initialization is now handled automatically when the server starts."
|
||||
@echo "The schema is embedded in the binary - no external files needed."
|
||||
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_x86"
|
||||
@echo "To manually recreate database: rm -f db/c_nostr_relay.db && ./build/c_relay_pg_x86"
|
||||
|
||||
# Clean build artifacts
|
||||
clean:
|
||||
@@ -180,7 +192,7 @@ install-deps:
|
||||
|
||||
# Help
|
||||
help:
|
||||
@echo "C-Relay Build System"
|
||||
@echo "C-Relay-PG Build System"
|
||||
@echo ""
|
||||
@echo "Targets:"
|
||||
@echo " all Build the relay for current architecture (default)"
|
||||
@@ -209,15 +221,15 @@ help:
|
||||
# Build fully static MUSL binaries using Docker
|
||||
static-musl-x86_64:
|
||||
@echo "Building fully static MUSL binary for x86_64..."
|
||||
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-x86_64 --load .
|
||||
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-x86_64 sh -c "cp /c_relay_static_musl_x86_64 /output/"
|
||||
@echo "Static binary created: build/c_relay_static_musl_x86_64"
|
||||
docker buildx build --platform linux/amd64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-x86_64 --load .
|
||||
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-x86_64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/"
|
||||
@echo "Static binary created: build/c_relay_pg_static_musl_x86_64"
|
||||
|
||||
static-musl-arm64:
|
||||
@echo "Building fully static MUSL binary for ARM64..."
|
||||
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-static-builder-arm64 --load .
|
||||
docker run --rm -v $(PWD)/build:/output c-relay-static-builder-arm64 sh -c "cp /c_relay_static_musl_x86_64 /output/c_relay_static_musl_arm64"
|
||||
@echo "Static binary created: build/c_relay_static_musl_arm64"
|
||||
docker buildx build --platform linux/arm64 -f examples/deployment/static-builder.Dockerfile -t c-relay-pg-static-builder-arm64 --load .
|
||||
docker run --rm -v $(PWD)/build:/output c-relay-pg-static-builder-arm64 sh -c "cp /c_relay_pg_static_musl_x86_64 /output/c_relay_pg_static_musl_arm64"
|
||||
@echo "Static binary created: build/c_relay_pg_static_musl_arm64"
|
||||
|
||||
static-musl: static-musl-x86_64 static-musl-arm64
|
||||
@echo "Built static MUSL binaries for both architectures"
|
||||
|
||||
+6
-6
@@ -45,21 +45,21 @@ Also included is a more standard administrative web front end. This front end co
|
||||
|
||||
## Screenshots
|
||||
|
||||

|
||||

|
||||
Main page with real time updates.
|
||||
|
||||

|
||||

|
||||
Set your configuration preferences.
|
||||
|
||||

|
||||

|
||||
View current subscriptions
|
||||
|
||||

|
||||

|
||||
Add npubs to white or black lists.
|
||||
|
||||

|
||||

|
||||
Run sql queries on the database.
|
||||
|
||||

|
||||

|
||||
Light mode.
|
||||
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# C-Relay: High-Performance Nostr Relay
|
||||
# C-Relay-PG: High-Performance Nostr Relay
|
||||
|
||||
A blazingly fast, production-ready Nostr relay implemented in C with an innovative event-based configuration system. Built for performance, security, and ease of deployment.
|
||||
|
||||
## 🚀 Why C-Relay?
|
||||
## 🚀 Why C-Relay-PG?
|
||||
|
||||
### Event-Based Configuration
|
||||
Unlike traditional relays that require config files, C-Relay uses **cryptographically signed Nostr events** for all configuration. This means:
|
||||
Unlike traditional relays that require config files, C-Relay-PG uses **cryptographically signed Nostr events** for all configuration. This means:
|
||||
- **Zero config files** - Everything stored in the database
|
||||
- **Real-time updates** - Changes applied instantly without restart
|
||||
- **Cryptographic security** - All changes must be signed by admin
|
||||
@@ -36,7 +36,7 @@ Control your relay by sending direct messages from any Nostr client:
|
||||
|
||||
## 📋 Supported NIPs
|
||||
|
||||
C-Relay implements a comprehensive set of Nostr Improvement Proposals:
|
||||
C-Relay-PG implements a comprehensive set of Nostr Improvement Proposals:
|
||||
|
||||
- ✅ **NIP-01**: Basic protocol flow implementation
|
||||
- ✅ **NIP-09**: Event deletion
|
||||
@@ -89,12 +89,12 @@ Download and run - no dependencies required:
|
||||
|
||||
```bash
|
||||
# Download the latest static release
|
||||
wget https://git.laantungir.net/laantungir/c-relay/releases/download/v0.6.0/c-relay-v0.6.0-linux-x86_64-static
|
||||
chmod +x c-relay-v0.6.0-linux-x86_64-static
|
||||
mv c-relay-v0.6.0-linux-x86_64-static c-relay
|
||||
wget https://git.laantungir.net/laantungir/c-relay-pg/releases/download/v0.6.0/c-relay-pg-v0.6.0-linux-x86_64-static
|
||||
chmod +x c-relay-pg-v0.6.0-linux-x86_64-static
|
||||
mv c-relay-pg-v0.6.0-linux-x86_64-static c-relay-pg
|
||||
|
||||
# Run the relay
|
||||
./c-relay
|
||||
./c-relay-pg
|
||||
```
|
||||
|
||||
**Important**: On first startup, save the **Admin Private Key** displayed in the console. You'll need it for all administrative operations.
|
||||
@@ -107,8 +107,8 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
|
||||
libwebsockets-dev libssl-dev libsecp256k1-dev libcurl4-openssl-dev zlib1g-dev
|
||||
|
||||
# Clone and build
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
git submodule update --init --recursive
|
||||
./make_and_restart_relay.sh
|
||||
```
|
||||
@@ -136,8 +136,8 @@ The web interface provides:
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
git submodule update --init --recursive
|
||||
|
||||
# Build
|
||||
@@ -147,25 +147,25 @@ make clean && make
|
||||
sudo systemd/install-service.sh
|
||||
|
||||
# Start and enable
|
||||
sudo systemctl start c-relay
|
||||
sudo systemctl enable c-relay
|
||||
sudo systemctl start c-relay-pg
|
||||
sudo systemctl enable c-relay-pg
|
||||
|
||||
# Capture admin keys from logs
|
||||
sudo journalctl -u c-relay | grep "Admin Private Key"
|
||||
sudo journalctl -u c-relay-pg | grep "Admin Private Key"
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
|
||||
```bash
|
||||
# Build Docker image
|
||||
docker build -f Dockerfile.alpine-musl -t c-relay .
|
||||
docker build -f Dockerfile.alpine-musl -t c-relay-pg .
|
||||
|
||||
# Run container
|
||||
docker run -d \
|
||||
--name c-relay \
|
||||
--name c-relay-pg \
|
||||
-p 8888:8888 \
|
||||
-v /path/to/data:/data \
|
||||
c-relay
|
||||
c-relay-pg
|
||||
```
|
||||
|
||||
### Cloud Deployment
|
||||
@@ -181,7 +181,7 @@ See [`docs/deployment_guide.md`](docs/deployment_guide.md) for detailed deployme
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
C-Relay uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
|
||||
C-Relay-PG uses an innovative event-based configuration system. All settings are managed through signed Nostr events.
|
||||
|
||||
### Basic Configuration
|
||||
|
||||
@@ -239,8 +239,8 @@ The admin private key is displayed **only once** during first startup. Store it
|
||||
|
||||
```bash
|
||||
# Save to secure location
|
||||
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-admin
|
||||
chmod 600 ~/.c-relay-admin
|
||||
echo "ADMIN_PRIVKEY=your_admin_private_key" > ~/.c-relay-pg-admin
|
||||
chmod 600 ~/.c-relay-pg-admin
|
||||
```
|
||||
|
||||
### Production Security
|
||||
@@ -269,9 +269,9 @@ Contributions are welcome! Please:
|
||||
|
||||
## 🔗 Links
|
||||
|
||||
- **Repository**: [https://github.com/your-org/c-relay](https://github.com/your-org/c-relay)
|
||||
- **Releases**: [https://git.laantungir.net/laantungir/c-relay/releases](https://git.laantungir.net/laantungir/c-relay/releases)
|
||||
- **Issues**: [https://github.com/your-org/c-relay/issues](https://github.com/your-org/c-relay/issues)
|
||||
- **Repository**: [https://github.com/your-org/c-relay-pg](https://github.com/your-org/c-relay-pg)
|
||||
- **Releases**: [https://git.laantungir.net/laantungir/c-relay-pg/releases](https://git.laantungir.net/laantungir/c-relay-pg/releases)
|
||||
- **Issues**: [https://github.com/your-org/c-relay-pg/issues](https://github.com/your-org/c-relay-pg/issues)
|
||||
- **Nostr Protocol**: [https://github.com/nostr-protocol/nostr](https://github.com/nostr-protocol/nostr)
|
||||
|
||||
## 💬 Support
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
# Real-Time Traffic Monitoring Commands (Direct Server Use)
|
||||
|
||||
Copy and paste these commands directly on your server.
|
||||
|
||||
## Quick Status Checks
|
||||
|
||||
### See IPs visiting in the last few minutes:
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
|
||||
```
|
||||
|
||||
### See what status codes they're getting:
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | awk '{print $1, $9}' | grep '216.73.216.38'
|
||||
```
|
||||
|
||||
### Count status codes (200 vs 403):
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
|
||||
```
|
||||
|
||||
## Real-Time Monitoring
|
||||
|
||||
### Watch live traffic (updates every 2 seconds):
|
||||
```bash
|
||||
watch -n 2 'sudo tail -200 /var/log/nginx/access.log | awk "{print \$1}" | sort | uniq -c | sort -rn | head -15'
|
||||
```
|
||||
|
||||
### See live log entries as they happen:
|
||||
```bash
|
||||
sudo tail -f /var/log/nginx/access.log
|
||||
```
|
||||
|
||||
### Live GoAccess dashboard:
|
||||
```bash
|
||||
sudo tail -f /var/log/nginx/access.log | goaccess -
|
||||
```
|
||||
|
||||
## Active Connections
|
||||
|
||||
### See who's connected RIGHT NOW:
|
||||
```bash
|
||||
sudo netstat -tn | grep ':443' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
|
||||
```
|
||||
|
||||
### Alternative (using ss command):
|
||||
```bash
|
||||
sudo ss -tn | grep ':443' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn
|
||||
```
|
||||
|
||||
## Detailed Analysis
|
||||
|
||||
### Last 100 requests with timestamps:
|
||||
```bash
|
||||
sudo tail -100 /var/log/nginx/access.log | awk '{print $4, $1}' | sed 's/\[//'
|
||||
```
|
||||
|
||||
### See what blocked IPs are trying to access:
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | grep '216.73.216.38' | awk '{print $7}' | head -10
|
||||
```
|
||||
|
||||
### Show all 403 (blocked) requests:
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | awk '$9==403 {print $1}' | sort | uniq -c | sort -rn
|
||||
```
|
||||
|
||||
### Show all successful (200) requests:
|
||||
```bash
|
||||
sudo tail -500 /var/log/nginx/access.log | awk '$9==200 {print $1}' | sort | uniq -c | sort -rn | head -10
|
||||
```
|
||||
|
||||
## Comprehensive Monitoring Script
|
||||
|
||||
### Create a monitoring script:
|
||||
```bash
|
||||
cat > /tmp/monitor-traffic.sh << 'EOF'
|
||||
#!/bin/bash
|
||||
echo "=== Traffic in last 5 minutes ==="
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
echo "Top IPs:"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
|
||||
echo ""
|
||||
echo "Blocked IPs (403 errors):"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '$9==403 {print $1}' | sort | uniq -c | sort -rn
|
||||
echo ""
|
||||
echo "Successful requests (200):"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '$9==200 {print $1}' | sort | uniq -c | sort -rn | head -5
|
||||
echo ""
|
||||
echo "Status Code Summary:"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn
|
||||
EOF
|
||||
chmod +x /tmp/monitor-traffic.sh
|
||||
```
|
||||
|
||||
### Run the monitoring script:
|
||||
```bash
|
||||
/tmp/monitor-traffic.sh
|
||||
```
|
||||
|
||||
## Auto-Refreshing Dashboard
|
||||
|
||||
### Live dashboard (refreshes every 5 seconds):
|
||||
```bash
|
||||
watch -n 5 'echo "=== Last 5 minutes ==="
|
||||
date
|
||||
echo ""
|
||||
echo "Top IPs:"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk "{print \$1}" | sort | uniq -c | sort -rn | head -10
|
||||
echo ""
|
||||
echo "Status Codes:"
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk "{print \$9}" | sort | uniq -c | sort -rn'
|
||||
```
|
||||
|
||||
Press `Ctrl+C` to exit.
|
||||
|
||||
## GoAccess HTML Report (Live Updating)
|
||||
|
||||
### Generate live HTML report:
|
||||
```bash
|
||||
sudo goaccess /var/log/nginx/access.log -o /var/www/html/live-stats.html --real-time-html --daemonize
|
||||
```
|
||||
|
||||
Then visit: https://git.laantungir.net/live-stats.html
|
||||
|
||||
### Stop the live report:
|
||||
```bash
|
||||
sudo pkill -f "goaccess.*live-stats"
|
||||
```
|
||||
|
||||
## Filter by Time
|
||||
|
||||
### Get timestamp from 5 minutes ago:
|
||||
```bash
|
||||
date -d '5 minutes ago' '+%d/%b/%Y:%H:%M'
|
||||
```
|
||||
|
||||
### Analyze only recent logs (replace timestamp):
|
||||
```bash
|
||||
sudo awk '/01\/Feb\/2026:19:09/,0' /var/log/nginx/access.log | goaccess -
|
||||
```
|
||||
|
||||
## Check Gitea CPU
|
||||
|
||||
### Current CPU usage:
|
||||
```bash
|
||||
ps aux | grep gitea | grep -v grep
|
||||
```
|
||||
|
||||
### Watch CPU in real-time:
|
||||
```bash
|
||||
watch -n 2 'ps aux | grep gitea | grep -v grep'
|
||||
```
|
||||
|
||||
## Most Useful Command for Quick Check
|
||||
|
||||
This one-liner shows everything you need:
|
||||
```bash
|
||||
|
||||
echo "=== Quick Status ===" && \
|
||||
echo "Time: $(date)" && \
|
||||
echo "" && \
|
||||
echo "Top 10 IPs (last 1000 requests):" && \
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10 && \
|
||||
echo "" && \
|
||||
echo "Status Codes:" && \
|
||||
sudo tail -1000 /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c && \
|
||||
echo "" && \
|
||||
echo "Gitea CPU:" && \
|
||||
ps aux | grep gitea | grep -v grep
|
||||
```
|
||||
|
||||
Copy any of these commands and run them directly on your server!
|
||||
@@ -41,7 +41,7 @@ This guide is specifically tailored for C programs that use:
|
||||
|
||||
```bash
|
||||
# 1. Copy the Dockerfile template (see below)
|
||||
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
|
||||
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
|
||||
|
||||
# 2. Customize for your project (see Customization section)
|
||||
vim Dockerfile.static
|
||||
@@ -466,13 +466,13 @@ docker build -t my-app:latest .
|
||||
docker run --rm my-app:latest --help
|
||||
```
|
||||
|
||||
## Reusing c-relay Files
|
||||
## Reusing c-relay-pg Files
|
||||
|
||||
You can directly copy these files from c-relay:
|
||||
You can directly copy these files from c-relay-pg:
|
||||
|
||||
### 1. Dockerfile.alpine-musl
|
||||
```bash
|
||||
cp /path/to/c-relay/Dockerfile.alpine-musl ./Dockerfile.static
|
||||
cp /path/to/c-relay-pg/Dockerfile.alpine-musl ./Dockerfile.static
|
||||
```
|
||||
|
||||
Then customize:
|
||||
@@ -482,7 +482,7 @@ Then customize:
|
||||
|
||||
### 2. build_static.sh
|
||||
```bash
|
||||
cp /path/to/c-relay/build_static.sh ./
|
||||
cp /path/to/c-relay-pg/build_static.sh ./
|
||||
```
|
||||
|
||||
Then customize:
|
||||
@@ -492,7 +492,7 @@ Then customize:
|
||||
|
||||
### 3. .dockerignore (Optional)
|
||||
```bash
|
||||
cp /path/to/c-relay/.dockerignore ./
|
||||
cp /path/to/c-relay-pg/.dockerignore ./
|
||||
```
|
||||
|
||||
Helps speed up Docker builds by excluding unnecessary files.
|
||||
@@ -522,7 +522,7 @@ Helps speed up Docker builds by excluding unnecessary files.
|
||||
- [Alpine Linux](https://alpinelinux.org/)
|
||||
- [nostr_core_lib](https://github.com/chebizarro/nostr_core_lib)
|
||||
- [Static Linking Best Practices](https://www.musl-libc.org/faq.html)
|
||||
- [c-relay Implementation](./docs/musl_static_build.md)
|
||||
- [c-relay-pg Implementation](./docs/musl_static_build.md)
|
||||
|
||||
## Example: Minimal Nostr Client
|
||||
|
||||
|
||||
+191
@@ -1291,6 +1291,184 @@ body.dark-mode .sql-results-table tbody tr:nth-child(even) {
|
||||
/* background-color: var(--secondary-color); */
|
||||
}
|
||||
|
||||
/* ================================
|
||||
WEB OF TRUST (WoT) STYLES
|
||||
================================ */
|
||||
|
||||
.wot-status-row, .wot-stats-row {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 8px 0;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.wot-indicator {
|
||||
padding: 2px 10px;
|
||||
border-radius: 4px;
|
||||
font-weight: bold;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.wot-indicator.wot-found { background: #28a745; color: white; }
|
||||
.wot-indicator.wot-missing { background: #dc3545; color: white; }
|
||||
.wot-indicator.wot-unknown { background: #6c757d; color: white; }
|
||||
|
||||
.wot-level-selector { padding: 10px 0; }
|
||||
.wot-level-selector label { display: block; margin-bottom: 5px; font-weight: bold; }
|
||||
|
||||
.wot-level-btn { min-width: 100px; }
|
||||
.wot-level-btn.active {
|
||||
background: var(--accent-color, #ff0000);
|
||||
color: white;
|
||||
border-color: var(--accent-color, #ff0000);
|
||||
}
|
||||
|
||||
.wot-level-description {
|
||||
font-size: 12px;
|
||||
color: var(--primary-color);
|
||||
margin-top: 5px;
|
||||
font-style: italic;
|
||||
opacity: 0.8;
|
||||
}
|
||||
|
||||
/* Dark mode adjustments for WoT */
|
||||
body.dark-mode .wot-indicator.wot-found { background: #28a745; }
|
||||
body.dark-mode .wot-indicator.wot-missing { background: #dc3545; }
|
||||
body.dark-mode .wot-indicator.wot-unknown { background: #6c757d; }
|
||||
|
||||
/* ================================
|
||||
ADMIN ACCESS GATE STYLES
|
||||
================================ */
|
||||
|
||||
/* Access Denied Overlay */
|
||||
.access-denied-overlay {
|
||||
position: fixed;
|
||||
top: 0;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: rgba(0, 0, 0, 0.85);
|
||||
z-index: 9999;
|
||||
display: none;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.access-denied-content {
|
||||
background: var(--card-bg);
|
||||
border: 2px solid #dc3545;
|
||||
border-radius: 12px;
|
||||
padding: 40px 60px;
|
||||
text-align: center;
|
||||
max-width: 500px;
|
||||
box-shadow: 0 10px 40px rgba(220, 53, 69, 0.3);
|
||||
}
|
||||
|
||||
.access-denied-icon {
|
||||
font-size: 64px;
|
||||
margin-bottom: 20px;
|
||||
}
|
||||
|
||||
.access-denied-content h2 {
|
||||
color: #dc3545;
|
||||
font-size: 32px;
|
||||
margin-bottom: 20px;
|
||||
letter-spacing: 2px;
|
||||
}
|
||||
|
||||
.access-denied-message {
|
||||
font-size: 16px;
|
||||
color: var(--primary-color);
|
||||
margin-bottom: 10px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.access-denied-submessage {
|
||||
font-size: 14px;
|
||||
color: var(--muted-color);
|
||||
margin-bottom: 30px;
|
||||
font-style: italic;
|
||||
}
|
||||
|
||||
.access-denied-logout-btn {
|
||||
background: #dc3545;
|
||||
color: white;
|
||||
border: none;
|
||||
padding: 12px 40px;
|
||||
font-size: 16px;
|
||||
font-weight: bold;
|
||||
border-radius: 6px;
|
||||
cursor: pointer;
|
||||
transition: all 0.3s ease;
|
||||
}
|
||||
|
||||
.access-denied-logout-btn:hover {
|
||||
background: #c82333;
|
||||
transform: translateY(-2px);
|
||||
box-shadow: 0 4px 12px rgba(220, 53, 69, 0.4);
|
||||
}
|
||||
|
||||
/* Admin Verification Loading Overlay */
|
||||
.admin-verification-overlay {
|
||||
position: fixed;
|
||||
top: 0;
|
||||
left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: rgba(0, 0, 0, 0.8);
|
||||
z-index: 9998;
|
||||
display: none;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
}
|
||||
|
||||
.admin-verification-content {
|
||||
background: var(--card-bg);
|
||||
border: 1px solid var(--border-color);
|
||||
border-radius: 12px;
|
||||
padding: 40px 60px;
|
||||
text-align: center;
|
||||
max-width: 450px;
|
||||
}
|
||||
|
||||
.admin-verification-content h3 {
|
||||
color: var(--accent-color);
|
||||
font-size: 20px;
|
||||
margin-bottom: 15px;
|
||||
}
|
||||
|
||||
.admin-verification-content p {
|
||||
color: var(--muted-color);
|
||||
font-size: 14px;
|
||||
margin-top: 15px;
|
||||
}
|
||||
|
||||
/* Spinner Animation */
|
||||
.spinner {
|
||||
width: 50px;
|
||||
height: 50px;
|
||||
border: 4px solid var(--border-color);
|
||||
border-top: 4px solid var(--accent-color);
|
||||
border-radius: 50%;
|
||||
animation: spin 1s linear infinite;
|
||||
margin: 0 auto 20px;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
0% { transform: rotate(0deg); }
|
||||
100% { transform: rotate(360deg); }
|
||||
}
|
||||
|
||||
/* Dark mode adjustments */
|
||||
body.dark-mode .access-denied-content {
|
||||
background: var(--card-bg);
|
||||
}
|
||||
|
||||
body.dark-mode .admin-verification-content {
|
||||
background: var(--card-bg);
|
||||
}
|
||||
|
||||
.subscription-detail-row:hover {
|
||||
background-color: var(--muted-color);
|
||||
}
|
||||
@@ -1308,3 +1486,16 @@ body.dark-mode .sql-results-table tbody tr:nth-child(even) {
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
|
||||
/* ================================
|
||||
IP BANS TABLE - compact rows
|
||||
================================ */
|
||||
#ip-bans-table td, #ip-bans-table th {
|
||||
padding: 4px 8px;
|
||||
line-height: 1.3;
|
||||
font-size: 13px;
|
||||
}
|
||||
#ip-bans-table button {
|
||||
padding: 2px 8px;
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
+336
-15
@@ -4,7 +4,7 @@
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>C-Relay Admin</title>
|
||||
<title>C-Relay-PG Admin</title>
|
||||
<link rel="stylesheet" href="/api/index.css">
|
||||
</head>
|
||||
|
||||
@@ -16,7 +16,9 @@
|
||||
<li><button class="nav-item" data-page="subscriptions">Subscriptions</button></li>
|
||||
<li><button class="nav-item" data-page="configuration">Configuration</button></li>
|
||||
<li><button class="nav-item" data-page="authorization">Authorization</button></li>
|
||||
<li><button class="nav-item" data-page="ip-bans">IP BANS</button></li>
|
||||
<li><button class="nav-item" data-page="relay-events">Relay Events</button></li>
|
||||
<li><button class="nav-item" data-page="caching">Caching</button></li>
|
||||
<li><button class="nav-item" data-page="dm">DM</button></li>
|
||||
<li><button class="nav-item" data-page="database">Database Query</button></li>
|
||||
</ul>
|
||||
@@ -41,7 +43,7 @@
|
||||
<span class="relay-letter" data-letter="Y">Y</span>
|
||||
</div>
|
||||
<div class="relay-info">
|
||||
<div id="relay-name" class="relay-name">C-Relay</div>
|
||||
<div id="relay-name" class="relay-name">C-Relay-PG</div>
|
||||
<div id="relay-description" class="relay-description">Loading...</div>
|
||||
<div id="relay-pubkey-container" class="relay-pubkey-container">
|
||||
<div id="relay-pubkey" class="relay-pubkey">Loading...</div>
|
||||
@@ -67,6 +69,17 @@
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Access Denied Overlay (shown when non-admin user logs in) -->
|
||||
<div id="access-denied-overlay" class="access-denied-overlay" style="display: none;">
|
||||
<div class="access-denied-content">
|
||||
<div class="access-denied-icon">⛔</div>
|
||||
<h2>ACCESS DENIED</h2>
|
||||
<p class="access-denied-message">This interface is restricted to the relay administrator.</p>
|
||||
<p class="access-denied-submessage">The logged-in account does not have admin privileges.</p>
|
||||
<button type="button" class="access-denied-logout-btn" onclick="logout()">LOGOUT</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- DATABASE STATISTICS Section -->
|
||||
<!-- Subscribe to kind 24567 events to receive real-time monitoring data -->
|
||||
<div class="section flex-section" id="databaseStatisticsSection" style="display: none;">
|
||||
@@ -100,6 +113,10 @@
|
||||
<td>Process ID</td>
|
||||
<td id="process-id">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>WebSocket Connections</td>
|
||||
<td id="websocket-connections">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Active Subscriptions</td>
|
||||
<td id="active-subscriptions">-</td>
|
||||
@@ -108,14 +125,14 @@
|
||||
<td>Memory Usage</td>
|
||||
<td id="memory-usage">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>CPU Core</td>
|
||||
<td id="cpu-core">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>CPU Usage</td>
|
||||
<td id="cpu-usage">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>CPU Core</td>
|
||||
<td id="cpu-core">-</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Oldest Event</td>
|
||||
<td id="oldest-event">-</td>
|
||||
@@ -251,6 +268,7 @@
|
||||
</div>
|
||||
|
||||
<!-- Auth Rules Management - Moved after configuration -->
|
||||
<!-- AUTH RULES MANAGEMENT SECTION -->
|
||||
<div class="section flex-section" id="authRulesSection" style="display: none;">
|
||||
<div class="section-header">
|
||||
AUTH RULES MANAGEMENT
|
||||
@@ -273,16 +291,11 @@ AUTH RULES MANAGEMENT
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<!-- Simplified Auth Rule Input Section -->
|
||||
<!-- Auth Rule Input Section -->
|
||||
<div id="authRuleInputSections" style="display: block;">
|
||||
|
||||
<!-- Combined Pubkey Auth Rule Section -->
|
||||
|
||||
|
||||
<div class="input-group">
|
||||
<label for="authRulePubkey">Pubkey (nsec or hex):</label>
|
||||
<input type="text" id="authRulePubkey" placeholder="nsec1... or 64-character hex pubkey">
|
||||
|
||||
<label for="authRulePubkey">Public Key (npub or hex):</label>
|
||||
<input type="text" id="authRulePubkey" placeholder="npub1... or 64-character hex pubkey">
|
||||
</div>
|
||||
<div id="whitelistWarning" class="warning-box" style="display: none;">
|
||||
<strong>⚠️ WARNING:</strong> Adding whitelist rules changes relay behavior to whitelist-only
|
||||
@@ -296,10 +309,51 @@ AUTH RULES MANAGEMENT
|
||||
BLACKLIST</button>
|
||||
<button type="button" id="refreshAuthRulesBtn">REFRESH</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- WEB OF TRUST SECTION -->
|
||||
<div class="section flex-section" id="wotSection" style="display: none;">
|
||||
<div class="section-header">
|
||||
WEB OF TRUST
|
||||
</div>
|
||||
|
||||
<!-- Kind 3 Status Indicator -->
|
||||
<div id="wotKind3Status" class="wot-status-row">
|
||||
<span>Admin Contact List (kind 3):</span>
|
||||
<span id="wotKind3Indicator" class="wot-indicator wot-unknown">Checking...</span>
|
||||
</div>
|
||||
|
||||
<!-- WoT Level Selector -->
|
||||
<div class="wot-level-selector">
|
||||
<label>WoT Level:</label>
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="wotLevel0Btn" class="wot-level-btn" onclick="setWotLevel(0)">
|
||||
OFF
|
||||
</button>
|
||||
<button type="button" id="wotLevel1Btn" class="wot-level-btn" onclick="setWotLevel(1)">
|
||||
WRITE ONLY
|
||||
</button>
|
||||
<button type="button" id="wotLevel2Btn" class="wot-level-btn" onclick="setWotLevel(2)">
|
||||
FULL
|
||||
</button>
|
||||
</div>
|
||||
<div class="wot-level-description" id="wotLevelDescription">
|
||||
Level 0: Open relay — anyone can read and write
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- WoT Stats -->
|
||||
<div class="wot-stats-row">
|
||||
<span>Whitelisted Pubkeys:</span>
|
||||
<span id="wotWhitelistCount">—</span>
|
||||
</div>
|
||||
|
||||
<!-- Sync Button -->
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="wotSyncBtn" onclick="syncWot()">SYNC FROM KIND 3</button>
|
||||
<button type="button" id="wotRefreshBtn" onclick="loadWotStatus()">REFRESH STATUS</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -334,12 +388,138 @@ AUTH RULES MANAGEMENT
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- IP BANS Section -->
|
||||
<div class="section" id="ipBansSection" style="display: none;">
|
||||
<div class="section-header">
|
||||
IP BAN MANAGEMENT
|
||||
</div>
|
||||
|
||||
<!-- Statistics Cards -->
|
||||
<div class="input-group">
|
||||
<div class="config-table-container">
|
||||
<table class="config-table" id="ip-bans-stats-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Total IPs Tracked</th>
|
||||
<th>Currently Banned</th>
|
||||
<th>Total Bans Issued</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td id="ip-bans-total">-</td>
|
||||
<td id="ip-bans-active">-</td>
|
||||
<td id="ip-bans-issued">-</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Add Ban Form -->
|
||||
<div class="input-group">
|
||||
<h3>Manually Ban IP Address</h3>
|
||||
<div class="form-group">
|
||||
<label for="ban-ip-input">IP Address:</label>
|
||||
<input type="text" id="ban-ip-input" placeholder="192.168.1.100">
|
||||
</div>
|
||||
<div class="form-group">
|
||||
<label for="ban-duration-select">Ban Duration:</label>
|
||||
<select id="ban-duration-select">
|
||||
<option value="3600">1 Hour</option>
|
||||
<option value="86400" selected>24 Hours</option>
|
||||
<option value="604800">7 Days</option>
|
||||
<option value="2592000">30 Days</option>
|
||||
<option value="31536000">1 Year</option>
|
||||
<option value="999999999">Permanent</option>
|
||||
</select>
|
||||
</div>
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="add-ban-btn">BAN IP</button>
|
||||
</div>
|
||||
<div id="add-ban-status" class="status-message"></div>
|
||||
</div>
|
||||
|
||||
<!-- Whitelist Management -->
|
||||
<div class="input-group">
|
||||
<h3>IP Whitelist (Never Banned)</h3>
|
||||
<p style="font-size:13px;opacity:0.8;">IPs in this list are never idle-banned. Comma-separated.</p>
|
||||
<div class="form-group">
|
||||
<label for="whitelist-ip-input">Add IP to Whitelist:</label>
|
||||
<input type="text" id="whitelist-ip-input" placeholder="103.81.231.220">
|
||||
</div>
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="add-whitelist-btn">ADD TO WHITELIST</button>
|
||||
</div>
|
||||
<div id="whitelist-status" class="status-message"></div>
|
||||
<div id="whitelist-current" style="margin-top:8px;font-size:13px;"></div>
|
||||
</div>
|
||||
|
||||
<!-- Filter Controls -->
|
||||
<div class="input-group">
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="ip-ban-filter-all" class="active">All IPs</button>
|
||||
<button type="button" id="ip-ban-filter-banned">Currently Banned</button>
|
||||
<button type="button" id="ip-ban-filter-expired">Expired</button>
|
||||
<button type="button" id="refresh-ip-bans-btn">REFRESH</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- IP Bans List -->
|
||||
<div class="input-group">
|
||||
<label>Banned IP Addresses:</label>
|
||||
<div class="config-table-container">
|
||||
<table class="config-table" id="ip-bans-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>IP Address</th>
|
||||
<th>Status</th>
|
||||
<th>Banned Until</th>
|
||||
<th>Failures</th>
|
||||
<th>Authed Successfully</th>
|
||||
<th>Connection Attempts</th>
|
||||
<th>Actions</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="ip-bans-tbody">
|
||||
<tr>
|
||||
<td colspan="7" style="text-align: center;">Click REFRESH to load IP bans</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- RELAY EVENTS Section -->
|
||||
<div class="section" id="relayEventsSection" style="display: none;">
|
||||
<div class="section-header">
|
||||
RELAY EVENTS MANAGEMENT
|
||||
</div>
|
||||
|
||||
<!-- Live Relay Event Feed -->
|
||||
<div class="input-group">
|
||||
<h3>Live Relay Event Feed (Normal Nostr Subscription)</h3>
|
||||
<div class="config-table-container">
|
||||
<table class="config-table" id="live-relay-events-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Time</th>
|
||||
<th>Kind</th>
|
||||
<th>Pubkey</th>
|
||||
<th>ID</th>
|
||||
<th>Content Preview</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="live-relay-events-table-body">
|
||||
<tr>
|
||||
<td colspan="5" style="text-align: center;">Waiting for live events...</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Kind 0: User Metadata -->
|
||||
<div class="input-group">
|
||||
<h3>Kind 0: User Metadata</h3>
|
||||
@@ -400,6 +580,143 @@ AUTH RULES MANAGEMENT
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- CACHING Section -->
|
||||
<div class="section" id="cachingSection" style="display: none;">
|
||||
<div class="section-header">CACHING</div>
|
||||
|
||||
<!-- Caching Service Status (read-only) -->
|
||||
<div class="input-group">
|
||||
<h3>Caching Service Status</h3>
|
||||
<div id="caching-service-status" class="status-display">
|
||||
<p>Service status will appear here after connecting.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Relay Inbox Status (read-only) -->
|
||||
<div class="input-group">
|
||||
<h3>Relay Inbox Status</h3>
|
||||
<div id="caching-inbox-status" class="status-display">
|
||||
<p>Inbox status will appear here after connecting.</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Caching Configuration (editable) -->
|
||||
<div class="input-group">
|
||||
<h3>Caching Configuration</h3>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-enabled">Enable Caching:</label>
|
||||
<select id="caching-enabled">
|
||||
<option value="false">Disabled</option>
|
||||
<option value="true">Enabled</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-inbox-enabled">Enable Inbox Consumer:</label>
|
||||
<select id="caching-inbox-enabled">
|
||||
<option value="false">Disabled</option>
|
||||
<option value="true">Enabled</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-root-npubs">Root Npubs (one per line):</label>
|
||||
<textarea id="caching-root-npubs" rows="4" placeholder="npub1...">npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks</textarea>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-bootstrap-relays">Bootstrap Relays (one per line):</label>
|
||||
<textarea id="caching-bootstrap-relays" rows="4" placeholder="wss://relay.damus.io">wss://relay.damus.io
|
||||
wss://nos.lol
|
||||
wss://relay.primal.net
|
||||
wss://laantungir.net/relay</textarea>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-kinds">Kinds (comma-separated):</label>
|
||||
<input type="text" id="caching-kinds" placeholder="1,3,6,10000,30023" value="0,1,3,6,10000,10002,30023">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-admin-kinds">Admin Kinds (comma-separated, * for all):</label>
|
||||
<input type="text" id="caching-admin-kinds" placeholder="*">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-live-enabled">Live Subscriptions:</label>
|
||||
<select id="caching-live-enabled">
|
||||
<option value="true">Enabled</option>
|
||||
<option value="false">Disabled</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-backfill-enabled">Backfill:</label>
|
||||
<select id="caching-backfill-enabled">
|
||||
<option value="true">Enabled</option>
|
||||
<option value="false">Disabled</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-backfill-windows">Backfill Windows (seconds, comma-separated):</label>
|
||||
<input type="text" id="caching-backfill-windows" placeholder="86400,604800,2592000,7776000,31536000">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-backfill-page-size">Backfill Page Size:</label>
|
||||
<input type="number" id="caching-backfill-page-size" placeholder="50">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-backfill-tick-interval">Backfill Tick Interval (ms):</label>
|
||||
<input type="number" id="caching-backfill-tick-interval" placeholder="5000">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-inbox-batch-size">Inbox Batch Size:</label>
|
||||
<input type="number" id="caching-inbox-batch-size" placeholder="25">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-inbox-active-poll">Inbox Active Poll (ms):</label>
|
||||
<input type="number" id="caching-inbox-active-poll" placeholder="200">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-inbox-idle-poll">Inbox Idle Poll (ms):</label>
|
||||
<input type="number" id="caching-inbox-idle-poll" placeholder="5000">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-service-binary-path">Caching Service Binary Path:</label>
|
||||
<input type="text" id="caching-service-binary-path" placeholder="./caching_relay" value="./caching_relay">
|
||||
</div>
|
||||
|
||||
<div class="form-group">
|
||||
<label for="caching-service-pg-conn">Caching Service PG Connection:</label>
|
||||
<input type="text" id="caching-service-pg-conn" placeholder="host=localhost port=5432 dbname=crelay user=crelay password=crelay" value="host=localhost port=5432 dbname=crelay user=crelay password=crelay">
|
||||
</div>
|
||||
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="caching-apply-btn">APPLY CONFIGURATION</button>
|
||||
<button type="button" id="caching-reset-progress-btn">RESET BACKFILL PROGRESS</button>
|
||||
</div>
|
||||
<div id="caching-config-status" class="status-message"></div>
|
||||
</div>
|
||||
|
||||
<div class="input-group">
|
||||
<h3>Caching Service Control</h3>
|
||||
<p>Start or stop the external caching service process. Set <code>caching_service_binary_path</code> and <code>caching_service_pg_conn</code> above and click APPLY CONFIGURATION before starting.</p>
|
||||
<div class="inline-buttons">
|
||||
<button type="button" id="caching-start-service-btn">START CACHING SERVICE</button>
|
||||
<button type="button" id="caching-stop-service-btn">STOP CACHING SERVICE</button>
|
||||
</div>
|
||||
<div id="caching-service-control-status" class="status-message"></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- SQL QUERY Section -->
|
||||
<div class="section" id="sqlQuerySection" style="display: none;">
|
||||
<div class="section-header">
|
||||
@@ -419,6 +736,10 @@ AUTH RULES MANAGEMENT
|
||||
<option value="event_kinds">Event Kinds Distribution</option>
|
||||
<option value="time_stats">Time-based Statistics</option>
|
||||
</optgroup>
|
||||
<optgroup label="IP Ban Queries">
|
||||
<option value="banned_ips_summary">Banned IPs Summary</option>
|
||||
<option value="banned_ips_list">All Banned IP Addresses</option>
|
||||
</optgroup>
|
||||
<optgroup label="Query History" id="history-group">
|
||||
<!-- Dynamically populated from localStorage -->
|
||||
</optgroup>
|
||||
|
||||
+1572
-104
File diff suppressed because it is too large
Load Diff
+95
-1
@@ -3297,8 +3297,18 @@ var NostrTools = (() => {
|
||||
this.enablePing = opts.enablePing;
|
||||
}
|
||||
async ensureRelay(url, params) {
|
||||
const rawUrl = url;
|
||||
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
|
||||
url = normalizeURL(url);
|
||||
let relay = this.relays.get(url);
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL ensureRelay START", {
|
||||
rawUrl,
|
||||
normalizedUrl: url,
|
||||
hadRelay: !!relay,
|
||||
relayMapKeysBefore: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
if (!relay) {
|
||||
relay = new AbstractRelay(url, {
|
||||
verifyEvent: this.trustedRelayURLs.has(url) ? alwaysTrue : this.verifyEvent,
|
||||
@@ -3306,13 +3316,38 @@ var NostrTools = (() => {
|
||||
enablePing: this.enablePing
|
||||
});
|
||||
relay.onclose = () => {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL ensureRelay onclose", {
|
||||
normalizedUrl: url,
|
||||
relayMapKeysBeforeDelete: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
this.relays.delete(url);
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL ensureRelay onclose complete", {
|
||||
normalizedUrl: url,
|
||||
relayMapKeysAfterDelete: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
};
|
||||
if (params?.connectionTimeout)
|
||||
relay.connectionTimeout = params.connectionTimeout;
|
||||
this.relays.set(url, relay);
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL ensureRelay created relay", {
|
||||
normalizedUrl: url,
|
||||
relayMapKeysAfterCreate: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
}
|
||||
await relay.connect();
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL ensureRelay connected", {
|
||||
normalizedUrl: url,
|
||||
relayConnected: relay.connected,
|
||||
relayMapKeysAfterConnect: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
return relay;
|
||||
}
|
||||
close(relays) {
|
||||
@@ -3334,16 +3369,26 @@ var NostrTools = (() => {
|
||||
}
|
||||
subscribeMany(relays, filters, params) {
|
||||
params.onauth = params.onauth || params.doauth;
|
||||
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
|
||||
const request = [];
|
||||
const uniqUrls = [];
|
||||
for (let i2 = 0; i2 < relays.length; i2++) {
|
||||
const url = normalizeURL(relays[i2]);
|
||||
if (uniqUrls.indexOf(url) === -1) {
|
||||
uniqUrls.push(url);
|
||||
for (let f2 = 0; f2 < filters.length; f2++) {
|
||||
request.push({ url, filter: filters[f2] });
|
||||
}
|
||||
}
|
||||
}
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL subscribeMany", {
|
||||
relaysInput: relays,
|
||||
uniqUrls,
|
||||
filtersCount: filters.length,
|
||||
requestsCount: request.length
|
||||
});
|
||||
}
|
||||
return this.subscribeMap(request, params);
|
||||
}
|
||||
subscribeMap(requests, params) {
|
||||
@@ -3391,14 +3436,31 @@ var NostrTools = (() => {
|
||||
_knownIds.add(id);
|
||||
return have;
|
||||
};
|
||||
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
|
||||
const allOpened = Promise.all(
|
||||
requests.map(async ({ url, filter }, i2) => {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL subscribeMap request", {
|
||||
index: i2,
|
||||
url,
|
||||
filterKinds: filter?.kinds,
|
||||
hasAuthorFilter: !!filter?.authors,
|
||||
hasPTagFilter: !!filter?.["#p"]
|
||||
});
|
||||
}
|
||||
let relay;
|
||||
try {
|
||||
relay = await this.ensureRelay(url, {
|
||||
connectionTimeout: params.maxWait ? Math.max(params.maxWait * 0.8, params.maxWait - 1e3) : void 0
|
||||
});
|
||||
} catch (err) {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL subscribeMap ensureRelay FAILED", {
|
||||
index: i2,
|
||||
url,
|
||||
error: err?.message || String(err)
|
||||
});
|
||||
}
|
||||
handleClose(i2, err?.message || String(err));
|
||||
return;
|
||||
}
|
||||
@@ -3480,18 +3542,50 @@ var NostrTools = (() => {
|
||||
return events[0] || null;
|
||||
}
|
||||
publish(relays, event, options) {
|
||||
return relays.map(normalizeURL).map(async (url, i2, arr) => {
|
||||
const debugEnabled = typeof window !== "undefined" && !!window.__SIMPLE_POOL_DEBUG__;
|
||||
const normalizedRelays = relays.map(normalizeURL);
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL publish START", {
|
||||
relaysInput: relays,
|
||||
normalizedRelays,
|
||||
eventKind: event?.kind,
|
||||
eventId: event?.id
|
||||
});
|
||||
}
|
||||
return normalizedRelays.map(async (url, i2, arr) => {
|
||||
if (arr.indexOf(url) !== i2) {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL publish duplicate URL", { url, index: i2 });
|
||||
}
|
||||
return Promise.reject("duplicate url");
|
||||
}
|
||||
let r = await this.ensureRelay(url);
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL publish ensured relay", {
|
||||
url,
|
||||
relayConnected: r?.connected,
|
||||
relayMapKeys: Array.from(this.relays.keys())
|
||||
});
|
||||
}
|
||||
return r.publish(event).catch(async (err) => {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL publish ERROR", {
|
||||
url,
|
||||
error: err?.message || String(err)
|
||||
});
|
||||
}
|
||||
if (err instanceof Error && err.message.startsWith("auth-required: ") && options?.onauth) {
|
||||
await r.auth(options.onauth);
|
||||
return r.publish(event);
|
||||
}
|
||||
throw err;
|
||||
}).then((reason) => {
|
||||
if (debugEnabled) {
|
||||
console.log("🔎 SIMPLE_POOL publish SUCCESS", {
|
||||
url,
|
||||
reason
|
||||
});
|
||||
}
|
||||
if (this.trackRelays) {
|
||||
let set = this.seenOn.get(event.id);
|
||||
if (!set) {
|
||||
|
||||
@@ -0,0 +1,813 @@
|
||||
<!DOCTYPE html>
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<html lang="en" dir="ltr">
|
||||
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>TEMPLATE</title>
|
||||
|
||||
<link rel="stylesheet" href="./css/client.css" />
|
||||
|
||||
<!-- Initialize theme BEFORE any components load -->
|
||||
<script>
|
||||
(function () {
|
||||
const savedTheme = localStorage.getItem('theme');
|
||||
if (savedTheme === 'dark') {
|
||||
document.documentElement.classList.add('dark-mode');
|
||||
if (document.body) {
|
||||
document.body.classList.add('dark-mode');
|
||||
}
|
||||
}
|
||||
})();
|
||||
</script>
|
||||
<link rel="shortcut icon" type="image/x-icon" href="./favicon/favicon-dots2.ico" />
|
||||
|
||||
<!-- SVG.js library (required by HamburgerMorphing) -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/@svgdotjs/svg.js@3.0/dist/svg.min.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<!-- ================================================================
|
||||
HAMBURGER BUTTON (Fixed, separate from header)
|
||||
================================================================
|
||||
The hamburger button is a fixed element outside the header
|
||||
to ensure it stays visible above the sidenav (z-index: 10 > 3).
|
||||
================================================================ -->
|
||||
<div id="divSvgHam" class="divHeaderButtons">
|
||||
<!-- HamburgerMorphing will be injected here -->
|
||||
</div>
|
||||
|
||||
<!-- ================================================================
|
||||
HEADER
|
||||
================================================================
|
||||
Standard header with title (center).
|
||||
================================================================ -->
|
||||
<div id="divHeader">
|
||||
<div id="divHeaderFlexLeft">
|
||||
<!-- Hamburger is now separate fixed element -->
|
||||
</div>
|
||||
|
||||
<div id="divHeaderFlexCenter">
|
||||
<div class="divHeaderText"></div>
|
||||
</div>
|
||||
|
||||
<div id="divHeaderFlexRight">
|
||||
<!-- No button in header right - logout is in sidenav footer -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ================================================================
|
||||
BODY
|
||||
================================================================
|
||||
Main content area. Add your page-specific content here.
|
||||
================================================================ -->
|
||||
<div id="divBody">
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<!-- ================================================================
|
||||
FOOTER
|
||||
================================================================
|
||||
Three-section footer layout:
|
||||
- Left: Relay status animations (HamburgerMorphing instances)
|
||||
- Center: General status information
|
||||
- Right: Additional information
|
||||
================================================================ -->
|
||||
<div id="divFooter">
|
||||
<div id="divFooterLeft" class="divFooterBox"></div>
|
||||
<div id="divFooterCenter" class="divFooterBox"></div>
|
||||
<div id="divFooterRight" class="divFooterBox"></div>
|
||||
<div id="divFooterBalance" class="divFooterBox">0 sats</div>
|
||||
</div>
|
||||
|
||||
<!-- ================================================================
|
||||
SIDENAV
|
||||
================================================================
|
||||
Slide-out navigation panel. Opens from left when hamburger clicked.
|
||||
Uses flexbox layout to pin version bar to bottom.
|
||||
Includes a version bar footer with theme toggle and logout buttons.
|
||||
================================================================ -->
|
||||
<div id="divSideNav">
|
||||
<div id="divSideNavHeader">
|
||||
<!-- No close button - use main hamburger to close -->
|
||||
</div>
|
||||
|
||||
<div id="divSideNavBody">
|
||||
<div id="divFiles"></div>
|
||||
</div>
|
||||
|
||||
<div id="divAiSection" class="sidenavSection">
|
||||
<div id="divAiSectionTitle" class="sidenavSectionTitle">AI</div>
|
||||
<div id="divAiList" class="sidenavSectionList">
|
||||
<div id="divAiProvidersList">No saved providers yet.</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div id="divRelaySection">
|
||||
<div id="divRelaySectionTitle">
|
||||
リレー
|
||||
</div>
|
||||
<div id="divRelayList">
|
||||
Loading relays...
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div id="divBlossomSection">
|
||||
|
||||
<div id="divBlossomSectionTitle">ブロッサム</div>
|
||||
|
||||
<div id="divBlossomList">Loading blossom servers...</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<div id="divVersionBar">
|
||||
<span id="versionDisplay">v0.0.1</span>
|
||||
<div id="divVersionBarButtons">
|
||||
<button id="themeToggleButton" title="Toggle Dark/Light Mode">
|
||||
<div id="themeToggleHamburgerContainer"></div>
|
||||
</button>
|
||||
<button id="logoutButton" title="Logout">
|
||||
<div id="logoutHamburgerContainer"></div>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- ================================================================
|
||||
REQUIRED SCRIPTS
|
||||
================================================================
|
||||
These scripts must be loaded in this order:
|
||||
1. nostr.bundle.js - Nostr tools library
|
||||
2. nostr-lite.js - Authentication modal (nostr-login-lite)
|
||||
================================================================ -->
|
||||
<script src="./nostr.bundle.js"></script>
|
||||
<script src="./nostr-lite.js"></script>
|
||||
|
||||
<script type="module">
|
||||
/* ================================================================
|
||||
IMPORTS
|
||||
================================================================
|
||||
Import shared NDK functionality from init-ndk.mjs:
|
||||
- initNDKPage() - Initialize authentication and worker
|
||||
- getPubkey() - Get current user's pubkey
|
||||
- subscribe() - Create NDK subscriptions
|
||||
- publishEvent() - Publish events via NDK
|
||||
- disconnect() - Disconnect from worker
|
||||
- getRelayData() - Get relay connection data
|
||||
- getRelayStats() - Get relay activity statistics
|
||||
|
||||
Import HamburgerMorphing for animated icons
|
||||
================================================================ */
|
||||
import {
|
||||
initNDKPage,
|
||||
getPubkey, injectHeaderAvatar,
|
||||
subscribe,
|
||||
publishEvent,
|
||||
disconnect,
|
||||
getVersion,
|
||||
updateVersionDisplay,
|
||||
getUserSettings,
|
||||
patchUserSettings,
|
||||
onUserSettings
|
||||
} from './js/init-ndk.mjs';
|
||||
import { HamburgerMorphing } from "./hamburger_morphing/hamburger.mjs";
|
||||
import { initFooterRelayStatus, updateFooterRelayStatus, initSidenavRelaySection, updateSidenavRelaySection, setRelayActivityState } from './js/relay-ui.mjs';
|
||||
|
||||
import { initBlossomSection, updateBlossomSection } from './js/blossom-ui.mjs';
|
||||
|
||||
import { initAiSectionWithLocalConfig } from './js/ai-ui.mjs';
|
||||
// Version will be loaded asynchronously
|
||||
const versionInfo = await getVersion();
|
||||
const VERSION = versionInfo.VERSION;
|
||||
console.log(`[template.html ${VERSION}] Loading...`);
|
||||
|
||||
/* ================================================================
|
||||
GLOBAL VARIABLES
|
||||
================================================================
|
||||
Track state for hamburger menu, relay status, and theme.
|
||||
================================================================ */
|
||||
let updateIntervalId = null;
|
||||
let currentPubkey = null;
|
||||
|
||||
/*
|
||||
AUTH STATE MODEL (Template reference)
|
||||
------------------------------------------------------------------
|
||||
This template now demonstrates three auth modes for standalone pages:
|
||||
|
||||
- required (default):
|
||||
Behaves like existing pages: login is required immediately.
|
||||
|
||||
- optional:
|
||||
Page can render public/read-only data without login, but can still
|
||||
prompt login later for user actions (publish, settings, etc).
|
||||
|
||||
- none:
|
||||
Never auto-login on load (pure public page).
|
||||
|
||||
URL behavior in this template:
|
||||
- If ?auth=required|optional|none is present, it wins.
|
||||
- Otherwise, if URL includes ?npub=... or ?pubkey=..., mode defaults
|
||||
to optional because pages with explicit profile targets are commonly
|
||||
public-readable.
|
||||
- Otherwise, mode defaults to required.
|
||||
*/
|
||||
let isAuthenticated = false;
|
||||
let authMode = 'required';
|
||||
let authedPageInitialized = false;
|
||||
let relayActivityListenersBound = false;
|
||||
|
||||
// Hamburger menu
|
||||
let hamburgerInstance = null;
|
||||
let isNavOpen = false;
|
||||
|
||||
// Version bar buttons
|
||||
let logoutHamburger = null;
|
||||
let themeToggleHamburger = null;
|
||||
let isDarkMode = false;
|
||||
|
||||
// App-wide user settings (NIP-78 kind 30078, d:user-settings)
|
||||
let pageSettings = {};
|
||||
let unsubscribeUserSettings = null;
|
||||
|
||||
/* ================================================================
|
||||
DOM VARIABLES
|
||||
================================================================
|
||||
Cache DOM element references for better performance.
|
||||
================================================================ */
|
||||
const divBody = document.getElementById("divBody");
|
||||
const divSideNav = document.getElementById("divSideNav");
|
||||
const divSideNavBody = document.getElementById("divSideNavBody");
|
||||
const divFooterCenter = document.getElementById("divFooterCenter");
|
||||
const divFooterRight = document.getElementById("divFooterRight");
|
||||
|
||||
/* ================================================================
|
||||
HAMBURGER MENU
|
||||
================================================================
|
||||
Initialize and control the animated hamburger menu.
|
||||
================================================================ */
|
||||
function initHamburgerMenu() {
|
||||
hamburgerInstance = new HamburgerMorphing('#divSvgHam', {
|
||||
foreground: 'var(--primary-color)',
|
||||
background: 'var(--secondary-color)',
|
||||
hover: 'var(--accent-color)'
|
||||
});
|
||||
hamburgerInstance.animateTo('burger');
|
||||
}
|
||||
|
||||
/* ================================================================
|
||||
SIDENAV FUNCTIONS
|
||||
================================================================
|
||||
Open/close sidenav with hamburger morphing animation.
|
||||
================================================================ */
|
||||
function openNav() {
|
||||
divSideNav.style.zIndex = 3;
|
||||
divSideNav.style.width = "clamp(400px, 50vw, 600px)";
|
||||
isNavOpen = true;
|
||||
if (hamburgerInstance) {
|
||||
hamburgerInstance.animateTo('arrow_left');
|
||||
}
|
||||
|
||||
|
||||
// Initialize version bar buttons when sidenav opens (lazy load)
|
||||
if (!logoutHamburger) {
|
||||
logoutHamburger = new HamburgerMorphing('#logoutHamburgerContainer', {
|
||||
size: 24,
|
||||
foreground: 'var(--primary-color)',
|
||||
background: 'var(--secondary-color)',
|
||||
hover: 'var(--accent-color)'
|
||||
});
|
||||
logoutHamburger.animateTo('x');
|
||||
}
|
||||
|
||||
if (!themeToggleHamburger) {
|
||||
themeToggleHamburger = new HamburgerMorphing('#themeToggleHamburgerContainer', {
|
||||
size: 24,
|
||||
foreground: 'var(--primary-color)',
|
||||
background: 'var(--secondary-color)',
|
||||
hover: 'var(--accent-color)'
|
||||
});
|
||||
|
||||
// Determine current theme
|
||||
const savedTheme = localStorage.getItem('theme');
|
||||
isDarkMode = savedTheme === 'dark' || document.body.classList.contains('dark-mode');
|
||||
const initialShape = isDarkMode ? 'moon' : 'circle';
|
||||
themeToggleHamburger.animateTo(initialShape);
|
||||
}
|
||||
}
|
||||
|
||||
function closeNav() {
|
||||
divSideNav.style.width = "0vw";
|
||||
divSideNav.style.zIndex = -1;
|
||||
isNavOpen = false;
|
||||
if (hamburgerInstance) {
|
||||
hamburgerInstance.animateTo('burger');
|
||||
}
|
||||
}
|
||||
|
||||
function toggleNav() {
|
||||
if (isNavOpen) {
|
||||
closeNav();
|
||||
} else {
|
||||
openNav();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/* ================================================================
|
||||
AUTH MODE HELPERS
|
||||
================================================================
|
||||
These functions are meant as reusable guidance for future pages.
|
||||
================================================================ */
|
||||
function hasTargetPubkeyInUrl() {
|
||||
const params = new URLSearchParams(window.location.search || '');
|
||||
const npub = String(params.get('npub') || '').trim();
|
||||
const pubkey = String(params.get('pubkey') || '').trim();
|
||||
return Boolean(npub || pubkey);
|
||||
}
|
||||
|
||||
function resolveAuthModeFromUrl() {
|
||||
const params = new URLSearchParams(window.location.search || '');
|
||||
const explicitAuth = String(params.get('auth') || '').trim().toLowerCase();
|
||||
if (explicitAuth === 'required' || explicitAuth === 'optional' || explicitAuth === 'none') {
|
||||
return explicitAuth;
|
||||
}
|
||||
|
||||
// Convention: explicit target profiles are public-readable by default.
|
||||
if (hasTargetPubkeyInUrl()) {
|
||||
return 'optional';
|
||||
}
|
||||
|
||||
return 'required';
|
||||
}
|
||||
|
||||
function isAuthRequiredError(error) {
|
||||
const message = String(error?.message || error || '').toLowerCase();
|
||||
return message.includes('authentication required');
|
||||
}
|
||||
|
||||
async function initializeAuthentication(mode) {
|
||||
// required: existing behavior, throw if auth fails.
|
||||
if (mode === 'required') {
|
||||
await initNDKPage();
|
||||
currentPubkey = await getPubkey();
|
||||
isAuthenticated = true;
|
||||
return;
|
||||
}
|
||||
|
||||
// none: public page, no login attempt on load.
|
||||
if (mode === 'none') {
|
||||
isAuthenticated = false;
|
||||
currentPubkey = null;
|
||||
return;
|
||||
}
|
||||
|
||||
// optional: try silent/normal init; if auth required, continue public.
|
||||
try {
|
||||
await initNDKPage();
|
||||
currentPubkey = await getPubkey();
|
||||
isAuthenticated = true;
|
||||
} catch (error) {
|
||||
if (isAuthRequiredError(error)) {
|
||||
console.log('[template.html] Optional auth mode: continuing unauthenticated');
|
||||
isAuthenticated = false;
|
||||
currentPubkey = null;
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function initializeAuthenticatedPageFeatures() {
|
||||
if (!isAuthenticated || authedPageInitialized) return;
|
||||
|
||||
await injectHeaderAvatar(currentPubkey);
|
||||
console.log('[template.html] Authenticated as:', currentPubkey);
|
||||
|
||||
// Hydrate app-wide user settings for this page
|
||||
try {
|
||||
pageSettings = await getUserSettings();
|
||||
} catch (error) {
|
||||
console.warn('[template.html] getUserSettings failed:', error);
|
||||
pageSettings = {};
|
||||
}
|
||||
|
||||
// Subscribe to live user settings updates (cross-tab + publish echoes)
|
||||
if (!unsubscribeUserSettings) {
|
||||
unsubscribeUserSettings = onUserSettings((settings) => {
|
||||
pageSettings = settings || {};
|
||||
// TODO: Re-render page-specific UI from pageSettings here.
|
||||
});
|
||||
}
|
||||
|
||||
// Initialize relay-dependent UI only once authenticated.
|
||||
initFooterRelayStatus();
|
||||
initSidenavRelaySection();
|
||||
await initBlossomSection();
|
||||
initAiSectionWithLocalConfig();
|
||||
await UpdateFooter();
|
||||
|
||||
if (!updateIntervalId) {
|
||||
updateIntervalId = setInterval(UpdateFooter, 1000);
|
||||
}
|
||||
|
||||
// Relay activity listeners only matter after worker init/auth.
|
||||
if (!relayActivityListenersBound) {
|
||||
window.addEventListener('ndkRelayActivity', (event) => {
|
||||
const { relayUrl, activity, stats } = event.detail;
|
||||
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`, stats);
|
||||
setRelayActivityState(relayUrl, activity);
|
||||
});
|
||||
|
||||
window.addEventListener('message', (event) => {
|
||||
if (event.data && event.data.type === 'relayActivity') {
|
||||
const { relayUrl, activity } = event.data;
|
||||
console.log(`[template.html] Relay activity: ${relayUrl} - ${activity}`);
|
||||
setRelayActivityState(relayUrl, activity);
|
||||
}
|
||||
});
|
||||
|
||||
relayActivityListenersBound = true;
|
||||
}
|
||||
|
||||
authedPageInitialized = true;
|
||||
}
|
||||
|
||||
async function promptLoginIfNeeded() {
|
||||
if (isAuthenticated) return true;
|
||||
|
||||
await initNDKPage();
|
||||
currentPubkey = await getPubkey();
|
||||
isAuthenticated = true;
|
||||
await initializeAuthenticatedPageFeatures();
|
||||
return true;
|
||||
}
|
||||
|
||||
/* ================================================================
|
||||
UPDATE FOOTER
|
||||
================================================================
|
||||
Update footer sections with relay status, pubkey, and other info.
|
||||
Called periodically by update loop.
|
||||
================================================================ */
|
||||
const UpdateFooter = async () => {
|
||||
|
||||
try {
|
||||
// Update relay status visuals in footer and sidenav
|
||||
await updateFooterRelayStatus();
|
||||
await updateSidenavRelaySection();
|
||||
|
||||
await updateBlossomSection();
|
||||
// Clear center and right sections
|
||||
divFooterCenter.innerHTML = '';
|
||||
divFooterRight.innerHTML = '';
|
||||
} catch (error) {
|
||||
console.error('[template.html] Error updating footer:', error);
|
||||
}
|
||||
};
|
||||
|
||||
/* ================================================================
|
||||
LOGOUT
|
||||
================================================================
|
||||
Complete logout process:
|
||||
1. Stop update loop
|
||||
2. Disconnect from NDK worker
|
||||
3. Logout from nostr-login-lite
|
||||
4. Clear all storage (localStorage, sessionStorage, IndexedDB)
|
||||
5. Reload page
|
||||
================================================================ */
|
||||
const Logout = async () => {
|
||||
console.log("[template.html] Starting logout process...");
|
||||
|
||||
// Stop the update loop
|
||||
if (updateIntervalId) {
|
||||
clearInterval(updateIntervalId);
|
||||
updateIntervalId = null;
|
||||
}
|
||||
|
||||
// Disconnect from worker
|
||||
disconnect();
|
||||
|
||||
// Logout from nostr-login-lite
|
||||
if (window.NOSTR_LOGIN_LITE && window.NOSTR_LOGIN_LITE.logout) {
|
||||
await window.NOSTR_LOGIN_LITE.logout();
|
||||
}
|
||||
|
||||
// Clear all storage
|
||||
localStorage.clear();
|
||||
sessionStorage.clear();
|
||||
|
||||
// Clear IndexedDB
|
||||
if (window.indexedDB) {
|
||||
const databases = await window.indexedDB.databases();
|
||||
for (const db of databases) {
|
||||
if (db.name) {
|
||||
window.indexedDB.deleteDatabase(db.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
console.log("[template.html] Logged out, reloading page");
|
||||
location.reload(true);
|
||||
};
|
||||
|
||||
/* ================================================================
|
||||
EVENT LISTENERS
|
||||
================================================================
|
||||
Wire up UI interactions.
|
||||
Main hamburger button click handler is set up in main() after initialization.
|
||||
================================================================ */
|
||||
|
||||
/* ================================================================
|
||||
SUBSCRIPTION EXAMPLE
|
||||
================================================================
|
||||
Example of how to subscribe to Nostr events:
|
||||
|
||||
const sub = subscribe(
|
||||
{ kinds: [1], authors: [pubkey], limit: 10 },
|
||||
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
|
||||
);
|
||||
|
||||
Cache usage options:
|
||||
- 'CACHE_FIRST' - Check cache first, then relays
|
||||
- 'ONLY_RELAY' - Only query relays
|
||||
- 'ONLY_CACHE' - Only query cache
|
||||
- 'PARALLEL' - Query cache and relays simultaneously
|
||||
|
||||
Listen for events via window events:
|
||||
window.addEventListener('ndkEvent', (event) => {
|
||||
const evt = event.detail;
|
||||
console.log('Received event:', evt);
|
||||
});
|
||||
================================================================ */
|
||||
|
||||
/* ================================================================
|
||||
PUBLISH EXAMPLE
|
||||
================================================================
|
||||
Example of how to publish a Nostr event:
|
||||
|
||||
const event = {
|
||||
created_at: Math.floor(Date.now() / 1000),
|
||||
kind: 1,
|
||||
tags: [],
|
||||
content: "Hello, Nostr!"
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await publishEvent(event);
|
||||
console.log("✅ Published to:", result.relayResults.successful);
|
||||
console.log("❌ Failed:", result.relayResults.failed);
|
||||
console.log("Total relays:", result.totalRelays);
|
||||
} catch (error) {
|
||||
console.error("Publish error:", error);
|
||||
}
|
||||
|
||||
Note: Events are automatically signed by the NDK worker using
|
||||
the message-based signer (which calls window.nostr.signEvent).
|
||||
================================================================ */
|
||||
|
||||
/* ================================================================
|
||||
USER SETTINGS EXAMPLE (NIP-78)
|
||||
================================================================
|
||||
Read/subscribe/write helper pattern for all pages:
|
||||
|
||||
// Read latest merged settings (cache + relay hydrated by worker)
|
||||
const settings = await getUserSettings();
|
||||
|
||||
// Subscribe to cross-tab updates
|
||||
const unsubscribe = onUserSettings((nextSettings) => {
|
||||
// Re-render page from nextSettings
|
||||
});
|
||||
|
||||
// Patch only your feature namespace
|
||||
await patchUserSettings({
|
||||
myFeature: {
|
||||
someFlag: true
|
||||
}
|
||||
});
|
||||
|
||||
// On page teardown (if applicable)
|
||||
// unsubscribe();
|
||||
================================================================ */
|
||||
|
||||
/* ================================================================
|
||||
INITIALIZATION
|
||||
================================================================
|
||||
Main initialization sequence:
|
||||
1. Initialize hamburger menu
|
||||
2. Set up hamburger click handler
|
||||
3. Resolve auth mode from URL/query policy
|
||||
4. Initialize authentication based on mode
|
||||
5. Initialize authenticated-only features (if signed in)
|
||||
6. Set up version bar button listeners
|
||||
7. Restore sidenav state
|
||||
8. Update version display
|
||||
|
||||
Notes:
|
||||
- required mode = existing behavior (prompt login on load)
|
||||
- optional mode = allow public load, login later on demand
|
||||
- none mode = no auto-login on load
|
||||
================================================================ */
|
||||
(async function main() {
|
||||
console.log("[template.html] Starting initialization...");
|
||||
|
||||
try {
|
||||
// Initialize hamburger menu first
|
||||
initHamburgerMenu();
|
||||
|
||||
// Add click handler to hamburger
|
||||
const divSvgHam = document.getElementById('divSvgHam');
|
||||
if (divSvgHam) {
|
||||
divSvgHam.addEventListener('click', toggleNav);
|
||||
}
|
||||
|
||||
// Initialize version bar buttons
|
||||
const themeToggleButton = document.getElementById('themeToggleButton');
|
||||
const logoutButton = document.getElementById('logoutButton');
|
||||
|
||||
if (themeToggleButton) {
|
||||
themeToggleButton.addEventListener('click', () => {
|
||||
isDarkMode = !isDarkMode;
|
||||
if (isDarkMode) {
|
||||
localStorage.setItem('theme', 'dark');
|
||||
} else {
|
||||
localStorage.setItem('theme', 'light');
|
||||
}
|
||||
// Save sidenav state before reload
|
||||
localStorage.setItem('sidenavWasOpen', isNavOpen ? 'true' : 'false');
|
||||
window.location.reload();
|
||||
});
|
||||
}
|
||||
|
||||
if (logoutButton) {
|
||||
logoutButton.addEventListener('click', async () => {
|
||||
try {
|
||||
// In optional/none modes this doubles as a "Sign in" entry point.
|
||||
if (!isAuthenticated) {
|
||||
await promptLoginIfNeeded();
|
||||
return;
|
||||
}
|
||||
await Logout();
|
||||
} catch (error) {
|
||||
console.error('Logout/login action failed:', error);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// Resolve and initialize page auth policy.
|
||||
authMode = resolveAuthModeFromUrl();
|
||||
console.log('[template.html] Resolved auth mode:', authMode);
|
||||
await initializeAuthentication(authMode);
|
||||
|
||||
// Initialize authenticated features only when signed in.
|
||||
await initializeAuthenticatedPageFeatures();
|
||||
|
||||
/* ============================================================
|
||||
EXAMPLE: Subscribe to events
|
||||
============================================================
|
||||
Uncomment to subscribe to user's notes:
|
||||
|
||||
const notesSub = subscribe(
|
||||
{ kinds: [1], authors: [currentPubkey], limit: 10 },
|
||||
{ closeOnEose: false, cacheUsage: 'CACHE_FIRST' }
|
||||
);
|
||||
console.log("[template.html] Subscribed to kind 1");
|
||||
============================================================ */
|
||||
|
||||
/* ============================================================
|
||||
EXAMPLE: Listen for events from worker
|
||||
============================================================
|
||||
Uncomment to handle incoming events:
|
||||
|
||||
window.addEventListener('ndkEvent', (event) => {
|
||||
const evt = event.detail;
|
||||
console.log("[template.html] Received event:", evt.kind, evt.pubkey);
|
||||
|
||||
if (evt.pubkey === currentPubkey) {
|
||||
if (evt.kind === 1) {
|
||||
// Handle note event
|
||||
console.log("Note:", evt.content);
|
||||
}
|
||||
}
|
||||
});
|
||||
============================================================ */
|
||||
|
||||
/* ============================================================
|
||||
EXAMPLE: Listen for cached profile
|
||||
============================================================
|
||||
Uncomment to handle cached profile data:
|
||||
|
||||
window.addEventListener('ndkProfile', (event) => {
|
||||
console.log("[template.html] Cached profile:", event.detail);
|
||||
// event.detail contains profile object (name, about, etc.)
|
||||
});
|
||||
============================================================ */
|
||||
|
||||
// Restore sidenav state if it was open before theme toggle
|
||||
const sidenavWasOpen = localStorage.getItem('sidenavWasOpen');
|
||||
if (sidenavWasOpen === 'true') {
|
||||
localStorage.removeItem('sidenavWasOpen');
|
||||
openNav();
|
||||
}
|
||||
|
||||
// Optional UX note for public mode pages.
|
||||
if (!isAuthenticated && (authMode === 'optional' || authMode === 'none')) {
|
||||
divFooterCenter.textContent = 'Public mode';
|
||||
divFooterRight.textContent = 'Sign in from side menu for private features';
|
||||
}
|
||||
|
||||
// Update version display
|
||||
await updateVersionDisplay();
|
||||
|
||||
console.log('[template.html] Initialization complete');
|
||||
} catch (error) {
|
||||
console.error('[template.html] Initialization failed:', error);
|
||||
divBody.innerHTML = `<div style="text-align: center; padding: 50px;">
|
||||
<div style="font-size: 24px; margin-bottom: 20px; color: red;">❌ Authentication Error</div>
|
||||
<div style="font-size: 16px; color: #666;">${error.message}</div>
|
||||
<div style="margin-top: 20px;">
|
||||
<button onclick="location.reload()" style="padding: 10px 20px; font-size: 16px;">Retry</button>
|
||||
</div>
|
||||
</div>`;
|
||||
}
|
||||
})();
|
||||
|
||||
/* ================================================================
|
||||
WORKER MESSAGE TYPES
|
||||
================================================================
|
||||
The NDK worker can send these message types:
|
||||
|
||||
1. 'response' - Response to init/subscribe/publish requests
|
||||
- data.profile - User profile (from init)
|
||||
- data.relays - User relays (from init)
|
||||
- data.success - Publish success status
|
||||
- data.relayResults - Relay publish results
|
||||
|
||||
2. 'event' - Nostr event from subscription
|
||||
- Dispatched as 'ndkEvent' window event
|
||||
- event.detail contains the Nostr event
|
||||
|
||||
3. 'eose' - End of stored events for subscription
|
||||
- Dispatched as 'ndkEose' window event
|
||||
- event.detail.subId contains subscription ID
|
||||
|
||||
4. 'signRequest' - Request to sign event/encrypt/decrypt
|
||||
- Handled automatically by init-ndk.mjs
|
||||
- Calls window.nostr methods and sends response
|
||||
|
||||
5. 'error' - Error from worker
|
||||
- Logged to console automatically
|
||||
|
||||
6. 'relayActivity' - Relay read/write activity notification
|
||||
- Dispatched as 'ndkRelayActivity' window event
|
||||
- Used to animate relay status icons in footer
|
||||
================================================================ */
|
||||
|
||||
/* ================================================================
|
||||
DISTRIBUTED ARCHITECTURE NOTES
|
||||
================================================================
|
||||
Each page is independently accessible and self-contained:
|
||||
|
||||
1. Authentication persists via nostr-login-lite localStorage
|
||||
- Login once on any page
|
||||
- All other pages automatically authenticated
|
||||
|
||||
2. NDK SharedWorker is shared across all tabs/pages
|
||||
- Single NDK instance manages all connections
|
||||
- Subscriptions from all pages handled by one worker
|
||||
- Events broadcast to all connected pages
|
||||
|
||||
3. Dexie cache is shared across all pages
|
||||
- IndexedDB persists across sessions
|
||||
- Cache-first queries are fast
|
||||
- Reduces relay load
|
||||
|
||||
4. User settings are centralized and shared
|
||||
- Worker hydrates kind 30078 (`d:user-settings`) on init
|
||||
- Pages read via getUserSettings()
|
||||
- Pages patch via patchUserSettings({ featureNamespace: ... })
|
||||
- Pages subscribe via onUserSettings() for live updates
|
||||
|
||||
5. Each page can be distributed independently
|
||||
- Copy template.html and customize
|
||||
- No dependencies on other pages
|
||||
- Works standalone or as part of suite
|
||||
|
||||
6. Message-based signer bridges worker and page
|
||||
- Worker's NDK uses MessageBasedSigner
|
||||
- Signer sends sign requests to page
|
||||
- Page calls window.nostr.signEvent()
|
||||
- Response sent back to worker
|
||||
- NDK completes signing and publishing
|
||||
|
||||
7. Relay status visualization
|
||||
- Footer left section shows connected relays
|
||||
- Each relay has animated icon (HamburgerMorphing)
|
||||
- Icons morph based on activity (read/write)
|
||||
- Temporary animations show real-time activity
|
||||
================================================================ */
|
||||
</script>
|
||||
</body>
|
||||
|
||||
</html>
|
||||
+54
-13
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Build fully static MUSL binaries for C-Relay using Alpine Docker
|
||||
# Build fully static MUSL binaries for C-Relay-PG using Alpine Docker
|
||||
# Produces truly portable binaries with zero runtime dependencies
|
||||
|
||||
set -e
|
||||
@@ -11,19 +11,52 @@ DOCKERFILE="$SCRIPT_DIR/Dockerfile.alpine-musl"
|
||||
|
||||
# Parse command line arguments
|
||||
DEBUG_BUILD=false
|
||||
if [[ "$1" == "--debug" ]]; then
|
||||
DEBUG_BUILD=true
|
||||
DB_BACKEND="${DB_BACKEND:-postgres}"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--debug)
|
||||
DEBUG_BUILD=true
|
||||
shift
|
||||
;;
|
||||
--db-backend)
|
||||
if [[ -z "$2" ]]; then
|
||||
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
|
||||
exit 1
|
||||
fi
|
||||
DB_BACKEND="$2"
|
||||
shift 2
|
||||
;;
|
||||
--db-backend=*)
|
||||
DB_BACKEND="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown argument: $1"
|
||||
echo "Usage: $0 [--debug] [--db-backend postgres|sqlite]"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ "$DB_BACKEND" != "sqlite" && "$DB_BACKEND" != "postgres" ]]; then
|
||||
echo "ERROR: Invalid DB backend '$DB_BACKEND'. Use sqlite or postgres."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$DEBUG_BUILD" == "true" ]]; then
|
||||
echo "=========================================="
|
||||
echo "C-Relay MUSL Static Binary Builder (DEBUG MODE)"
|
||||
echo "C-Relay-PG MUSL Static Binary Builder (DEBUG MODE)"
|
||||
echo "=========================================="
|
||||
else
|
||||
echo "=========================================="
|
||||
echo "C-Relay MUSL Static Binary Builder (PRODUCTION MODE)"
|
||||
echo "C-Relay-PG MUSL Static Binary Builder (PRODUCTION MODE)"
|
||||
echo "=========================================="
|
||||
fi
|
||||
echo "Project directory: $SCRIPT_DIR"
|
||||
echo "Build directory: $BUILD_DIR"
|
||||
echo "Debug build: $DEBUG_BUILD"
|
||||
echo "DB backend: $DB_BACKEND"
|
||||
echo ""
|
||||
|
||||
# Create build directory
|
||||
@@ -63,20 +96,25 @@ ARCH=$(uname -m)
|
||||
case "$ARCH" in
|
||||
x86_64)
|
||||
PLATFORM="linux/amd64"
|
||||
OUTPUT_NAME="c_relay_static_x86_64"
|
||||
OUTPUT_NAME="c_relay_pg_static_x86_64"
|
||||
;;
|
||||
aarch64|arm64)
|
||||
PLATFORM="linux/arm64"
|
||||
OUTPUT_NAME="c_relay_static_arm64"
|
||||
OUTPUT_NAME="c_relay_pg_static_arm64"
|
||||
;;
|
||||
*)
|
||||
echo "WARNING: Unknown architecture: $ARCH"
|
||||
echo "Defaulting to linux/amd64"
|
||||
PLATFORM="linux/amd64"
|
||||
OUTPUT_NAME="c_relay_static_${ARCH}"
|
||||
OUTPUT_NAME="c_relay_pg_static_${ARCH}"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Append _debug suffix to output name for debug builds so production binary is never overwritten
|
||||
if [ "$DEBUG_BUILD" = true ]; then
|
||||
OUTPUT_NAME="${OUTPUT_NAME}_debug"
|
||||
fi
|
||||
|
||||
echo "Building for platform: $PLATFORM"
|
||||
echo "Output binary: $OUTPUT_NAME"
|
||||
echo ""
|
||||
@@ -111,14 +149,15 @@ echo "=========================================="
|
||||
echo "This will:"
|
||||
echo " - Use Alpine Linux (native MUSL)"
|
||||
echo " - Build all dependencies statically"
|
||||
echo " - Compile c-relay with full static linking"
|
||||
echo " - Compile c-relay-pg with full static linking"
|
||||
echo ""
|
||||
|
||||
$DOCKER_CMD build \
|
||||
--platform "$PLATFORM" \
|
||||
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
|
||||
--build-arg DB_BACKEND=$DB_BACKEND \
|
||||
-f "$DOCKERFILE" \
|
||||
-t c-relay-musl-builder:latest \
|
||||
-t c-relay-pg-musl-builder:latest \
|
||||
--progress=plain \
|
||||
. || {
|
||||
echo ""
|
||||
@@ -140,16 +179,17 @@ echo "=========================================="
|
||||
$DOCKER_CMD build \
|
||||
--platform "$PLATFORM" \
|
||||
--build-arg DEBUG_BUILD=$DEBUG_BUILD \
|
||||
--build-arg DB_BACKEND=$DB_BACKEND \
|
||||
--target builder \
|
||||
-f "$DOCKERFILE" \
|
||||
-t c-relay-static-builder-stage:latest \
|
||||
-t c-relay-pg-static-builder-stage:latest \
|
||||
. > /dev/null 2>&1
|
||||
|
||||
# Create a temporary container to copy the binary
|
||||
CONTAINER_ID=$($DOCKER_CMD create c-relay-static-builder-stage:latest)
|
||||
CONTAINER_ID=$($DOCKER_CMD create c-relay-pg-static-builder-stage:latest)
|
||||
|
||||
# Copy binary from container
|
||||
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_static" "$BUILD_DIR/$OUTPUT_NAME" || {
|
||||
$DOCKER_CMD cp "$CONTAINER_ID:/build/c_relay_pg_static" "$BUILD_DIR/$OUTPUT_NAME" || {
|
||||
echo "ERROR: Failed to extract binary from container"
|
||||
$DOCKER_CMD rm "$CONTAINER_ID" 2>/dev/null
|
||||
exit 1
|
||||
@@ -219,6 +259,7 @@ if [ "$DEBUG_BUILD" = true ]; then
|
||||
else
|
||||
echo "Build Type: PRODUCTION (optimized, stripped)"
|
||||
fi
|
||||
echo "DB Backend: $DB_BACKEND"
|
||||
if [ "$TRULY_STATIC" = true ]; then
|
||||
echo "Linkage: Fully static binary (Alpine MUSL-based)"
|
||||
echo "Portability: Works on ANY Linux distribution"
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,31 @@
|
||||
# Database files (from c-relay running in this directory)
|
||||
*.db
|
||||
*.db-shm
|
||||
*.db-wal
|
||||
|
||||
# Build artifacts
|
||||
*.o
|
||||
*.a
|
||||
|
||||
# Binary (built into root, not version-controlled)
|
||||
caching_relay
|
||||
caching_relay_static_*
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
crelay.log
|
||||
|
||||
# Test configs (not for version control)
|
||||
test_config*.jsonc
|
||||
|
||||
# Note: caching_relay_config.jsonc IS version-controlled (it's the default
|
||||
# config + state store). Don't ignore it.
|
||||
|
||||
# Docker build context temp (copied nostr_core_lib during static build)
|
||||
nostr_core_lib/
|
||||
|
||||
# Editor
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.vscode/
|
||||
@@ -0,0 +1,125 @@
|
||||
# Alpine-based MUSL static binary builder for caching_relay
|
||||
# Produces a truly portable binary with zero runtime dependencies.
|
||||
# Adapted from c-relay's Dockerfile.alpine-musl.
|
||||
|
||||
ARG DEBUG_BUILD=false
|
||||
|
||||
FROM alpine:3.19 AS builder
|
||||
|
||||
ARG DEBUG_BUILD=false
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache \
|
||||
build-base \
|
||||
musl-dev \
|
||||
git \
|
||||
cmake \
|
||||
pkgconfig \
|
||||
autoconf \
|
||||
automake \
|
||||
libtool \
|
||||
openssl-dev \
|
||||
openssl-libs-static \
|
||||
zlib-dev \
|
||||
zlib-static \
|
||||
curl-dev \
|
||||
curl-static \
|
||||
sqlite-dev \
|
||||
linux-headers \
|
||||
wget \
|
||||
bash
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Build libsecp256k1 static
|
||||
RUN cd /tmp && \
|
||||
git clone https://github.com/bitcoin-core/secp256k1.git && \
|
||||
cd secp256k1 && \
|
||||
./autogen.sh && \
|
||||
./configure --enable-static --disable-shared --prefix=/usr \
|
||||
CFLAGS="-fPIC" && \
|
||||
make -j$(nproc) && \
|
||||
make install && \
|
||||
rm -rf /tmp/secp256k1
|
||||
|
||||
# Build libwebsockets static with minimal features
|
||||
RUN cd /tmp && \
|
||||
git clone --depth 1 --branch v4.3.3 https://github.com/warmcat/libwebsockets.git && \
|
||||
cd libwebsockets && \
|
||||
mkdir build && cd build && \
|
||||
cmake .. \
|
||||
-DLWS_WITH_STATIC=ON \
|
||||
-DLWS_WITH_SHARED=OFF \
|
||||
-DLWS_WITH_SSL=ON \
|
||||
-DLWS_WITHOUT_TESTAPPS=ON \
|
||||
-DLWS_WITHOUT_TEST_SERVER=ON \
|
||||
-DLWS_WITHOUT_TEST_CLIENT=ON \
|
||||
-DLWS_WITHOUT_TEST_PING=ON \
|
||||
-DLWS_WITH_HTTP2=OFF \
|
||||
-DLWS_WITH_LIBUV=OFF \
|
||||
-DLWS_WITH_LIBEVENT=OFF \
|
||||
-DLWS_IPV6=ON \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_INSTALL_PREFIX=/usr \
|
||||
-DCMAKE_C_FLAGS="-fPIC" && \
|
||||
make -j$(nproc) && \
|
||||
make install && \
|
||||
rm -rf /tmp/libwebsockets
|
||||
|
||||
# Copy nostr_core_lib source (sibling project)
|
||||
COPY nostr_core_lib /build/nostr_core_lib/
|
||||
|
||||
# Build nostr_core_lib with the NIPs needed by the relay pool + signer:
|
||||
# 1 (basic), 6 (keys), 19 (npub bech32), 4 (legacy encryption, signer dep),
|
||||
# 42 (auth, relay pool dep), 44 (modern encryption, signer dep)
|
||||
RUN cd nostr_core_lib && \
|
||||
chmod +x build.sh && \
|
||||
sed -i 's/CFLAGS="-Wall -Wextra -std=c99 -fPIC -O2"/CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"/' build.sh && \
|
||||
rm -f *.o *.a 2>/dev/null || true && \
|
||||
./build.sh --nips=1,4,6,19,42,44 && \
|
||||
if [ -f libnostr_core_arm64.a ]; then \
|
||||
cp libnostr_core_arm64.a libnostr_core.a; \
|
||||
elif [ -f libnostr_core_x64.a ]; then \
|
||||
cp libnostr_core_x64.a libnostr_core.a; \
|
||||
else \
|
||||
echo "ERROR: No supported nostr_core static library produced"; \
|
||||
ls -la *.a 2>/dev/null || true; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Copy caching_relay source LAST (only this layer rebuilds on source changes)
|
||||
COPY src/ /build/src/
|
||||
|
||||
# Build caching_relay with full static linking.
|
||||
# No sqlite (the daemon doesn't use it), no c_utils (not needed).
|
||||
RUN if [ "$DEBUG_BUILD" = "true" ]; then \
|
||||
CFLAGS="-g -O2 -DDEBUG -fno-omit-frame-pointer"; \
|
||||
STRIP_CMD="echo 'Keeping debug symbols'"; \
|
||||
else \
|
||||
CFLAGS="-O2"; \
|
||||
STRIP_CMD="strip /build/caching_relay_static"; \
|
||||
fi && \
|
||||
gcc -static $CFLAGS -Wall -Wextra -std=c99 \
|
||||
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
|
||||
-I. -Isrc -Inostr_core_lib -Inostr_core_lib/nostr_core \
|
||||
-Inostr_core_lib/cjson -Inostr_core_lib/nostr_websocket \
|
||||
src/main.c src/debug.c src/jsonc_strip.c src/config.c src/state.c \
|
||||
src/follow_graph.c src/relay_sink.c src/live_subscriber.c \
|
||||
src/backfill.c src/relay_discovery.c \
|
||||
-o /build/caching_relay_static \
|
||||
nostr_core_lib/libnostr_core.a \
|
||||
-lwebsockets -lssl -lcrypto -lsecp256k1 \
|
||||
-lcurl -lz -lpthread -lm -ldl && \
|
||||
eval "$STRIP_CMD"
|
||||
|
||||
# Verify it's truly static
|
||||
RUN echo "=== Binary Information ===" && \
|
||||
file /build/caching_relay_static && \
|
||||
ls -lh /build/caching_relay_static && \
|
||||
echo "=== Checking for dynamic dependencies ===" && \
|
||||
(ldd /build/caching_relay_static 2>&1 || echo "Binary is static") && \
|
||||
echo "=== Build complete ==="
|
||||
|
||||
# Output stage - just the binary
|
||||
FROM scratch AS output
|
||||
COPY --from=builder /build/caching_relay_static /caching_relay_static
|
||||
@@ -0,0 +1,67 @@
|
||||
# caching_relay Makefile - statically linked C99 binary, c-relay style
|
||||
|
||||
# Use bash for reliable glob expansion in recipes (dash handles globs differently).
|
||||
SHELL := /bin/bash
|
||||
|
||||
CC = gcc
|
||||
CFLAGS = -Wall -Wextra -std=c99 -g -O2
|
||||
|
||||
# nostr_core_lib is a sibling of c-relay-pg (one dir up from caching/).
|
||||
NOSTR_CORE_DIR = ../nostr_core_lib
|
||||
|
||||
INCLUDES = -I. -Isrc -I$(NOSTR_CORE_DIR) -I$(NOSTR_CORE_DIR)/nostr_core \
|
||||
-I$(NOSTR_CORE_DIR)/cjson -I$(NOSTR_CORE_DIR)/nostr_websocket \
|
||||
-I/usr/include/postgresql
|
||||
|
||||
# -lsqlite3: nostr_core_lib's request_validator.x64.o references SQLite symbols;
|
||||
# we use --whole-archive so the linker pulls in all objects (needed for NIP-42
|
||||
# cross-references within the archive), which means SQLite must be satisfied.
|
||||
# No c_utils_lib: nostr_core_lib does not depend on it for the NIPs we use.
|
||||
# -lpq: PostgreSQL client library for the caching_event_inbox integration.
|
||||
LIBS = -lwebsockets -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm -lpq -lsqlite3
|
||||
|
||||
MAIN_SRC = src/main.c src/debug.c src/jsonc_strip.c src/config.c src/state.c \
|
||||
src/follow_graph.c src/relay_sink.c src/live_subscriber.c \
|
||||
src/backfill.c src/relay_discovery.c \
|
||||
src/pg_inbox.c src/pg_config.c
|
||||
|
||||
# Architecture detection
|
||||
ARCH = $(shell uname -m)
|
||||
ifeq ($(ARCH),x86_64)
|
||||
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
|
||||
else ifeq ($(ARCH),aarch64)
|
||||
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
|
||||
else ifeq ($(ARCH),arm64)
|
||||
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
|
||||
else
|
||||
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
|
||||
endif
|
||||
|
||||
# Binary goes in the c-relay-pg build/ directory so the launcher can find it.
|
||||
TARGET = ../build/caching_relay
|
||||
|
||||
all: $(TARGET)
|
||||
|
||||
# Build nostr_core_lib with the NIPs needed by the relay pool + signer:
|
||||
# 1 (basic), 6 (keys), 19 (npub bech32), 4 (legacy encryption, signer dep),
|
||||
# 42 (auth, relay pool dep), 44 (modern encryption, signer dep)
|
||||
$(NOSTR_CORE_LIB):
|
||||
@echo "Building nostr_core_lib with required NIPs..."
|
||||
cd $(NOSTR_CORE_DIR) && ./build.sh --nips=1,4,6,19,42,44
|
||||
|
||||
$(TARGET): $(MAIN_SRC) $(NOSTR_CORE_LIB)
|
||||
@echo "Compiling caching_relay for architecture: $(ARCH)"
|
||||
@# Extract all objects from the static library and link them directly.
|
||||
@# This avoids archive symbol resolution ordering issues (NIP-42 cross-refs).
|
||||
@# Use a fixed temp dir and chain all commands with && so failures stop the build.
|
||||
@rm -rf /tmp/cr_lib && mkdir -p /tmp/cr_lib && \
|
||||
ar x $(NOSTR_CORE_LIB) --output=/tmp/cr_lib && \
|
||||
echo "Extracted $$(ls /tmp/cr_lib/*.o | wc -l) objects" && \
|
||||
$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) /tmp/cr_lib/*.o -o $(TARGET) $(LIBS) && \
|
||||
rm -rf /tmp/cr_lib && \
|
||||
echo "Build complete: $(TARGET)"
|
||||
|
||||
clean:
|
||||
rm -f $(TARGET)
|
||||
|
||||
.PHONY: all clean
|
||||
@@ -0,0 +1,297 @@
|
||||
# caching_relay
|
||||
|
||||
A C99 daemon that caches Nostr events from people you follow into a local relay,
|
||||
so your Nostr client can point at a single fast local relay instead of fanning
|
||||
out to dozens of upstream relays.
|
||||
|
||||
## What it does
|
||||
|
||||
1. Reads a `.jsonc` config file listing your **root npub(s)**, upstream relays,
|
||||
a local relay URL, and the event kinds to cache.
|
||||
2. For each root npub, fetches its kind-3 contact list to discover **followed
|
||||
pubkeys** (your follows + their follows).
|
||||
3. **Live-subscribes** to new events of the configured kinds from the union of
|
||||
followed pubkeys.
|
||||
4. **Backfills** historical events using a progressive window-expansion strategy
|
||||
(24h → 7d → 30d → 90d → 365d), round-robin per pubkey, throttled to be polite
|
||||
to upstream relays.
|
||||
5. **Re-publishes** every fetched event to the local relay via a plain WebSocket
|
||||
`EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg,
|
||||
or any Nostr relay).
|
||||
6. Persists its backfill progress **in the config file itself** so a restart
|
||||
resumes where it left off.
|
||||
|
||||
## Build
|
||||
|
||||
### Static binary (recommended, c-relay style)
|
||||
|
||||
Produces a truly portable statically-linked MUSL binary with zero runtime
|
||||
dependencies:
|
||||
|
||||
```bash
|
||||
./build_static.sh
|
||||
# or with debug symbols:
|
||||
./build_static.sh --debug
|
||||
# or cross-compile for arm64:
|
||||
./build_static.sh --arch arm64
|
||||
```
|
||||
|
||||
Output: `caching_relay` (in the project root).
|
||||
|
||||
Requires Docker. The build runs in an Alpine container that compiles
|
||||
libsecp256k1, libwebsockets, and nostr_core_lib from source, then statically
|
||||
links everything.
|
||||
|
||||
### Local build (if you have the shared libs installed)
|
||||
|
||||
```bash
|
||||
make
|
||||
```
|
||||
|
||||
Output: `caching_relay` (in the project root).
|
||||
|
||||
Requires: `libwebsockets`, `openssl`, `libsecp256k1`, `libcurl`, `zlib` shared
|
||||
libraries, and a pre-built `../nostr_core_lib/libnostr_core_x64.a`.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
./caching_relay -d 3
|
||||
```
|
||||
|
||||
The daemon looks for `caching_relay_config.jsonc` in the current directory by default.
|
||||
You can override with `-c`:
|
||||
|
||||
```bash
|
||||
./caching_relay -c /path/to/my_config.jsonc -d 3
|
||||
```
|
||||
|
||||
Options:
|
||||
- `-c, --config <file>` - Path to `.jsonc` config file (default: `./caching_relay_config.jsonc`)
|
||||
- `-d, --debug <level>` - Log level 1-4 (1=error, 2=warn, 3=info, 4=debug). Default 3.
|
||||
- `-h, --help` - Show help
|
||||
|
||||
Signals:
|
||||
- `SIGINT` / `SIGTERM` - Graceful shutdown (saves state, closes connections)
|
||||
- `SIGHUP` - Reload config (preserves backfill state)
|
||||
|
||||
## Config file
|
||||
|
||||
The config file [`caching_relay_config.jsonc`](caching_relay_config.jsonc) lives in the
|
||||
project root. It is JSONC (JSON with `//` and `/* */` comments). The daemon
|
||||
rewrites it as plain JSON when saving state (comments are not preserved on
|
||||
rewrite).
|
||||
|
||||
Key fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `root_npubs` | npubs whose kind-3 follows list we crawl |
|
||||
| `upstream_relays` | bootstrap relays for initial discovery (kind-3, kind-10002). Outbox relays are discovered dynamically via NIP-65. |
|
||||
| `local_relay` | relay to publish cached events into (never queried) |
|
||||
| `kinds` | event kinds to cache for followed people (e.g. `[1, 3, 6, 10000, 30023]`) |
|
||||
| `admin_kinds` | kinds to follow specifically for root (admin) npubs. Use `["*"]` for all kinds. If omitted, admin uses same `kinds` as everyone else. |
|
||||
| `backfill.window_schedule_seconds` | progressive window sizes in seconds |
|
||||
| `backfill.events_per_tick` | max events per pubkey per backfill tick |
|
||||
| `backfill.tick_interval_seconds` | delay between pubkey backfills (throttle) |
|
||||
| `live.enabled` | enable live subscription |
|
||||
| `follow_graph_refresh_seconds` | how often to re-resolve the follow graph |
|
||||
| `state.*` | managed by the daemon - do not hand-edit |
|
||||
|
||||
## Architecture
|
||||
|
||||
See [`plans/plan.md`](plans/plan.md) for the full architecture document with
|
||||
Mermaid diagrams.
|
||||
|
||||
The daemon uses a **NIP-65 outbox model**: instead of querying a fixed set of
|
||||
bootstrap relays for everything, it discovers which relays each followed pubkey
|
||||
actually posts to (via kind 10002 relay lists) and connects to those relays
|
||||
dynamically. The `upstream_relays` in the config serve only as **bootstrap
|
||||
relays** -- the initial set used to discover kind-3 and kind-10002 events before
|
||||
the outbox relay map is built.
|
||||
|
||||
### Relay pool selection: minimum covering set
|
||||
|
||||
After discovering each followed pubkey's outbox relays (from their kind 10002),
|
||||
the daemon computes the **minimum set of relays that covers all followed
|
||||
pubkeys**. This is the classic set cover problem, solved with a greedy
|
||||
approximation:
|
||||
|
||||
1. Build a map: `{relay_url -> set of pubkeys that list it in their 10002}`
|
||||
2. Greedily pick the relay that covers the most uncovered pubkeys
|
||||
3. Repeat until all pubkeys are covered (or no more relays to pick)
|
||||
4. Always include the bootstrap relays in the final set (they may have events
|
||||
from pubkeys that don't publish a kind 10002)
|
||||
|
||||
This minimizes the number of WebSocket connections while ensuring every
|
||||
followed pubkey is reachable. The daemon logs the selected relay set and which
|
||||
pubkeys each relay covers.
|
||||
|
||||
Pubkeys that have no kind 10002 (or whose 10002 lists no relays) are covered by
|
||||
the bootstrap relays as a fallback.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ caching_relay │
|
||||
│ │
|
||||
upstream relays │ upstream_pool sink_pool │ local relay
|
||||
(damus, nos.lol) │ (query + subscribe) (publish only) │ (c-relay)
|
||||
│ │ │ │ │ │
|
||||
│ │ follow_graph relay_sink │ │
|
||||
└──────────►│ live_subscriber ──────►│ ├─────►│
|
||||
│ backfill ──────────────►│ │ │
|
||||
│ │ │ │ │
|
||||
│ state + seen ring │ │ │
|
||||
└──────────────────────────────────────────┘ │
|
||||
```
|
||||
|
||||
Two `nostr_relay_pool_t` instances:
|
||||
- **upstream_pool** - holds bootstrap relays initially, then dynamically adds
|
||||
outbox relays discovered from kind 10002. Used for `query_sync` (kind-3 fetch,
|
||||
kind-10002 fetch, backfill) and the long-lived live subscription.
|
||||
- **sink_pool** - holds only the local relay; used exclusively for
|
||||
`publish_async`. Never queried.
|
||||
|
||||
## Startup Flow
|
||||
|
||||
### First-time startup (empty local relay)
|
||||
|
||||
The daemon has never run before. The local relay has no cached events. The
|
||||
daemon must bootstrap from the config's `upstream_relays` to discover the
|
||||
outbox relays for each followed pubkey.
|
||||
|
||||
```
|
||||
START
|
||||
|
|
||||
v
|
||||
Load config (caching_relay_config.jsonc)
|
||||
| - root_npubs, upstream_relays (bootstrap), kinds, admin_kinds
|
||||
| - state.backfilled_until == 0 => first-time startup
|
||||
|
|
||||
v
|
||||
Create upstream_pool with bootstrap relays only
|
||||
Create sink_pool with local_relay
|
||||
|
|
||||
v
|
||||
Phase 1: Resolve follow graph (from bootstrap relays)
|
||||
| - For each root npub: query_sync kind=3 from bootstrap relays
|
||||
| - Parse "p" tags => followed pubkey set
|
||||
| - Log: "follow: resolved N followed pubkeys"
|
||||
|
|
||||
v
|
||||
Phase 2: Discover outbox relays (NIP-65, kind 10002) from bootstrap relays
|
||||
| - For each followed pubkey: query_sync kind=10002 from bootstrap relays
|
||||
| - Parse "r" tags => per-pubkey relay list
|
||||
| - Publish all kind-10002 events to local relay (cache them)
|
||||
| - Dynamically add discovered relays to upstream_pool
|
||||
| - Log: "relay_discovery: found N relays from M pubkeys"
|
||||
| - Log: "upstream_pool: now connected to N relays" + list them
|
||||
|
|
||||
v
|
||||
Phase 3: Open live subscriptions on upstream_pool
|
||||
| - follows_sub: non-admin pubkeys + regular kinds
|
||||
| - admin_sub: admin npubs + admin_kinds (or all kinds)
|
||||
|
|
||||
v
|
||||
Phase 4: Begin progressive backfill
|
||||
| - Window 0: 24h -> query each pubkey from their outbox relays
|
||||
| - Window 1: 7d -> ...
|
||||
| - Window 2: 30d -> ...
|
||||
| - Publish all fetched events to local relay
|
||||
| - Save state to config file as each window completes
|
||||
|
|
||||
v
|
||||
Steady-state: live sub + periodic follow-graph refresh + periodic
|
||||
kind-10002 refresh + backfill re-cycle
|
||||
|
|
||||
v
|
||||
SHUTDOWN (SIGINT/SIGTERM) -> save state -> clean exit
|
||||
```
|
||||
|
||||
### Subsequent startup (local relay already has cached events)
|
||||
|
||||
The daemon has run before. The local relay already has kind-3 and kind-10002
|
||||
events cached from the previous run. The daemon can read these from the local
|
||||
relay directly (fast, no network round-trip to bootstrap relays) and only falls
|
||||
back to bootstrap relays for pubkeys it cannot find locally.
|
||||
|
||||
```
|
||||
START
|
||||
|
|
||||
v
|
||||
Load config (caching_relay_config.jsonc)
|
||||
| - state.backfilled_until > 0 => subsequent startup
|
||||
| - state.current_window_index, backfill_cursor preserved
|
||||
|
|
||||
v
|
||||
Create upstream_pool with bootstrap relays
|
||||
Create sink_pool with local_relay
|
||||
|
|
||||
v
|
||||
Phase 1: Resolve follow graph (from LOCAL relay first, bootstrap fallback)
|
||||
| - Query local relay for kind=3 per root npub
|
||||
| - If found locally: use it (fast, no upstream query)
|
||||
| - If not found: fall back to bootstrap relays
|
||||
| - Parse "p" tags => followed pubkey set
|
||||
| - Log: "follow: resolved N followed pubkeys (L local, B bootstrap)"
|
||||
|
|
||||
v
|
||||
Phase 2: Discover outbox relays (from LOCAL relay first, bootstrap fallback)
|
||||
| - Query local relay for kind=10002 per followed pubkey
|
||||
| - If found locally: use it (fast)
|
||||
| - If not found: fall back to bootstrap relays, cache result to local
|
||||
| - Parse "r" tags => per-pubkey relay list
|
||||
| - Dynamically add discovered relays to upstream_pool
|
||||
| - Log: "relay_discovery: found N relays (L local, B bootstrap)"
|
||||
| - Log: "upstream_pool: now connected to N relays" + list them
|
||||
|
|
||||
v
|
||||
Phase 3: Open live subscriptions on upstream_pool
|
||||
| - (same as first-time)
|
||||
|
|
||||
v
|
||||
Phase 4: Resume backfill from saved state
|
||||
| - Resume at state.current_window_index, state.backfill_cursor
|
||||
| - No re-pull of already-backfilled windows
|
||||
| - Continue progressive window expansion from where it left off
|
||||
|
|
||||
v
|
||||
Steady-state (same as first-time)
|
||||
|
|
||||
v
|
||||
SHUTDOWN -> save state -> clean exit
|
||||
```
|
||||
|
||||
### Relay logging
|
||||
|
||||
The daemon logs all relay activity so you can see exactly which relays are
|
||||
being used:
|
||||
|
||||
- `upstream: added wss://relay.damus.io` -- each relay added to the pool
|
||||
- `relay_discovery: found 47 relays from 195 pubkeys` -- outbox discovery summary
|
||||
- `upstream_pool: 50 relays connected:` -- full relay list at startup
|
||||
- `relay_discovery: pubkey X -> wss://relay.example.com (local)` -- per-pubkey
|
||||
relay source (local cache vs bootstrap)
|
||||
- `backfill: pubkey[3/195] abc123 -> wss://relay.example.com (12 events)` --
|
||||
which relay served each backfill query
|
||||
|
||||
## Dependencies
|
||||
|
||||
- [nostr_core_lib](../nostr_core_lib) - built with NIPs 1, 4, 6, 19, 42, 44
|
||||
- libwebsockets, openssl, libsecp256k1, libcurl, zlib (all statically linked
|
||||
in the Docker build)
|
||||
|
||||
## Testing
|
||||
|
||||
Start a local c-relay, then run the daemon:
|
||||
|
||||
```bash
|
||||
# Start local relay (from c-relay project)
|
||||
cd ../c-relay/build && ./c_relay_static_x86_64 -p 8888 &
|
||||
|
||||
# Run the caching daemon (uses ./caching_relay_config.jsonc by default)
|
||||
./caching_relay -d 3
|
||||
|
||||
# Verify events are cached (from another terminal)
|
||||
nak req -k 1 -l 10 ws://127.0.0.1:8888
|
||||
```
|
||||
@@ -0,0 +1 @@
|
||||
0.0.2
|
||||
Executable
+148
@@ -0,0 +1,148 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Build fully static MUSL binary for caching_relay using Alpine Docker.
|
||||
# Produces a truly portable binary with zero runtime dependencies.
|
||||
# Adapted from c-relay's build_static.sh.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DOCKERFILE="$SCRIPT_DIR/Dockerfile.alpine-musl"
|
||||
|
||||
# Parse command line arguments
|
||||
DEBUG_BUILD=false
|
||||
TARGET_ARCH=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--debug)
|
||||
DEBUG_BUILD=true
|
||||
shift
|
||||
;;
|
||||
--arch)
|
||||
if [[ -z "$2" ]]; then
|
||||
echo "ERROR: --arch requires a value"
|
||||
echo "Usage: $0 [--debug] [--arch <arm64|x86_64>]"
|
||||
exit 1
|
||||
fi
|
||||
case "$2" in
|
||||
arm64|x86_64)
|
||||
TARGET_ARCH="$2"
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unsupported architecture '$2'"
|
||||
echo "Supported values: arm64, x86_64"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift 2
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown argument '$1'"
|
||||
echo "Usage: $0 [--debug] [--arch <arm64|x86_64>]"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ "$DEBUG_BUILD" = true ]; then
|
||||
echo "=========================================="
|
||||
echo "caching_relay MUSL Static Binary Builder (DEBUG MODE)"
|
||||
echo "=========================================="
|
||||
else
|
||||
echo "=========================================="
|
||||
echo "caching_relay MUSL Static Binary Builder (PRODUCTION MODE)"
|
||||
echo "=========================================="
|
||||
fi
|
||||
echo "Project directory: $SCRIPT_DIR"
|
||||
echo ""
|
||||
|
||||
if ! command -v docker &> /dev/null; then
|
||||
echo "ERROR: Docker is not installed or not in PATH"
|
||||
exit 1
|
||||
fi
|
||||
if ! docker info &> /dev/null; then
|
||||
echo "ERROR: Docker daemon is not running or user not in docker group"
|
||||
exit 1
|
||||
fi
|
||||
echo "Docker is available and running"
|
||||
echo ""
|
||||
|
||||
# Detect host architecture
|
||||
ARCH=$(uname -m)
|
||||
case "$ARCH" in
|
||||
x86_64) ARCH="x86_64";;
|
||||
aarch64) ARCH="arm64";;
|
||||
arm64) ARCH="arm64";;
|
||||
esac
|
||||
if [[ -n "$TARGET_ARCH" ]]; then
|
||||
ARCH="$TARGET_ARCH"
|
||||
echo "Using target architecture: $ARCH"
|
||||
else
|
||||
echo "Detected host architecture: $ARCH"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# nostr_core_lib is a sibling directory; copy it into the build context.
|
||||
# Docker cannot follow symlinks outside the context, so we do a real copy
|
||||
# (excluding .git and build artifacts) and clean up afterwards.
|
||||
NOSTR_CORE_DIR="$SCRIPT_DIR/../nostr_core_lib"
|
||||
if [ ! -d "$NOSTR_CORE_DIR" ]; then
|
||||
echo "ERROR: nostr_core_lib not found at $NOSTR_CORE_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NEEDS_COPY=1
|
||||
if [ -d "$SCRIPT_DIR/nostr_core_lib" ] && [ ! -L "$SCRIPT_DIR/nostr_core_lib" ]; then
|
||||
# Already a real directory (e.g. from a previous run); assume it's good.
|
||||
NEEDS_COPY=0
|
||||
fi
|
||||
if [ "$NEEDS_COPY" = "1" ]; then
|
||||
echo "Copying nostr_core_lib into build context..."
|
||||
rm -rf "$SCRIPT_DIR/nostr_core_lib"
|
||||
mkdir -p "$SCRIPT_DIR/nostr_core_lib"
|
||||
rsync -a --exclude='.git' --exclude='*.a' --exclude='*.o' \
|
||||
--exclude='.venv*' --exclude='backups' --exclude='verify_*' \
|
||||
--exclude='rewrite_mirror' --exclude='websocket_debug' \
|
||||
"$NOSTR_CORE_DIR/" "$SCRIPT_DIR/nostr_core_lib/"
|
||||
COPIED_NOSTR_CORE=1
|
||||
fi
|
||||
|
||||
# Build args
|
||||
BUILD_ARGS="--build-arg DEBUG_BUILD=$DEBUG_BUILD"
|
||||
if [ "$ARCH" = "arm64" ] && [ "$(uname -m)" != "aarch64" ] && [ "$(uname -m)" != "arm64" ]; then
|
||||
BUILD_ARGS="$BUILD_ARGS --platform linux/arm64"
|
||||
fi
|
||||
|
||||
IMAGE_TAG="caching_relay_builder:latest"
|
||||
|
||||
echo "Building Docker image..."
|
||||
docker build $BUILD_ARGS \
|
||||
-f "$DOCKERFILE" \
|
||||
-t "$IMAGE_TAG" \
|
||||
"$SCRIPT_DIR" 2>&1
|
||||
|
||||
BUILD_RC=$?
|
||||
|
||||
# Clean up the copied nostr_core_lib to keep the project dir tidy.
|
||||
if [ "${COPIED_NOSTR_CORE:-0}" = "1" ]; then
|
||||
echo "Cleaning up copied nostr_core_lib from build context..."
|
||||
rm -rf "$SCRIPT_DIR/nostr_core_lib"
|
||||
fi
|
||||
|
||||
if [ $BUILD_RC -ne 0 ]; then
|
||||
exit $BUILD_RC
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Extracting binary from image..."
|
||||
# The output stage is FROM scratch with no CMD, so pass an empty command.
|
||||
CONTAINER_ID=$(docker create "$IMAGE_TAG" "")
|
||||
docker cp "$CONTAINER_ID:/caching_relay_static" "$SCRIPT_DIR/caching_relay"
|
||||
docker rm "$CONTAINER_ID" >/dev/null
|
||||
chmod +x "$SCRIPT_DIR/caching_relay"
|
||||
|
||||
echo ""
|
||||
echo "=== Build complete ==="
|
||||
ls -lh "$SCRIPT_DIR/caching_relay"
|
||||
file "$SCRIPT_DIR/caching_relay"
|
||||
@@ -0,0 +1,432 @@
|
||||
/*
|
||||
* caching_relay - progressive window-expansion backfill
|
||||
*
|
||||
* Uses until-based pagination so authors with more events than the page size
|
||||
* are fully drained across multiple ticks instead of being skipped. In
|
||||
* PostgreSQL mode, per-author/window progress is persisted to the
|
||||
* caching_backfill_progress table so the service can resume after restart.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "backfill.h"
|
||||
#include "follow_graph.h"
|
||||
#include "pg_inbox.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include "../nostr_core_lib/cjson/cJSON.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
#include <time.h>
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Helpers */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/* Returns 1 if the PostgreSQL inbox is initialized (PG mode active).
|
||||
* Probes the config table; a non-NULL result means the connection is up. */
|
||||
static int pg_mode_active(void) {
|
||||
char *v = pg_inbox_get_config_value("caching_root_npubs");
|
||||
if (v) { free(v); return 1; }
|
||||
char *v2 = pg_inbox_get_config_value("caching_kinds");
|
||||
if (v2) { free(v2); return 1; }
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Find the oldest (minimum) created_at among an array of event JSON objects.
|
||||
* Returns 1 and sets *out_oldest, or 0 if no events / no valid created_at. */
|
||||
static int find_oldest_created_at(cJSON **events, int count, long *out_oldest) {
|
||||
long oldest = 0;
|
||||
int found = 0;
|
||||
for (int i = 0; i < count; i++) {
|
||||
cJSON *ca = cJSON_GetObjectItem(events[i], "created_at");
|
||||
if (!ca || !cJSON_IsNumber(ca)) continue;
|
||||
long ts = (long)ca->valuedouble;
|
||||
if (!found || ts < oldest) {
|
||||
oldest = ts;
|
||||
found = 1;
|
||||
}
|
||||
}
|
||||
if (found && out_oldest) *out_oldest = oldest;
|
||||
return found;
|
||||
}
|
||||
|
||||
/* Count how many events in the array have created_at == ts. */
|
||||
static int count_at_timestamp(cJSON **events, int count, long ts) {
|
||||
int n = 0;
|
||||
for (int i = 0; i < count; i++) {
|
||||
cJSON *ca = cJSON_GetObjectItem(events[i], "created_at");
|
||||
if (ca && cJSON_IsNumber(ca) && (long)ca->valuedouble == ts) n++;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
/* Build the kinds array for a given pubkey (admin vs regular).
|
||||
* Returns NULL if no kinds filter should be applied (admin_all_kinds). */
|
||||
static cJSON *build_kinds(cr_config_t *cfg, const char *pk) {
|
||||
int is_admin = cr_follow_is_root(cfg, pk);
|
||||
if (is_admin && cfg->admin_all_kinds) {
|
||||
return NULL;
|
||||
}
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
if (is_admin && cfg->admin_kind_count > 0) {
|
||||
for (int i = 0; i < cfg->admin_kind_count; i++)
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
|
||||
} else {
|
||||
for (int i = 0; i < cfg->kind_count; i++)
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
|
||||
}
|
||||
return kinds;
|
||||
}
|
||||
|
||||
/* Resolve the relay URL list to query for a given pubkey. Sets *out_urls and
|
||||
* *out_n. Caller frees *out_urls (but not the strings, which alias internal
|
||||
* storage). Returns 0 on success, -1 if no relays available. */
|
||||
static int resolve_relays(nostr_relay_pool_t *upstream,
|
||||
const cr_relay_map_t *relay_map,
|
||||
const char *pk,
|
||||
const char ***out_urls,
|
||||
int *out_n) {
|
||||
*out_urls = NULL;
|
||||
*out_n = 0;
|
||||
|
||||
if (relay_map) {
|
||||
const cr_outbox_entry_t *oe = cr_relay_map_get_outbox(relay_map, pk);
|
||||
if (oe && oe->relay_count > 0) {
|
||||
const char **urls = malloc(oe->relay_count * sizeof(char *));
|
||||
if (!urls) return -1;
|
||||
for (int j = 0; j < oe->relay_count; j++) urls[j] = oe->relays[j];
|
||||
*out_urls = urls;
|
||||
*out_n = oe->relay_count;
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/* Fallback: all upstream relays. */
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
if (n > 0) {
|
||||
const char **urls = malloc(n * sizeof(char *));
|
||||
if (!urls) { free(listed); free(statuses); return -1; }
|
||||
for (int j = 0; j < n; j++) urls[j] = listed[j];
|
||||
*out_urls = urls;
|
||||
*out_n = n;
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Issue a query_sync for one author with the given since/until/limit.
|
||||
* Returns the events array (caller frees each event + the array) or NULL.
|
||||
* Sets *out_count. */
|
||||
static cJSON **query_author(nostr_relay_pool_t *upstream,
|
||||
const cr_relay_map_t *relay_map,
|
||||
cr_config_t *cfg, const char *pk,
|
||||
long since, long until, int limit,
|
||||
int *out_count) {
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(pk));
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
|
||||
cJSON *kinds = build_kinds(cfg, pk);
|
||||
if (kinds) cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
|
||||
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)since));
|
||||
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)until));
|
||||
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)limit));
|
||||
|
||||
const char **urls = NULL;
|
||||
int n = 0;
|
||||
resolve_relays(upstream, relay_map, pk, &urls, &n);
|
||||
|
||||
cJSON **events = nostr_relay_pool_query_sync(upstream, urls, n, filter,
|
||||
out_count, 15000);
|
||||
free(urls);
|
||||
cJSON_Delete(filter);
|
||||
return events;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Window advancement */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
static void advance_window(cr_backfill_t *bf, cr_config_t *cfg) {
|
||||
/* Record that we've backfilled up to (now - current_window). */
|
||||
cfg->state.backfilled_until = (long)time(NULL) - bf->current_window_s;
|
||||
cfg->state.current_window_index = bf->window_index + 1;
|
||||
cfg->state.backfill_cursor = 0;
|
||||
if (!pg_mode_active()) {
|
||||
cr_config_save_state(cfg);
|
||||
}
|
||||
|
||||
DEBUG_INFO("backfill: window %d complete (%ld events). backfilled_until=%ld",
|
||||
bf->window_index, bf->events_this_window, cfg->state.backfilled_until);
|
||||
|
||||
bf->window_index++;
|
||||
if (bf->window_index >= cfg->backfill.window_count) {
|
||||
DEBUG_INFO("backfill: all windows complete, entering steady-state");
|
||||
bf->in_progress = 0;
|
||||
return;
|
||||
}
|
||||
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
|
||||
bf->cursor = 0;
|
||||
bf->until_cursor = 0;
|
||||
bf->author_complete = 0;
|
||||
bf->blocked = 0;
|
||||
bf->blocked_until_ts = 0;
|
||||
bf->events_this_window = 0;
|
||||
bf->window_started = time(NULL);
|
||||
DEBUG_INFO("backfill: advancing to window %d (%ld seconds)",
|
||||
bf->window_index, bf->current_window_s);
|
||||
}
|
||||
|
||||
/* Persist the current cursor state. In PG mode we rely on the per-author
|
||||
* progress table; in legacy mode we write the .jsonc state. */
|
||||
static void persist_cursor(cr_backfill_t *bf, cr_config_t *cfg) {
|
||||
cfg->state.backfill_cursor = bf->cursor;
|
||||
if (!pg_mode_active()) {
|
||||
cr_config_save_state(cfg);
|
||||
}
|
||||
}
|
||||
|
||||
/* Save progress for the current author to PG (no-op outside PG mode). */
|
||||
static void pg_save_current(cr_backfill_t *bf, cr_config_t *cfg,
|
||||
const char *pk, int complete) {
|
||||
(void)cfg; /* window anchor is derived from bf + time(), cfg not needed */
|
||||
if (!pg_mode_active() || !pk) return;
|
||||
long anchor = (long)time(NULL) - bf->current_window_s;
|
||||
pg_inbox_save_backfill_progress(pk, bf->window_index, anchor,
|
||||
bf->until_cursor, complete);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Init */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg) {
|
||||
memset(bf, 0, sizeof(*bf));
|
||||
bf->window_index = cfg->state.current_window_index;
|
||||
if (bf->window_index >= cfg->backfill.window_count) {
|
||||
/* Already done; steady state. */
|
||||
bf->in_progress = 0;
|
||||
DEBUG_INFO("backfill: already complete (window index %d >= %d), steady-state",
|
||||
bf->window_index, cfg->backfill.window_count);
|
||||
return;
|
||||
}
|
||||
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
|
||||
bf->cursor = cfg->state.backfill_cursor;
|
||||
bf->until_cursor = 0;
|
||||
bf->author_complete = 0;
|
||||
bf->blocked = 0;
|
||||
bf->blocked_until_ts = 0;
|
||||
bf->in_progress = 1;
|
||||
bf->window_started = time(NULL);
|
||||
|
||||
/* In PG mode, per-author restore is deferred to the first tick (the
|
||||
* followed set is not resolved yet at init time). The tick loop skips
|
||||
* already-complete authors via the progress table. */
|
||||
DEBUG_INFO("backfill: starting at window %d (%ld seconds), cursor %d",
|
||||
bf->window_index, bf->current_window_s, bf->cursor);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Tick */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
|
||||
cr_sink_t *sink, const cr_relay_map_t *relay_map) {
|
||||
if (!cfg->backfill.enabled) return -2;
|
||||
if (!bf->in_progress) return -2;
|
||||
if (followed->count == 0) return 0;
|
||||
|
||||
/* Throttle: only one query per tick_interval. */
|
||||
time_t now = time(NULL);
|
||||
if (bf->last_tick && (now - bf->last_tick) < cfg->backfill.tick_interval_seconds) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* In PG mode, skip past authors already marked complete for this window
|
||||
* (resume case). Restore the until_cursor for the first incomplete author. */
|
||||
if (pg_mode_active()) {
|
||||
while (bf->cursor < followed->count) {
|
||||
const char *pk = followed->items[bf->cursor];
|
||||
long uc = 0;
|
||||
int complete = 0, found = 0;
|
||||
if (pg_inbox_load_backfill_progress(pk, bf->window_index,
|
||||
&uc, &complete, &found) == 0
|
||||
&& found && complete) {
|
||||
DEBUG_TRACE("backfill: author[%d] %s already complete, skipping",
|
||||
bf->cursor, pk);
|
||||
bf->cursor++;
|
||||
cfg->state.backfill_cursor = bf->cursor;
|
||||
continue;
|
||||
}
|
||||
if (found && uc > 0 && bf->until_cursor == 0) {
|
||||
bf->until_cursor = uc;
|
||||
DEBUG_TRACE("backfill: resumed author[%d] %s at until=%ld",
|
||||
bf->cursor, pk, bf->until_cursor);
|
||||
}
|
||||
break;
|
||||
}
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
const char *pk = followed->items[bf->cursor];
|
||||
|
||||
/* Determine the until cursor for this query. First query for an author:
|
||||
* until = now. Subsequent queries: until = oldest event created_at - 1
|
||||
* from the previous page (set at the end of the previous tick). */
|
||||
long until = bf->until_cursor;
|
||||
if (until == 0) {
|
||||
until = (long)now;
|
||||
}
|
||||
long since = (long)now - bf->current_window_s;
|
||||
|
||||
int page_size = cfg->backfill.events_per_tick;
|
||||
if (page_size < 1) page_size = 50;
|
||||
int max_cap = CR_BACKFILL_MAX_PAGE_CAP;
|
||||
if (max_cap < page_size) max_cap = page_size;
|
||||
|
||||
bf->last_tick = now;
|
||||
|
||||
DEBUG_TRACE("backfill: querying %s (window %ld, since %ld, until %ld, limit %d)",
|
||||
pk, bf->current_window_s, since, until, page_size);
|
||||
|
||||
int ev_count = 0;
|
||||
cJSON **events = query_author(upstream, relay_map, cfg, pk,
|
||||
since, until, page_size, &ev_count);
|
||||
|
||||
/* Saturation handling: if we got exactly page_size events, the page may
|
||||
* be truncated. Bump the limit up to max_cap and retry once. */
|
||||
int limit_used = page_size;
|
||||
if (events && ev_count == page_size && page_size < max_cap) {
|
||||
for (int k = 0; k < ev_count; k++) cJSON_Delete(events[k]);
|
||||
free(events);
|
||||
|
||||
DEBUG_TRACE("backfill: saturated page for %s, retrying with limit %d",
|
||||
pk, max_cap);
|
||||
events = query_author(upstream, relay_map, cfg, pk,
|
||||
since, until, max_cap, &ev_count);
|
||||
limit_used = max_cap;
|
||||
}
|
||||
|
||||
/* If still saturated at the cap, we cannot safely paginate past the
|
||||
* oldest timestamp (the relay may have more events at that exact ts).
|
||||
* Mark the author blocked at that timestamp and move on; it will be
|
||||
* retried in a later window sweep. */
|
||||
if (events && ev_count == max_cap && max_cap > page_size) {
|
||||
long oldest = 0;
|
||||
find_oldest_created_at(events, ev_count, &oldest);
|
||||
int at_oldest = count_at_timestamp(events, ev_count, oldest);
|
||||
|
||||
DEBUG_WARN("backfill: author %s saturated at limit %d (oldest ts %ld, "
|
||||
"%d events at that ts) - marking blocked, will retry next window",
|
||||
pk, max_cap, oldest, at_oldest);
|
||||
|
||||
/* Publish what we have. */
|
||||
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
|
||||
for (int k = 0; k < ev_count; k++) {
|
||||
cr_sink_publish(sink, events[k]);
|
||||
cJSON_Delete(events[k]);
|
||||
bf->events_this_window++;
|
||||
bf->events_total++;
|
||||
}
|
||||
free(events);
|
||||
|
||||
bf->blocked = 1;
|
||||
bf->blocked_until_ts = oldest;
|
||||
bf->until_cursor = oldest; /* resume point for next window */
|
||||
pg_save_current(bf, cfg, pk, 0); /* not complete - retry next window */
|
||||
|
||||
bf->cursor++;
|
||||
persist_cursor(bf, cfg);
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Normal page processing. Capture the oldest timestamp BEFORE publishing
|
||||
* so we can paginate if saturated. */
|
||||
long oldest_ts = 0;
|
||||
int have_oldest = 0;
|
||||
if (events && ev_count > 0) {
|
||||
have_oldest = find_oldest_created_at(events, ev_count, &oldest_ts);
|
||||
}
|
||||
|
||||
if (events && ev_count > 0) {
|
||||
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
|
||||
for (int k = 0; k < ev_count; k++) {
|
||||
cr_sink_publish(sink, events[k]);
|
||||
cJSON_Delete(events[k]);
|
||||
bf->events_this_window++;
|
||||
bf->events_total++;
|
||||
}
|
||||
free(events);
|
||||
}
|
||||
|
||||
DEBUG_LOG("backfill: pubkey[%d/%d] %s -> %d events (until=%ld)",
|
||||
bf->cursor, followed->count, pk, ev_count, until);
|
||||
|
||||
int saturated = (ev_count == limit_used) && (ev_count > 0);
|
||||
|
||||
if (ev_count == 0) {
|
||||
/* No more events for this author in the window. */
|
||||
bf->author_complete = 1;
|
||||
bf->until_cursor = 0;
|
||||
pg_save_current(bf, cfg, pk, 1);
|
||||
bf->cursor++;
|
||||
persist_cursor(bf, cfg);
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
}
|
||||
} else if (!saturated) {
|
||||
/* Last page for this author. */
|
||||
bf->author_complete = 1;
|
||||
bf->until_cursor = 0;
|
||||
pg_save_current(bf, cfg, pk, 1);
|
||||
bf->cursor++;
|
||||
persist_cursor(bf, cfg);
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
}
|
||||
} else {
|
||||
/* Saturated: paginate. Set until = oldest - 1 and keep the same
|
||||
* author for the next tick. The `until` parameter is inclusive, so
|
||||
* subtracting 1 avoids re-fetching the oldest event. */
|
||||
if (have_oldest) {
|
||||
bf->until_cursor = oldest_ts - 1;
|
||||
if (bf->until_cursor < since) {
|
||||
/* Walked back past the window start; author is done. */
|
||||
bf->author_complete = 1;
|
||||
bf->until_cursor = 0;
|
||||
pg_save_current(bf, cfg, pk, 1);
|
||||
bf->cursor++;
|
||||
persist_cursor(bf, cfg);
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
}
|
||||
} else {
|
||||
pg_save_current(bf, cfg, pk, 0);
|
||||
}
|
||||
} else {
|
||||
/* Could not determine oldest (no valid created_at); mark done. */
|
||||
bf->author_complete = 1;
|
||||
bf->until_cursor = 0;
|
||||
pg_save_current(bf, cfg, pk, 1);
|
||||
bf->cursor++;
|
||||
persist_cursor(bf, cfg);
|
||||
if (bf->cursor >= followed->count) {
|
||||
advance_window(bf, cfg);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return 1;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/*
|
||||
* caching_relay - progressive window-expansion backfill.
|
||||
*
|
||||
* Round-robin through the followed-pubkey set, issuing query_sync per pubkey
|
||||
* for the current window (since = now - window_seconds). When a full pass
|
||||
* completes, advance the window and persist state to the config file.
|
||||
*
|
||||
* Pagination: each author is queried with an `until` cursor that walks
|
||||
* backwards in time. If a query returns exactly `page_size` events (saturated),
|
||||
* the limit is bumped up to a configurable cap and retried; if still saturated
|
||||
* the author is marked "blocked" at that timestamp and retried in a later
|
||||
* window sweep. Otherwise the until cursor advances to
|
||||
* (oldest_event_created_at - 1) and the same author is queried again on the
|
||||
* next tick until no more events are returned.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_BACKFILL_H
|
||||
#define CACHING_RELAY_BACKFILL_H
|
||||
|
||||
#include "config.h"
|
||||
#include "state.h"
|
||||
#include "relay_sink.h"
|
||||
#include "relay_discovery.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
|
||||
/* Maximum limit we will bump a saturated page to before marking an author
|
||||
* blocked at a timestamp. Keeps a single query bounded. */
|
||||
#define CR_BACKFILL_MAX_PAGE_CAP 200
|
||||
|
||||
typedef struct {
|
||||
int window_index;
|
||||
long current_window_s;
|
||||
int cursor; /* current index into followed set */
|
||||
long until_cursor; /* current until timestamp for the current author */
|
||||
int in_progress; /* 1 if a window pass is in progress */
|
||||
long events_this_window; /* events pulled in current window */
|
||||
long events_total; /* events pulled across all windows */
|
||||
time_t last_tick; /* last time a pubkey was queried */
|
||||
time_t window_started; /* when current window pass started */
|
||||
int author_complete; /* 1 when current author is done, advance cursor */
|
||||
int blocked; /* 1 when current author is blocked at a saturated timestamp */
|
||||
long blocked_until_ts; /* timestamp where the author is blocked */
|
||||
} cr_backfill_t;
|
||||
|
||||
/* Initialize backfill state from the persisted config state.
|
||||
* In PostgreSQL mode (pg_inbox initialized), restores per-author progress
|
||||
* for the current window from the caching_backfill_progress table. */
|
||||
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg);
|
||||
|
||||
/* Perform one backfill tick. This queries ONE followed pubkey (the one at
|
||||
* cfg->state.backfill_cursor) and publishes results to the sink. Uses
|
||||
* until-based pagination so authors with more events than the page size are
|
||||
* fully drained across multiple ticks. When a full round-robin pass completes,
|
||||
* advances the window and saves state.
|
||||
*
|
||||
* Returns:
|
||||
* 1 if a tick was performed (a pubkey was queried)
|
||||
* 0 if throttled (tick_interval not elapsed) - caller should pump pools
|
||||
* -1 on hard error
|
||||
* -2 if backfill is complete (all windows done) - caller should go steady-state
|
||||
*/
|
||||
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
|
||||
cr_sink_t *sink, const cr_relay_map_t *relay_map);
|
||||
|
||||
#endif /* CACHING_RELAY_BACKFILL_H */
|
||||
@@ -0,0 +1,315 @@
|
||||
/*
|
||||
* caching_relay - config parsing + persistent state (in the .jsonc file itself)
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "config.h"
|
||||
#include "jsonc_strip.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include "../nostr_core_lib/cjson/cJSON.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <errno.h>
|
||||
#include <unistd.h>
|
||||
#include <fcntl.h>
|
||||
|
||||
static int read_file(const char *path, char **out, size_t *out_len) {
|
||||
FILE *f = fopen(path, "rb");
|
||||
if (!f) {
|
||||
DEBUG_ERROR("cannot open config '%s': %s", path, strerror(errno));
|
||||
return -1;
|
||||
}
|
||||
if (fseek(f, 0, SEEK_END) != 0) { fclose(f); return -1; }
|
||||
long sz = ftell(f);
|
||||
if (sz < 0) { fclose(f); return -1; }
|
||||
rewind(f);
|
||||
char *buf = malloc(sz + 1);
|
||||
if (!buf) { fclose(f); return -1; }
|
||||
size_t n = fread(buf, 1, sz, f);
|
||||
fclose(f);
|
||||
buf[n] = '\0';
|
||||
*out = buf;
|
||||
if (out_len) *out_len = n;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void copy_string_array(cJSON *arr, char *dst, int max, int *count_out,
|
||||
int elem_len) {
|
||||
int count = 0;
|
||||
if (arr && cJSON_IsArray(arr)) {
|
||||
cJSON *item;
|
||||
cJSON_ArrayForEach(item, arr) {
|
||||
if (count >= max) break;
|
||||
if (!cJSON_IsString(item)) continue;
|
||||
const char *s = cJSON_GetStringValue(item);
|
||||
if (!s) continue;
|
||||
strncpy(dst + count * elem_len, s, elem_len - 1);
|
||||
dst[count * elem_len + (elem_len - 1)] = '\0';
|
||||
count++;
|
||||
}
|
||||
}
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
static void copy_int_array(cJSON *arr, int *dst, int max, int *count_out) {
|
||||
int count = 0;
|
||||
if (arr && cJSON_IsArray(arr)) {
|
||||
cJSON *item;
|
||||
cJSON_ArrayForEach(item, arr) {
|
||||
if (count >= max) break;
|
||||
if (cJSON_IsNumber(item)) {
|
||||
dst[count++] = (int)cJSON_GetNumberValue(item);
|
||||
}
|
||||
}
|
||||
}
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
static void copy_long_array(cJSON *arr, long *dst, int max, int *count_out) {
|
||||
int count = 0;
|
||||
if (arr && cJSON_IsArray(arr)) {
|
||||
cJSON *item;
|
||||
cJSON_ArrayForEach(item, arr) {
|
||||
if (count >= max) break;
|
||||
if (cJSON_IsNumber(item)) {
|
||||
dst[count++] = (long)cJSON_GetNumberValue(item);
|
||||
}
|
||||
}
|
||||
}
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
int cr_config_load(cr_config_t *cfg, const char *path) {
|
||||
memset(cfg, 0, sizeof(*cfg));
|
||||
strncpy(cfg->path, path, sizeof(cfg->path) - 1);
|
||||
|
||||
char *raw = NULL;
|
||||
size_t raw_len = 0;
|
||||
if (read_file(path, &raw, &raw_len) != 0) return -1;
|
||||
|
||||
char *stripped = jsonc_strip_comments(raw, raw_len);
|
||||
free(raw);
|
||||
if (!stripped) {
|
||||
DEBUG_ERROR("failed to strip comments from config");
|
||||
return -1;
|
||||
}
|
||||
|
||||
cJSON *root = cJSON_Parse(stripped);
|
||||
free(stripped);
|
||||
if (!root) {
|
||||
DEBUG_ERROR("failed to parse config JSON");
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* root_npubs */
|
||||
copy_string_array(cJSON_GetObjectItem(root, "root_npubs"),
|
||||
(char *)cfg->root_npubs, CR_MAX_ROOT_NPUBS,
|
||||
&cfg->root_npub_count, CR_NPUB_LEN);
|
||||
|
||||
/* upstream_relays */
|
||||
copy_string_array(cJSON_GetObjectItem(root, "upstream_relays"),
|
||||
(char *)cfg->upstream_relays, CR_MAX_UPSTREAM,
|
||||
&cfg->upstream_count, CR_URL_LEN);
|
||||
|
||||
/* local_relay */
|
||||
cJSON *local = cJSON_GetObjectItem(root, "local_relay");
|
||||
if (local && cJSON_IsString(local)) {
|
||||
strncpy(cfg->local_relay, cJSON_GetStringValue(local), CR_URL_LEN - 1);
|
||||
}
|
||||
|
||||
/* kinds */
|
||||
copy_int_array(cJSON_GetObjectItem(root, "kinds"),
|
||||
cfg->kinds, CR_MAX_KINDS, &cfg->kind_count);
|
||||
|
||||
/* admin_kinds - kinds to follow specifically for root (admin) npubs.
|
||||
* Supports [*] to mean "all kinds" (no kind filter for admin). */
|
||||
cJSON *ak = cJSON_GetObjectItem(root, "admin_kinds");
|
||||
if (ak && cJSON_IsArray(ak)) {
|
||||
cJSON *item;
|
||||
cJSON_ArrayForEach(item, ak) {
|
||||
if (cJSON_IsString(item)) {
|
||||
const char *s = cJSON_GetStringValue(item);
|
||||
if (s && strcmp(s, "*") == 0) {
|
||||
cfg->admin_all_kinds = 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!cfg->admin_all_kinds) {
|
||||
copy_int_array(ak, cfg->admin_kinds, CR_MAX_KINDS, &cfg->admin_kind_count);
|
||||
}
|
||||
}
|
||||
|
||||
/* backfill */
|
||||
cJSON *bf = cJSON_GetObjectItem(root, "backfill");
|
||||
if (bf) {
|
||||
cfg->backfill.enabled = cJSON_IsTrue(cJSON_GetObjectItem(bf, "enabled"));
|
||||
copy_long_array(cJSON_GetObjectItem(bf, "window_schedule_seconds"),
|
||||
cfg->backfill.window_schedule_seconds, CR_MAX_WINDOWS,
|
||||
&cfg->backfill.window_count);
|
||||
cJSON *ept = cJSON_GetObjectItem(bf, "events_per_tick");
|
||||
if (ept) cfg->backfill.events_per_tick = (int)cJSON_GetNumberValue(ept);
|
||||
cJSON *tis = cJSON_GetObjectItem(bf, "tick_interval_seconds");
|
||||
if (tis) cfg->backfill.tick_interval_seconds = (int)cJSON_GetNumberValue(tis);
|
||||
cJSON *wcs = cJSON_GetObjectItem(bf, "window_cooldown_seconds");
|
||||
if (wcs) cfg->backfill.window_cooldown_seconds = (int)cJSON_GetNumberValue(wcs);
|
||||
}
|
||||
|
||||
/* live */
|
||||
cJSON *lv = cJSON_GetObjectItem(root, "live");
|
||||
if (lv) {
|
||||
cfg->live.enabled = cJSON_IsTrue(cJSON_GetObjectItem(lv, "enabled"));
|
||||
cJSON *rsi = cJSON_GetObjectItem(lv, "resubscribe_interval_seconds");
|
||||
if (rsi) cfg->live.resubscribe_interval_seconds = (int)cJSON_GetNumberValue(rsi);
|
||||
}
|
||||
|
||||
/* follow_graph_refresh_seconds */
|
||||
cJSON *fgr = cJSON_GetObjectItem(root, "follow_graph_refresh_seconds");
|
||||
if (fgr) cfg->follow_graph_refresh_seconds = (int)cJSON_GetNumberValue(fgr);
|
||||
|
||||
/* state */
|
||||
cJSON *st = cJSON_GetObjectItem(root, "state");
|
||||
if (st) {
|
||||
cJSON *bu = cJSON_GetObjectItem(st, "backfilled_until");
|
||||
if (bu) cfg->state.backfilled_until = (long)cJSON_GetNumberValue(bu);
|
||||
cJSON *cwi = cJSON_GetObjectItem(st, "current_window_index");
|
||||
if (cwi) cfg->state.current_window_index = (int)cJSON_GetNumberValue(cwi);
|
||||
cJSON *bc = cJSON_GetObjectItem(st, "backfill_cursor");
|
||||
if (bc) cfg->state.backfill_cursor = (int)cJSON_GetNumberValue(bc);
|
||||
}
|
||||
|
||||
cJSON_Delete(root);
|
||||
|
||||
/* Defaults if missing. */
|
||||
if (cfg->backfill.window_count == 0) {
|
||||
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
|
||||
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
|
||||
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
|
||||
for (int i = 0; i < n; i++) cfg->backfill.window_schedule_seconds[i] = def_windows[i];
|
||||
cfg->backfill.window_count = n;
|
||||
}
|
||||
if (cfg->backfill.events_per_tick == 0) cfg->backfill.events_per_tick = 50;
|
||||
if (cfg->backfill.tick_interval_seconds == 0) cfg->backfill.tick_interval_seconds = 5;
|
||||
if (cfg->backfill.window_cooldown_seconds == 0) cfg->backfill.window_cooldown_seconds = 60;
|
||||
if (cfg->live.resubscribe_interval_seconds == 0) cfg->live.resubscribe_interval_seconds = 300;
|
||||
if (cfg->follow_graph_refresh_seconds == 0) cfg->follow_graph_refresh_seconds = 600;
|
||||
|
||||
/* Validation. */
|
||||
if (cfg->root_npub_count == 0) {
|
||||
DEBUG_ERROR("config: at least one root_npub required");
|
||||
return -1;
|
||||
}
|
||||
if (cfg->upstream_count == 0) {
|
||||
DEBUG_ERROR("config: at least one upstream_relay required");
|
||||
return -1;
|
||||
}
|
||||
if (cfg->local_relay[0] == '\0') {
|
||||
DEBUG_ERROR("config: local_relay required");
|
||||
return -1;
|
||||
}
|
||||
if (cfg->kind_count == 0) {
|
||||
DEBUG_ERROR("config: at least one kind required");
|
||||
return -1;
|
||||
}
|
||||
|
||||
DEBUG_INFO("config loaded: %d root npubs, %d upstream relays, %d kinds",
|
||||
cfg->root_npub_count, cfg->upstream_count, cfg->kind_count);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Build a fresh cJSON tree from the in-memory config and write it out.
|
||||
* We re-emit plain JSON (comments are not preserved on rewrite). */
|
||||
int cr_config_save_state(cr_config_t *cfg) {
|
||||
cJSON *root = cJSON_CreateObject();
|
||||
|
||||
cJSON *npubs = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->root_npub_count; i++)
|
||||
cJSON_AddItemToArray(npubs, cJSON_CreateString(cfg->root_npubs[i]));
|
||||
cJSON_AddItemToObject(root, "root_npubs", npubs);
|
||||
|
||||
cJSON *upstream = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->upstream_count; i++)
|
||||
cJSON_AddItemToArray(upstream, cJSON_CreateString(cfg->upstream_relays[i]));
|
||||
cJSON_AddItemToObject(root, "upstream_relays", upstream);
|
||||
|
||||
cJSON_AddItemToObject(root, "local_relay", cJSON_CreateString(cfg->local_relay));
|
||||
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->kind_count; i++)
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
|
||||
cJSON_AddItemToObject(root, "kinds", kinds);
|
||||
|
||||
/* admin_kinds */
|
||||
cJSON *akinds = cJSON_CreateArray();
|
||||
if (cfg->admin_all_kinds) {
|
||||
cJSON_AddItemToArray(akinds, cJSON_CreateString("*"));
|
||||
} else {
|
||||
for (int i = 0; i < cfg->admin_kind_count; i++)
|
||||
cJSON_AddItemToArray(akinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
|
||||
}
|
||||
cJSON_AddItemToObject(root, "admin_kinds", akinds);
|
||||
|
||||
cJSON *bf = cJSON_CreateObject();
|
||||
cJSON_AddItemToObject(bf, "enabled", cJSON_CreateBool(cfg->backfill.enabled));
|
||||
cJSON *ws = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->backfill.window_count; i++)
|
||||
cJSON_AddItemToArray(ws, cJSON_CreateNumber(cfg->backfill.window_schedule_seconds[i]));
|
||||
cJSON_AddItemToObject(bf, "window_schedule_seconds", ws);
|
||||
cJSON_AddItemToObject(bf, "events_per_tick", cJSON_CreateNumber(cfg->backfill.events_per_tick));
|
||||
cJSON_AddItemToObject(bf, "tick_interval_seconds", cJSON_CreateNumber(cfg->backfill.tick_interval_seconds));
|
||||
cJSON_AddItemToObject(bf, "window_cooldown_seconds", cJSON_CreateNumber(cfg->backfill.window_cooldown_seconds));
|
||||
cJSON_AddItemToObject(root, "backfill", bf);
|
||||
|
||||
cJSON *lv = cJSON_CreateObject();
|
||||
cJSON_AddItemToObject(lv, "enabled", cJSON_CreateBool(cfg->live.enabled));
|
||||
cJSON_AddItemToObject(lv, "resubscribe_interval_seconds", cJSON_CreateNumber(cfg->live.resubscribe_interval_seconds));
|
||||
cJSON_AddItemToObject(root, "live", lv);
|
||||
|
||||
cJSON_AddItemToObject(root, "follow_graph_refresh_seconds",
|
||||
cJSON_CreateNumber(cfg->follow_graph_refresh_seconds));
|
||||
|
||||
cJSON *st = cJSON_CreateObject();
|
||||
cJSON_AddItemToObject(st, "backfilled_until", cJSON_CreateNumber(cfg->state.backfilled_until));
|
||||
cJSON_AddItemToObject(st, "current_window_index", cJSON_CreateNumber(cfg->state.current_window_index));
|
||||
cJSON_AddItemToObject(st, "backfill_cursor", cJSON_CreateNumber(cfg->state.backfill_cursor));
|
||||
cJSON_AddItemToObject(root, "state", st);
|
||||
|
||||
char *json = cJSON_Print(root);
|
||||
cJSON_Delete(root);
|
||||
if (!json) {
|
||||
DEBUG_ERROR("config save: failed to serialize");
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* Atomic write: temp file + rename. */
|
||||
char tmp[1100];
|
||||
snprintf(tmp, sizeof(tmp), "%s.tmp", cfg->path);
|
||||
int fd = open(tmp, O_WRONLY | O_CREAT | O_TRUNC, 0644);
|
||||
if (fd < 0) {
|
||||
DEBUG_ERROR("config save: cannot open tmp '%s': %s", tmp, strerror(errno));
|
||||
free(json);
|
||||
return -1;
|
||||
}
|
||||
size_t jlen = strlen(json);
|
||||
ssize_t w = write(fd, json, jlen);
|
||||
close(fd);
|
||||
free(json);
|
||||
if (w < 0 || (size_t)w != jlen) {
|
||||
DEBUG_ERROR("config save: short write");
|
||||
return -1;
|
||||
}
|
||||
if (rename(tmp, cfg->path) != 0) {
|
||||
DEBUG_ERROR("config save: rename failed: %s", strerror(errno));
|
||||
return -1;
|
||||
}
|
||||
DEBUG_LOG("config state saved: backfilled_until=%ld window=%d cursor=%d",
|
||||
cfg->state.backfilled_until, cfg->state.current_window_index,
|
||||
cfg->state.backfill_cursor);
|
||||
return 0;
|
||||
}
|
||||
|
||||
void cr_config_free(cr_config_t *cfg) {
|
||||
(void)cfg;
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
/*
|
||||
* caching_relay - config parsing + persistent state (in the .jsonc file itself)
|
||||
*/
|
||||
#ifndef CACHING_RELAY_CONFIG_H
|
||||
#define CACHING_RELAY_CONFIG_H
|
||||
|
||||
#include <stddef.h>
|
||||
#include <time.h>
|
||||
|
||||
/* Maximums to keep things statically sized and simple. */
|
||||
#define CR_MAX_ROOT_NPUBS 16
|
||||
#define CR_MAX_UPSTREAM 32
|
||||
#define CR_MAX_KINDS 32
|
||||
#define CR_MAX_WINDOWS 16
|
||||
#define CR_NPUB_LEN 64 /* npub1... bech32, generous */
|
||||
#define CR_URL_LEN 256
|
||||
#define CR_HEX_PUBKEY_LEN 65 /* 64 hex chars + NUL */
|
||||
|
||||
typedef struct {
|
||||
int enabled;
|
||||
long window_schedule_seconds[CR_MAX_WINDOWS];
|
||||
int window_count;
|
||||
int events_per_tick;
|
||||
int tick_interval_seconds;
|
||||
int window_cooldown_seconds;
|
||||
} cr_backfill_config_t;
|
||||
|
||||
typedef struct {
|
||||
int enabled;
|
||||
int resubscribe_interval_seconds;
|
||||
} cr_live_config_t;
|
||||
|
||||
typedef struct {
|
||||
long backfilled_until; /* unix ts; 0 = nothing backfilled yet */
|
||||
int current_window_index; /* index into window_schedule */
|
||||
int backfill_cursor; /* round-robin cursor over followed pubkeys */
|
||||
} cr_state_t;
|
||||
|
||||
typedef struct {
|
||||
char root_npubs[CR_MAX_ROOT_NPUBS][CR_NPUB_LEN];
|
||||
int root_npub_count;
|
||||
/* Decoded hex pubkeys for root npubs (filled by follow_graph). */
|
||||
char root_hex[CR_MAX_ROOT_NPUBS][CR_HEX_PUBKEY_LEN];
|
||||
int root_hex_ready;
|
||||
|
||||
char upstream_relays[CR_MAX_UPSTREAM][CR_URL_LEN];
|
||||
int upstream_count;
|
||||
|
||||
char local_relay[CR_URL_LEN];
|
||||
|
||||
int kinds[CR_MAX_KINDS];
|
||||
int kind_count;
|
||||
|
||||
/* Kinds to follow specifically for the root (admin) npubs.
|
||||
* If admin_all_kinds is 1, grab everything (no kind filter). */
|
||||
int admin_kinds[CR_MAX_KINDS];
|
||||
int admin_kind_count;
|
||||
int admin_all_kinds;
|
||||
|
||||
cr_backfill_config_t backfill;
|
||||
cr_live_config_t live;
|
||||
int follow_graph_refresh_seconds;
|
||||
|
||||
cr_state_t state;
|
||||
|
||||
/* Path the config was loaded from (for state write-back). */
|
||||
char path[1024];
|
||||
} cr_config_t;
|
||||
|
||||
/* Load + validate config from a .jsonc file. Returns 0 on success, -1 on error.
|
||||
* On success cfg->path is set and cfg->state is populated from the file. */
|
||||
int cr_config_load(cr_config_t *cfg, const char *path);
|
||||
|
||||
/* Persist the state sub-object back into the config file (temp + rename).
|
||||
* Preserves all other config fields. Returns 0 on success, -1 on error. */
|
||||
int cr_config_save_state(cr_config_t *cfg);
|
||||
|
||||
/* Free any heap resources held by cfg (currently none, but kept for future). */
|
||||
void cr_config_free(cr_config_t *cfg);
|
||||
|
||||
#endif /* CACHING_RELAY_CONFIG_H */
|
||||
@@ -0,0 +1,71 @@
|
||||
#include "debug.h"
|
||||
#include <stdarg.h>
|
||||
#include <string.h>
|
||||
|
||||
/**
|
||||
* @file debug.c
|
||||
* @brief Debug and logging system implementation
|
||||
*
|
||||
* Provides a configurable logging system with timestamp formatting,
|
||||
* level-based filtering, and optional file:line information.
|
||||
*/
|
||||
|
||||
// Global debug level (default: no debug output)
|
||||
debug_level_t g_debug_level = DEBUG_LEVEL_NONE;
|
||||
|
||||
/**
|
||||
* @brief Initialize the debug system with a specific level
|
||||
* @param level Debug level (0-5, clamped to valid range)
|
||||
*/
|
||||
void debug_init(int level) {
|
||||
if (level < 0) level = 0;
|
||||
if (level > 5) level = 5;
|
||||
g_debug_level = (debug_level_t)level;
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Core logging function
|
||||
* @param level Debug level for this message
|
||||
* @param file Source file name (__FILE__)
|
||||
* @param line Source line number (__LINE__)
|
||||
* @param format printf-style format string
|
||||
* @param ... Variable arguments for format string
|
||||
*/
|
||||
void debug_log(debug_level_t level, const char* file, int line, const char* format, ...) {
|
||||
// Get timestamp
|
||||
time_t now = time(NULL);
|
||||
struct tm* tm_info = localtime(&now);
|
||||
char timestamp[32];
|
||||
strftime(timestamp, sizeof(timestamp), "%Y-%m-%d %H:%M:%S", tm_info);
|
||||
|
||||
// Get level string
|
||||
const char* level_str = "UNKNOWN";
|
||||
switch (level) {
|
||||
case DEBUG_LEVEL_ERROR: level_str = "ERROR"; break;
|
||||
case DEBUG_LEVEL_WARN: level_str = "WARN "; break;
|
||||
case DEBUG_LEVEL_INFO: level_str = "INFO "; break;
|
||||
case DEBUG_LEVEL_DEBUG: level_str = "DEBUG"; break;
|
||||
case DEBUG_LEVEL_TRACE: level_str = "TRACE"; break;
|
||||
default: break;
|
||||
}
|
||||
|
||||
// Print prefix with timestamp and level
|
||||
printf("[%s] [%s] ", timestamp, level_str);
|
||||
|
||||
// Print source location when debug level is TRACE (5) or higher
|
||||
if (file && g_debug_level >= DEBUG_LEVEL_TRACE) {
|
||||
// Extract just the filename (not full path)
|
||||
const char* filename = strrchr(file, '/');
|
||||
filename = filename ? filename + 1 : file;
|
||||
printf("[%s:%d] ", filename, line);
|
||||
}
|
||||
|
||||
// Print message
|
||||
va_list args;
|
||||
va_start(args, format);
|
||||
vprintf(format, args);
|
||||
va_end(args);
|
||||
|
||||
printf("\n");
|
||||
fflush(stdout);
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
#ifndef C_UTILS_DEBUG_H
|
||||
#define C_UTILS_DEBUG_H
|
||||
|
||||
/**
|
||||
* @file debug.h
|
||||
* @brief Debug and logging system with configurable verbosity levels
|
||||
*
|
||||
* Provides a simple, efficient logging system with 5 levels:
|
||||
* - ERROR: Critical errors
|
||||
* - WARN: Warnings
|
||||
* - INFO: Informational messages
|
||||
* - DEBUG: Debug messages
|
||||
* - TRACE: Detailed trace with file:line info
|
||||
*/
|
||||
|
||||
#include <stdio.h>
|
||||
#include <time.h>
|
||||
|
||||
// Debug levels
|
||||
typedef enum {
|
||||
DEBUG_LEVEL_NONE = 0, /**< No debug output */
|
||||
DEBUG_LEVEL_ERROR = 1, /**< Critical errors only */
|
||||
DEBUG_LEVEL_WARN = 2, /**< Warnings and above */
|
||||
DEBUG_LEVEL_INFO = 3, /**< Informational messages and above */
|
||||
DEBUG_LEVEL_DEBUG = 4, /**< Debug messages and above */
|
||||
DEBUG_LEVEL_TRACE = 5 /**< Detailed trace with file:line info */
|
||||
} debug_level_t;
|
||||
|
||||
// Global debug level (set at runtime via CLI)
|
||||
extern debug_level_t g_debug_level;
|
||||
|
||||
/**
|
||||
* @brief Initialize the debug system
|
||||
* @param level Debug level (0-5, clamped to valid range)
|
||||
*/
|
||||
void debug_init(int level);
|
||||
|
||||
/**
|
||||
* @brief Core logging function
|
||||
* @param level Debug level for this message
|
||||
* @param file Source file name (__FILE__)
|
||||
* @param line Source line number (__LINE__)
|
||||
* @param format printf-style format string
|
||||
* @param ... Variable arguments for format string
|
||||
*/
|
||||
void debug_log(debug_level_t level, const char* file, int line, const char* format, ...);
|
||||
|
||||
// Convenience macros that check level before calling
|
||||
// Note: TRACE level (5) and above include file:line information for ALL messages
|
||||
#define DEBUG_ERROR(...) \
|
||||
do { if (g_debug_level >= DEBUG_LEVEL_ERROR) debug_log(DEBUG_LEVEL_ERROR, __FILE__, __LINE__, __VA_ARGS__); } while(0)
|
||||
|
||||
#define DEBUG_WARN(...) \
|
||||
do { if (g_debug_level >= DEBUG_LEVEL_WARN) debug_log(DEBUG_LEVEL_WARN, __FILE__, __LINE__, __VA_ARGS__); } while(0)
|
||||
|
||||
#define DEBUG_INFO(...) \
|
||||
do { if (g_debug_level >= DEBUG_LEVEL_INFO) debug_log(DEBUG_LEVEL_INFO, __FILE__, __LINE__, __VA_ARGS__); } while(0)
|
||||
|
||||
#define DEBUG_LOG(...) \
|
||||
do { if (g_debug_level >= DEBUG_LEVEL_DEBUG) debug_log(DEBUG_LEVEL_DEBUG, __FILE__, __LINE__, __VA_ARGS__); } while(0)
|
||||
|
||||
#define DEBUG_TRACE(...) \
|
||||
do { if (g_debug_level >= DEBUG_LEVEL_TRACE) debug_log(DEBUG_LEVEL_TRACE, __FILE__, __LINE__, __VA_ARGS__); } while(0)
|
||||
|
||||
#endif /* C_UTILS_DEBUG_H */
|
||||
@@ -0,0 +1,110 @@
|
||||
/*
|
||||
* caching_relay - follow graph resolution
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "follow_graph.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
int cr_follow_is_root(const cr_config_t *cfg, const char *hex) {
|
||||
if (!cfg->root_hex_ready) return 0;
|
||||
for (int i = 0; i < cfg->root_npub_count; i++) {
|
||||
if (strcmp(cfg->root_hex[i], hex) == 0) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cr_follow_decode_roots(cr_config_t *cfg) {
|
||||
for (int i = 0; i < cfg->root_npub_count; i++) {
|
||||
unsigned char pubkey[32];
|
||||
if (nostr_decode_npub(cfg->root_npubs[i], pubkey) != NOSTR_SUCCESS) {
|
||||
DEBUG_ERROR("follow: failed to decode npub '%s'", cfg->root_npubs[i]);
|
||||
return -1;
|
||||
}
|
||||
nostr_bytes_to_hex(pubkey, 32, cfg->root_hex[i]);
|
||||
}
|
||||
cfg->root_hex_ready = 1;
|
||||
DEBUG_INFO("follow: decoded %d root npubs", cfg->root_npub_count);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Parse "p" tags from a kind-3 event and add pubkeys to the set. */
|
||||
static int parse_p_tags(cJSON *event, cr_pubkey_set_t *followed) {
|
||||
cJSON *tags = cJSON_GetObjectItem(event, "tags");
|
||||
if (!tags || !cJSON_IsArray(tags)) return 0;
|
||||
int added = 0;
|
||||
cJSON *tag;
|
||||
cJSON_ArrayForEach(tag, tags) {
|
||||
if (!cJSON_IsArray(tag)) continue;
|
||||
cJSON *name = cJSON_GetArrayItem(tag, 0);
|
||||
if (!name || !cJSON_IsString(name)) continue;
|
||||
if (strcmp(cJSON_GetStringValue(name), "p") != 0) continue;
|
||||
cJSON *pk = cJSON_GetArrayItem(tag, 1);
|
||||
if (!pk || !cJSON_IsString(pk)) continue;
|
||||
const char *hex = cJSON_GetStringValue(pk);
|
||||
if (strlen(hex) != 64) continue;
|
||||
if (cr_pubkey_set_add(followed, hex) == 1) added++;
|
||||
}
|
||||
return added;
|
||||
}
|
||||
|
||||
int cr_follow_resolve(cr_config_t *cfg, nostr_relay_pool_t *upstream,
|
||||
cr_pubkey_set_t *followed) {
|
||||
if (!cfg->root_hex_ready) {
|
||||
if (cr_follow_decode_roots(cfg) != 0) return -1;
|
||||
}
|
||||
|
||||
/* Seed the followed set with the root pubkeys themselves. */
|
||||
for (int i = 0; i < cfg->root_npub_count; i++) {
|
||||
cr_pubkey_set_add(followed, cfg->root_hex[i]);
|
||||
}
|
||||
|
||||
int total_follows = 0;
|
||||
for (int i = 0; i < cfg->root_npub_count; i++) {
|
||||
/* Build filter: authors=[root], kinds=[3], limit=1 (most recent). */
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(cfg->root_hex[i]));
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(3));
|
||||
cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber(1));
|
||||
|
||||
const char **urls = NULL;
|
||||
int n = 0;
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int listed_count = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
if (listed_count > 0) {
|
||||
urls = malloc(listed_count * sizeof(char *));
|
||||
for (int j = 0; j < listed_count; j++) { urls[n++] = listed[j]; }
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
|
||||
int ev_count = 0;
|
||||
cJSON **events = nostr_relay_pool_query_sync(upstream, urls, n,
|
||||
filter, &ev_count, 15000);
|
||||
free(urls);
|
||||
cJSON_Delete(filter);
|
||||
|
||||
if (events && ev_count > 0) {
|
||||
int added = parse_p_tags(events[0], followed);
|
||||
total_follows += added;
|
||||
DEBUG_INFO("follow: root %d: kind-3 found, +%d follows (total set %d)",
|
||||
i, added, followed->count);
|
||||
for (int k = 0; k < ev_count; k++) cJSON_Delete(events[k]);
|
||||
free(events);
|
||||
} else {
|
||||
DEBUG_WARN("follow: root %d: no kind-3 found", i);
|
||||
free(events);
|
||||
}
|
||||
}
|
||||
|
||||
DEBUG_INFO("follow: resolved %d total followed pubkeys (set size %d)",
|
||||
total_follows, followed->count);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
/*
|
||||
* caching_relay - follow graph resolution.
|
||||
*
|
||||
* Decodes root npubs (NIP-19) to hex, queries their most recent kind-3
|
||||
* contact list from the upstream pool, parses the "p" tags, and populates
|
||||
* the in-memory followed-pubkey set (root pubkeys + their follows).
|
||||
*/
|
||||
#ifndef CACHING_RELAY_FOLLOW_GRAPH_H
|
||||
#define CACHING_RELAY_FOLLOW_GRAPH_H
|
||||
|
||||
#include "config.h"
|
||||
#include "state.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
|
||||
/* Check if a hex pubkey is one of the root (admin) npubs. Returns 1 if yes. */
|
||||
int cr_follow_is_root(const cr_config_t *cfg, const char *hex);
|
||||
|
||||
/* Decode all root_npubs in cfg into cfg->root_hex[]. Returns 0 on success. */
|
||||
int cr_follow_decode_roots(cr_config_t *cfg);
|
||||
|
||||
/* Resolve the follow graph: for each root hex pubkey, query the most recent
|
||||
* kind-3 from the upstream pool, parse "p" tags, and add root + follows to
|
||||
* the followed set. Returns 0 on success, -1 on hard failure. */
|
||||
int cr_follow_resolve(cr_config_t *cfg, nostr_relay_pool_t *upstream,
|
||||
cr_pubkey_set_t *followed);
|
||||
|
||||
#endif /* CACHING_RELAY_FOLLOW_GRAPH_H */
|
||||
@@ -0,0 +1,69 @@
|
||||
/*
|
||||
* caching_relay - strip JSONC comments so cJSON can parse it.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "jsonc_strip.h"
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
char *jsonc_strip_comments(const char *input, size_t len) {
|
||||
if (!input) return NULL;
|
||||
if (len == 0) len = strlen(input);
|
||||
|
||||
/* Worst case: output is same size as input (no comments). +1 for NUL. */
|
||||
char *out = malloc(len + 1);
|
||||
if (!out) return NULL;
|
||||
|
||||
size_t i = 0, o = 0;
|
||||
int in_string = 0;
|
||||
int escape = 0;
|
||||
|
||||
while (i < len) {
|
||||
char c = input[i];
|
||||
|
||||
if (in_string) {
|
||||
out[o++] = c;
|
||||
if (escape) {
|
||||
escape = 0;
|
||||
} else if (c == '\\') {
|
||||
escape = 1;
|
||||
} else if (c == '"') {
|
||||
in_string = 0;
|
||||
}
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Not in a string. */
|
||||
if (c == '"') {
|
||||
in_string = 1;
|
||||
out[o++] = c;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Line comment // */
|
||||
if (c == '/' && i + 1 < len && input[i + 1] == '/') {
|
||||
i += 2;
|
||||
while (i < len && input[i] != '\n') i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
/* Block comment / * ... * / */
|
||||
if (c == '/' && i + 1 < len && input[i + 1] == '*') {
|
||||
i += 2;
|
||||
while (i < len && !(input[i] == '*' && i + 1 < len && input[i + 1] == '/')) i++;
|
||||
if (i < len) i += 2; /* skip closing */
|
||||
/* Preserve a space so tokens don't merge. */
|
||||
out[o++] = ' ';
|
||||
continue;
|
||||
}
|
||||
|
||||
out[o++] = c;
|
||||
i++;
|
||||
}
|
||||
|
||||
out[o] = '\0';
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
/*
|
||||
* caching_relay - strip JSONC comments so cJSON can parse it.
|
||||
*
|
||||
* cJSON does not natively understand JSONC. This produces a heap buffer the
|
||||
* caller must free() containing the comment-stripped JSON. String literals are
|
||||
* respected so a double-slash inside a string is preserved.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_JSONC_STRIP_H
|
||||
#define CACHING_RELAY_JSONC_STRIP_H
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
/* Returns malloc'd buffer (null-terminated) or NULL on alloc failure. */
|
||||
char *jsonc_strip_comments(const char *input, size_t len);
|
||||
|
||||
#endif /* CACHING_RELAY_JSONC_STRIP_H */
|
||||
@@ -0,0 +1,171 @@
|
||||
/*
|
||||
* caching_relay - live subscription on the upstream pool
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "live_subscriber.h"
|
||||
#include "follow_graph.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
#include <time.h>
|
||||
|
||||
/* Context passed as user_data to the subscription callbacks. */
|
||||
typedef struct {
|
||||
cr_sink_t *sink;
|
||||
cr_live_t *live;
|
||||
} cr_live_ctx_t;
|
||||
|
||||
static cr_live_ctx_t g_live_ctx;
|
||||
|
||||
static void live_on_event(cJSON *event, const char *relay_url, void *user_data) {
|
||||
cr_live_ctx_t *ctx = (cr_live_ctx_t *)user_data;
|
||||
(void)relay_url;
|
||||
ctx->live->events_received++;
|
||||
cr_sink_publish(ctx->sink, event);
|
||||
}
|
||||
|
||||
static void live_on_eose(cJSON **events, int event_count, void *user_data) {
|
||||
(void)events; (void)event_count; (void)user_data;
|
||||
DEBUG_LOG("live: EOSE (stored events flushed), staying open");
|
||||
}
|
||||
|
||||
/* Build a filter for non-admin followed pubkeys with regular kinds. */
|
||||
static cJSON *build_follows_filter(cr_config_t *cfg, cr_pubkey_set_t *followed) {
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
/* Skip admin pubkeys - they get their own subscription. */
|
||||
if (cr_follow_is_root(cfg, followed->items[i])) continue;
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(followed->items[i]));
|
||||
}
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->kind_count; i++)
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->kinds[i]));
|
||||
cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
|
||||
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)time(NULL)));
|
||||
return filter;
|
||||
}
|
||||
|
||||
/* Build a filter for admin (root) pubkeys with admin_kinds (or all kinds). */
|
||||
static cJSON *build_admin_filter(cr_config_t *cfg) {
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->root_npub_count; i++)
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(cfg->root_hex[i]));
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
|
||||
/* If admin_all_kinds, omit the kinds filter entirely (grab everything). */
|
||||
if (!cfg->admin_all_kinds && cfg->admin_kind_count > 0) {
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
for (int i = 0; i < cfg->admin_kind_count; i++)
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(cfg->admin_kinds[i]));
|
||||
cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
}
|
||||
|
||||
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)time(NULL)));
|
||||
return filter;
|
||||
}
|
||||
|
||||
static int get_relay_urls(nostr_relay_pool_t *upstream, const char ***urls_out) {
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
const char **urls = NULL;
|
||||
if (n > 0) {
|
||||
urls = malloc(n * sizeof(char *));
|
||||
for (int j = 0; j < n; j++) urls[j] = listed[j];
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
*urls_out = urls;
|
||||
return n;
|
||||
}
|
||||
|
||||
static nostr_pool_subscription_t *open_subscription(nostr_relay_pool_t *upstream,
|
||||
cJSON *filter) {
|
||||
const char **urls = NULL;
|
||||
int n = get_relay_urls(upstream, &urls);
|
||||
|
||||
nostr_pool_subscription_t *sub = nostr_relay_pool_subscribe(
|
||||
upstream, urls, n, filter,
|
||||
live_on_event, live_on_eose, &g_live_ctx,
|
||||
0, 1, NOSTR_POOL_EOSE_FULL_SET, 0, 0);
|
||||
free(urls);
|
||||
return sub;
|
||||
}
|
||||
|
||||
static int open_subs(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
|
||||
cr_pubkey_set_t *followed, cr_sink_t *sink) {
|
||||
g_live_ctx.sink = sink;
|
||||
g_live_ctx.live = live;
|
||||
|
||||
int admin_count = 0;
|
||||
int follows_count = 0;
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (cr_follow_is_root(cfg, followed->items[i])) admin_count++;
|
||||
else follows_count++;
|
||||
}
|
||||
|
||||
/* Follows subscription (non-admin pubkeys, regular kinds). */
|
||||
if (follows_count > 0) {
|
||||
cJSON *filter = build_follows_filter(cfg, followed);
|
||||
live->follows_sub = open_subscription(upstream, filter);
|
||||
cJSON_Delete(filter);
|
||||
if (!live->follows_sub) {
|
||||
DEBUG_ERROR("live: follows subscribe failed");
|
||||
} else {
|
||||
DEBUG_INFO("live: follows sub - %d authors, %d kinds",
|
||||
follows_count, cfg->kind_count);
|
||||
}
|
||||
}
|
||||
|
||||
/* Admin subscription (root npubs, admin_kinds or all kinds). */
|
||||
if (admin_count > 0 || cfg->root_npub_count > 0) {
|
||||
cJSON *filter = build_admin_filter(cfg);
|
||||
live->admin_sub = open_subscription(upstream, filter);
|
||||
cJSON_Delete(filter);
|
||||
if (!live->admin_sub) {
|
||||
DEBUG_ERROR("live: admin subscribe failed");
|
||||
} else {
|
||||
if (cfg->admin_all_kinds) {
|
||||
DEBUG_INFO("live: admin sub - %d authors, ALL kinds",
|
||||
cfg->root_npub_count);
|
||||
} else {
|
||||
DEBUG_INFO("live: admin sub - %d authors, %d kinds",
|
||||
cfg->root_npub_count, cfg->admin_kind_count);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
live->last_resubscribe = time(NULL);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cr_live_open(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
|
||||
cr_pubkey_set_t *followed, cr_sink_t *sink) {
|
||||
memset(live, 0, sizeof(*live));
|
||||
return open_subs(live, cfg, upstream, followed, sink);
|
||||
}
|
||||
|
||||
int cr_live_resubscribe(cr_live_t *live, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
|
||||
cr_sink_t *sink) {
|
||||
cr_live_close(live);
|
||||
DEBUG_INFO("live: resubscribing (events so far: %ld)", live->events_received);
|
||||
return open_subs(live, cfg, upstream, followed, sink);
|
||||
}
|
||||
|
||||
void cr_live_close(cr_live_t *live) {
|
||||
if (live->follows_sub) {
|
||||
nostr_pool_subscription_close(live->follows_sub);
|
||||
live->follows_sub = NULL;
|
||||
}
|
||||
if (live->admin_sub) {
|
||||
nostr_pool_subscription_close(live->admin_sub);
|
||||
live->admin_sub = NULL;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
/*
|
||||
* caching_relay - live subscription on the upstream pool.
|
||||
*
|
||||
* Builds filters from the followed-pubkey set + configured kinds, opens
|
||||
* long-lived subscriptions, and routes on_event to the relay sink.
|
||||
*
|
||||
* Two subscriptions are opened:
|
||||
* 1. "follows" sub: non-admin followed pubkeys + regular kinds
|
||||
* 2. "admin" sub: root (admin) npubs + admin_kinds (or all kinds if [*])
|
||||
*/
|
||||
#ifndef CACHING_RELAY_LIVE_SUBSCRIBER_H
|
||||
#define CACHING_RELAY_LIVE_SUBSCRIBER_H
|
||||
|
||||
#include "config.h"
|
||||
#include "state.h"
|
||||
#include "relay_sink.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
|
||||
typedef struct {
|
||||
nostr_pool_subscription_t *follows_sub; /* non-admin pubkeys, regular kinds */
|
||||
nostr_pool_subscription_t *admin_sub; /* admin pubkeys, admin_kinds */
|
||||
long events_received;
|
||||
time_t last_resubscribe;
|
||||
} cr_live_t;
|
||||
|
||||
/* Open the live subscription(s). Returns 0 on success. */
|
||||
int cr_live_open(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upstream,
|
||||
cr_pubkey_set_t *followed, cr_sink_t *sink);
|
||||
|
||||
/* Close + reopen the subscription(s). Returns 0 on success. */
|
||||
int cr_live_resubscribe(cr_live_t *live, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
|
||||
cr_sink_t *sink);
|
||||
|
||||
void cr_live_close(cr_live_t *live);
|
||||
|
||||
#endif /* CACHING_RELAY_LIVE_SUBSCRIBER_H */
|
||||
@@ -0,0 +1,463 @@
|
||||
/*
|
||||
* caching_relay - a daemon that caches Nostr events from followed people
|
||||
* into a local relay.
|
||||
*
|
||||
* Architecture: see plans/plan.md. Two nostr_relay_pool_t instances:
|
||||
* - upstream_pool: query/subscribe to upstream relays
|
||||
* - sink_pool (in cr_sink_t): publish-only to the local relay
|
||||
*
|
||||
* Build: see Makefile. Statically linked C99 binary, c-relay style.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "main.h"
|
||||
#include "debug.h"
|
||||
#include "config.h"
|
||||
#include "state.h"
|
||||
#include "follow_graph.h"
|
||||
#include "relay_sink.h"
|
||||
#include "live_subscriber.h"
|
||||
#include "backfill.h"
|
||||
#include "relay_discovery.h"
|
||||
#include "pg_inbox.h"
|
||||
#include "pg_config.h"
|
||||
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_log.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <signal.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
#include <getopt.h>
|
||||
|
||||
/* Forward nostr_core_lib's internal logging to our debug system. */
|
||||
static void nostr_log_forwarder(int level, const char *component,
|
||||
const char *message, void *user_data) {
|
||||
(void)user_data;
|
||||
/* Map nostr log levels to our debug levels (they're the same 1-5). */
|
||||
if (level >= 5) {
|
||||
DEBUG_TRACE("[nostr:%s] %s", component ? component : "?", message ? message : "");
|
||||
} else if (level >= 4) {
|
||||
DEBUG_LOG("[nostr:%s] %s", component ? component : "?", message ? message : "");
|
||||
} else if (level >= 3) {
|
||||
DEBUG_INFO("[nostr:%s] %s", component ? component : "?", message ? message : "");
|
||||
} else if (level >= 2) {
|
||||
DEBUG_WARN("[nostr:%s] %s", component ? component : "?", message ? message : "");
|
||||
} else {
|
||||
DEBUG_ERROR("[nostr:%s] %s", component ? component : "?", message ? message : "");
|
||||
}
|
||||
}
|
||||
|
||||
volatile sig_atomic_t g_shutdown = 0;
|
||||
static volatile sig_atomic_t g_reload = 0;
|
||||
|
||||
static void on_signal(int sig) {
|
||||
if (sig == SIGINT || sig == SIGTERM) g_shutdown = 1;
|
||||
else if (sig == SIGHUP) g_reload = 1;
|
||||
}
|
||||
|
||||
static void usage(const char *prog) {
|
||||
fprintf(stderr,
|
||||
"caching_relay %s - cache Nostr events from followed people into a local relay\n"
|
||||
"\n"
|
||||
"Usage: %s [-c <config.jsonc>] [-p <pg-conn>] [options]\n"
|
||||
"\n"
|
||||
"Options:\n"
|
||||
" -c, --config <file> Path to .jsonc config file (default: ./caching_relay_config.jsonc)\n"
|
||||
" -p, --pg-conn <str> PostgreSQL connection string (libpq format). When provided,\n"
|
||||
" config is read from the c-relay-pg config table and fetched\n"
|
||||
" events are inserted into the caching_event_inbox table instead\n"
|
||||
" of being published via WebSocket to a local relay.\n"
|
||||
" -d, --debug <level> Log level 0-5 (0=none, 1=error, 2=warn, 3=info, 4=debug, 5=trace). Default 3.\n"
|
||||
" -r, --restart Reset state to first-time startup (ignore local relay cache,\n"
|
||||
" re-discover all kind-10002 from bootstrap relays, reset backfill)\n"
|
||||
" -h, --help Show this help\n"
|
||||
"\n"
|
||||
"Without -p, the daemon uses the .jsonc config file and publishes to a local relay\n"
|
||||
"via WebSocket (legacy mode). With -p, it uses PostgreSQL for config and inbox.\n"
|
||||
"The config file is also the persistent state store in legacy mode; the daemon\n"
|
||||
"rewrites it as backfill progresses. See plans/plan.md and caching_relay_config.jsonc.\n",
|
||||
CR_VERSION, prog);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
const char *config_path = NULL;
|
||||
const char *pg_conn = NULL;
|
||||
int log_level = DEBUG_LEVEL_INFO;
|
||||
int restart = 0;
|
||||
|
||||
static struct option longopts[] = {
|
||||
{"config", required_argument, 0, 'c'},
|
||||
{"pg-conn", required_argument, 0, 'p'},
|
||||
{"debug", required_argument, 0, 'd'},
|
||||
{"restart", no_argument, 0, 'r'},
|
||||
{"help", no_argument, 0, 'h'},
|
||||
{0, 0, 0, 0}
|
||||
};
|
||||
int opt;
|
||||
while ((opt = getopt_long(argc, argv, "c:p:d:rh", longopts, NULL)) != -1) {
|
||||
switch (opt) {
|
||||
case 'c': config_path = optarg; break;
|
||||
case 'p': pg_conn = optarg; break;
|
||||
case 'd': log_level = atoi(optarg); break;
|
||||
case 'r': restart = 1; break;
|
||||
case 'h': usage(argv[0]); return 0;
|
||||
default: usage(argv[0]); return 1;
|
||||
}
|
||||
}
|
||||
/* In PostgreSQL mode, the .jsonc config file is optional. */
|
||||
if (!config_path && !pg_conn) {
|
||||
/* Default: look for caching_relay.jsonc in the current directory. */
|
||||
config_path = "caching_relay_config.jsonc";
|
||||
if (access(config_path, F_OK) != 0) {
|
||||
fprintf(stderr, "ERROR: no config specified and default '%s' not found\n\n", config_path);
|
||||
usage(argv[0]);
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
if (log_level < 0) log_level = 0;
|
||||
if (log_level > 5) log_level = 5;
|
||||
|
||||
debug_init(log_level);
|
||||
|
||||
/* Enable nostr_core_lib internal logging and forward to our debug system.
|
||||
* This shows the actual WebSocket messages (REQ filters, EVENT responses)
|
||||
* at debug level 5 (trace). */
|
||||
nostr_set_log_callback(nostr_log_forwarder, NULL);
|
||||
nostr_set_log_level((nostr_log_level_t)log_level);
|
||||
|
||||
DEBUG_INFO("caching_relay %s starting (config=%s, pg-conn=%s, loglevel=%d%s)",
|
||||
CR_VERSION,
|
||||
config_path ? config_path : "(none)",
|
||||
pg_conn ? "yes" : "no",
|
||||
log_level, restart ? ", RESTART" : "");
|
||||
|
||||
/* Signals. */
|
||||
struct sigaction sa;
|
||||
memset(&sa, 0, sizeof(sa));
|
||||
sa.sa_handler = on_signal;
|
||||
sigaction(SIGINT, &sa, NULL);
|
||||
sigaction(SIGTERM, &sa, NULL);
|
||||
sigaction(SIGHUP, &sa, NULL);
|
||||
/* Ignore SIGPIPE - relay pool handles its own socket errors. */
|
||||
signal(SIGPIPE, SIG_IGN);
|
||||
|
||||
/* PostgreSQL inbox mode: connect and load config from the config table. */
|
||||
long config_generation = -1;
|
||||
if (pg_conn) {
|
||||
if (pg_inbox_init(pg_conn) != 0) {
|
||||
DEBUG_ERROR("failed to connect to PostgreSQL");
|
||||
return 1;
|
||||
}
|
||||
config_generation = pg_inbox_get_config_generation();
|
||||
if (config_generation < 0) {
|
||||
DEBUG_WARN("could not read caching_config_generation (defaulting to 0)");
|
||||
config_generation = 0;
|
||||
}
|
||||
DEBUG_INFO("config generation: %ld", config_generation);
|
||||
}
|
||||
|
||||
/* Load config. */
|
||||
cr_config_t cfg;
|
||||
if (pg_conn) {
|
||||
if (pg_config_load(&cfg) != 0) {
|
||||
DEBUG_ERROR("failed to load config from PostgreSQL");
|
||||
pg_inbox_shutdown();
|
||||
return 1;
|
||||
}
|
||||
} else {
|
||||
if (cr_config_load(&cfg, config_path) != 0) return 1;
|
||||
}
|
||||
|
||||
/* --restart: reset state to first-time startup. */
|
||||
if (restart) {
|
||||
DEBUG_INFO("RESTART: resetting state to first-time startup");
|
||||
cfg.state.backfilled_until = 0;
|
||||
cfg.state.current_window_index = 0;
|
||||
cfg.state.backfill_cursor = 0;
|
||||
if (pg_conn) {
|
||||
/* Clear per-author backfill progress so the next run starts fresh. */
|
||||
pg_inbox_reset_backfill_progress();
|
||||
} else {
|
||||
cr_config_save_state(&cfg);
|
||||
}
|
||||
}
|
||||
|
||||
/* Init crypto. */
|
||||
if (nostr_crypto_init() != NOSTR_SUCCESS) {
|
||||
DEBUG_ERROR("nostr_crypto_init failed");
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* In-memory state. */
|
||||
cr_seen_ring_t seen;
|
||||
cr_seen_ring_init(&seen);
|
||||
cr_pubkey_set_t followed;
|
||||
cr_pubkey_set_init(&followed);
|
||||
|
||||
/* Upstream pool. */
|
||||
nostr_relay_pool_t *upstream = nostr_relay_pool_create(nostr_pool_reconnect_config_default());
|
||||
if (!upstream) {
|
||||
DEBUG_ERROR("failed to create upstream pool");
|
||||
return 1;
|
||||
}
|
||||
for (int i = 0; i < cfg.upstream_count; i++) {
|
||||
if (nostr_relay_pool_add_relay(upstream, cfg.upstream_relays[i]) != NOSTR_SUCCESS) {
|
||||
DEBUG_WARN("failed to add upstream relay %s (continuing)", cfg.upstream_relays[i]);
|
||||
} else {
|
||||
DEBUG_INFO("upstream: added %s", cfg.upstream_relays[i]);
|
||||
}
|
||||
}
|
||||
|
||||
/* Sink: WebSocket pool (legacy) or PostgreSQL inbox. */
|
||||
cr_sink_t sink;
|
||||
if (pg_conn) {
|
||||
if (cr_sink_init_pg(&sink, &seen) != 0) {
|
||||
DEBUG_ERROR("failed to init sink (pg)");
|
||||
return 1;
|
||||
}
|
||||
} else {
|
||||
if (cr_sink_init(&sink, cfg.local_relay, &seen) != 0) {
|
||||
DEBUG_ERROR("failed to init sink");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
/* Give the sink pool a moment to connect to the local relay before we
|
||||
* start publishing (relay discovery caches kind-10002 events to local).
|
||||
* No-op in PostgreSQL inbox mode (cr_sink_pump does nothing). */
|
||||
if (!pg_conn) {
|
||||
DEBUG_INFO("waiting for sink connection to establish...");
|
||||
for (int i = 0; i < 30 && !g_shutdown; i++) {
|
||||
cr_sink_pump(&sink, 100);
|
||||
}
|
||||
}
|
||||
|
||||
/* Resolve follow graph. */
|
||||
if (g_shutdown) goto shutdown;
|
||||
if (cr_follow_resolve(&cfg, upstream, &followed) != 0) {
|
||||
DEBUG_ERROR("follow graph resolution failed");
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (g_shutdown) goto shutdown;
|
||||
|
||||
/* Discover outbox relays (NIP-65) and compute minimum covering set. */
|
||||
int is_first_time = (cfg.state.backfilled_until == 0);
|
||||
cr_relay_map_t relay_map;
|
||||
if (cr_relay_discovery_run(&relay_map, &cfg, upstream, &sink, &followed,
|
||||
is_first_time) != 0) {
|
||||
DEBUG_WARN("relay discovery failed, continuing with bootstrap relays only");
|
||||
memset(&relay_map, 0, sizeof(relay_map));
|
||||
}
|
||||
|
||||
/* Add discovered outbox relays to the upstream pool. */
|
||||
for (int i = 0; i < relay_map.selected_count; i++) {
|
||||
/* Check if already in the pool (bootstrap relays may already be there). */
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
int already = 0;
|
||||
for (int j = 0; j < n; j++) {
|
||||
if (strcmp(listed[j], relay_map.selected_relays[i]) == 0) {
|
||||
already = 1; break;
|
||||
}
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
if (!already) {
|
||||
if (nostr_relay_pool_add_relay(upstream, relay_map.selected_relays[i]) == NOSTR_SUCCESS) {
|
||||
DEBUG_INFO("upstream: added outbox relay %s", relay_map.selected_relays[i]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Log final upstream pool. */
|
||||
{
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
DEBUG_INFO("upstream_pool: %d relays connected:", n);
|
||||
for (int j = 0; j < n; j++) {
|
||||
DEBUG_INFO(" %s", listed[j]);
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
}
|
||||
|
||||
/* Open live subscription. */
|
||||
cr_live_t live;
|
||||
if (cfg.live.enabled) {
|
||||
if (cr_live_open(&live, &cfg, upstream, &followed, &sink) != 0) {
|
||||
DEBUG_WARN("live subscription failed to open (will retry on resubscribe)");
|
||||
}
|
||||
} else {
|
||||
memset(&live, 0, sizeof(live));
|
||||
}
|
||||
|
||||
/* Init backfill. */
|
||||
cr_backfill_t bf;
|
||||
cr_backfill_init(&bf, &cfg);
|
||||
|
||||
time_t last_follow_refresh = time(NULL);
|
||||
time_t last_state_save = time(NULL);
|
||||
time_t last_status_heartbeat = 0;
|
||||
|
||||
/* Initial status heartbeat in PostgreSQL mode. */
|
||||
if (pg_conn) {
|
||||
pg_inbox_update_status("starting", config_generation, (long)time(NULL),
|
||||
followed.count, relay_map.selected_count, 0,
|
||||
cfg.state.current_window_index,
|
||||
cfg.state.backfill_cursor, 0, 0, NULL, 0);
|
||||
}
|
||||
|
||||
DEBUG_INFO("entering main loop");
|
||||
|
||||
while (!g_shutdown) {
|
||||
if (g_reload) {
|
||||
g_reload = 0;
|
||||
DEBUG_INFO("SIGHUP: reloading config (state preserved)");
|
||||
cr_config_t newcfg;
|
||||
int reload_ok = 0;
|
||||
if (pg_conn) {
|
||||
if (pg_config_load(&newcfg) == 0) reload_ok = 1;
|
||||
} else if (config_path) {
|
||||
if (cr_config_load(&newcfg, config_path) == 0) reload_ok = 1;
|
||||
}
|
||||
if (reload_ok) {
|
||||
/* Preserve runtime state across reload. */
|
||||
newcfg.state = cfg.state;
|
||||
cr_config_free(&cfg);
|
||||
cfg = newcfg;
|
||||
DEBUG_INFO("config reloaded");
|
||||
} else {
|
||||
DEBUG_ERROR("config reload failed, keeping old config");
|
||||
}
|
||||
}
|
||||
|
||||
/* PostgreSQL: check for config generation change and reload. */
|
||||
if (pg_conn) {
|
||||
int changed = pg_config_generation_changed(config_generation);
|
||||
if (changed == 1) {
|
||||
long new_gen = pg_inbox_get_config_generation();
|
||||
DEBUG_INFO("config generation changed (%ld -> %ld), reloading",
|
||||
config_generation, new_gen);
|
||||
cr_config_t newcfg;
|
||||
if (pg_config_load(&newcfg) == 0) {
|
||||
newcfg.state = cfg.state;
|
||||
cr_config_free(&cfg);
|
||||
cfg = newcfg;
|
||||
config_generation = new_gen;
|
||||
DEBUG_INFO("config reloaded from PostgreSQL");
|
||||
/* Resubscribe live with new config. */
|
||||
if (cfg.live.enabled) {
|
||||
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
|
||||
}
|
||||
} else {
|
||||
DEBUG_ERROR("PostgreSQL config reload failed, keeping old config");
|
||||
}
|
||||
} else if (changed < 0) {
|
||||
DEBUG_WARN("config generation check failed");
|
||||
}
|
||||
}
|
||||
|
||||
/* Pump upstream pool (drives live subscription callbacks). */
|
||||
nostr_relay_pool_run(upstream, 100);
|
||||
|
||||
/* Pump sink pool (flush publish callbacks). */
|
||||
cr_sink_pump(&sink, 50);
|
||||
|
||||
/* Backfill tick. */
|
||||
int brc = cr_backfill_tick(&bf, &cfg, upstream, &followed, &sink, &relay_map);
|
||||
(void)brc;
|
||||
|
||||
/* Periodic follow-graph refresh. */
|
||||
time_t now = time(NULL);
|
||||
if (cfg.follow_graph_refresh_seconds > 0 &&
|
||||
(now - last_follow_refresh) >= cfg.follow_graph_refresh_seconds) {
|
||||
DEBUG_INFO("refreshing follow graph");
|
||||
cr_pubkey_set_t new_followed;
|
||||
cr_pubkey_set_init(&new_followed);
|
||||
if (cr_follow_resolve(&cfg, upstream, &new_followed) == 0) {
|
||||
/* If the set changed, resubscribe live. */
|
||||
int changed = (new_followed.count != followed.count);
|
||||
if (!changed) {
|
||||
for (int i = 0; i < followed.count; i++) {
|
||||
if (!cr_pubkey_set_contains(&new_followed, followed.items[i])) {
|
||||
changed = 1; break;
|
||||
}
|
||||
}
|
||||
}
|
||||
cr_pubkey_set_free(&followed);
|
||||
followed = new_followed;
|
||||
if (changed && cfg.live.enabled) {
|
||||
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
|
||||
}
|
||||
} else {
|
||||
DEBUG_WARN("follow graph refresh failed");
|
||||
cr_pubkey_set_free(&new_followed);
|
||||
}
|
||||
last_follow_refresh = now;
|
||||
}
|
||||
|
||||
/* Periodic live resubscribe. */
|
||||
if (cfg.live.enabled && cfg.live.resubscribe_interval_seconds > 0 &&
|
||||
(now - live.last_resubscribe) >= cfg.live.resubscribe_interval_seconds) {
|
||||
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
|
||||
}
|
||||
|
||||
/* Periodic state save (in case backfill didn't just save). */
|
||||
if ((now - last_state_save) >= 60) {
|
||||
if (!pg_conn) cr_config_save_state(&cfg);
|
||||
last_state_save = now;
|
||||
}
|
||||
|
||||
/* PostgreSQL: periodic status heartbeat. */
|
||||
if (pg_conn && (now - last_status_heartbeat) >= 15) {
|
||||
long events_fetched = live.events_received + bf.events_total;
|
||||
long inbox_inserts = sink.published_ok;
|
||||
int connected = 0;
|
||||
{
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
|
||||
for (int j = 0; j < n; j++) {
|
||||
if (statuses[j] == NOSTR_POOL_RELAY_CONNECTED) connected++;
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
}
|
||||
pg_inbox_update_status("running", config_generation, (long)now,
|
||||
followed.count, relay_map.selected_count,
|
||||
connected, cfg.state.current_window_index,
|
||||
cfg.state.backfill_cursor,
|
||||
events_fetched, inbox_inserts, NULL, 0);
|
||||
last_status_heartbeat = now;
|
||||
}
|
||||
}
|
||||
|
||||
/* Graceful shutdown. */
|
||||
shutdown:
|
||||
DEBUG_INFO("shutting down...");
|
||||
cr_live_close(&live);
|
||||
if (!pg_conn) cr_config_save_state(&cfg);
|
||||
if (pg_conn) {
|
||||
pg_inbox_update_status("stopped", config_generation, (long)time(NULL),
|
||||
followed.count, relay_map.selected_count, 0,
|
||||
cfg.state.current_window_index,
|
||||
cfg.state.backfill_cursor,
|
||||
live.events_received + bf.events_total,
|
||||
sink.published_ok, NULL, 0);
|
||||
}
|
||||
cr_sink_destroy(&sink);
|
||||
nostr_relay_pool_destroy(upstream);
|
||||
cr_pubkey_set_free(&followed);
|
||||
cr_relay_map_free(&relay_map);
|
||||
cr_config_free(&cfg);
|
||||
if (pg_conn) pg_inbox_shutdown();
|
||||
nostr_crypto_cleanup();
|
||||
DEBUG_INFO("clean exit");
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
/*
|
||||
* caching_relay - main header
|
||||
*
|
||||
* Version information is auto-updated by increment_and_push.sh.
|
||||
* Git tags are the source of truth for versioning.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_MAIN_H
|
||||
#define CACHING_RELAY_MAIN_H
|
||||
|
||||
// Version information (auto-updated by increment_and_push.sh)
|
||||
#define CR_VERSION_MAJOR 0
|
||||
#define CR_VERSION_MINOR 0
|
||||
#define CR_VERSION_PATCH 2
|
||||
#define CR_VERSION "v0.0.2"
|
||||
|
||||
#endif /* CACHING_RELAY_MAIN_H */
|
||||
@@ -0,0 +1,236 @@
|
||||
/*
|
||||
* caching_relay - read caching configuration from the c-relay-pg PostgreSQL
|
||||
* config table and populate the existing cr_config_t structure.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "pg_config.h"
|
||||
#include "pg_inbox.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Helpers */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/* Split a comma-separated string into a string array (fixed-size rows).
|
||||
* Trims leading/trailing whitespace from each token. */
|
||||
static void split_csv_str(const char *s, char *dst, int max, int elem_len,
|
||||
int *count_out) {
|
||||
int count = 0;
|
||||
if (!s) { *count_out = 0; return; }
|
||||
|
||||
/* Work on a mutable copy because strtok_r modifies its input. */
|
||||
char *copy = strdup(s);
|
||||
if (!copy) { *count_out = 0; return; }
|
||||
|
||||
char *saveptr = NULL;
|
||||
char *tok = strtok_r(copy, ",", &saveptr);
|
||||
while (tok && count < max) {
|
||||
/* Trim leading spaces. */
|
||||
while (*tok == ' ' || *tok == '\t') tok++;
|
||||
/* Trim trailing spaces. */
|
||||
char *end = tok + strlen(tok) - 1;
|
||||
while (end > tok && (*end == ' ' || *end == '\t' || *end == '\r' || *end == '\n')) {
|
||||
*end-- = '\0';
|
||||
}
|
||||
if (*tok != '\0') {
|
||||
strncpy(dst + count * elem_len, tok, elem_len - 1);
|
||||
dst[count * elem_len + (elem_len - 1)] = '\0';
|
||||
count++;
|
||||
}
|
||||
tok = strtok_r(NULL, ",", &saveptr);
|
||||
}
|
||||
free(copy);
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
/* Split a comma-separated string of integers into an int array. */
|
||||
static void split_csv_int(const char *s, int *dst, int max, int *count_out) {
|
||||
int count = 0;
|
||||
if (!s) { *count_out = 0; return; }
|
||||
|
||||
char *copy = strdup(s);
|
||||
if (!copy) { *count_out = 0; return; }
|
||||
|
||||
char *saveptr = NULL;
|
||||
char *tok = strtok_r(copy, ",", &saveptr);
|
||||
while (tok && count < max) {
|
||||
while (*tok == ' ' || *tok == '\t') tok++;
|
||||
if (*tok != '\0') {
|
||||
dst[count++] = atoi(tok);
|
||||
}
|
||||
tok = strtok_r(NULL, ",", &saveptr);
|
||||
}
|
||||
free(copy);
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
/* Split a comma-separated string of longs into a long array. */
|
||||
static void split_csv_long(const char *s, long *dst, int max, int *count_out) {
|
||||
int count = 0;
|
||||
if (!s) { *count_out = 0; return; }
|
||||
|
||||
char *copy = strdup(s);
|
||||
if (!copy) { *count_out = 0; return; }
|
||||
|
||||
char *saveptr = NULL;
|
||||
char *tok = strtok_r(copy, ",", &saveptr);
|
||||
while (tok && count < max) {
|
||||
while (*tok == ' ' || *tok == '\t') tok++;
|
||||
if (*tok != '\0') {
|
||||
dst[count++] = strtol(tok, NULL, 10);
|
||||
}
|
||||
tok = strtok_r(NULL, ",", &saveptr);
|
||||
}
|
||||
free(copy);
|
||||
*count_out = count;
|
||||
}
|
||||
|
||||
/* Read a config key as a freshly-allocated string. Returns NULL if missing. */
|
||||
static char *get_key(const char *key) {
|
||||
return pg_inbox_get_config_value(key);
|
||||
}
|
||||
|
||||
/* Read a config key as a boolean ("true"/"1"/"yes" -> 1). */
|
||||
static int get_bool(const char *key, int default_val) {
|
||||
char *v = get_key(key);
|
||||
if (!v) return default_val;
|
||||
int r = (strcmp(v, "true") == 0 || strcmp(v, "1") == 0 ||
|
||||
strcmp(v, "yes") == 0 || strcmp(v, "on") == 0);
|
||||
free(v);
|
||||
return r;
|
||||
}
|
||||
|
||||
/* Read a config key as a long. */
|
||||
static long get_long(const char *key, long default_val) {
|
||||
char *v = get_key(key);
|
||||
if (!v) return default_val;
|
||||
long r = strtol(v, NULL, 10);
|
||||
free(v);
|
||||
return r;
|
||||
}
|
||||
|
||||
/* Read a config key as an int. */
|
||||
static int get_int(const char *key, int default_val) {
|
||||
return (int)get_long(key, (long)default_val);
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Public API */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int pg_config_load(cr_config_t *cfg) {
|
||||
if (!cfg) return -1;
|
||||
memset(cfg, 0, sizeof(*cfg));
|
||||
|
||||
/* root_npubs */
|
||||
char *npubs = get_key("caching_root_npubs");
|
||||
split_csv_str(npubs, (char *)cfg->root_npubs, CR_MAX_ROOT_NPUBS,
|
||||
CR_NPUB_LEN, &cfg->root_npub_count);
|
||||
free(npubs);
|
||||
|
||||
/* bootstrap_relays -> upstream_relays */
|
||||
char *relays = get_key("caching_bootstrap_relays");
|
||||
split_csv_str(relays, (char *)cfg->upstream_relays, CR_MAX_UPSTREAM,
|
||||
CR_URL_LEN, &cfg->upstream_count);
|
||||
free(relays);
|
||||
|
||||
/* local_relay - no longer required for PostgreSQL mode, but keep a
|
||||
* placeholder for compatibility. The caller may override it. */
|
||||
cfg->local_relay[0] = '\0';
|
||||
|
||||
/* kinds */
|
||||
char *kinds = get_key("caching_kinds");
|
||||
split_csv_int(kinds, cfg->kinds, CR_MAX_KINDS, &cfg->kind_count);
|
||||
free(kinds);
|
||||
|
||||
/* admin_kinds - supports "*" to mean all kinds */
|
||||
char *akinds = get_key("caching_admin_kinds");
|
||||
if (akinds) {
|
||||
/* Check for "*" anywhere in the list. */
|
||||
if (strstr(akinds, "*") != NULL) {
|
||||
cfg->admin_all_kinds = 1;
|
||||
} else {
|
||||
split_csv_int(akinds, cfg->admin_kinds, CR_MAX_KINDS,
|
||||
&cfg->admin_kind_count);
|
||||
}
|
||||
free(akinds);
|
||||
}
|
||||
|
||||
/* live */
|
||||
cfg->live.enabled = get_bool("caching_live_enabled", 1);
|
||||
cfg->live.resubscribe_interval_seconds =
|
||||
get_int("caching_live_resubscribe_seconds", 300);
|
||||
|
||||
/* backfill */
|
||||
cfg->backfill.enabled = get_bool("caching_backfill_enabled", 1);
|
||||
char *windows = get_key("caching_backfill_windows");
|
||||
split_csv_long(windows, cfg->backfill.window_schedule_seconds,
|
||||
CR_MAX_WINDOWS, &cfg->backfill.window_count);
|
||||
free(windows);
|
||||
cfg->backfill.events_per_tick =
|
||||
get_int("caching_backfill_page_size", 50);
|
||||
/* tick interval is stored in ms in the config table; convert to seconds. */
|
||||
int tick_ms = get_int("caching_backfill_tick_interval_ms", 5000);
|
||||
cfg->backfill.tick_interval_seconds = (tick_ms + 999) / 1000;
|
||||
if (cfg->backfill.tick_interval_seconds < 1)
|
||||
cfg->backfill.tick_interval_seconds = 1;
|
||||
cfg->backfill.window_cooldown_seconds =
|
||||
get_int("caching_backfill_window_cooldown_seconds", 60);
|
||||
|
||||
/* follow graph refresh */
|
||||
cfg->follow_graph_refresh_seconds =
|
||||
get_int("caching_follow_graph_refresh_seconds", 600);
|
||||
|
||||
/* Defaults if missing (mirror cr_config_load). */
|
||||
if (cfg->backfill.window_count == 0) {
|
||||
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
|
||||
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
|
||||
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
|
||||
for (int i = 0; i < n; i++)
|
||||
cfg->backfill.window_schedule_seconds[i] = def_windows[i];
|
||||
cfg->backfill.window_count = n;
|
||||
}
|
||||
if (cfg->backfill.events_per_tick == 0)
|
||||
cfg->backfill.events_per_tick = 50;
|
||||
if (cfg->backfill.tick_interval_seconds == 0)
|
||||
cfg->backfill.tick_interval_seconds = 5;
|
||||
if (cfg->backfill.window_cooldown_seconds == 0)
|
||||
cfg->backfill.window_cooldown_seconds = 60;
|
||||
if (cfg->live.resubscribe_interval_seconds == 0)
|
||||
cfg->live.resubscribe_interval_seconds = 300;
|
||||
if (cfg->follow_graph_refresh_seconds == 0)
|
||||
cfg->follow_graph_refresh_seconds = 600;
|
||||
|
||||
/* State: initialize to defaults for this phase. Phase 7 will read
|
||||
* caching_backfill_progress / caching_service_state. */
|
||||
cfg->state.backfilled_until = 0;
|
||||
cfg->state.current_window_index = 0;
|
||||
cfg->state.backfill_cursor = 0;
|
||||
|
||||
/* Validation. */
|
||||
if (cfg->root_npub_count == 0) {
|
||||
DEBUG_ERROR("pg_config: at least one caching_root_npubs required");
|
||||
return -1;
|
||||
}
|
||||
if (cfg->upstream_count == 0) {
|
||||
DEBUG_ERROR("pg_config: at least one caching_bootstrap_relays required");
|
||||
return -1;
|
||||
}
|
||||
if (cfg->kind_count == 0) {
|
||||
DEBUG_ERROR("pg_config: at least one caching_kinds required");
|
||||
return -1;
|
||||
}
|
||||
|
||||
DEBUG_INFO("pg_config loaded: %d root npubs, %d upstream relays, %d kinds",
|
||||
cfg->root_npub_count, cfg->upstream_count, cfg->kind_count);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int pg_config_generation_changed(long last_generation) {
|
||||
long cur = pg_inbox_get_config_generation();
|
||||
if (cur < 0) return -1;
|
||||
return (cur != last_generation) ? 1 : 0;
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
/*
|
||||
* caching_relay - read caching configuration from the c-relay-pg PostgreSQL
|
||||
* config table and populate the existing cr_config_t structure.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_PG_CONFIG_H
|
||||
#define CACHING_RELAY_PG_CONFIG_H
|
||||
|
||||
#include "config.h" /* cr_config_t */
|
||||
|
||||
/* Load caching configuration from the c-relay-pg PostgreSQL config table.
|
||||
* Populates the cr_config_t fields from the config table.
|
||||
* Returns 0 on success, -1 on error. */
|
||||
int pg_config_load(cr_config_t *cfg);
|
||||
|
||||
/* Check if the config generation has changed since the last load.
|
||||
* Returns 1 if changed, 0 if not, -1 on error. */
|
||||
int pg_config_generation_changed(long last_generation);
|
||||
|
||||
#endif /* CACHING_RELAY_PG_CONFIG_H */
|
||||
@@ -0,0 +1,488 @@
|
||||
/*
|
||||
* caching_relay - PostgreSQL inbox connection and operations.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "pg_inbox.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <libpq-fe.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#include "../nostr_core_lib/cjson/cJSON.h"
|
||||
|
||||
static PGconn *g_pg = NULL;
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int pg_inbox_init(const char *connection_string) {
|
||||
if (g_pg) {
|
||||
PQfinish(g_pg);
|
||||
g_pg = NULL;
|
||||
}
|
||||
g_pg = PQconnectdb(connection_string);
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: PQconnectdb returned NULL (out of memory)");
|
||||
return -1;
|
||||
}
|
||||
if (PQstatus(g_pg) != CONNECTION_OK) {
|
||||
DEBUG_ERROR("pg_inbox: connection failed: %s",
|
||||
PQerrorMessage(g_pg));
|
||||
PQfinish(g_pg);
|
||||
g_pg = NULL;
|
||||
return -1;
|
||||
}
|
||||
DEBUG_INFO("pg_inbox: connected to PostgreSQL (%s)",
|
||||
PQdb(g_pg) ? PQdb(g_pg) : "?");
|
||||
return 0;
|
||||
}
|
||||
|
||||
void pg_inbox_shutdown(void) {
|
||||
if (g_pg) {
|
||||
PQfinish(g_pg);
|
||||
g_pg = NULL;
|
||||
DEBUG_INFO("pg_inbox: connection closed");
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Inbox insert */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int pg_inbox_insert_event(const char *event_json, const char *source_relay,
|
||||
const char *source_class, int priority) {
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return -1;
|
||||
}
|
||||
if (!event_json) {
|
||||
DEBUG_WARN("pg_inbox: insert called with NULL event_json");
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* Extract the event id from the JSON for the event_id column.
|
||||
* The schema CHECK requires event_id = event_json->>'id'. */
|
||||
cJSON *ev = cJSON_Parse(event_json);
|
||||
if (!ev) {
|
||||
DEBUG_WARN("pg_inbox: event_json is not valid JSON, skipping");
|
||||
return -1;
|
||||
}
|
||||
cJSON *id_node = cJSON_GetObjectItem(ev, "id");
|
||||
const char *event_id = NULL;
|
||||
if (id_node && cJSON_IsString(id_node)) {
|
||||
event_id = cJSON_GetStringValue(id_node);
|
||||
}
|
||||
if (!event_id || strlen(event_id) != 64) {
|
||||
DEBUG_WARN("pg_inbox: event missing valid 64-char id, skipping");
|
||||
cJSON_Delete(ev);
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* priority is an integer; pass as a separate param via a small buffer. */
|
||||
char prio_buf[16];
|
||||
snprintf(prio_buf, sizeof(prio_buf), "%d", priority);
|
||||
|
||||
const char *vals[5];
|
||||
int lens[5];
|
||||
int fmts[5];
|
||||
|
||||
/* $1: event_id (text) */
|
||||
vals[0] = event_id;
|
||||
lens[0] = (int)strlen(event_id);
|
||||
fmts[0] = 0;
|
||||
|
||||
/* $2: event_json (cast to jsonb in SQL) */
|
||||
vals[1] = event_json;
|
||||
lens[1] = (int)strlen(event_json);
|
||||
fmts[1] = 0;
|
||||
|
||||
/* $3: source_relay (text, may be NULL) */
|
||||
vals[2] = source_relay;
|
||||
lens[2] = source_relay ? (int)strlen(source_relay) : 0;
|
||||
fmts[2] = 0;
|
||||
|
||||
/* $4: source_class (text) */
|
||||
vals[3] = source_class ? source_class : "backfill";
|
||||
lens[3] = (int)strlen(vals[3]);
|
||||
fmts[3] = 0;
|
||||
|
||||
/* $5: priority (smallint, passed as text) */
|
||||
vals[4] = prio_buf;
|
||||
lens[4] = (int)strlen(prio_buf);
|
||||
fmts[4] = 0;
|
||||
|
||||
const char *sql =
|
||||
"INSERT INTO caching_event_inbox "
|
||||
"(event_id, event_json, source_relay, source_class, priority) "
|
||||
"VALUES ($1, $2::jsonb, $3, $4, $5) "
|
||||
"ON CONFLICT (event_id) DO NOTHING";
|
||||
|
||||
PGresult *res = PQexecParams(g_pg, sql, 5, NULL, vals, lens, fmts, 0);
|
||||
cJSON_Delete(ev);
|
||||
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: insert returned NULL result for %s", event_id);
|
||||
return -1;
|
||||
}
|
||||
ExecStatusType st = PQresultStatus(res);
|
||||
if (st != PGRES_COMMAND_OK) {
|
||||
DEBUG_WARN("pg_inbox: insert failed for %s: %s",
|
||||
event_id, PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return -1;
|
||||
}
|
||||
/* ON CONFLICT DO NOTHING returns COMMAND_OK either way; the row count
|
||||
* is not directly available without RETURNING, so we treat both
|
||||
* inserted and conflict-skipped as success (0). */
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Service state update */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int pg_inbox_update_status(const char *service_state,
|
||||
long config_generation,
|
||||
long heartbeat_at,
|
||||
int followed_author_count,
|
||||
int selected_relay_count,
|
||||
int connected_relay_count,
|
||||
int current_window_index,
|
||||
int backfill_cursor,
|
||||
long events_fetched,
|
||||
long inbox_inserts,
|
||||
const char *last_error,
|
||||
long last_error_at) {
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return -1;
|
||||
}
|
||||
|
||||
char gen_buf[32], hb_buf[32], fac_buf[16], src_buf[16], crc_buf[16];
|
||||
char cwi_buf[16], bc_buf[16], ef_buf[32], ii_buf[32], lea_buf[32];
|
||||
|
||||
snprintf(gen_buf, sizeof(gen_buf), "%ld", config_generation);
|
||||
snprintf(hb_buf, sizeof(hb_buf), "%ld", heartbeat_at);
|
||||
snprintf(fac_buf, sizeof(fac_buf), "%d", followed_author_count);
|
||||
snprintf(src_buf, sizeof(src_buf), "%d", selected_relay_count);
|
||||
snprintf(crc_buf, sizeof(crc_buf), "%d", connected_relay_count);
|
||||
snprintf(cwi_buf, sizeof(cwi_buf), "%d", current_window_index);
|
||||
snprintf(bc_buf, sizeof(bc_buf), "%d", backfill_cursor);
|
||||
snprintf(ef_buf, sizeof(ef_buf), "%ld", events_fetched);
|
||||
snprintf(ii_buf, sizeof(ii_buf), "%ld", inbox_inserts);
|
||||
if (last_error_at > 0) {
|
||||
snprintf(lea_buf, sizeof(lea_buf), "%ld", last_error_at);
|
||||
} else {
|
||||
lea_buf[0] = '\0';
|
||||
}
|
||||
|
||||
const char *vals[12];
|
||||
int lens[12];
|
||||
int fmts[12];
|
||||
int n = 0;
|
||||
|
||||
/* $1 service_state (may be NULL) */
|
||||
vals[n] = service_state; lens[n] = service_state ? (int)strlen(service_state) : 0; fmts[n] = 0; n++;
|
||||
/* $2 config_generation */
|
||||
vals[n] = gen_buf; lens[n] = (int)strlen(gen_buf); fmts[n] = 0; n++;
|
||||
/* $3 heartbeat_at */
|
||||
vals[n] = hb_buf; lens[n] = (int)strlen(hb_buf); fmts[n] = 0; n++;
|
||||
/* $4 followed_author_count */
|
||||
vals[n] = fac_buf; lens[n] = (int)strlen(fac_buf); fmts[n] = 0; n++;
|
||||
/* $5 selected_relay_count */
|
||||
vals[n] = src_buf; lens[n] = (int)strlen(src_buf); fmts[n] = 0; n++;
|
||||
/* $6 connected_relay_count */
|
||||
vals[n] = crc_buf; lens[n] = (int)strlen(crc_buf); fmts[n] = 0; n++;
|
||||
/* $7 current_window_index */
|
||||
vals[n] = cwi_buf; lens[n] = (int)strlen(cwi_buf); fmts[n] = 0; n++;
|
||||
/* $8 backfill_cursor */
|
||||
vals[n] = bc_buf; lens[n] = (int)strlen(bc_buf); fmts[n] = 0; n++;
|
||||
/* $9 events_fetched */
|
||||
vals[n] = ef_buf; lens[n] = (int)strlen(ef_buf); fmts[n] = 0; n++;
|
||||
/* $10 inbox_inserts */
|
||||
vals[n] = ii_buf; lens[n] = (int)strlen(ii_buf); fmts[n] = 0; n++;
|
||||
/* $11 last_error (may be NULL) */
|
||||
vals[n] = last_error; lens[n] = last_error ? (int)strlen(last_error) : 0; fmts[n] = 0; n++;
|
||||
/* $12 last_error_at (empty string -> NULL) */
|
||||
vals[n] = lea_buf; lens[n] = (int)strlen(lea_buf); fmts[n] = 0; n++;
|
||||
|
||||
const char *sql =
|
||||
"INSERT INTO caching_service_state "
|
||||
"(id, service_state, config_generation, heartbeat_at, "
|
||||
" followed_author_count, selected_relay_count, connected_relay_count, "
|
||||
" current_window_index, backfill_cursor, events_fetched, inbox_inserts, "
|
||||
" last_error, last_error_at, updated_at) "
|
||||
"VALUES (1, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, "
|
||||
" CASE WHEN $12 = '' THEN NULL ELSE $12::BIGINT END, "
|
||||
" EXTRACT(EPOCH FROM NOW())::BIGINT) "
|
||||
"ON CONFLICT (id) DO UPDATE SET "
|
||||
" service_state = COALESCE(EXCLUDED.service_state, caching_service_state.service_state), "
|
||||
" config_generation = EXCLUDED.config_generation, "
|
||||
" heartbeat_at = EXCLUDED.heartbeat_at, "
|
||||
" followed_author_count= EXCLUDED.followed_author_count, "
|
||||
" selected_relay_count = EXCLUDED.selected_relay_count, "
|
||||
" connected_relay_count= EXCLUDED.connected_relay_count, "
|
||||
" current_window_index = EXCLUDED.current_window_index, "
|
||||
" backfill_cursor = EXCLUDED.backfill_cursor, "
|
||||
" events_fetched = EXCLUDED.events_fetched, "
|
||||
" inbox_inserts = EXCLUDED.inbox_inserts, "
|
||||
" last_error = COALESCE(EXCLUDED.last_error, caching_service_state.last_error), "
|
||||
" last_error_at = CASE WHEN EXCLUDED.last_error_at IS NULL "
|
||||
" THEN caching_service_state.last_error_at "
|
||||
" ELSE EXCLUDED.last_error_at END, "
|
||||
" updated_at = EXCLUDED.updated_at";
|
||||
|
||||
PGresult *res = PQexecParams(g_pg, sql, n, NULL, vals, lens, fmts, 0);
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: update_status returned NULL result");
|
||||
return -1;
|
||||
}
|
||||
ExecStatusType st = PQresultStatus(res);
|
||||
if (st != PGRES_COMMAND_OK) {
|
||||
DEBUG_ERROR("pg_inbox: update_status failed: %s",
|
||||
PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return -1;
|
||||
}
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Config reads */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
char *pg_inbox_get_config_value(const char *key) {
|
||||
if (!g_pg || !key) return NULL;
|
||||
|
||||
const char *vals[1] = { key };
|
||||
int lens[1] = { (int)strlen(key) };
|
||||
int fmts[1] = { 0 };
|
||||
|
||||
PGresult *res = PQexecParams(g_pg,
|
||||
"SELECT value FROM config WHERE key = $1",
|
||||
1, NULL, vals, lens, fmts, 0);
|
||||
if (!res) return NULL;
|
||||
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
|
||||
DEBUG_WARN("pg_inbox: config select failed for '%s': %s",
|
||||
key, PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return NULL;
|
||||
}
|
||||
if (PQntuples(res) < 1 || PQgetisnull(res, 0, 0)) {
|
||||
PQclear(res);
|
||||
return NULL;
|
||||
}
|
||||
char *v = strdup(PQgetvalue(res, 0, 0));
|
||||
PQclear(res);
|
||||
return v;
|
||||
}
|
||||
|
||||
long pg_inbox_get_config_generation(void) {
|
||||
char *v = pg_inbox_get_config_value("caching_config_generation");
|
||||
if (!v) return -1;
|
||||
long gen = strtol(v, NULL, 10);
|
||||
free(v);
|
||||
return gen;
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Backfill progress persistence */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int pg_inbox_save_backfill_progress(const char *author_pubkey,
|
||||
int window_index,
|
||||
long window_anchor,
|
||||
long until_cursor,
|
||||
int complete) {
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return -1;
|
||||
}
|
||||
if (!author_pubkey) {
|
||||
DEBUG_WARN("pg_inbox: save_backfill_progress called with NULL author");
|
||||
return -1;
|
||||
}
|
||||
|
||||
char wi_buf[16], wa_buf[32], uc_buf[32], cmp_buf[8];
|
||||
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
|
||||
snprintf(wa_buf, sizeof(wa_buf), "%ld", window_anchor);
|
||||
snprintf(uc_buf, sizeof(uc_buf), "%ld", until_cursor);
|
||||
snprintf(cmp_buf, sizeof(cmp_buf), "%d", complete ? 1 : 0);
|
||||
|
||||
const char *vals[5] = { author_pubkey, wi_buf, wa_buf, uc_buf, cmp_buf };
|
||||
int lens[5] = {
|
||||
(int)strlen(author_pubkey),
|
||||
(int)strlen(wi_buf),
|
||||
(int)strlen(wa_buf),
|
||||
(int)strlen(uc_buf),
|
||||
(int)strlen(cmp_buf),
|
||||
};
|
||||
int fmts[5] = { 0, 0, 0, 0, 0 };
|
||||
|
||||
const char *sql =
|
||||
"INSERT INTO caching_backfill_progress "
|
||||
"(author_pubkey, window_index, window_anchor, until_cursor, complete, updated_at) "
|
||||
"VALUES ($1, $2, $3, $4, $5::BOOLEAN, EXTRACT(EPOCH FROM NOW())::BIGINT) "
|
||||
"ON CONFLICT (author_pubkey, window_index) DO UPDATE SET "
|
||||
" until_cursor = EXCLUDED.until_cursor, "
|
||||
" complete = EXCLUDED.complete, "
|
||||
" updated_at = EXCLUDED.updated_at";
|
||||
|
||||
PGresult *res = PQexecParams(g_pg, sql, 5, NULL, vals, lens, fmts, 0);
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: save_backfill_progress returned NULL result");
|
||||
return -1;
|
||||
}
|
||||
ExecStatusType st = PQresultStatus(res);
|
||||
if (st != PGRES_COMMAND_OK) {
|
||||
DEBUG_ERROR("pg_inbox: save_backfill_progress failed: %s",
|
||||
PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return -1;
|
||||
}
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int pg_inbox_load_backfill_progress(const char *author_pubkey,
|
||||
int window_index,
|
||||
long *out_until_cursor,
|
||||
int *out_complete,
|
||||
int *out_found) {
|
||||
if (out_found) *out_found = 0;
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return -1;
|
||||
}
|
||||
if (!author_pubkey) return -1;
|
||||
|
||||
char wi_buf[16];
|
||||
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
|
||||
|
||||
const char *vals[2] = { author_pubkey, wi_buf };
|
||||
int lens[2] = { (int)strlen(author_pubkey), (int)strlen(wi_buf) };
|
||||
int fmts[2] = { 0, 0 };
|
||||
|
||||
const char *sql =
|
||||
"SELECT until_cursor, complete FROM caching_backfill_progress "
|
||||
"WHERE author_pubkey = $1 AND window_index = $2";
|
||||
|
||||
PGresult *res = PQexecParams(g_pg, sql, 2, NULL, vals, lens, fmts, 0);
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: load_backfill_progress returned NULL result");
|
||||
return -1;
|
||||
}
|
||||
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
|
||||
DEBUG_ERROR("pg_inbox: load_backfill_progress failed: %s",
|
||||
PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return -1;
|
||||
}
|
||||
if (PQntuples(res) < 1) {
|
||||
/* Not found - not an error. */
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (out_until_cursor) {
|
||||
if (PQgetisnull(res, 0, 0)) {
|
||||
*out_until_cursor = 0;
|
||||
} else {
|
||||
*out_until_cursor = strtol(PQgetvalue(res, 0, 0), NULL, 10);
|
||||
}
|
||||
}
|
||||
if (out_complete) {
|
||||
if (PQgetisnull(res, 0, 1)) {
|
||||
*out_complete = 0;
|
||||
} else {
|
||||
const char *v = PQgetvalue(res, 0, 1);
|
||||
/* boolean comes back as 't'/'f' or '1'/'0' depending on output mode. */
|
||||
*out_complete = (v && (v[0] == 't' || v[0] == 'T' || v[0] == '1')) ? 1 : 0;
|
||||
}
|
||||
}
|
||||
if (out_found) *out_found = 1;
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
cJSON *pg_inbox_load_incomplete_progress(int window_index) {
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return NULL;
|
||||
}
|
||||
|
||||
char wi_buf[16];
|
||||
snprintf(wi_buf, sizeof(wi_buf), "%d", window_index);
|
||||
|
||||
const char *vals[1] = { wi_buf };
|
||||
int lens[1] = { (int)strlen(wi_buf) };
|
||||
int fmts[1] = { 0 };
|
||||
|
||||
const char *sql =
|
||||
"SELECT author_pubkey, until_cursor, complete FROM caching_backfill_progress "
|
||||
"WHERE window_index = $1 AND complete = FALSE";
|
||||
|
||||
PGresult *res = PQexecParams(g_pg, sql, 1, NULL, vals, lens, fmts, 0);
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: load_incomplete_progress returned NULL result");
|
||||
return NULL;
|
||||
}
|
||||
if (PQresultStatus(res) != PGRES_TUPLES_OK) {
|
||||
DEBUG_ERROR("pg_inbox: load_incomplete_progress failed: %s",
|
||||
PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
int n = PQntuples(res);
|
||||
if (n <= 0) {
|
||||
PQclear(res);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
cJSON *arr = cJSON_CreateArray();
|
||||
if (!arr) {
|
||||
PQclear(res);
|
||||
return NULL;
|
||||
}
|
||||
for (int i = 0; i < n; i++) {
|
||||
cJSON *obj = cJSON_CreateObject();
|
||||
if (!obj) continue;
|
||||
const char *pk = PQgetisnull(res, i, 0) ? "" : PQgetvalue(res, i, 0);
|
||||
long uc = PQgetisnull(res, i, 1) ? 0 : strtol(PQgetvalue(res, i, 1), NULL, 10);
|
||||
int cmp = 0;
|
||||
if (!PQgetisnull(res, i, 1)) {
|
||||
const char *v = PQgetvalue(res, i, 2);
|
||||
cmp = (v && (v[0] == 't' || v[0] == 'T' || v[0] == '1')) ? 1 : 0;
|
||||
}
|
||||
cJSON_AddStringToObject(obj, "author_pubkey", pk);
|
||||
cJSON_AddNumberToObject(obj, "until_cursor", (double)uc);
|
||||
cJSON_AddBoolToObject(obj, "complete", cmp ? 1 : 0);
|
||||
cJSON_AddItemToArray(arr, obj);
|
||||
}
|
||||
PQclear(res);
|
||||
return arr;
|
||||
}
|
||||
|
||||
int pg_inbox_reset_backfill_progress(void) {
|
||||
if (!g_pg) {
|
||||
DEBUG_ERROR("pg_inbox: not initialized");
|
||||
return -1;
|
||||
}
|
||||
PGresult *res = PQexec(g_pg, "DELETE FROM caching_backfill_progress");
|
||||
if (!res) {
|
||||
DEBUG_ERROR("pg_inbox: reset_backfill_progress returned NULL result");
|
||||
return -1;
|
||||
}
|
||||
ExecStatusType st = PQresultStatus(res);
|
||||
if (st != PGRES_COMMAND_OK) {
|
||||
DEBUG_ERROR("pg_inbox: reset_backfill_progress failed: %s",
|
||||
PQresultErrorMessage(res));
|
||||
PQclear(res);
|
||||
return -1;
|
||||
}
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
/*
|
||||
* caching_relay - PostgreSQL inbox connection and operations.
|
||||
*
|
||||
* Manages the libpq connection to the c-relay-pg PostgreSQL database and
|
||||
* provides functions to insert fetched events into the caching_event_inbox
|
||||
* table, update the caching_service_state singleton, and read configuration
|
||||
* from the shared config table.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_PG_INBOX_H
|
||||
#define CACHING_RELAY_PG_INBOX_H
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
/* Forward declaration so the header does not pull in cJSON everywhere. */
|
||||
typedef struct cJSON cJSON;
|
||||
|
||||
/* Initialize PostgreSQL connection using the given connection string.
|
||||
* Returns 0 on success, -1 on error. */
|
||||
int pg_inbox_init(const char *connection_string);
|
||||
|
||||
/* Close the PostgreSQL connection. */
|
||||
void pg_inbox_shutdown(void);
|
||||
|
||||
/* Insert an event into the caching_event_inbox table.
|
||||
* event_json is the raw Nostr event JSON string.
|
||||
* source_relay may be NULL. source_class is "live", "discovery", or "backfill".
|
||||
* priority is 0 (live/discovery) or 1 (backfill).
|
||||
* Returns 0 on success (including conflict-skip), -1 on error. */
|
||||
int pg_inbox_insert_event(const char *event_json, const char *source_relay,
|
||||
const char *source_class, int priority);
|
||||
|
||||
/* Update the caching_service_state singleton row.
|
||||
* All string parameters may be NULL (left unchanged).
|
||||
* Returns 0 on success, -1 on error. */
|
||||
int pg_inbox_update_status(const char *service_state,
|
||||
long config_generation,
|
||||
long heartbeat_at,
|
||||
int followed_author_count,
|
||||
int selected_relay_count,
|
||||
int connected_relay_count,
|
||||
int current_window_index,
|
||||
int backfill_cursor,
|
||||
long events_fetched,
|
||||
long inbox_inserts,
|
||||
const char *last_error,
|
||||
long last_error_at);
|
||||
|
||||
/* Read a config value from the c-relay-pg config table.
|
||||
* Returns a malloc'd string (caller must free) or NULL if not found/error. */
|
||||
char *pg_inbox_get_config_value(const char *key);
|
||||
|
||||
/* Read the current config generation.
|
||||
* Returns the generation number, or -1 on error. */
|
||||
long pg_inbox_get_config_generation(void);
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Backfill progress persistence */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
/* Save backfill progress for a specific author+window (UPSERT).
|
||||
* Returns 0 on success, -1 on error. */
|
||||
int pg_inbox_save_backfill_progress(const char *author_pubkey,
|
||||
int window_index,
|
||||
long window_anchor,
|
||||
long until_cursor,
|
||||
int complete);
|
||||
|
||||
/* Load backfill progress for a specific author+window.
|
||||
* Returns 0 on success (found), -1 on error.
|
||||
* Sets *out_until_cursor and *out_complete. If not found, sets *out_found = 0
|
||||
* and returns 0. */
|
||||
int pg_inbox_load_backfill_progress(const char *author_pubkey,
|
||||
int window_index,
|
||||
long *out_until_cursor,
|
||||
int *out_complete,
|
||||
int *out_found);
|
||||
|
||||
/* Load all incomplete backfill progress rows for a given window.
|
||||
* Returns a cJSON array of objects with "author_pubkey", "until_cursor" and
|
||||
* "complete" fields. Caller must cJSON_Delete the result.
|
||||
* Returns NULL on error or empty. */
|
||||
cJSON *pg_inbox_load_incomplete_progress(int window_index);
|
||||
|
||||
/* Reset all backfill progress (delete all rows).
|
||||
* Returns 0 on success, -1 on error. */
|
||||
int pg_inbox_reset_backfill_progress(void);
|
||||
|
||||
#endif /* CACHING_RELAY_PG_INBOX_H */
|
||||
@@ -0,0 +1,471 @@
|
||||
/*
|
||||
* caching_relay - NIP-65 outbox relay discovery
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "relay_discovery.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
#include <signal.h>
|
||||
|
||||
/* Access the shutdown flag from main.c so we can abort discovery early. */
|
||||
extern volatile sig_atomic_t g_shutdown;
|
||||
|
||||
/* Normalize a relay URL: strip trailing slash for consistency. */
|
||||
static void normalize_url(char *url, size_t maxlen) {
|
||||
size_t len = strlen(url);
|
||||
while (len > 0 && url[len - 1] == '/') {
|
||||
url[--len] = '\0';
|
||||
}
|
||||
(void)maxlen;
|
||||
}
|
||||
|
||||
/* ---- helpers ---- */
|
||||
|
||||
static int get_pool_urls(nostr_relay_pool_t *pool, const char ***urls_out) {
|
||||
char **listed = NULL;
|
||||
nostr_pool_relay_status_t *statuses = NULL;
|
||||
int n = nostr_relay_pool_list_relays(pool, &listed, &statuses);
|
||||
const char **urls = NULL;
|
||||
if (n > 0) {
|
||||
urls = malloc(n * sizeof(char *));
|
||||
for (int j = 0; j < n; j++) urls[j] = listed[j];
|
||||
}
|
||||
free(listed);
|
||||
free(statuses);
|
||||
*urls_out = urls;
|
||||
return n;
|
||||
}
|
||||
|
||||
/* Query the most recent kind 10002 for a single pubkey from a pool.
|
||||
* Returns the event (caller must cJSON_Delete) or NULL. */
|
||||
static cJSON *query_kind10002(nostr_relay_pool_t *pool, const char *hex) {
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(hex));
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(10002));
|
||||
cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber(1));
|
||||
|
||||
const char **urls = NULL;
|
||||
int n = get_pool_urls(pool, &urls);
|
||||
int ev_count = 0;
|
||||
cJSON **events = nostr_relay_pool_query_sync(pool, urls, n, filter, &ev_count, 10000);
|
||||
free(urls);
|
||||
cJSON_Delete(filter);
|
||||
|
||||
cJSON *result = NULL;
|
||||
if (events && ev_count > 0) {
|
||||
result = events[0]; /* take first (most recent) */
|
||||
for (int k = 1; k < ev_count; k++) cJSON_Delete(events[k]);
|
||||
}
|
||||
free(events);
|
||||
return result;
|
||||
}
|
||||
|
||||
/* Parse "r" tags from a kind 10002 event into an outbox entry.
|
||||
* Only includes relays with "read" marker or no marker (assume both).
|
||||
* Skips "write" only relays. */
|
||||
static int parse_r_tags(cJSON *event, cr_outbox_entry_t *entry) {
|
||||
memset(entry, 0, sizeof(*entry));
|
||||
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
|
||||
if (pubkey && cJSON_IsString(pubkey)) {
|
||||
strncpy(entry->pubkey, cJSON_GetStringValue(pubkey), CR_HEX_LEN - 1);
|
||||
}
|
||||
|
||||
cJSON *tags = cJSON_GetObjectItem(event, "tags");
|
||||
if (!tags || !cJSON_IsArray(tags)) return 0;
|
||||
|
||||
int added = 0;
|
||||
cJSON *tag;
|
||||
cJSON_ArrayForEach(tag, tags) {
|
||||
if (!cJSON_IsArray(tag)) continue;
|
||||
cJSON *name = cJSON_GetArrayItem(tag, 0);
|
||||
if (!name || !cJSON_IsString(name)) continue;
|
||||
if (strcmp(cJSON_GetStringValue(name), "r") != 0) continue;
|
||||
cJSON *url = cJSON_GetArrayItem(tag, 1);
|
||||
if (!url || !cJSON_IsString(url)) continue;
|
||||
const char *relay_url = cJSON_GetStringValue(url);
|
||||
if (!relay_url || !relay_url[0]) continue;
|
||||
|
||||
/* Check marker (3rd element): "read", "write", or absent. */
|
||||
cJSON *marker = cJSON_GetArrayItem(tag, 2);
|
||||
if (marker && cJSON_IsString(marker)) {
|
||||
const char *m = cJSON_GetStringValue(marker);
|
||||
if (strcmp(m, "write") == 0) continue; /* skip write-only relays */
|
||||
}
|
||||
/* No marker or "read" => include. */
|
||||
|
||||
if (entry->relay_count >= CR_MAX_RELAYS_PER_PUBKEY) break;
|
||||
strncpy(entry->relays[entry->relay_count], relay_url, CR_URL_LEN - 1);
|
||||
entry->relays[entry->relay_count][CR_URL_LEN - 1] = '\0';
|
||||
normalize_url(entry->relays[entry->relay_count], CR_URL_LEN);
|
||||
entry->relay_count++;
|
||||
added++;
|
||||
}
|
||||
return added;
|
||||
}
|
||||
|
||||
/* ---- greedy set cover ---- */
|
||||
|
||||
/* Build relay -> pubkey coverage map and compute minimum covering set. */
|
||||
static void compute_covering_set(cr_relay_map_t *map, cr_config_t *cfg,
|
||||
const cr_pubkey_set_t *followed) {
|
||||
/* Build a temporary relay -> pubkeys map. */
|
||||
typedef struct {
|
||||
char url[CR_URL_LEN];
|
||||
char pubkeys[CR_MAX_PUBKEYS_PER_RELAY][CR_HEX_LEN];
|
||||
int pubkey_count;
|
||||
} relay_cov_t;
|
||||
|
||||
relay_cov_t *relays = calloc(CR_MAX_DISCOVERED_RELAYS, sizeof(relay_cov_t));
|
||||
int relay_count = 0;
|
||||
|
||||
/* Always include bootstrap relays in the coverage map. */
|
||||
for (int i = 0; i < cfg->upstream_count && relay_count < CR_MAX_DISCOVERED_RELAYS; i++) {
|
||||
strncpy(relays[relay_count].url, cfg->upstream_relays[i], CR_URL_LEN - 1);
|
||||
relays[relay_count].url[CR_URL_LEN - 1] = '\0';
|
||||
normalize_url(relays[relay_count].url, CR_URL_LEN);
|
||||
relay_count++;
|
||||
}
|
||||
|
||||
/* Add outbox relays and build coverage. */
|
||||
for (int i = 0; i < map->outbox_count; i++) {
|
||||
cr_outbox_entry_t *oe = &map->outboxes[i];
|
||||
for (int r = 0; r < oe->relay_count; r++) {
|
||||
/* Find or create relay entry. */
|
||||
int found = -1;
|
||||
for (int j = 0; j < relay_count; j++) {
|
||||
if (strcmp(relays[j].url, oe->relays[r]) == 0) { found = j; break; }
|
||||
}
|
||||
if (found < 0) {
|
||||
if (relay_count >= CR_MAX_DISCOVERED_RELAYS) break;
|
||||
strncpy(relays[relay_count].url, oe->relays[r], CR_URL_LEN - 1);
|
||||
found = relay_count++;
|
||||
}
|
||||
/* Add this pubkey to the relay's coverage. */
|
||||
if (relays[found].pubkey_count < CR_MAX_PUBKEYS_PER_RELAY) {
|
||||
strncpy(relays[found].pubkeys[relays[found].pubkey_count],
|
||||
oe->pubkey, CR_HEX_LEN - 1);
|
||||
relays[found].pubkey_count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Greedy set cover. */
|
||||
char *covered = calloc(followed->count, 1); /* 1 if pubkey is covered */
|
||||
map->selected_count = 0;
|
||||
map->coverage = calloc(map->outbox_count, sizeof(int));
|
||||
for (int i = 0; i < map->outbox_count; i++) map->coverage[i] = -1;
|
||||
|
||||
int uncovered_count = followed->count;
|
||||
|
||||
/* Helper: find index of a pubkey in the followed set. */
|
||||
/* (linear search, fine for our scale) */
|
||||
|
||||
while (uncovered_count > 0 && map->selected_count < CR_MAX_DISCOVERED_RELAYS) {
|
||||
int best_relay = -1;
|
||||
int best_count = 0;
|
||||
|
||||
for (int r = 0; r < relay_count; r++) {
|
||||
/* Count how many uncovered pubkeys this relay covers. */
|
||||
int count = 0;
|
||||
for (int p = 0; p < relays[r].pubkey_count; p++) {
|
||||
/* Find this pubkey in followed set. */
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (strcmp(followed->items[i], relays[r].pubkeys[p]) == 0) {
|
||||
if (!covered[i]) count++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (count > best_count) {
|
||||
best_count = count;
|
||||
best_relay = r;
|
||||
}
|
||||
}
|
||||
|
||||
if (best_relay < 0 || best_count == 0) break; /* no more coverage possible */
|
||||
|
||||
/* Select this relay. */
|
||||
strncpy(map->selected_relays[map->selected_count],
|
||||
relays[best_relay].url, CR_URL_LEN - 1);
|
||||
map->selected_count++;
|
||||
|
||||
/* Mark covered pubkeys and record coverage. */
|
||||
for (int p = 0; p < relays[best_relay].pubkey_count; p++) {
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (strcmp(followed->items[i], relays[best_relay].pubkeys[p]) == 0) {
|
||||
if (!covered[i]) {
|
||||
covered[i] = 1;
|
||||
uncovered_count--;
|
||||
/* Record which selected relay covers this outbox entry. */
|
||||
for (int oe_i = 0; oe_i < map->outbox_count; oe_i++) {
|
||||
if (strcmp(map->outboxes[oe_i].pubkey,
|
||||
relays[best_relay].pubkeys[p]) == 0) {
|
||||
if (map->coverage[oe_i] < 0) {
|
||||
map->coverage[oe_i] = map->selected_count - 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/* Log results. */
|
||||
DEBUG_INFO("relay_discovery: selected %d relays to cover %d/%d pubkeys",
|
||||
map->selected_count, followed->count - uncovered_count, followed->count);
|
||||
for (int i = 0; i < map->selected_count; i++) {
|
||||
DEBUG_INFO(" relay[%d]: %s", i, map->selected_relays[i]);
|
||||
}
|
||||
if (uncovered_count > 0) {
|
||||
DEBUG_WARN("relay_discovery: %d pubkeys have no outbox relays (covered by bootstrap)",
|
||||
uncovered_count);
|
||||
}
|
||||
|
||||
free(covered);
|
||||
free(relays);
|
||||
}
|
||||
|
||||
/* ---- batched kind-10002 query ---- */
|
||||
|
||||
/* Query kind 10002 for a batch of up to 100 pubkeys from a pool.
|
||||
* Returns a cJSON array of events (caller must cJSON_Delete each + free).
|
||||
* Sets *out_count to the number of events returned. */
|
||||
static cJSON **query_kind10002_batch(nostr_relay_pool_t *pool,
|
||||
const char **pubkeys, int pubkey_count,
|
||||
int *out_count) {
|
||||
cJSON *filter = cJSON_CreateObject();
|
||||
cJSON *authors = cJSON_CreateArray();
|
||||
for (int i = 0; i < pubkey_count; i++)
|
||||
cJSON_AddItemToArray(authors, cJSON_CreateString(pubkeys[i]));
|
||||
cJSON_AddItemToObject(filter, "authors", authors);
|
||||
cJSON *kinds = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(kinds, cJSON_CreateNumber(10002));
|
||||
cJSON_AddItemToObject(filter, "kinds", kinds);
|
||||
|
||||
const char **urls = NULL;
|
||||
int n = get_pool_urls(pool, &urls);
|
||||
|
||||
/* TRACE: log the query being sent. */
|
||||
if (g_debug_level >= DEBUG_LEVEL_TRACE) {
|
||||
char *filter_str = cJSON_PrintUnformatted(filter);
|
||||
DEBUG_TRACE("query_kind10002_batch: querying %d relays for %d pubkeys", n, pubkey_count);
|
||||
if (filter_str) {
|
||||
/* Only print first 200 chars of filter to avoid huge output */
|
||||
if (strlen(filter_str) > 200) filter_str[200] = '\0';
|
||||
DEBUG_TRACE(" filter: %s...", filter_str);
|
||||
free(filter_str);
|
||||
}
|
||||
}
|
||||
|
||||
int ev_count = 0;
|
||||
cJSON **events = nostr_relay_pool_query_sync(pool, urls, n, filter, &ev_count, 30000);
|
||||
free(urls);
|
||||
cJSON_Delete(filter);
|
||||
|
||||
DEBUG_TRACE("query_kind10002_batch: returned %d events", ev_count);
|
||||
|
||||
*out_count = ev_count;
|
||||
return events;
|
||||
}
|
||||
|
||||
/* Process a batch of kind-10002 events: parse r-tags, populate outbox entries,
|
||||
* publish to local relay. Returns count of events processed. */
|
||||
static int process_10002_batch(cJSON **events, int ev_count,
|
||||
cr_relay_map_t *map, cr_sink_t *sink,
|
||||
const char *source) {
|
||||
int processed = 0;
|
||||
for (int k = 0; k < ev_count; k++) {
|
||||
cJSON *event = events[k];
|
||||
if (!event) continue;
|
||||
|
||||
/* Get pubkey from event. */
|
||||
cJSON *pk = cJSON_GetObjectItem(event, "pubkey");
|
||||
if (!pk || !cJSON_IsString(pk)) { cJSON_Delete(event); continue; }
|
||||
const char *hex = cJSON_GetStringValue(pk);
|
||||
|
||||
/* Find or create outbox entry for this pubkey. */
|
||||
cr_outbox_entry_t *oe = NULL;
|
||||
for (int i = 0; i < map->outbox_count; i++) {
|
||||
if (strcmp(map->outboxes[i].pubkey, hex) == 0) { oe = &map->outboxes[i]; break; }
|
||||
}
|
||||
if (!oe) {
|
||||
if (map->outbox_count >= /* followed->count passed in via map size */ 99999) {
|
||||
cJSON_Delete(event); continue;
|
||||
}
|
||||
oe = &map->outboxes[map->outbox_count];
|
||||
strncpy(oe->pubkey, hex, CR_HEX_LEN - 1);
|
||||
map->outbox_count++;
|
||||
}
|
||||
|
||||
int n = parse_r_tags(event, oe);
|
||||
(void)n;
|
||||
|
||||
if (strcmp(source, "bootstrap") == 0) {
|
||||
cr_sink_publish(sink, event);
|
||||
/* Pump sink pool every 16 events to avoid pending publish overflow
|
||||
* (NOSTR_POOL_MAX_PENDING_PUBLISHES = 32). */
|
||||
if (processed > 0 && (processed % 16) == 0) {
|
||||
cr_sink_pump(sink, 200);
|
||||
}
|
||||
}
|
||||
|
||||
processed++;
|
||||
cJSON_Delete(event);
|
||||
}
|
||||
free(events);
|
||||
|
||||
/* Log sink stats after each batch. */
|
||||
DEBUG_INFO("relay_discovery: batch processed %d events (sink: ok=%ld reject=%ld dup=%ld err=%ld)",
|
||||
processed, sink->published_ok, sink->published_failed,
|
||||
sink->published_dup, 0L);
|
||||
|
||||
return processed;
|
||||
}
|
||||
|
||||
/* ---- main discovery function ---- */
|
||||
|
||||
#define CR_10002_BATCH_SIZE 100
|
||||
|
||||
int cr_relay_discovery_run(cr_relay_map_t *map, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_sink_t *sink,
|
||||
const cr_pubkey_set_t *followed, int is_first_time) {
|
||||
memset(map, 0, sizeof(*map));
|
||||
map->outboxes = calloc(followed->count, sizeof(cr_outbox_entry_t));
|
||||
if (!map->outboxes) return -1;
|
||||
map->outbox_count = 0;
|
||||
|
||||
DEBUG_INFO("relay_discovery: discovering outbox relays for %d pubkeys (%s)",
|
||||
followed->count, is_first_time ? "first-time" : "local-first");
|
||||
|
||||
int found_local = 0, found_bootstrap = 0, found_none = 0;
|
||||
|
||||
/* Pre-populate outbox entries for all followed pubkeys (empty, for coverage
|
||||
* tracking of pubkeys with no 10002). */
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
strncpy(map->outboxes[i].pubkey, followed->items[i], CR_HEX_LEN - 1);
|
||||
}
|
||||
map->outbox_count = followed->count;
|
||||
|
||||
/* Track which pubkeys we've found 10002 for. */
|
||||
char *found = calloc(followed->count, 1);
|
||||
|
||||
/* Phase 1: If subsequent startup, batch-query local relay first. */
|
||||
if (!is_first_time && sink && sink->pool) {
|
||||
DEBUG_INFO("relay_discovery: querying local relay for kind-10002 (batched)...");
|
||||
|
||||
for (int start = 0; start < followed->count && !g_shutdown; start += CR_10002_BATCH_SIZE) {
|
||||
int batch = followed->count - start;
|
||||
if (batch > CR_10002_BATCH_SIZE) batch = CR_10002_BATCH_SIZE;
|
||||
|
||||
int ev_count = 0;
|
||||
cJSON **events = query_kind10002_batch(sink->pool,
|
||||
(const char **)&followed->items[start],
|
||||
batch, &ev_count);
|
||||
if (events && ev_count > 0) {
|
||||
for (int k = 0; k < ev_count; k++) {
|
||||
cJSON *pk = cJSON_GetObjectItem(events[k], "pubkey");
|
||||
if (pk && cJSON_IsString(pk)) {
|
||||
const char *hex = cJSON_GetStringValue(pk);
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (strcmp(followed->items[i], hex) == 0) {
|
||||
if (!found[i]) { found[i] = 1; found_local++; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
process_10002_batch(events, ev_count, map, sink, "local");
|
||||
/* Pump sink pool to flush published events. */
|
||||
cr_sink_pump(sink, 500);
|
||||
} else {
|
||||
free(events);
|
||||
}
|
||||
}
|
||||
DEBUG_INFO("relay_discovery: local relay provided %d kind-10002 events", found_local);
|
||||
}
|
||||
|
||||
/* Phase 2: Batch-query bootstrap relays for pubkeys not found locally. */
|
||||
int remaining = 0;
|
||||
for (int i = 0; i < followed->count; i++) if (!found[i]) remaining++;
|
||||
|
||||
if (remaining > 0) {
|
||||
DEBUG_INFO("relay_discovery: querying bootstrap relays for %d remaining pubkeys (batched)...",
|
||||
remaining);
|
||||
|
||||
/* Build list of remaining pubkeys. */
|
||||
const char **remaining_pk = malloc(remaining * sizeof(char *));
|
||||
int ridx = 0;
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (!found[i]) remaining_pk[ridx++] = followed->items[i];
|
||||
}
|
||||
|
||||
for (int start = 0; start < remaining && !g_shutdown; start += CR_10002_BATCH_SIZE) {
|
||||
int batch = remaining - start;
|
||||
if (batch > CR_10002_BATCH_SIZE) batch = CR_10002_BATCH_SIZE;
|
||||
|
||||
int ev_count = 0;
|
||||
cJSON **events = query_kind10002_batch(upstream,
|
||||
&remaining_pk[start],
|
||||
batch, &ev_count);
|
||||
if (events && ev_count > 0) {
|
||||
for (int k = 0; k < ev_count; k++) {
|
||||
cJSON *pk = cJSON_GetObjectItem(events[k], "pubkey");
|
||||
if (pk && cJSON_IsString(pk)) {
|
||||
const char *hex = cJSON_GetStringValue(pk);
|
||||
for (int i = 0; i < followed->count; i++) {
|
||||
if (strcmp(followed->items[i], hex) == 0) {
|
||||
if (!found[i]) { found[i] = 1; found_bootstrap++; }
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
process_10002_batch(events, ev_count, map, sink, "bootstrap");
|
||||
/* Pump sink pool to flush published events. */
|
||||
cr_sink_pump(sink, 500);
|
||||
} else {
|
||||
free(events);
|
||||
}
|
||||
}
|
||||
free(remaining_pk);
|
||||
}
|
||||
|
||||
found_none = followed->count - found_local - found_bootstrap;
|
||||
DEBUG_INFO("relay_discovery: %d local, %d bootstrap, %d none",
|
||||
found_local, found_bootstrap, found_none);
|
||||
|
||||
free(found);
|
||||
|
||||
/* Compute minimum covering set. */
|
||||
compute_covering_set(map, cfg, followed);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
void cr_relay_map_free(cr_relay_map_t *map) {
|
||||
if (map->outboxes) {
|
||||
free(map->outboxes);
|
||||
map->outboxes = NULL;
|
||||
}
|
||||
if (map->coverage) {
|
||||
free(map->coverage);
|
||||
map->coverage = NULL;
|
||||
}
|
||||
map->outbox_count = 0;
|
||||
map->selected_count = 0;
|
||||
}
|
||||
|
||||
const cr_outbox_entry_t *cr_relay_map_get_outbox(const cr_relay_map_t *map, const char *hex) {
|
||||
for (int i = 0; i < map->outbox_count; i++) {
|
||||
if (strcmp(map->outboxes[i].pubkey, hex) == 0) return &map->outboxes[i];
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
/*
|
||||
* caching_relay - NIP-65 outbox relay discovery.
|
||||
*
|
||||
* For each followed pubkey, fetches their kind 10002 (relay list) and parses
|
||||
* the "r" tags to build a per-pubkey outbox relay map. Then computes the
|
||||
* minimum covering set of relays (greedy set cover) that covers all followed
|
||||
* pubkeys.
|
||||
*
|
||||
* On first-time startup, queries bootstrap relays for kind 10002.
|
||||
* On subsequent startups, queries the local relay first (fast), falls back
|
||||
* to bootstrap relays for any pubkey not found locally.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_RELAY_DISCOVERY_H
|
||||
#define CACHING_RELAY_RELAY_DISCOVERY_H
|
||||
|
||||
#include "config.h"
|
||||
#include "state.h"
|
||||
#include "relay_sink.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
|
||||
#define CR_MAX_RELAYS_PER_PUBKEY 8
|
||||
#define CR_MAX_PUBKEYS_PER_RELAY 512
|
||||
#define CR_MAX_DISCOVERED_RELAYS 128
|
||||
|
||||
/* Per-pubkey outbox relay list. */
|
||||
typedef struct {
|
||||
char pubkey[CR_HEX_LEN];
|
||||
char relays[CR_MAX_RELAYS_PER_PUBKEY][CR_URL_LEN];
|
||||
int relay_count;
|
||||
} cr_outbox_entry_t;
|
||||
|
||||
/* Result of relay discovery. */
|
||||
typedef struct {
|
||||
cr_outbox_entry_t *outboxes; /* per-pubkey relay lists (heap) */
|
||||
int outbox_count;
|
||||
|
||||
/* The selected minimum covering set of relay URLs. */
|
||||
char selected_relays[CR_MAX_DISCOVERED_RELAYS][CR_URL_LEN];
|
||||
int selected_count;
|
||||
|
||||
/* Per-pubkey: which selected relay covers this pubkey (index into selected_relays).
|
||||
* -1 means covered by bootstrap relays only. */
|
||||
int *coverage; /* heap, outbox_count entries */
|
||||
} cr_relay_map_t;
|
||||
|
||||
/* Discover outbox relays for all followed pubkeys.
|
||||
*
|
||||
* cfg - config (provides bootstrap relays, local_relay, root_hex)
|
||||
* upstream - upstream pool (has bootstrap relays; discovered relays will be added)
|
||||
* sink - sink pool (local relay, for publishing 10002 events + querying on subsequent startup)
|
||||
* followed - the followed pubkey set
|
||||
* is_first_time - 1 if first startup (query bootstrap only), 0 if subsequent (local-first)
|
||||
*
|
||||
* Returns 0 on success, -1 on hard error. On success, map is populated.
|
||||
* Caller must call cr_relay_map_free() when done.
|
||||
*/
|
||||
int cr_relay_discovery_run(cr_relay_map_t *map, cr_config_t *cfg,
|
||||
nostr_relay_pool_t *upstream, cr_sink_t *sink,
|
||||
const cr_pubkey_set_t *followed, int is_first_time);
|
||||
|
||||
/* Free heap resources in a relay map. */
|
||||
void cr_relay_map_free(cr_relay_map_t *map);
|
||||
|
||||
/* Get the outbox relays for a specific pubkey. Returns NULL if not found. */
|
||||
const cr_outbox_entry_t *cr_relay_map_get_outbox(const cr_relay_map_t *map, const char *hex);
|
||||
|
||||
#endif /* CACHING_RELAY_RELAY_DISCOVERY_H */
|
||||
@@ -0,0 +1,161 @@
|
||||
/*
|
||||
* caching_relay - relay sink: publishes fetched events to the local relay.
|
||||
*
|
||||
* See relay_sink.h for the dual-mode (WebSocket / PostgreSQL inbox) design.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "relay_sink.h"
|
||||
#include "pg_inbox.h"
|
||||
#include "debug.h"
|
||||
|
||||
#include <string.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* WebSocket mode callback */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
static void sink_publish_cb(const char *relay_url, const char *event_id,
|
||||
int success, const char *message, void *user_data) {
|
||||
cr_sink_t *sink = (cr_sink_t *)user_data;
|
||||
if (success) {
|
||||
sink->published_ok++;
|
||||
DEBUG_TRACE("sink: OK %s from %s", event_id ? event_id : "?",
|
||||
relay_url ? relay_url : "?");
|
||||
} else {
|
||||
sink->published_failed++;
|
||||
/* REJECT is important - show at INFO level so it's visible at level 3. */
|
||||
DEBUG_INFO("sink: REJECT %s from %s: %s", event_id ? event_id : "?",
|
||||
relay_url ? relay_url : "?",
|
||||
message ? message : "(no message)");
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Init */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int cr_sink_init(cr_sink_t *sink, const char *local_relay_url, cr_seen_ring_t *seen) {
|
||||
memset(sink, 0, sizeof(*sink));
|
||||
sink->seen = seen;
|
||||
sink->source_class = CR_SINK_CLASS_LIVE;
|
||||
strncpy(sink->url, local_relay_url, sizeof(sink->url) - 1);
|
||||
|
||||
sink->pool = nostr_relay_pool_create(nostr_pool_reconnect_config_default());
|
||||
if (!sink->pool) {
|
||||
DEBUG_ERROR("sink: failed to create pool");
|
||||
return -1;
|
||||
}
|
||||
if (nostr_relay_pool_add_relay(sink->pool, local_relay_url) != NOSTR_SUCCESS) {
|
||||
DEBUG_ERROR("sink: failed to add relay %s", local_relay_url);
|
||||
nostr_relay_pool_destroy(sink->pool);
|
||||
sink->pool = NULL;
|
||||
return -1;
|
||||
}
|
||||
DEBUG_INFO("sink: initialized (WebSocket) for %s", local_relay_url);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cr_sink_init_pg(cr_sink_t *sink, cr_seen_ring_t *seen) {
|
||||
memset(sink, 0, sizeof(*sink));
|
||||
sink->seen = seen;
|
||||
sink->source_class = CR_SINK_CLASS_LIVE;
|
||||
sink->pool = NULL; /* PostgreSQL inbox mode - no WebSocket pool. */
|
||||
DEBUG_INFO("sink: initialized (PostgreSQL inbox)");
|
||||
return 0;
|
||||
}
|
||||
|
||||
void cr_sink_set_source_class(cr_sink_t *sink, const char *class_ptr) {
|
||||
if (sink) sink->source_class = class_ptr ? class_ptr : CR_SINK_CLASS_LIVE;
|
||||
}
|
||||
|
||||
void cr_sink_destroy(cr_sink_t *sink) {
|
||||
if (sink && sink->pool) {
|
||||
nostr_relay_pool_destroy(sink->pool);
|
||||
sink->pool = NULL;
|
||||
}
|
||||
}
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
/* Publish */
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
||||
int cr_sink_publish_from(cr_sink_t *sink, cJSON *event, const char *source_relay) {
|
||||
if (!sink || !event) return -1;
|
||||
|
||||
cJSON *id = cJSON_GetObjectItem(event, "id");
|
||||
if (!id || !cJSON_IsString(id)) {
|
||||
DEBUG_WARN("sink: event missing id, skipping");
|
||||
return -1;
|
||||
}
|
||||
const char *eid = cJSON_GetStringValue(id);
|
||||
|
||||
if (cr_seen_ring_add(sink->seen, eid) == 0) {
|
||||
sink->published_dup++;
|
||||
return 0; /* already seen recently */
|
||||
}
|
||||
|
||||
/* PostgreSQL inbox mode. */
|
||||
if (!sink->pool) {
|
||||
char *json = cJSON_PrintUnformatted(event);
|
||||
if (!json) {
|
||||
DEBUG_WARN("sink: failed to serialize event %s", eid);
|
||||
sink->published_failed++;
|
||||
return -1;
|
||||
}
|
||||
const char *klass = sink->source_class ? sink->source_class
|
||||
: CR_SINK_CLASS_LIVE;
|
||||
int priority = (strcmp(klass, CR_SINK_CLASS_BACKFILL) == 0) ? 1 : 0;
|
||||
|
||||
cJSON *kind = cJSON_GetObjectItem(event, "kind");
|
||||
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
|
||||
int kind_num = (kind && cJSON_IsNumber(kind)) ? (int)cJSON_GetNumberValue(kind) : -1;
|
||||
DEBUG_TRACE("sink: INBOX event id=%s kind=%d pubkey=%s class=%s",
|
||||
eid, kind_num,
|
||||
pubkey && cJSON_IsString(pubkey) ? cJSON_GetStringValue(pubkey) : "?",
|
||||
klass);
|
||||
|
||||
int rc = pg_inbox_insert_event(json, source_relay, klass, priority);
|
||||
free(json);
|
||||
if (rc != 0) {
|
||||
sink->published_failed++;
|
||||
return -1;
|
||||
}
|
||||
sink->published_ok++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* WebSocket mode. */
|
||||
const char *urls[1] = { sink->url };
|
||||
cJSON *kind = cJSON_GetObjectItem(event, "kind");
|
||||
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
|
||||
int kind_num = (kind && cJSON_IsNumber(kind)) ? (int)cJSON_GetNumberValue(kind) : -1;
|
||||
DEBUG_TRACE("sink: SEND event id=%s kind=%d pubkey=%s to %s",
|
||||
eid, kind_num,
|
||||
pubkey && cJSON_IsString(pubkey) ? cJSON_GetStringValue(pubkey) : "?",
|
||||
sink->url);
|
||||
|
||||
int rc = nostr_relay_pool_publish_async(sink->pool, urls, 1, event,
|
||||
sink_publish_cb, sink);
|
||||
if (rc < 0) {
|
||||
DEBUG_WARN("sink: publish_async error rc=%d for %s", rc, eid);
|
||||
return -1;
|
||||
}
|
||||
if (rc == 0) {
|
||||
DEBUG_INFO("sink: publish SKIPPED (not connected) for %s", eid);
|
||||
return 0;
|
||||
}
|
||||
DEBUG_TRACE("sink: publish_async rc=%d for %s", rc, eid);
|
||||
return 1;
|
||||
}
|
||||
|
||||
int cr_sink_publish(cr_sink_t *sink, cJSON *event) {
|
||||
return cr_sink_publish_from(sink, event, NULL);
|
||||
}
|
||||
|
||||
void cr_sink_pump(cr_sink_t *sink, int timeout_ms) {
|
||||
if (sink && sink->pool) {
|
||||
nostr_relay_pool_run(sink->pool, timeout_ms);
|
||||
}
|
||||
/* No-op in PostgreSQL inbox mode. */
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
/*
|
||||
* caching_relay - relay sink: publishes fetched events to the local relay.
|
||||
*
|
||||
* Two modes:
|
||||
* - WebSocket mode (legacy): a dedicated single-relay pool publishes via
|
||||
* nostr_relay_pool_publish_async(). Used when no PostgreSQL connection
|
||||
* string is provided.
|
||||
* - PostgreSQL inbox mode: events are inserted directly into the
|
||||
* caching_event_inbox table via pg_inbox_insert_event(). Used when the
|
||||
* daemon is started with --pg-conn.
|
||||
*
|
||||
* The public function signatures are identical in both modes so that callers
|
||||
* (live_subscriber, backfill, relay_discovery) do not need to change.
|
||||
*/
|
||||
#ifndef CACHING_RELAY_RELAY_SINK_H
|
||||
#define CACHING_RELAY_RELAY_SINK_H
|
||||
|
||||
#include "state.h"
|
||||
#include "../nostr_core_lib/nostr_core/nostr_core.h"
|
||||
|
||||
/* Source class labels for the inbox priority field. */
|
||||
#define CR_SINK_CLASS_LIVE "live"
|
||||
#define CR_SINK_CLASS_DISCOVERY "discovery"
|
||||
#define CR_SINK_CLASS_BACKFILL "backfill"
|
||||
|
||||
typedef struct {
|
||||
/* WebSocket mode only (NULL in PostgreSQL inbox mode). */
|
||||
nostr_relay_pool_t *pool;
|
||||
char url[256];
|
||||
|
||||
/* PostgreSQL inbox mode: source class used for priority assignment.
|
||||
* Defaults to "live". Set to "backfill" by the backfill module.
|
||||
* Ignored in WebSocket mode. */
|
||||
const char *source_class;
|
||||
|
||||
cr_seen_ring_t *seen; /* shared seen ring for dedup */
|
||||
long published_ok;
|
||||
long published_failed;
|
||||
long published_dup;
|
||||
} cr_sink_t;
|
||||
|
||||
/* Create the sink.
|
||||
*
|
||||
* In WebSocket mode (pg_mode = 0): creates a relay pool and adds
|
||||
* local_relay_url.
|
||||
* In PostgreSQL inbox mode (pg_mode != 0): no pool is created; the URL is
|
||||
* ignored. Events are inserted via pg_inbox_insert_event() (the pg_inbox
|
||||
* module must be initialized beforehand).
|
||||
*
|
||||
* The original two-argument signature is preserved via cr_sink_init() below
|
||||
* for backward compatibility; it defaults to WebSocket mode. */
|
||||
int cr_sink_init(cr_sink_t *sink, const char *local_relay_url, cr_seen_ring_t *seen);
|
||||
|
||||
/* Initialize the sink in PostgreSQL inbox mode. */
|
||||
int cr_sink_init_pg(cr_sink_t *sink, cr_seen_ring_t *seen);
|
||||
|
||||
/* Set the source class for priority assignment in PostgreSQL inbox mode.
|
||||
* class_ptr must point to a static string literal (e.g. CR_SINK_CLASS_LIVE). */
|
||||
void cr_sink_set_source_class(cr_sink_t *sink, const char *class_ptr);
|
||||
|
||||
void cr_sink_destroy(cr_sink_t *sink);
|
||||
|
||||
/* Publish a cJSON event. Dedups against the seen ring.
|
||||
* Returns 1 if enqueued/inserted, 0 if dup (skipped), -1 on error. */
|
||||
int cr_sink_publish(cr_sink_t *sink, cJSON *event);
|
||||
|
||||
/* Variant that records the originating relay URL in PostgreSQL inbox mode.
|
||||
* In WebSocket mode the source_relay is ignored. */
|
||||
int cr_sink_publish_from(cr_sink_t *sink, cJSON *event, const char *source_relay);
|
||||
|
||||
/* Pump the sink pool briefly to flush pending publish callbacks.
|
||||
* No-op in PostgreSQL inbox mode. */
|
||||
void cr_sink_pump(cr_sink_t *sink, int timeout_ms);
|
||||
|
||||
#endif /* CACHING_RELAY_RELAY_SINK_H */
|
||||
@@ -0,0 +1,76 @@
|
||||
/*
|
||||
* caching_relay - in-memory runtime state
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include "state.h"
|
||||
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
/* ---- pubkey set ---- */
|
||||
|
||||
void cr_pubkey_set_init(cr_pubkey_set_t *s) {
|
||||
s->items = NULL;
|
||||
s->count = 0;
|
||||
s->capacity = 0;
|
||||
}
|
||||
|
||||
void cr_pubkey_set_free(cr_pubkey_set_t *s) {
|
||||
for (int i = 0; i < s->count; i++) free(s->items[i]);
|
||||
free(s->items);
|
||||
s->items = NULL;
|
||||
s->count = 0;
|
||||
s->capacity = 0;
|
||||
}
|
||||
|
||||
void cr_pubkey_set_clear(cr_pubkey_set_t *s) {
|
||||
cr_pubkey_set_free(s);
|
||||
cr_pubkey_set_init(s);
|
||||
}
|
||||
|
||||
int cr_pubkey_set_contains(const cr_pubkey_set_t *s, const char *hex) {
|
||||
for (int i = 0; i < s->count; i++) {
|
||||
if (strcmp(s->items[i], hex) == 0) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cr_pubkey_set_add(cr_pubkey_set_t *s, const char *hex) {
|
||||
if (!hex || !hex[0]) return 0;
|
||||
if (cr_pubkey_set_contains(s, hex)) return 0;
|
||||
if (s->count >= s->capacity) {
|
||||
int newcap = s->capacity ? s->capacity * 2 : 64;
|
||||
char **ni = realloc(s->items, newcap * sizeof(char *));
|
||||
if (!ni) return -1;
|
||||
s->items = ni;
|
||||
s->capacity = newcap;
|
||||
}
|
||||
s->items[s->count] = strdup(hex);
|
||||
if (!s->items[s->count]) return -1;
|
||||
s->count++;
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* ---- seen-event ring buffer ---- */
|
||||
|
||||
void cr_seen_ring_init(cr_seen_ring_t *r) {
|
||||
memset(r, 0, sizeof(*r));
|
||||
}
|
||||
|
||||
int cr_seen_ring_contains(const cr_seen_ring_t *r, const char *id) {
|
||||
for (int i = 0; i < r->count; i++) {
|
||||
int idx = (r->head - 1 - i + CR_SEEN_RING_SIZE) % CR_SEEN_RING_SIZE;
|
||||
if (strcmp(r->ids[idx], id) == 0) return 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cr_seen_ring_add(cr_seen_ring_t *r, const char *id) {
|
||||
if (!id || !id[0]) return 0;
|
||||
if (cr_seen_ring_contains(r, id)) return 0;
|
||||
strncpy(r->ids[r->head], id, CR_HEX_LEN - 1);
|
||||
r->ids[r->head][CR_HEX_LEN - 1] = '\0';
|
||||
r->head = (r->head + 1) % CR_SEEN_RING_SIZE;
|
||||
if (r->count < CR_SEEN_RING_SIZE) r->count++;
|
||||
return 1;
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
/*
|
||||
* caching_relay - in-memory runtime state:
|
||||
* - followed-pubkey set (hex pubkeys)
|
||||
* - seen-event-id ring buffer (to avoid redundant publish attempts)
|
||||
*/
|
||||
#ifndef CACHING_RELAY_STATE_H
|
||||
#define CACHING_RELAY_STATE_H
|
||||
|
||||
#include <stddef.h>
|
||||
|
||||
/* Hex pubkey is 64 chars + NUL. Event id is 64 hex chars + NUL. */
|
||||
#define CR_HEX_LEN 65
|
||||
|
||||
/* A simple dynamic string set for followed pubkeys. */
|
||||
typedef struct {
|
||||
char **items;
|
||||
int count;
|
||||
int capacity;
|
||||
} cr_pubkey_set_t;
|
||||
|
||||
void cr_pubkey_set_init(cr_pubkey_set_t *s);
|
||||
void cr_pubkey_set_free(cr_pubkey_set_t *s);
|
||||
int cr_pubkey_set_add(cr_pubkey_set_t *s, const char *hex); /* 1 if added, 0 if dup */
|
||||
int cr_pubkey_set_contains(const cr_pubkey_set_t *s, const char *hex);
|
||||
void cr_pubkey_set_clear(cr_pubkey_set_t *s);
|
||||
|
||||
/* Fixed-size ring buffer of event ids for dedup. */
|
||||
#define CR_SEEN_RING_SIZE 4096
|
||||
|
||||
typedef struct {
|
||||
char ids[CR_SEEN_RING_SIZE][CR_HEX_LEN];
|
||||
int head;
|
||||
int count;
|
||||
} cr_seen_ring_t;
|
||||
|
||||
void cr_seen_ring_init(cr_seen_ring_t *r);
|
||||
/* Returns 1 if the id was newly inserted, 0 if it was already present. */
|
||||
int cr_seen_ring_add(cr_seen_ring_t *r, const char *id);
|
||||
int cr_seen_ring_contains(const cr_seen_ring_t *r, const char *id);
|
||||
|
||||
#endif /* CACHING_RELAY_STATE_H */
|
||||
@@ -0,0 +1 @@
|
||||
key,value,data_type,description,category,requires_restart,created_at,updated_at
|
||||
|
+5
-12
@@ -1,19 +1,12 @@
|
||||
#!/bin/bash
|
||||
# Restart the service
|
||||
sudo systemctl stop c-relay-pg.service
|
||||
|
||||
# Copy the binary to the deployment location
|
||||
cp build/c_relay_x86 ~/Storage/c_relay/crelay
|
||||
|
||||
# Copy the service file to systemd (use the main service file)
|
||||
sudo cp systemd/c-relay.service /etc/systemd/system/c-relay-local.service
|
||||
|
||||
# Reload systemd daemon to pick up the new service
|
||||
sudo systemctl daemon-reload
|
||||
|
||||
# Enable the service (if not already enabled)
|
||||
sudo systemctl enable c-relay-local.service
|
||||
cp build/c_relay_pg_static_x86_64 ~/Storage/c_relay_pg/crelay
|
||||
|
||||
# Restart the service
|
||||
sudo systemctl restart c-relay-local.service
|
||||
sudo systemctl restart c-relay-pg.service
|
||||
|
||||
# Show service status
|
||||
sudo systemctl status c-relay-local.service --no-pager -l
|
||||
sudo systemctl status c-relay-pg.service --no-pager -l
|
||||
|
||||
+88
-18
@@ -1,28 +1,98 @@
|
||||
#!/bin/bash
|
||||
|
||||
# C-Relay Static Binary Deployment Script
|
||||
# Deploys build/c_relay_static_x86_64 to server via ssh
|
||||
# C-Relay-PG PostgreSQL Deployment Script
|
||||
# Deploys static relay binary and configures PostgreSQL 18 + systemd service on remote server.
|
||||
|
||||
set -e
|
||||
set -euo pipefail
|
||||
|
||||
# Configuration
|
||||
LOCAL_BINARY="build/c_relay_static_x86_64"
|
||||
REMOTE_BINARY_PATH="/usr/local/bin/c_relay/c_relay"
|
||||
SERVICE_NAME="c-relay"
|
||||
REMOTE_HOST="ubuntu@laantungir.net"
|
||||
LOCAL_BINARY="build/c_relay_pg_static_x86_64"
|
||||
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
|
||||
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
|
||||
SERVICE_NAME="c-relay-pg"
|
||||
|
||||
# Create backup
|
||||
ssh ubuntu@laantungir.com "sudo cp '$REMOTE_BINARY_PATH' '${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)'" 2>/dev/null || true
|
||||
LOCAL_SERVICE_FILE="systemd/c-relay.service"
|
||||
LOCAL_PG_SETUP_SCRIPT="systemd/setup_postgres_18.sh"
|
||||
|
||||
# Upload binary to temp location
|
||||
scp "$LOCAL_BINARY" "ubuntu@laantungir.com:/tmp/c_relay.tmp"
|
||||
if [ ! -f "$LOCAL_BINARY" ]; then
|
||||
echo "ERROR: Binary not found: $LOCAL_BINARY"
|
||||
echo "Build it first (e.g. ./make_and_restart_relay.sh)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Install binary
|
||||
ssh ubuntu@laantungir.com "sudo mv '/tmp/c_relay.tmp' '$REMOTE_BINARY_PATH'"
|
||||
ssh ubuntu@laantungir.com "sudo chown c-relay:c-relay '$REMOTE_BINARY_PATH'"
|
||||
ssh ubuntu@laantungir.com "sudo chmod +x '$REMOTE_BINARY_PATH'"
|
||||
if [ ! -f "$LOCAL_SERVICE_FILE" ]; then
|
||||
echo "ERROR: Service file not found: $LOCAL_SERVICE_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Reload systemd and restart service
|
||||
ssh ubuntu@laantungir.com "sudo systemctl daemon-reload"
|
||||
ssh ubuntu@laantungir.com "sudo systemctl restart '$SERVICE_NAME'"
|
||||
if [ ! -f "$LOCAL_PG_SETUP_SCRIPT" ]; then
|
||||
echo "ERROR: PostgreSQL setup script not found: $LOCAL_PG_SETUP_SCRIPT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Deployment complete!"
|
||||
echo "==> Uploading artifacts to $REMOTE_HOST"
|
||||
scp "$LOCAL_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg.tmp"
|
||||
scp "$LOCAL_SERVICE_FILE" "$REMOTE_HOST:/tmp/c-relay-pg.service"
|
||||
scp "$LOCAL_PG_SETUP_SCRIPT" "$REMOTE_HOST:/tmp/setup_postgres_18.sh"
|
||||
|
||||
echo "==> Running remote install/configuration"
|
||||
ssh "$REMOTE_HOST" 'bash -s' <<'EOF'
|
||||
set -euo pipefail
|
||||
|
||||
SERVICE_NAME="c-relay-pg"
|
||||
RELAY_USER="c-relay-pg"
|
||||
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
|
||||
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
|
||||
|
||||
echo "[remote] Ensuring service user exists"
|
||||
if ! id "$RELAY_USER" >/dev/null 2>&1; then
|
||||
sudo useradd --system --home-dir /opt/c-relay-pg --shell /usr/sbin/nologin "$RELAY_USER"
|
||||
fi
|
||||
|
||||
echo "[remote] Ensuring required directories exist"
|
||||
sudo mkdir -p "$REMOTE_BINARY_DIR" /opt/c-relay-pg /etc/c-relay-pg
|
||||
sudo chown "$RELAY_USER:$RELAY_USER" /opt/c-relay-pg
|
||||
|
||||
echo "[remote] Installing PostgreSQL 18 (PGDG) if missing"
|
||||
if ! dpkg -s postgresql-18 >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y curl ca-certificates lsb-release gnupg
|
||||
sudo install -d /usr/share/postgresql-common/pgdg
|
||||
sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc
|
||||
echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list >/dev/null
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y postgresql-18
|
||||
fi
|
||||
|
||||
sudo systemctl enable postgresql
|
||||
sudo systemctl restart postgresql
|
||||
|
||||
echo "[remote] Configuring PostgreSQL role/database"
|
||||
sudo chmod +x /tmp/setup_postgres_18.sh
|
||||
sudo /tmp/setup_postgres_18.sh
|
||||
|
||||
echo "[remote] Installing relay binary"
|
||||
if [ -f "$REMOTE_BINARY_PATH" ]; then
|
||||
sudo cp "$REMOTE_BINARY_PATH" "${REMOTE_BINARY_PATH}.backup.$(date +%Y%m%d_%H%M%S)"
|
||||
fi
|
||||
sudo mv /tmp/c_relay_pg.tmp "$REMOTE_BINARY_PATH"
|
||||
sudo chown "$RELAY_USER:$RELAY_USER" "$REMOTE_BINARY_PATH"
|
||||
sudo chmod +x "$REMOTE_BINARY_PATH"
|
||||
|
||||
echo "[remote] Installing systemd unit"
|
||||
sudo mv /tmp/c-relay-pg.service /etc/systemd/system/c-relay-pg.service
|
||||
sudo chown root:root /etc/systemd/system/c-relay-pg.service
|
||||
sudo chmod 644 /etc/systemd/system/c-relay-pg.service
|
||||
|
||||
echo "[remote] Reloading and restarting service"
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable "$SERVICE_NAME"
|
||||
sudo systemctl restart "$SERVICE_NAME"
|
||||
|
||||
echo "[remote] Health checks"
|
||||
sudo systemctl --no-pager --full status "$SERVICE_NAME" | sed -n '1,25p'
|
||||
sudo -u "$RELAY_USER" psql -d crelay -c "SELECT current_user, current_database();"
|
||||
EOF
|
||||
|
||||
echo "Deployment complete: $REMOTE_HOST"
|
||||
|
||||
Executable
+122
@@ -0,0 +1,122 @@
|
||||
#!/bin/bash
|
||||
|
||||
# C-Relay-PG Debug Binary Deployment Script
|
||||
# Deploys build/c_relay_pg_static_x86_64_debug to server for CPU profiling
|
||||
#
|
||||
# Usage:
|
||||
# ./deploy_lt_debug.sh -- deploy debug binary and restart
|
||||
# ./deploy_lt_debug.sh --profile -- deploy, then run perf and fetch results
|
||||
#
|
||||
# After deploying, profile with:
|
||||
# sudo perf record -g -p $(pgrep c_relay_pg) -- sleep 30
|
||||
# sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -80
|
||||
#
|
||||
# Restore production binary:
|
||||
# ./deploy_lt.sh
|
||||
|
||||
set -e
|
||||
|
||||
LOCAL_DEBUG_BINARY="build/c_relay_pg_static_x86_64_debug"
|
||||
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
|
||||
REMOTE_PROD_BACKUP="/usr/local/bin/c_relay_pg/c_relay_pg.production"
|
||||
SERVICE_NAME="c-relay-pg"
|
||||
REMOTE_HOST="ubuntu@laantungir.com"
|
||||
|
||||
# Check debug binary exists
|
||||
if [ ! -f "$LOCAL_DEBUG_BINARY" ]; then
|
||||
echo "ERROR: Debug binary not found: $LOCAL_DEBUG_BINARY"
|
||||
echo ""
|
||||
echo "Build it first with:"
|
||||
echo " ./build_static.sh --debug"
|
||||
echo ""
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=========================================="
|
||||
echo "C-Relay-PG Debug Deployment"
|
||||
echo "=========================================="
|
||||
echo "Binary: $LOCAL_DEBUG_BINARY ($(du -h "$LOCAL_DEBUG_BINARY" | cut -f1))"
|
||||
echo "Target: $REMOTE_HOST:$REMOTE_BINARY_PATH"
|
||||
echo ""
|
||||
echo "WARNING: Debug binary has symbols and is larger than production."
|
||||
echo " It is safe to run but should not be left deployed long-term."
|
||||
echo ""
|
||||
|
||||
# Backup production binary (only if not already backed up)
|
||||
echo "Backing up production binary..."
|
||||
ssh "$REMOTE_HOST" "
|
||||
if [ ! -f '$REMOTE_PROD_BACKUP' ]; then
|
||||
sudo cp '$REMOTE_BINARY_PATH' '$REMOTE_PROD_BACKUP'
|
||||
echo 'Production binary backed up to $REMOTE_PROD_BACKUP'
|
||||
else
|
||||
echo 'Production backup already exists, skipping'
|
||||
fi
|
||||
"
|
||||
|
||||
# Upload debug binary
|
||||
echo "Uploading debug binary..."
|
||||
scp "$LOCAL_DEBUG_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg_debug.tmp"
|
||||
|
||||
# Install debug binary
|
||||
echo "Installing debug binary..."
|
||||
ssh "$REMOTE_HOST" "
|
||||
sudo mv '/tmp/c_relay_pg_debug.tmp' '$REMOTE_BINARY_PATH'
|
||||
sudo chown c-relay-pg:c-relay-pg '$REMOTE_BINARY_PATH'
|
||||
sudo chmod +x '$REMOTE_BINARY_PATH'
|
||||
"
|
||||
|
||||
# Restart service
|
||||
echo "Restarting c-relay-pg service..."
|
||||
ssh "$REMOTE_HOST" "sudo systemctl daemon-reload && sudo systemctl restart '$SERVICE_NAME'"
|
||||
|
||||
echo ""
|
||||
echo "✓ Debug binary deployed and service restarted"
|
||||
echo ""
|
||||
|
||||
# If --profile flag, run perf automatically
|
||||
if [ "$1" = "--profile" ]; then
|
||||
echo "=========================================="
|
||||
echo "Running perf profile (30 seconds)..."
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Wait for relay to start
|
||||
sleep 3
|
||||
|
||||
ssh "$REMOTE_HOST" "
|
||||
PID=\$(pgrep c_relay_pg)
|
||||
if [ -z \"\$PID\" ]; then
|
||||
echo 'ERROR: c_relay_pg not running'
|
||||
exit 1
|
||||
fi
|
||||
echo \"Profiling PID \$PID for 30 seconds...\"
|
||||
sudo perf record -g -p \$PID -- sleep 30
|
||||
echo ''
|
||||
echo '=== TOP FUNCTIONS BY CPU ==='
|
||||
sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -60
|
||||
"
|
||||
else
|
||||
echo "=========================================="
|
||||
echo "Next Steps: Profile the relay"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "SSH to server and run:"
|
||||
echo " sudo perf record -g -p \$(pgrep c_relay_pg) -- sleep 30"
|
||||
echo " sudo perf report --stdio --sort=symbol --no-children -n 2>/dev/null | head -60"
|
||||
echo ""
|
||||
echo "Or run with --profile to do it automatically:"
|
||||
echo " ./deploy_lt_debug.sh --profile"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo "Restore Production Binary When Done"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
echo "Run this to restore the production binary:"
|
||||
echo " ./deploy_lt.sh"
|
||||
echo ""
|
||||
echo "Or manually on the server:"
|
||||
echo " sudo cp '$REMOTE_PROD_BACKUP' '$REMOTE_BINARY_PATH'"
|
||||
echo " sudo systemctl restart $SERVICE_NAME"
|
||||
echo ""
|
||||
@@ -1,4 +1,4 @@
|
||||
# C-Relay Administrator API Implementation Plan
|
||||
# C-Relay-PG Administrator API Implementation Plan
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
@@ -161,7 +161,7 @@ Would require changing schema, migration scripts, and storage logic.
|
||||
#### README.md Documentation Format:
|
||||
|
||||
```markdown
|
||||
# C-Relay Administrator API
|
||||
# C-Relay-PG Administrator API
|
||||
|
||||
## Authentication
|
||||
All admin commands require signing with the admin private key generated during first startup.
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
# Agent Browser Testing Guide for C-Relay-PG
|
||||
|
||||
This document explains how to use the `agent-browser` CLI tool to test the c-relay-pg admin web UI from an AI agent context (no physical display required).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **agent-browser** installed globally: `npm install -g agent-browser`
|
||||
- Binary location: `/home/user/.nvm/versions/node/v24.14.1/bin/agent-browser`
|
||||
- The relay must be running locally (default port 8888)
|
||||
- Test keys configured in `.test_keys`
|
||||
|
||||
## Starting the Relay for Local Testing
|
||||
|
||||
```bash
|
||||
./make_and_restart_relay.sh -t
|
||||
```
|
||||
|
||||
This reads `.test_keys` and starts the relay with:
|
||||
- `ADMIN_PUBKEY` — the hex public key of the admin account
|
||||
- `ADMIN_PRIVKEY` — the hex secret key (used for browser login, not by the relay itself)
|
||||
- `SERVER_PRIVKEY` — the relay's own private key
|
||||
|
||||
## Key URLs
|
||||
|
||||
| URL | Purpose |
|
||||
|-----|---------|
|
||||
| `http://127.0.0.1:8888/api/index.html` | Admin web UI (the correct entry point) |
|
||||
| `http://127.0.0.1:8888/` | Returns 406 without NIP-11 Accept header — **do not use for browser testing** |
|
||||
| `http://127.0.0.1:8888/` with `Accept: application/nostr+json` | NIP-11 relay info JSON |
|
||||
|
||||
**Important:** The root URL `/` is a WebSocket/NIP-11 endpoint, not an HTML page. Always use `/api/index.html` for browser testing.
|
||||
|
||||
## Admin Login Credentials
|
||||
|
||||
The admin nsec for the current `.test_keys` configuration:
|
||||
|
||||
```
|
||||
nsec: nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm
|
||||
hex: 15a6fbfd759330c9948ba07d9ed77e49d414c4377a2516d47fc0ab0d54d411f5
|
||||
npub: npub1tlyzk6slzt8d989f4pn3synk98fea64ea3jmrdc7dtd0kw8uxlas9a9fwr
|
||||
hex pubkey: 5fc82b6a1f12ced29ca9a86718127629d39eeab9ec65b1b71e6adafb38fc37fb
|
||||
```
|
||||
|
||||
## Complete Login Flow with agent-browser
|
||||
|
||||
### Step 1: Open the admin UI
|
||||
|
||||
```bash
|
||||
agent-browser open http://127.0.0.1:8888/api/index.html
|
||||
agent-browser wait --load networkidle
|
||||
```
|
||||
|
||||
### Step 2: Take an interactive snapshot to see what is on screen
|
||||
|
||||
```bash
|
||||
agent-browser snapshot -i
|
||||
```
|
||||
|
||||
You will see a login modal with buttons like:
|
||||
- `"Browser Extension"` — skip this
|
||||
- `"Local Key"` — **use this one**
|
||||
- `"Seed Phrase"` — skip
|
||||
- `"Nostr Connect"` — skip
|
||||
- `"Read Only"` — skip
|
||||
|
||||
### Step 3: Click "Local Key"
|
||||
|
||||
```bash
|
||||
agent-browser click @e15
|
||||
```
|
||||
|
||||
(The ref number may vary — use the ref from the snapshot output for the "Local Key" button.)
|
||||
|
||||
After clicking, a text input appears asking for the secret key.
|
||||
|
||||
### Step 4: Enter the admin nsec
|
||||
|
||||
```bash
|
||||
agent-browser fill @e14 "nsec1zkn0hlt4jvcvn9yt5p7ea4m7f82pf3ph0gj3d4rlcz4s64x5z86satv9qm"
|
||||
```
|
||||
|
||||
(Use the ref from the snapshot for the textbox element.)
|
||||
|
||||
### Step 5: Click "Import Key"
|
||||
|
||||
```bash
|
||||
agent-browser snapshot -i
|
||||
```
|
||||
|
||||
Check the snapshot — the "Import Key" button should now be enabled. Click it:
|
||||
|
||||
```bash
|
||||
agent-browser click @e15
|
||||
```
|
||||
|
||||
### Step 6: Click "Continue" on the success screen
|
||||
|
||||
After import, a success screen appears with "Continue" button:
|
||||
|
||||
```bash
|
||||
agent-browser wait 800
|
||||
agent-browser snapshot -i
|
||||
agent-browser click @e19
|
||||
```
|
||||
|
||||
(Use the ref from the snapshot for the "Continue" button.)
|
||||
|
||||
### Step 7: Wait for admin UI to load
|
||||
|
||||
```bash
|
||||
agent-browser wait 5000
|
||||
agent-browser snapshot -i
|
||||
```
|
||||
|
||||
You should now see the admin dashboard with:
|
||||
- Statistics table (Database Size, Total Events, PID, etc.)
|
||||
- Navigation buttons (Statistics, Subscriptions, Configuration, Authorization, etc.)
|
||||
- Admin profile area showing "admin" label
|
||||
|
||||
## Navigating Admin Sections
|
||||
|
||||
After login, use the sidebar navigation buttons:
|
||||
|
||||
```bash
|
||||
# View configuration
|
||||
agent-browser click @e8 # Configuration button ref
|
||||
|
||||
# View authorization rules
|
||||
agent-browser click @e9 # Authorization button ref
|
||||
|
||||
# View statistics
|
||||
agent-browser click @e6 # Statistics button ref
|
||||
|
||||
# Always snapshot after navigation to see results
|
||||
agent-browser wait 2500
|
||||
agent-browser snapshot -i
|
||||
```
|
||||
|
||||
## Checking for Errors
|
||||
|
||||
```bash
|
||||
# View browser console logs
|
||||
agent-browser console
|
||||
|
||||
# View JavaScript errors
|
||||
agent-browser errors
|
||||
|
||||
# View relay server logs
|
||||
tail -n 100 relay.log
|
||||
```
|
||||
|
||||
## Chained Command Example (Full Login in One Shot)
|
||||
|
||||
```bash
|
||||
agent-browser open http://127.0.0.1:8888/api/index.html && \
|
||||
agent-browser wait --load networkidle && \
|
||||
agent-browser snapshot -i
|
||||
```
|
||||
|
||||
Then use refs from the snapshot to complete login steps.
|
||||
|
||||
## Tips for AI Agents
|
||||
|
||||
1. **Always use `/api/index.html`** — never the root URL
|
||||
2. **Use `snapshot -i`** after every action to see the current interactive elements and their refs
|
||||
3. **Refs change** between snapshots — always re-snapshot before clicking
|
||||
4. **Wait after clicks** — use `agent-browser wait 2000` (milliseconds) between actions that trigger async operations
|
||||
5. **The login flow has 3 screens**: method selection → key input → success confirmation
|
||||
6. **Console logs are cumulative** — they show all logs since page load, which is useful for debugging admin API responses
|
||||
7. **The relay log** at `relay.log` shows server-side processing of admin commands
|
||||
8. **Command chaining** with `&&` works — the browser daemon persists between commands
|
||||
|
||||
## Verifying Admin API is Working
|
||||
|
||||
After login, the statistics page should show populated data:
|
||||
- Database Size (e.g., "4 KB")
|
||||
- Process ID (the relay PID)
|
||||
- WebSocket Connections count
|
||||
- Memory Usage
|
||||
|
||||
If these show "-" or "Loading...", check:
|
||||
1. `relay.log` for errors
|
||||
2. `agent-browser console` for JavaScript errors
|
||||
3. `agent-browser errors` for page-level errors
|
||||
|
||||
## Closing the Browser
|
||||
|
||||
```bash
|
||||
agent-browser close --all
|
||||
```
|
||||
@@ -324,7 +324,7 @@ int main() {
|
||||
}
|
||||
```
|
||||
|
||||
## Migration Plan for c-relay
|
||||
## Migration Plan for c-relay-pg
|
||||
|
||||
### Phase 1: Extract Debug System
|
||||
1. Create `c_utils_lib` repository
|
||||
@@ -333,16 +333,16 @@ int main() {
|
||||
4. Add basic tests
|
||||
|
||||
### Phase 2: Add Versioning System
|
||||
1. Extract version generation logic from c-relay
|
||||
1. Extract version generation logic from c-relay-pg
|
||||
2. Create reusable version utilities
|
||||
3. Update c-relay to use new system
|
||||
3. Update c-relay-pg to use new system
|
||||
4. Update nostr_core_lib to use new system
|
||||
|
||||
### Phase 3: Add as Submodule
|
||||
1. Add `c_utils_lib` as submodule to c-relay
|
||||
2. Update c-relay Makefile
|
||||
3. Update includes in c-relay source files
|
||||
4. Remove old debug files from c-relay
|
||||
1. Add `c_utils_lib` as submodule to c-relay-pg
|
||||
2. Update c-relay-pg Makefile
|
||||
3. Update includes in c-relay-pg source files
|
||||
4. Remove old debug files from c-relay-pg
|
||||
|
||||
### Phase 4: Documentation & Examples
|
||||
1. Create comprehensive README
|
||||
@@ -352,7 +352,7 @@ int main() {
|
||||
|
||||
## Benefits
|
||||
|
||||
### For c-relay
|
||||
### For c-relay-pg
|
||||
- Cleaner separation of concerns
|
||||
- Reusable utilities across projects
|
||||
- Easier to maintain and test
|
||||
@@ -379,7 +379,7 @@ int main() {
|
||||
- Memory leak detection (valgrind)
|
||||
|
||||
### Integration Tests
|
||||
- Test with real projects (c-relay, nostr_core_lib)
|
||||
- Test with real projects (c-relay-pg, nostr_core_lib)
|
||||
- Cross-platform testing
|
||||
- Performance benchmarks
|
||||
|
||||
@@ -433,7 +433,7 @@ MIT License - permissive and suitable for learning and commercial use.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. ✅ Successfully integrated into c-relay
|
||||
1. ✅ Successfully integrated into c-relay-pg
|
||||
2. ✅ Successfully integrated into nostr_core_lib
|
||||
3. ✅ All tests passing
|
||||
4. ✅ Documentation complete
|
||||
@@ -449,7 +449,7 @@ MIT License - permissive and suitable for learning and commercial use.
|
||||
4. Create build system
|
||||
5. Write tests
|
||||
6. Create documentation
|
||||
7. Integrate into c-relay
|
||||
7. Integrate into c-relay-pg
|
||||
8. Publish to GitHub
|
||||
|
||||
---
|
||||
|
||||
@@ -2,13 +2,13 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay project.
|
||||
This document provides a step-by-step implementation plan for creating the `c_utils_lib` library and integrating it into the c-relay-pg project.
|
||||
|
||||
## Phase 1: Repository Setup & Structure
|
||||
|
||||
### Step 1.1: Create Repository Structure
|
||||
|
||||
**Location**: Create outside c-relay project (sibling directory)
|
||||
**Location**: Create outside c-relay-pg project (sibling directory)
|
||||
|
||||
```bash
|
||||
# Create directory structure
|
||||
@@ -42,7 +42,7 @@ git branch -M main
|
||||
|
||||
### Step 2.1: Move Debug Files
|
||||
|
||||
**Source files** (from c-relay):
|
||||
**Source files** (from c-relay-pg):
|
||||
- `src/debug.c` → `c_utils_lib/src/debug.c`
|
||||
- `src/debug.h` → `c_utils_lib/include/c_utils/debug.h`
|
||||
|
||||
@@ -484,20 +484,20 @@ How to integrate into projects:
|
||||
3. Code examples
|
||||
4. Migration from standalone utilities
|
||||
|
||||
## Phase 7: Integration with c-relay
|
||||
## Phase 7: Integration with c-relay-pg
|
||||
|
||||
### Step 7.1: Add as Submodule
|
||||
|
||||
```bash
|
||||
cd /path/to/c-relay
|
||||
cd /path/to/c-relay-pg
|
||||
git submodule add <repo-url> c_utils_lib
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
### Step 7.2: Update c-relay Makefile
|
||||
### Step 7.2: Update c-relay-pg Makefile
|
||||
|
||||
```makefile
|
||||
# Add to c-relay Makefile
|
||||
# Add to c-relay-pg Makefile
|
||||
C_UTILS_LIB = c_utils_lib/libc_utils.a
|
||||
|
||||
# Update includes
|
||||
@@ -514,7 +514,7 @@ $(C_UTILS_LIB):
|
||||
$(TARGET): $(C_UTILS_LIB) ...
|
||||
```
|
||||
|
||||
### Step 7.3: Update c-relay Source Files
|
||||
### Step 7.3: Update c-relay-pg Source Files
|
||||
|
||||
**Changes needed**:
|
||||
|
||||
@@ -543,7 +543,7 @@ $(TARGET): $(C_UTILS_LIB) ...
|
||||
### Step 7.4: Test Integration
|
||||
|
||||
```bash
|
||||
cd c-relay
|
||||
cd c-relay-pg
|
||||
make clean
|
||||
make
|
||||
./make_and_restart_relay.sh
|
||||
@@ -556,7 +556,7 @@ Verify:
|
||||
|
||||
## Phase 8: Version System Integration
|
||||
|
||||
### Step 8.1: Update c-relay Makefile for Versioning
|
||||
### Step 8.1: Update c-relay-pg Makefile for Versioning
|
||||
|
||||
```makefile
|
||||
# Add version generation
|
||||
@@ -567,7 +567,7 @@ src/version.h: .git/refs/tags/*
|
||||
$(TARGET): src/version.h ...
|
||||
```
|
||||
|
||||
### Step 8.2: Update c-relay to Use Generated Version
|
||||
### Step 8.2: Update c-relay-pg to Use Generated Version
|
||||
|
||||
Replace hardcoded version in `src/main.h` with:
|
||||
```c
|
||||
@@ -583,7 +583,7 @@ Replace hardcoded version in `src/main.h` with:
|
||||
- **Phase 4**: Build System - 2 hours
|
||||
- **Phase 5**: Examples & Tests - 3 hours
|
||||
- **Phase 6**: Documentation - 3 hours
|
||||
- **Phase 7**: c-relay Integration - 2 hours
|
||||
- **Phase 7**: c-relay-pg Integration - 2 hours
|
||||
- **Phase 8**: Version Integration - 2 hours
|
||||
|
||||
**Total**: ~19 hours
|
||||
@@ -593,11 +593,11 @@ Replace hardcoded version in `src/main.h` with:
|
||||
- [ ] c_utils_lib builds successfully
|
||||
- [ ] All tests pass
|
||||
- [ ] Examples compile and run
|
||||
- [ ] c-relay integrates successfully
|
||||
- [ ] Debug output works in c-relay
|
||||
- [ ] c-relay-pg integrates successfully
|
||||
- [ ] Debug output works in c-relay-pg
|
||||
- [ ] Version generation works
|
||||
- [ ] Documentation complete
|
||||
- [ ] No regressions in c-relay functionality
|
||||
- [ ] No regressions in c-relay-pg functionality
|
||||
|
||||
## Next Steps
|
||||
|
||||
@@ -608,7 +608,7 @@ Replace hardcoded version in `src/main.h` with:
|
||||
5. Create build system
|
||||
6. Write tests and examples
|
||||
7. Create documentation
|
||||
8. Integrate into c-relay
|
||||
8. Integrate into c-relay-pg
|
||||
9. Test thoroughly
|
||||
10. Publish to GitHub
|
||||
|
||||
|
||||
@@ -78,9 +78,9 @@ Configuration events follow the standard Nostr event format with kind 33334:
|
||||
|
||||
#### `relay_software`
|
||||
- **Description**: Software identifier for NIP-11
|
||||
- **Default**: `"c-relay"`
|
||||
- **Default**: `"c-relay-pg"`
|
||||
- **Format**: String, max 64 characters
|
||||
- **Example**: `"c-relay v1.0.0"`
|
||||
- **Example**: `"c-relay-pg v1.0.0"`
|
||||
|
||||
#### `relay_version`
|
||||
- **Description**: Software version string
|
||||
@@ -366,7 +366,7 @@ sqlite3 relay.nrdb "SELECT json_pretty(json_object(
|
||||
#### Invalid Parameter Values
|
||||
```bash
|
||||
# Check relay logs for validation errors
|
||||
journalctl -u c-relay | grep "Configuration.*invalid\|Invalid.*configuration"
|
||||
journalctl -u c-relay-pg | grep "Configuration.*invalid\|Invalid.*configuration"
|
||||
|
||||
# Common issues:
|
||||
# - Numeric values outside valid ranges
|
||||
@@ -395,10 +395,10 @@ ORDER BY date DESC;"
|
||||
#### Resource Usage After Changes
|
||||
```bash
|
||||
# Monitor system resources after configuration updates
|
||||
top -p $(pgrep c_relay)
|
||||
top -p $(pgrep c_relay_pg)
|
||||
|
||||
# Check for memory leaks
|
||||
ps aux | grep c_relay | awk '{print $6}' # RSS memory
|
||||
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory
|
||||
```
|
||||
|
||||
### Emergency Recovery
|
||||
@@ -425,7 +425,7 @@ nostrtool event \
|
||||
# If database is corrupted, backup and recreate
|
||||
cp relay.nrdb relay.nrdb.backup
|
||||
rm relay.nrdb*
|
||||
./build/c_relay_x86 # Creates fresh database with new keys
|
||||
./build/c_relay_pg_x86 # Creates fresh database with new keys
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+14
-14
@@ -21,22 +21,22 @@ typedef enum {
|
||||
|
||||
```bash
|
||||
# Production (default - no debug output)
|
||||
./c_relay_x86
|
||||
./c_relay_pg_x86
|
||||
|
||||
# Show errors only
|
||||
./c_relay_x86 --debug-level=1
|
||||
./c_relay_pg_x86 --debug-level=1
|
||||
|
||||
# Show errors and warnings
|
||||
./c_relay_x86 --debug-level=2
|
||||
./c_relay_pg_x86 --debug-level=2
|
||||
|
||||
# Show errors, warnings, and info (recommended for development)
|
||||
./c_relay_x86 --debug-level=3
|
||||
./c_relay_pg_x86 --debug-level=3
|
||||
|
||||
# Show all debug messages
|
||||
./c_relay_x86 --debug-level=4
|
||||
./c_relay_pg_x86 --debug-level=4
|
||||
|
||||
# Show everything including trace with file:line (very verbose)
|
||||
./c_relay_x86 --debug-level=5
|
||||
./c_relay_pg_x86 --debug-level=5
|
||||
```
|
||||
|
||||
## Implementation
|
||||
@@ -475,7 +475,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
|
||||
|
||||
### Level 3 (Errors + Warnings + Info)
|
||||
```
|
||||
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
|
||||
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
|
||||
[2025-01-12 14:30:15] [INFO ] Loading configuration from database
|
||||
[2025-01-12 14:30:15] [ERROR] Failed to open database: permission denied
|
||||
[2025-01-12 14:30:16] [WARN ] Port 8888 unavailable, trying 8889
|
||||
@@ -484,7 +484,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
|
||||
|
||||
### Level 4 (All Debug Messages)
|
||||
```
|
||||
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay v0.4.6
|
||||
[2025-01-12 14:30:15] [INFO ] Initializing C-Relay-PG v0.4.6
|
||||
[2025-01-12 14:30:15] [DEBUG] Opening database: build/abc123...def.db
|
||||
[2025-01-12 14:30:15] [DEBUG] Executing schema initialization
|
||||
[2025-01-12 14:30:15] [INFO ] SQLite WAL mode enabled
|
||||
@@ -496,7 +496,7 @@ When `g_debug_level = 0` (constant), you'll see the compiler has removed all deb
|
||||
|
||||
### Level 5 (Everything Including file:line for ALL messages)
|
||||
```
|
||||
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay v0.4.6
|
||||
[2025-01-12 14:30:15] [INFO ] [main.c:1607] Initializing C-Relay-PG v0.4.6
|
||||
[2025-01-12 14:30:15] [DEBUG] [main.c:348] Opening database: build/abc123...def.db
|
||||
[2025-01-12 14:30:15] [TRACE] [main.c:330] Entering init_database()
|
||||
[2025-01-12 14:30:15] [ERROR] [config.c:125] Database locked
|
||||
@@ -525,11 +525,11 @@ Update the existing `log_*` functions to use the new debug macros
|
||||
make clean && make
|
||||
|
||||
# Test different levels
|
||||
./build/c_relay_x86 # No output
|
||||
./build/c_relay_x86 --debug-level=1 # Errors only
|
||||
./build/c_relay_x86 --debug-level=3 # Info + warnings + errors
|
||||
./build/c_relay_x86 --debug-level=4 # All debug messages
|
||||
./build/c_relay_x86 --debug-level=5 # Everything with file:line on TRACE
|
||||
./build/c_relay_pg_x86 # No output
|
||||
./build/c_relay_pg_x86 --debug-level=1 # Errors only
|
||||
./build/c_relay_pg_x86 --debug-level=3 # Info + warnings + errors
|
||||
./build/c_relay_pg_x86 --debug-level=4 # All debug messages
|
||||
./build/c_relay_pg_x86 --debug-level=5 # Everything with file:line on TRACE
|
||||
```
|
||||
|
||||
### Step 5: Gradual Migration (Ongoing)
|
||||
|
||||
@@ -34,7 +34,7 @@ static const struct {
|
||||
// NIP-11 Relay Information (relay keys will be populated at runtime)
|
||||
{"relay_description", "High-performance C Nostr relay with SQLite storage"},
|
||||
{"relay_contact", ""},
|
||||
{"relay_software", "https://git.laantungir.net/laantungir/c-relay.git"},
|
||||
{"relay_software", "https://git.laantungir.net/laantungir/c-relay-pg.git"},
|
||||
{"relay_version", "v1.0.0"},
|
||||
|
||||
// NIP-13 Proof of Work (pow_min_difficulty = 0 means PoW disabled)
|
||||
|
||||
+51
-51
@@ -52,11 +52,11 @@ sudo apt install -y build-essential git sqlite3 libsqlite3-dev \
|
||||
#### User and Directory Setup
|
||||
```bash
|
||||
# Create dedicated system user
|
||||
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
|
||||
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
|
||||
|
||||
# Create application directory
|
||||
sudo mkdir -p /opt/c-relay
|
||||
sudo chown c-relay:c-relay /opt/c-relay
|
||||
sudo mkdir -p /opt/c-relay-pg
|
||||
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg
|
||||
```
|
||||
|
||||
### Build and Installation
|
||||
@@ -64,8 +64,8 @@ sudo chown c-relay:c-relay /opt/c-relay
|
||||
#### Automated Installation (Recommended)
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
git submodule update --init --recursive
|
||||
|
||||
# Build
|
||||
@@ -81,12 +81,12 @@ sudo systemd/install-service.sh
|
||||
make clean && make
|
||||
|
||||
# Install binary
|
||||
sudo cp build/c_relay_x86 /opt/c-relay/
|
||||
sudo chown c-relay:c-relay /opt/c-relay/c_relay_x86
|
||||
sudo chmod +x /opt/c-relay/c_relay_x86
|
||||
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
|
||||
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
|
||||
sudo chmod +x /opt/c-relay-pg/c_relay_pg_x86
|
||||
|
||||
# Install systemd service
|
||||
sudo cp systemd/c-relay.service /etc/systemd/system/
|
||||
sudo cp systemd/c-relay-pg.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
@@ -95,22 +95,22 @@ sudo systemctl daemon-reload
|
||||
#### Start and Enable Service
|
||||
```bash
|
||||
# Start the service
|
||||
sudo systemctl start c-relay
|
||||
sudo systemctl start c-relay-pg
|
||||
|
||||
# Enable auto-start on boot
|
||||
sudo systemctl enable c-relay
|
||||
sudo systemctl enable c-relay-pg
|
||||
|
||||
# Check status
|
||||
sudo systemctl status c-relay
|
||||
sudo systemctl status c-relay-pg
|
||||
```
|
||||
|
||||
#### Capture Admin Keys (CRITICAL)
|
||||
```bash
|
||||
# View startup logs to get admin keys
|
||||
sudo journalctl -u c-relay --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
|
||||
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep -A 10 "IMPORTANT: SAVE THIS ADMIN PRIVATE KEY"
|
||||
|
||||
# Or check the full log
|
||||
sudo journalctl -u c-relay --no-pager | grep "Admin Private Key"
|
||||
sudo journalctl -u c-relay-pg --no-pager | grep "Admin Private Key"
|
||||
```
|
||||
|
||||
⚠️ **CRITICAL**: Save the admin private key immediately - it's only shown once and is needed for all configuration updates!
|
||||
@@ -151,8 +151,8 @@ sudo iptables-save > /etc/iptables/rules.v4
|
||||
ssh -i your-key.pem ubuntu@your-instance-ip
|
||||
|
||||
# Use the simple deployment script
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
sudo examples/deployment/simple-vps/deploy.sh
|
||||
```
|
||||
|
||||
@@ -168,10 +168,10 @@ sudo examples/deployment/simple-vps/deploy.sh
|
||||
sudo mkfs.ext4 /dev/xvdf
|
||||
sudo mkdir /data
|
||||
sudo mount /dev/xvdf /data
|
||||
sudo chown c-relay:c-relay /data
|
||||
sudo chown c-relay-pg:c-relay-pg /data
|
||||
|
||||
# Update systemd service to use /data
|
||||
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay/WorkingDirectory=\/data/' /etc/systemd/system/c-relay.service
|
||||
sudo sed -i 's/WorkingDirectory=\/opt\/c-relay-pg/WorkingDirectory=\/data/' /etc/systemd/system/c-relay-pg.service
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
@@ -180,7 +180,7 @@ sudo systemctl daemon-reload
|
||||
#### Compute Engine Setup
|
||||
```bash
|
||||
# Create VM instance (e2-micro or larger)
|
||||
gcloud compute instances create c-relay-instance \
|
||||
gcloud compute instances create c-relay-pg-instance \
|
||||
--image-family=ubuntu-2204-lts \
|
||||
--image-project=ubuntu-os-cloud \
|
||||
--machine-type=e2-micro \
|
||||
@@ -193,9 +193,9 @@ gcloud compute firewall-rules create allow-nostr-relay \
|
||||
--target-tags nostr-relay
|
||||
|
||||
# SSH and deploy
|
||||
gcloud compute ssh c-relay-instance
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
gcloud compute ssh c-relay-pg-instance
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
sudo examples/deployment/simple-vps/deploy.sh
|
||||
```
|
||||
|
||||
@@ -203,13 +203,13 @@ sudo examples/deployment/simple-vps/deploy.sh
|
||||
```bash
|
||||
# Create and attach persistent disk
|
||||
gcloud compute disks create relay-data --size=50GB
|
||||
gcloud compute instances attach-disk c-relay-instance --disk=relay-data
|
||||
gcloud compute instances attach-disk c-relay-pg-instance --disk=relay-data
|
||||
|
||||
# Format and mount
|
||||
sudo mkfs.ext4 /dev/sdb
|
||||
sudo mkdir /data
|
||||
sudo mount /dev/sdb /data
|
||||
sudo chown c-relay:c-relay /data
|
||||
sudo chown c-relay-pg:c-relay-pg /data
|
||||
```
|
||||
|
||||
### DigitalOcean
|
||||
@@ -223,8 +223,8 @@ sudo chown c-relay:c-relay /data
|
||||
ssh root@your-droplet-ip
|
||||
|
||||
# Deploy relay
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
examples/deployment/simple-vps/deploy.sh
|
||||
```
|
||||
|
||||
@@ -245,8 +245,8 @@ The `examples/deployment/` directory contains ready-to-use scripts:
|
||||
### Simple VPS Deployment
|
||||
```bash
|
||||
# Clone repository and run automated deployment
|
||||
git clone https://github.com/your-org/c-relay.git
|
||||
cd c-relay
|
||||
git clone https://github.com/your-org/c-relay-pg.git
|
||||
cd c-relay-pg
|
||||
sudo examples/deployment/simple-vps/deploy.sh
|
||||
```
|
||||
|
||||
@@ -417,9 +417,9 @@ LOG_FILE="/var/log/relay-monitor.log"
|
||||
DATE=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
|
||||
# Check if relay is running
|
||||
if ! pgrep -f "c_relay_x86" > /dev/null; then
|
||||
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
|
||||
echo "[$DATE] ERROR: Relay process not running" >> $LOG_FILE
|
||||
systemctl restart c-relay
|
||||
systemctl restart c-relay-pg
|
||||
fi
|
||||
|
||||
# Check port availability
|
||||
@@ -428,14 +428,14 @@ if ! netstat -tln | grep -q ":8888"; then
|
||||
fi
|
||||
|
||||
# Check database file
|
||||
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
|
||||
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
|
||||
if [[ -n "$RELAY_DB" ]]; then
|
||||
DB_SIZE=$(du -h "$RELAY_DB" | cut -f1)
|
||||
echo "[$DATE] INFO: Database size: $DB_SIZE" >> $LOG_FILE
|
||||
fi
|
||||
|
||||
# Check memory usage
|
||||
MEM_USAGE=$(ps aux | grep c_relay_x86 | grep -v grep | awk '{print $6}')
|
||||
MEM_USAGE=$(ps aux | grep c_relay_pg_x86 | grep -v grep | awk '{print $6}')
|
||||
if [[ -n "$MEM_USAGE" ]]; then
|
||||
echo "[$DATE] INFO: Memory usage: ${MEM_USAGE}KB" >> $LOG_FILE
|
||||
fi
|
||||
@@ -454,8 +454,8 @@ sudo chmod +x /usr/local/bin/relay-monitor.sh
|
||||
|
||||
#### Centralized Logging with rsyslog
|
||||
```bash
|
||||
# /etc/rsyslog.d/50-c-relay.conf
|
||||
if $programname == 'c-relay' then /var/log/c-relay.log
|
||||
# /etc/rsyslog.d/50-c-relay-pg.conf
|
||||
if $programname == 'c-relay-pg' then /var/log/c-relay-pg.log
|
||||
& stop
|
||||
```
|
||||
|
||||
@@ -465,7 +465,7 @@ if $programname == 'c-relay' then /var/log/c-relay.log
|
||||
```yaml
|
||||
# /etc/prometheus/prometheus.yml
|
||||
scrape_configs:
|
||||
- job_name: 'c-relay'
|
||||
- job_name: 'c-relay-pg'
|
||||
static_configs:
|
||||
- targets: ['localhost:8888']
|
||||
metrics_path: '/metrics' # If implemented
|
||||
@@ -479,30 +479,30 @@ scrape_configs:
|
||||
#### Service User Restrictions
|
||||
```bash
|
||||
# Restrict service user
|
||||
sudo usermod -s /bin/false c-relay
|
||||
sudo usermod -d /opt/c-relay c-relay
|
||||
sudo usermod -s /bin/false c-relay-pg
|
||||
sudo usermod -d /opt/c-relay-pg c-relay-pg
|
||||
|
||||
# Set proper permissions
|
||||
sudo chmod 700 /opt/c-relay
|
||||
sudo chown -R c-relay:c-relay /opt/c-relay
|
||||
sudo chmod 700 /opt/c-relay-pg
|
||||
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
|
||||
```
|
||||
|
||||
#### File System Restrictions
|
||||
```bash
|
||||
# Mount data directory with appropriate options
|
||||
echo "/dev/sdb /opt/c-relay ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
|
||||
echo "/dev/sdb /opt/c-relay-pg ext4 defaults,noexec,nosuid,nodev 0 2" >> /etc/fstab
|
||||
```
|
||||
|
||||
### Network Security
|
||||
|
||||
#### Fail2Ban Configuration
|
||||
```ini
|
||||
# /etc/fail2ban/jail.d/c-relay.conf
|
||||
[c-relay-dos]
|
||||
# /etc/fail2ban/jail.d/c-relay-pg.conf
|
||||
[c-relay-pg-dos]
|
||||
enabled = true
|
||||
port = 8888
|
||||
filter = c-relay-dos
|
||||
logpath = /var/log/c-relay.log
|
||||
filter = c-relay-pg-dos
|
||||
logpath = /var/log/c-relay-pg.log
|
||||
maxretry = 10
|
||||
findtime = 60
|
||||
bantime = 300
|
||||
@@ -534,9 +534,9 @@ sudo mkfs.ext4 /dev/mapper/relay-data
|
||||
#!/bin/bash
|
||||
# /usr/local/bin/backup-relay.sh
|
||||
|
||||
BACKUP_DIR="/backup/c-relay"
|
||||
BACKUP_DIR="/backup/c-relay-pg"
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
RELAY_DB=$(find /opt/c-relay -name "*.nrdb" | head -1)
|
||||
RELAY_DB=$(find /opt/c-relay-pg -name "*.nrdb" | head -1)
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
@@ -574,7 +574,7 @@ sudo apt install -y awscli
|
||||
aws configure
|
||||
|
||||
# Sync backups to S3
|
||||
aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
|
||||
aws s3 sync /backup/c-relay-pg/ s3://your-backup-bucket/c-relay-pg/ --delete
|
||||
```
|
||||
|
||||
### Disaster Recovery
|
||||
@@ -583,16 +583,16 @@ aws s3 sync /backup/c-relay/ s3://your-backup-bucket/c-relay/ --delete
|
||||
```bash
|
||||
# 1. Restore from backup
|
||||
gunzip backup/relay_backup_20231201_020000.nrdb.gz
|
||||
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay/
|
||||
cp backup/relay_backup_20231201_020000.nrdb /opt/c-relay-pg/
|
||||
|
||||
# 2. Fix permissions
|
||||
sudo chown c-relay:c-relay /opt/c-relay/*.nrdb
|
||||
sudo chown c-relay-pg:c-relay-pg /opt/c-relay-pg/*.nrdb
|
||||
|
||||
# 3. Restart service
|
||||
sudo systemctl restart c-relay
|
||||
sudo systemctl restart c-relay-pg
|
||||
|
||||
# 4. Verify recovery
|
||||
sudo journalctl -u c-relay --since "1 minute ago"
|
||||
sudo journalctl -u c-relay-pg --since "1 minute ago"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+25
-25
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This guide explains how to build truly portable MUSL-based static binaries of c-relay using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
|
||||
This guide explains how to build truly portable MUSL-based static binaries of c-relay-pg using Alpine Linux Docker containers. These binaries have **zero runtime dependencies** and work on any Linux distribution.
|
||||
|
||||
## Why MUSL?
|
||||
|
||||
@@ -36,8 +36,8 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
|
||||
./build_static.sh
|
||||
|
||||
# The binary will be created at:
|
||||
# build/c_relay_static_musl_x86_64 (on x86_64)
|
||||
# build/c_relay_static_musl_arm64 (on ARM64)
|
||||
# build/c_relay_pg_static_musl_x86_64 (on x86_64)
|
||||
# build/c_relay_pg_static_musl_arm64 (on ARM64)
|
||||
```
|
||||
|
||||
### What Happens During Build
|
||||
@@ -56,7 +56,7 @@ This guide explains how to build truly portable MUSL-based static binaries of c-
|
||||
- Includes required NIPs: 001, 006, 013, 017, 019, 044, 059
|
||||
- Produces static library (~316KB)
|
||||
|
||||
4. **c-relay Compilation**: Links everything statically:
|
||||
4. **c-relay-pg Compilation**: Links everything statically:
|
||||
- All source files compiled with `-static` flag
|
||||
- Fortification disabled to avoid `__*_chk` symbols
|
||||
- Results in ~7.6MB stripped binary
|
||||
@@ -78,7 +78,7 @@ FROM alpine:3.19 AS builder
|
||||
- Install build tools and static libraries
|
||||
- Build dependencies from source
|
||||
- Compile nostr_core_lib with MUSL flags
|
||||
- Compile c-relay with full static linking
|
||||
- Compile c-relay-pg with full static linking
|
||||
- Strip binary to reduce size
|
||||
|
||||
# Stage 2: Output (scratch)
|
||||
@@ -93,7 +93,7 @@ FROM scratch AS output
|
||||
CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"
|
||||
```
|
||||
|
||||
**For c-relay:**
|
||||
**For c-relay-pg:**
|
||||
```bash
|
||||
gcc -static -O2 -Wall -Wextra -std=c99 \
|
||||
-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 \
|
||||
@@ -110,7 +110,7 @@ gcc -static -O2 -Wall -Wextra -std=c99 \
|
||||
- `-U_FORTIFY_SOURCE` (undefine any existing definition)
|
||||
- `-D_FORTIFY_SOURCE=0` (set to 0)
|
||||
|
||||
This must be applied to **both** nostr_core_lib and c-relay compilation.
|
||||
This must be applied to **both** nostr_core_lib and c-relay-pg compilation.
|
||||
|
||||
### NIP Dependencies
|
||||
|
||||
@@ -129,26 +129,26 @@ The build includes these NIPs in nostr_core_lib:
|
||||
|
||||
```bash
|
||||
# Should show "statically linked"
|
||||
file build/c_relay_static_musl_x86_64
|
||||
file build/c_relay_pg_static_musl_x86_64
|
||||
|
||||
# Should show "not a dynamic executable"
|
||||
ldd build/c_relay_static_musl_x86_64
|
||||
ldd build/c_relay_pg_static_musl_x86_64
|
||||
|
||||
# Check size (should be ~7.6MB)
|
||||
ls -lh build/c_relay_static_musl_x86_64
|
||||
ls -lh build/c_relay_pg_static_musl_x86_64
|
||||
```
|
||||
|
||||
### Test Execution
|
||||
|
||||
```bash
|
||||
# Show help
|
||||
./build/c_relay_static_musl_x86_64 --help
|
||||
./build/c_relay_pg_static_musl_x86_64 --help
|
||||
|
||||
# Show version
|
||||
./build/c_relay_static_musl_x86_64 --version
|
||||
./build/c_relay_pg_static_musl_x86_64 --version
|
||||
|
||||
# Run relay
|
||||
./build/c_relay_static_musl_x86_64 --port 8888
|
||||
./build/c_relay_pg_static_musl_x86_64 --port 8888
|
||||
```
|
||||
|
||||
### Cross-Distribution Testing
|
||||
@@ -157,16 +157,16 @@ Test the binary on different distributions to verify portability:
|
||||
|
||||
```bash
|
||||
# Alpine Linux
|
||||
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_static_musl_x86_64 --version
|
||||
docker run --rm -v $(pwd)/build:/app alpine:latest /app/c_relay_pg_static_musl_x86_64 --version
|
||||
|
||||
# Ubuntu
|
||||
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_static_musl_x86_64 --version
|
||||
docker run --rm -v $(pwd)/build:/app ubuntu:latest /app/c_relay_pg_static_musl_x86_64 --version
|
||||
|
||||
# Debian
|
||||
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_static_musl_x86_64 --version
|
||||
docker run --rm -v $(pwd)/build:/app debian:latest /app/c_relay_pg_static_musl_x86_64 --version
|
||||
|
||||
# CentOS
|
||||
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_static_musl_x86_64 --version
|
||||
docker run --rm -v $(pwd)/build:/app centos:latest /app/c_relay_pg_static_musl_x86_64 --version
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -187,7 +187,7 @@ newgrp docker # Or logout and login again
|
||||
|
||||
**Solution**: Ensure fortification is disabled in both:
|
||||
1. nostr_core_lib build.sh (line 534)
|
||||
2. c-relay compilation flags in Dockerfile
|
||||
2. c-relay-pg compilation flags in Dockerfile
|
||||
|
||||
### Binary Won't Execute
|
||||
|
||||
@@ -213,26 +213,26 @@ newgrp docker # Or logout and login again
|
||||
|
||||
```bash
|
||||
# Copy binary to server
|
||||
scp build/c_relay_static_musl_x86_64 user@server:/opt/c-relay/
|
||||
scp build/c_relay_pg_static_musl_x86_64 user@server:/opt/c-relay-pg/
|
||||
|
||||
# Run on server (no dependencies needed!)
|
||||
ssh user@server
|
||||
cd /opt/c-relay
|
||||
./c_relay_static_musl_x86_64 --port 8888
|
||||
cd /opt/c-relay-pg
|
||||
./c_relay_pg_static_musl_x86_64 --port 8888
|
||||
```
|
||||
|
||||
### SystemD Service
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=C-Relay Nostr Relay (MUSL Static)
|
||||
Description=C-Relay-PG Nostr Relay (MUSL Static)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=c-relay
|
||||
WorkingDirectory=/opt/c-relay
|
||||
ExecStart=/opt/c-relay/c_relay_static_musl_x86_64
|
||||
User=c-relay-pg
|
||||
WorkingDirectory=/opt/c-relay-pg
|
||||
ExecStart=/opt/c-relay-pg/c_relay_pg_static_musl_x86_64
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ We chose Option B because:
|
||||
|
||||
## Detailed Implementation Steps
|
||||
|
||||
### Phase 1: Configuration Setup in c-relay
|
||||
### Phase 1: Configuration Setup in c-relay-pg
|
||||
|
||||
#### 1.1 Add Configuration Parameter
|
||||
**File:** `src/default_config_event.h`
|
||||
@@ -293,7 +293,7 @@ int nostr_nip17_send_dm(cJSON* dm_event,
|
||||
|
||||
---
|
||||
|
||||
### Phase 4: Update c-relay Call Sites
|
||||
### Phase 4: Update c-relay-pg Call Sites
|
||||
|
||||
#### 4.1 Update src/api.c
|
||||
**Location:** Line 1319
|
||||
@@ -470,7 +470,7 @@ causing compatibility issues.
|
||||
- [ ] Update `nostr_nip17_send_dm()` signature and implementation
|
||||
- [ ] Update `nip017.h` function declaration and documentation
|
||||
|
||||
### c-relay Changes
|
||||
### c-relay-pg Changes
|
||||
- [ ] Add `nip59_timestamp_max_delay_sec` to `default_config_event.h`
|
||||
- [ ] Add validation in `config.c` for new parameter
|
||||
- [ ] Update `src/api.c` call site to pass `max_delay_sec`
|
||||
@@ -494,7 +494,7 @@ causing compatibility issues.
|
||||
|
||||
If issues arise:
|
||||
1. Revert nostr_core_lib changes (git revert in submodule)
|
||||
2. Revert c-relay changes
|
||||
2. Revert c-relay-pg changes
|
||||
3. Configuration parameter will be ignored if not used
|
||||
4. Default behavior (0) provides maximum compatibility
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the design for a general-purpose SQL query interface for the C-Relay admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
|
||||
This document describes the design for a general-purpose SQL query interface for the C-Relay-PG admin API. This allows administrators to execute read-only SQL queries against the relay database through cryptographically signed kind 23456 events with NIP-44 encrypted command arrays.
|
||||
|
||||
## Security Model
|
||||
|
||||
@@ -271,7 +271,7 @@ const SQL_QUERY_TEMPLATES = {
|
||||
};
|
||||
|
||||
// Query history management (localStorage)
|
||||
const QUERY_HISTORY_KEY = 'c_relay_sql_history';
|
||||
const QUERY_HISTORY_KEY = 'c_relay_pg_sql_history';
|
||||
const MAX_HISTORY_ITEMS = 20;
|
||||
|
||||
// Load query history from localStorage
|
||||
|
||||
@@ -224,7 +224,7 @@ The script should:
|
||||
- `curl` or `websocat` for WebSocket communication
|
||||
- `jq` for JSON parsing
|
||||
- Nostr CLI tools (optional, for event signing)
|
||||
- Running c-relay instance
|
||||
- Running c-relay-pg instance
|
||||
|
||||
## Example Output
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# C-Relay Complete Startup Flow Documentation
|
||||
# C-Relay-PG Complete Startup Flow Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
C-Relay has two distinct startup paths:
|
||||
C-Relay-PG has two distinct startup paths:
|
||||
1. **First-Time Startup**: No database exists, generates keys and initializes system
|
||||
2. **Existing Relay Startup**: Database exists, loads configuration and resumes operation
|
||||
|
||||
@@ -220,7 +220,7 @@ int populate_default_config_values(void) {
|
||||
|
||||
struct config_default defaults[] = {
|
||||
{"relay_port", "8888", "integer", "WebSocket port", "network", 1},
|
||||
{"relay_name", "C-Relay", "string", "Relay name", "info", 0},
|
||||
{"relay_name", "C-Relay-PG", "string", "Relay name", "info", 0},
|
||||
{"relay_description", "High-performance C Nostr relay", "string", "Description", "info", 0},
|
||||
{"max_subscriptions_per_client", "25", "integer", "Max subs per client", "limits", 0},
|
||||
{"pow_min_difficulty", "0", "integer", "Minimum PoW difficulty", "security", 0},
|
||||
@@ -983,7 +983,7 @@ if (current_version < LATEST_VERSION) {
|
||||
|
||||
**Solutions**:
|
||||
- Use `--port <number>` to specify different port
|
||||
- Kill existing process: `pkill -f c_relay_`
|
||||
- Kill existing process: `pkill -f c_relay_pg_`
|
||||
- Force kill port: `fuser -k 8888/tcp`
|
||||
- Use `--strict-port` to fail fast instead of trying fallback ports
|
||||
|
||||
@@ -994,7 +994,7 @@ if (current_version < LATEST_VERSION) {
|
||||
**Solutions**:
|
||||
- Kill existing relay processes
|
||||
- Remove WAL files: `rm build/*.db-wal build/*.db-shm`
|
||||
- Check for stale processes: `ps aux | grep c_relay_`
|
||||
- Check for stale processes: `ps aux | grep c_relay_pg_`
|
||||
|
||||
#### 3. Missing Admin Private Key
|
||||
|
||||
@@ -1071,7 +1071,7 @@ Typical startup times (on modern hardware):
|
||||
|
||||
## Summary
|
||||
|
||||
The c-relay startup system is designed for:
|
||||
The c-relay-pg startup system is designed for:
|
||||
|
||||
1. **Security**: Admin keys never stored, relay keys encrypted
|
||||
2. **Reliability**: Automatic port fallback, schema migrations
|
||||
|
||||
@@ -18,10 +18,10 @@ The script now attempts to build with `musl-gcc` for truly portable static binar
|
||||
|
||||
SQLite is now built once and cached for future builds:
|
||||
|
||||
- **Cache location**: `~/.cache/c-relay-sqlite/`
|
||||
- **Cache location**: `~/.cache/c-relay-pg-sqlite/`
|
||||
- **Version-specific**: Each SQLite version gets its own cache directory
|
||||
- **Significant speedup**: Subsequent builds skip the SQLite compilation step
|
||||
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-sqlite` to clear cache
|
||||
- **Manual cleanup**: `rm -rf ~/.cache/c-relay-pg-sqlite` to clear cache
|
||||
|
||||
### 3. Smart Package Installation
|
||||
|
||||
@@ -51,13 +51,13 @@ The script will:
|
||||
## Binary Types
|
||||
|
||||
### MUSL Static Binary (Ideal - Currently Not Achievable)
|
||||
- **Filename**: `build/c_relay_static_musl_x86_64`
|
||||
- **Filename**: `build/c_relay_pg_static_musl_x86_64`
|
||||
- **Dependencies**: None (truly static)
|
||||
- **Portability**: Works on any Linux distribution
|
||||
- **Status**: Requires MUSL-compiled libwebsockets and other dependencies (not available by default)
|
||||
|
||||
### Glibc Static Binary (Current Output)
|
||||
- **Filename**: `build/c_relay_static_x86_64` or `build/c_relay_static_glibc_x86_64`
|
||||
- **Filename**: `build/c_relay_pg_static_x86_64` or `build/c_relay_pg_static_glibc_x86_64`
|
||||
- **Dependencies**: None - fully statically linked with glibc
|
||||
- **Portability**: Works on most Linux distributions (glibc is statically included)
|
||||
- **Note**: Despite using glibc, this is a **fully static binary** with no runtime dependencies
|
||||
@@ -68,11 +68,11 @@ The script automatically verifies binaries using `ldd` and `file`:
|
||||
|
||||
```bash
|
||||
# For MUSL binary
|
||||
ldd build/c_relay_static_musl_x86_64
|
||||
ldd build/c_relay_pg_static_musl_x86_64
|
||||
# Output: "not a dynamic executable" (good!)
|
||||
|
||||
# For glibc binary
|
||||
ldd build/c_relay_static_glibc_x86_64
|
||||
ldd build/c_relay_pg_static_glibc_x86_64
|
||||
# Output: Shows glibc dependencies
|
||||
```
|
||||
|
||||
@@ -112,7 +112,7 @@ The script attempts MUSL compilation but falls back to glibc:
|
||||
|
||||
### Clear SQLite Cache
|
||||
```bash
|
||||
rm -rf ~/.cache/c-relay-sqlite
|
||||
rm -rf ~/.cache/c-relay-pg-sqlite
|
||||
```
|
||||
|
||||
### Force Package Reinstall
|
||||
@@ -127,8 +127,8 @@ cat /tmp/musl_build.log
|
||||
|
||||
### Verify Binary Type
|
||||
```bash
|
||||
file build/c_relay_static_*
|
||||
ldd build/c_relay_static_* 2>&1
|
||||
file build/c_relay_pg_static_*
|
||||
ldd build/c_relay_pg_static_* 2>&1
|
||||
```
|
||||
|
||||
## Performance Impact
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Problem Summary
|
||||
|
||||
The c-relay Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
|
||||
The c-relay-pg Nostr relay experienced severe performance degradation (90-100% CPU) due to subscription accumulation in the database. Investigation revealed **323,644 orphaned subscriptions** that were never properly closed when WebSocket connections dropped.
|
||||
|
||||
## Solution: Two-Component Approach
|
||||
|
||||
|
||||
@@ -586,7 +586,7 @@ int add_pubkeys_to_config_table(void) {
|
||||
rm -f *.db
|
||||
|
||||
# Start relay with defaults
|
||||
./build/c_relay_x86
|
||||
./build/c_relay_pg_x86
|
||||
|
||||
# Verify config table complete
|
||||
sqlite3 <relay_pubkey>.db "SELECT COUNT(*) FROM config;"
|
||||
@@ -602,7 +602,7 @@ int add_pubkeys_to_config_table(void) {
|
||||
rm -f *.db
|
||||
|
||||
# Start relay with port override
|
||||
./build/c_relay_x86 --port 9999
|
||||
./build/c_relay_pg_x86 --port 9999
|
||||
|
||||
# Verify port override applied
|
||||
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
|
||||
@@ -612,13 +612,13 @@ int add_pubkeys_to_config_table(void) {
|
||||
3. **Restart with Existing Database**
|
||||
```bash
|
||||
# Start relay (creates database)
|
||||
./build/c_relay_x86
|
||||
./build/c_relay_pg_x86
|
||||
|
||||
# Stop relay
|
||||
pkill -f c_relay_
|
||||
pkill -f c_relay_pg_
|
||||
|
||||
# Restart relay
|
||||
./build/c_relay_x86
|
||||
./build/c_relay_pg_x86
|
||||
|
||||
# Verify config unchanged
|
||||
# Check relay.log for validation message
|
||||
@@ -627,13 +627,13 @@ int add_pubkeys_to_config_table(void) {
|
||||
4. **Restart with CLI Overrides**
|
||||
```bash
|
||||
# Start relay (creates database)
|
||||
./build/c_relay_x86
|
||||
./build/c_relay_pg_x86
|
||||
|
||||
# Stop relay
|
||||
pkill -f c_relay_
|
||||
pkill -f c_relay_pg_
|
||||
|
||||
# Restart with port override
|
||||
./build/c_relay_x86 --port 9999
|
||||
./build/c_relay_pg_x86 --port 9999
|
||||
|
||||
# Verify port override applied
|
||||
sqlite3 <relay_pubkey>.db "SELECT value FROM config WHERE key='relay_port';"
|
||||
|
||||
+32
-32
@@ -19,12 +19,12 @@ Complete guide for deploying, configuring, and managing the C Nostr Relay with e
|
||||
```bash
|
||||
# Clone and build
|
||||
git clone <repository-url>
|
||||
cd c-relay
|
||||
cd c-relay-pg
|
||||
git submodule update --init --recursive
|
||||
make
|
||||
|
||||
# Start relay (zero configuration needed)
|
||||
./build/c_relay_x86
|
||||
./build/c_relay_pg_x86
|
||||
```
|
||||
|
||||
### 2. First Startup - Save Keys
|
||||
@@ -78,7 +78,7 @@ brew install git sqlite libwebsockets openssl libsecp256k1 curl zlib
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone <repository-url>
|
||||
cd c-relay
|
||||
cd c-relay-pg
|
||||
|
||||
# Initialize submodules
|
||||
git submodule update --init --recursive
|
||||
@@ -87,7 +87,7 @@ git submodule update --init --recursive
|
||||
make clean && make
|
||||
|
||||
# Verify build
|
||||
ls -la build/c_relay_x86
|
||||
ls -la build/c_relay_pg_x86
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
@@ -98,27 +98,27 @@ ls -la build/c_relay_x86
|
||||
sudo systemd/install-service.sh
|
||||
|
||||
# Start service
|
||||
sudo systemctl start c-relay
|
||||
sudo systemctl start c-relay-pg
|
||||
|
||||
# Enable auto-start
|
||||
sudo systemctl enable c-relay
|
||||
sudo systemctl enable c-relay-pg
|
||||
|
||||
# Check status
|
||||
sudo systemctl status c-relay
|
||||
sudo systemctl status c-relay-pg
|
||||
```
|
||||
|
||||
#### Manual Deployment
|
||||
```bash
|
||||
# Create dedicated user
|
||||
sudo useradd --system --home-dir /opt/c-relay --shell /bin/false c-relay
|
||||
sudo useradd --system --home-dir /opt/c-relay-pg --shell /bin/false c-relay-pg
|
||||
|
||||
# Install binary
|
||||
sudo mkdir -p /opt/c-relay
|
||||
sudo cp build/c_relay_x86 /opt/c-relay/
|
||||
sudo chown -R c-relay:c-relay /opt/c-relay
|
||||
sudo mkdir -p /opt/c-relay-pg
|
||||
sudo cp build/c_relay_pg_x86 /opt/c-relay-pg/
|
||||
sudo chown -R c-relay-pg:c-relay-pg /opt/c-relay-pg
|
||||
|
||||
# Run as service user
|
||||
sudo -u c-relay /opt/c-relay/c_relay_x86
|
||||
sudo -u c-relay-pg /opt/c-relay-pg/c_relay_pg_x86
|
||||
```
|
||||
|
||||
## Configuration Management
|
||||
@@ -188,7 +188,7 @@ Send this to your relay via WebSocket, and changes are applied immediately.
|
||||
|-----------|-------------|---------|---------|
|
||||
| `relay_description` | Relay description for NIP-11 | "C Nostr Relay" | "My awesome relay" |
|
||||
| `relay_contact` | Admin contact information | "" | "admin@example.com" |
|
||||
| `relay_software` | Software identifier | "c-relay" | "c-relay v1.0" |
|
||||
| `relay_software` | Software identifier | "c-relay-pg" | "c-relay-pg v1.0" |
|
||||
|
||||
#### Client Limits
|
||||
| Parameter | Description | Default | Range |
|
||||
@@ -273,7 +273,7 @@ chmod 600 admin_keys_backup_*.txt
|
||||
#### Key Recovery
|
||||
If you lose your admin private key:
|
||||
|
||||
1. **Stop the relay**: `pkill c_relay` or `sudo systemctl stop c-relay`
|
||||
1. **Stop the relay**: `pkill c_relay_pg` or `sudo systemctl stop c-relay-pg`
|
||||
2. **Backup events**: `cp <relay_pubkey>.nrdb backup_$(date +%Y%m%d).nrdb`
|
||||
3. **Remove database**: `rm <relay_pubkey>.nrdb*`
|
||||
4. **Restart relay**: This creates new database with new keys
|
||||
@@ -300,7 +300,7 @@ sudo ufw allow 8888/tcp
|
||||
```bash
|
||||
# Secure database file permissions
|
||||
chmod 600 <relay_pubkey>.nrdb
|
||||
chown c-relay:c-relay <relay_pubkey>.nrdb
|
||||
chown c-relay-pg:c-relay-pg <relay_pubkey>.nrdb
|
||||
|
||||
# Regular backups
|
||||
cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
|
||||
@@ -311,10 +311,10 @@ cp <relay_pubkey>.nrdb backup/relay_backup_$(date +%Y%m%d_%H%M%S).nrdb
|
||||
### Service Status
|
||||
```bash
|
||||
# Check if relay is running
|
||||
ps aux | grep c_relay
|
||||
ps aux | grep c_relay_pg
|
||||
|
||||
# SystemD status
|
||||
sudo systemctl status c-relay
|
||||
sudo systemctl status c-relay-pg
|
||||
|
||||
# Network connections
|
||||
netstat -tln | grep 8888
|
||||
@@ -324,16 +324,16 @@ sudo ss -tlpn | grep 8888
|
||||
### Log Monitoring
|
||||
```bash
|
||||
# Real-time logs (systemd)
|
||||
sudo journalctl -u c-relay -f
|
||||
sudo journalctl -u c-relay-pg -f
|
||||
|
||||
# Recent logs
|
||||
sudo journalctl -u c-relay --since "1 hour ago"
|
||||
sudo journalctl -u c-relay-pg --since "1 hour ago"
|
||||
|
||||
# Error logs only
|
||||
sudo journalctl -u c-relay -p err
|
||||
sudo journalctl -u c-relay-pg -p err
|
||||
|
||||
# Configuration changes
|
||||
sudo journalctl -u c-relay | grep "Configuration updated via kind 33334"
|
||||
sudo journalctl -u c-relay-pg | grep "Configuration updated via kind 33334"
|
||||
```
|
||||
|
||||
### Database Analytics
|
||||
@@ -365,13 +365,13 @@ ORDER BY created_at DESC;
|
||||
du -sh <relay_pubkey>.nrdb*
|
||||
|
||||
# Memory usage
|
||||
ps aux | grep c_relay | awk '{print $6}' # RSS memory in KB
|
||||
ps aux | grep c_relay_pg | awk '{print $6}' # RSS memory in KB
|
||||
|
||||
# Connection count (approximate)
|
||||
netstat -an | grep :8888 | grep ESTABLISHED | wc -l
|
||||
|
||||
# System resources
|
||||
top -p $(pgrep c_relay)
|
||||
top -p $(pgrep c_relay_pg)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
@@ -385,11 +385,11 @@ netstat -tln | grep 8888
|
||||
# If port in use, find process: sudo lsof -i :8888
|
||||
|
||||
# Check binary permissions
|
||||
ls -la build/c_relay_x86
|
||||
chmod +x build/c_relay_x86
|
||||
ls -la build/c_relay_pg_x86
|
||||
chmod +x build/c_relay_pg_x86
|
||||
|
||||
# Check dependencies
|
||||
ldd build/c_relay_x86
|
||||
ldd build/c_relay_pg_x86
|
||||
```
|
||||
|
||||
#### Configuration Not Updating
|
||||
@@ -442,7 +442,7 @@ sqlite3 recovered.nrdb < recovered.sql
|
||||
|
||||
# If repair fails, start fresh (loses all events)
|
||||
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.corrupted
|
||||
./build/c_relay_x86 # Creates new database
|
||||
./build/c_relay_pg_x86 # Creates new database
|
||||
```
|
||||
|
||||
#### Lost Configuration Recovery
|
||||
@@ -455,12 +455,12 @@ If configuration is lost but database is intact:
|
||||
#### Emergency Restart
|
||||
```bash
|
||||
# Quick restart with clean state
|
||||
sudo systemctl stop c-relay
|
||||
sudo systemctl stop c-relay-pg
|
||||
mv <relay_pubkey>.nrdb <relay_pubkey>.nrdb.backup
|
||||
sudo systemctl start c-relay
|
||||
sudo systemctl start c-relay-pg
|
||||
|
||||
# Check logs for new admin keys
|
||||
sudo journalctl -u c-relay --since "5 minutes ago" | grep "Admin Private Key"
|
||||
sudo journalctl -u c-relay-pg --since "5 minutes ago" | grep "Admin Private Key"
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
@@ -503,7 +503,7 @@ ws.on('open', function() {
|
||||
# backup-relay.sh
|
||||
DATE=$(date +%Y%m%d_%H%M%S)
|
||||
DB_FILE=$(ls *.nrdb | head -1)
|
||||
BACKUP_DIR="/backup/c-relay"
|
||||
BACKUP_DIR="/backup/c-relay-pg"
|
||||
|
||||
mkdir -p $BACKUP_DIR
|
||||
cp $DB_FILE $BACKUP_DIR/relay_backup_$DATE.nrdb
|
||||
@@ -533,7 +533,7 @@ tar czf relay_migration.tar.gz *.nrdb* relay.log
|
||||
|
||||
# Target server
|
||||
tar xzf relay_migration.tar.gz
|
||||
./build/c_relay_x86 # Will detect existing database and continue
|
||||
./build/c_relay_pg_x86 # Will detect existing database and continue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Script to embed web files into C headers for the C-Relay admin interface
|
||||
# Script to embed web files into C headers for the C-Relay-PG admin interface
|
||||
# Converts HTML, CSS, and JS files from api/ directory into C byte arrays
|
||||
|
||||
set -e
|
||||
|
||||
@@ -60,7 +60,7 @@ All examples assume the event-based configuration system where:
|
||||
- **Save Admin Keys**: All deployment examples emphasize capturing the admin private key on first startup
|
||||
- **Firewall Configuration**: Examples include proper firewall rules
|
||||
- **SSL/TLS**: Production examples include HTTPS configuration
|
||||
- **User Isolation**: Service runs as dedicated `c-relay` system user
|
||||
- **User Isolation**: Service runs as dedicated `c-relay-pg` system user
|
||||
|
||||
## Support
|
||||
|
||||
|
||||
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Default configuration
|
||||
RELAY_DIR="/opt/c-relay"
|
||||
BACKUP_DIR="/backup/c-relay"
|
||||
RELAY_DIR="/opt/c-relay-pg"
|
||||
BACKUP_DIR="/backup/c-relay-pg"
|
||||
RETENTION_DAYS="30"
|
||||
COMPRESS="true"
|
||||
REMOTE_BACKUP=""
|
||||
@@ -47,8 +47,8 @@ show_help() {
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo
|
||||
echo "Options:"
|
||||
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
|
||||
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay)"
|
||||
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
|
||||
echo " -b, --backup-dir DIR Backup directory (default: /backup/c-relay-pg)"
|
||||
echo " -r, --retention DAYS Retention period in days (default: 30)"
|
||||
echo " -n, --no-compress Don't compress backups"
|
||||
echo " -s, --s3-bucket BUCKET Upload to S3 bucket"
|
||||
@@ -231,7 +231,7 @@ upload_to_s3() {
|
||||
|
||||
print_step "Uploading backup to S3..."
|
||||
|
||||
local s3_path="s3://$S3_BUCKET/c-relay/$(date +%Y)/$(date +%m)/"
|
||||
local s3_path="s3://$S3_BUCKET/c-relay-pg/$(date +%Y)/$(date +%m)/"
|
||||
|
||||
if aws s3 cp "$BACKUP_FILE" "$s3_path" --storage-class STANDARD_IA; then
|
||||
print_success "Backup uploaded to S3: $s3_path"
|
||||
@@ -264,7 +264,7 @@ cleanup_old_backups() {
|
||||
print_step "Cleaning S3 backups older than $cutoff_date..."
|
||||
|
||||
# Note: This is a simplified approach. In production, use S3 lifecycle policies
|
||||
aws s3 ls "s3://$S3_BUCKET/c-relay/" --recursive | \
|
||||
aws s3 ls "s3://$S3_BUCKET/c-relay-pg/" --recursive | \
|
||||
awk '$1 < "'$cutoff_date'" {print $4}' | \
|
||||
while read -r key; do
|
||||
aws s3 rm "s3://$S3_BUCKET/$key"
|
||||
|
||||
@@ -13,8 +13,8 @@ BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Configuration
|
||||
RELAY_DIR="/opt/c-relay"
|
||||
SERVICE_NAME="c-relay"
|
||||
RELAY_DIR="/opt/c-relay-pg"
|
||||
SERVICE_NAME="c-relay-pg"
|
||||
RELAY_PORT="8888"
|
||||
LOG_FILE="/var/log/relay-monitor.log"
|
||||
ALERT_EMAIL=""
|
||||
@@ -60,7 +60,7 @@ show_help() {
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo
|
||||
echo "Options:"
|
||||
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay)"
|
||||
echo " -d, --relay-dir DIR Relay directory (default: /opt/c-relay-pg)"
|
||||
echo " -p, --port PORT Relay port (default: 8888)"
|
||||
echo " -i, --interval SECONDS Check interval (default: 60)"
|
||||
echo " -e, --email EMAIL Alert email address"
|
||||
@@ -134,7 +134,7 @@ parse_args() {
|
||||
check_process_running() {
|
||||
print_step "Checking if relay process is running..."
|
||||
|
||||
if pgrep -f "c_relay_x86" > /dev/null; then
|
||||
if pgrep -f "c_relay_pg_x86" > /dev/null; then
|
||||
print_success "Relay process is running"
|
||||
return 0
|
||||
else
|
||||
@@ -172,7 +172,7 @@ check_service_status() {
|
||||
check_memory_usage() {
|
||||
print_step "Checking memory usage..."
|
||||
|
||||
local memory_kb=$(ps aux | grep "c_relay_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
|
||||
local memory_kb=$(ps aux | grep "c_relay_pg_x86" | grep -v grep | awk '{sum+=$6} END {print sum}')
|
||||
|
||||
if [[ -z "$memory_kb" ]]; then
|
||||
print_warning "Could not determine memory usage"
|
||||
|
||||
@@ -56,7 +56,7 @@ http {
|
||||
}
|
||||
|
||||
# Upstream for the relay
|
||||
upstream c_relay_backend {
|
||||
upstream c_relay_pg_backend {
|
||||
server 127.0.0.1:8888;
|
||||
keepalive 32;
|
||||
}
|
||||
@@ -108,7 +108,7 @@ http {
|
||||
# Main proxy location for WebSocket and HTTP
|
||||
location / {
|
||||
# Proxy settings
|
||||
proxy_pass http://c_relay_backend;
|
||||
proxy_pass http://c_relay_pg_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
|
||||
@@ -144,7 +144,7 @@ http {
|
||||
|
||||
# Health check endpoint (if implemented)
|
||||
location /health {
|
||||
proxy_pass http://c_relay_backend/health;
|
||||
proxy_pass http://c_relay_pg_backend/health;
|
||||
access_log off;
|
||||
}
|
||||
|
||||
@@ -157,7 +157,7 @@ http {
|
||||
|
||||
# Optional: Metrics endpoint (if implemented)
|
||||
location /metrics {
|
||||
proxy_pass http://c_relay_backend/metrics;
|
||||
proxy_pass http://c_relay_pg_backend/metrics;
|
||||
# Restrict access to monitoring systems
|
||||
allow 10.0.0.0/8;
|
||||
allow 172.16.0.0/12;
|
||||
|
||||
@@ -96,9 +96,9 @@ check_root() {
|
||||
check_relay_running() {
|
||||
print_step "Checking if C Nostr Relay is running..."
|
||||
|
||||
if ! pgrep -f "c_relay_x86" > /dev/null; then
|
||||
if ! pgrep -f "c_relay_pg_x86" > /dev/null; then
|
||||
print_error "C Nostr Relay is not running"
|
||||
print_error "Please start the relay first with: sudo systemctl start c-relay"
|
||||
print_error "Please start the relay first with: sudo systemctl start c-relay-pg"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -13,9 +13,9 @@ BLUE='\033[0;34m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Configuration
|
||||
RELAY_USER="c-relay"
|
||||
INSTALL_DIR="/opt/c-relay"
|
||||
SERVICE_NAME="c-relay"
|
||||
RELAY_USER="c-relay-pg"
|
||||
INSTALL_DIR="/opt/c-relay-pg"
|
||||
SERVICE_NAME="c-relay-pg"
|
||||
RELAY_PORT="8888"
|
||||
|
||||
# Functions
|
||||
@@ -99,7 +99,7 @@ build_relay() {
|
||||
|
||||
# Check if we're in the source directory
|
||||
if [[ ! -f "Makefile" ]]; then
|
||||
print_error "Makefile not found. Please run this script from the c-relay source directory."
|
||||
print_error "Makefile not found. Please run this script from the c-relay-pg source directory."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -107,7 +107,7 @@ build_relay() {
|
||||
make clean
|
||||
make
|
||||
|
||||
if [[ ! -f "build/c_relay_x86" ]]; then
|
||||
if [[ ! -f "build/c_relay_pg_x86" ]]; then
|
||||
print_error "Build failed - binary not found"
|
||||
exit 1
|
||||
fi
|
||||
@@ -118,9 +118,9 @@ build_relay() {
|
||||
install_binary() {
|
||||
print_step "Installing relay binary..."
|
||||
|
||||
cp build/c_relay_x86 "$INSTALL_DIR/"
|
||||
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_x86"
|
||||
chmod +x "$INSTALL_DIR/c_relay_x86"
|
||||
cp build/c_relay_pg_x86 "$INSTALL_DIR/"
|
||||
chown "$RELAY_USER:$RELAY_USER" "$INSTALL_DIR/c_relay_pg_x86"
|
||||
chmod +x "$INSTALL_DIR/c_relay_pg_x86"
|
||||
|
||||
print_success "Binary installed to $INSTALL_DIR"
|
||||
}
|
||||
@@ -129,14 +129,14 @@ install_service() {
|
||||
print_step "Installing systemd service..."
|
||||
|
||||
# Use the existing systemd service file
|
||||
if [[ -f "systemd/c-relay.service" ]]; then
|
||||
cp systemd/c-relay.service /etc/systemd/system/
|
||||
if [[ -f "systemd/c-relay-pg.service" ]]; then
|
||||
cp systemd/c-relay-pg.service /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
print_success "Systemd service installed"
|
||||
else
|
||||
print_warning "Systemd service file not found, creating basic one..."
|
||||
|
||||
cat > /etc/systemd/system/c-relay.service << EOF
|
||||
cat > /etc/systemd/system/c-relay-pg.service << EOF
|
||||
[Unit]
|
||||
Description=C Nostr Relay
|
||||
After=network.target
|
||||
@@ -146,7 +146,7 @@ Type=simple
|
||||
User=$RELAY_USER
|
||||
Group=$RELAY_USER
|
||||
WorkingDirectory=$INSTALL_DIR
|
||||
ExecStart=$INSTALL_DIR/c_relay_x86
|
||||
ExecStart=$INSTALL_DIR/c_relay_pg_x86
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# MUSL-based fully static C-Relay builder
|
||||
# MUSL-based fully static C-Relay-PG builder
|
||||
# Produces portable binaries with zero runtime dependencies
|
||||
|
||||
FROM alpine:latest AS builder
|
||||
@@ -125,7 +125,7 @@ RUN cd /tmp && \
|
||||
--prefix=/usr && \
|
||||
make && make install
|
||||
|
||||
# Copy c-relay source
|
||||
# Copy c-relay-pg source
|
||||
COPY . /build/
|
||||
|
||||
# Initialize submodules
|
||||
@@ -134,7 +134,7 @@ RUN git submodule update --init --recursive
|
||||
# Build nostr_core_lib
|
||||
RUN cd nostr_core_lib && ./build.sh
|
||||
|
||||
# Build c-relay static
|
||||
# Build c-relay-pg static
|
||||
RUN make clean && \
|
||||
CC="musl-gcc -static" \
|
||||
CFLAGS="-O2 -Wall -Wextra -std=c99 -g" \
|
||||
@@ -143,8 +143,8 @@ RUN make clean && \
|
||||
make
|
||||
|
||||
# Strip binary for size
|
||||
RUN strip build/c_relay_x86
|
||||
RUN strip build/c_relay_pg_x86
|
||||
|
||||
# Multi-stage build to produce minimal output
|
||||
FROM scratch AS output
|
||||
COPY --from=builder /build/build/c_relay_x86 /c_relay_static_musl_x86_64
|
||||
COPY --from=builder /build/build/c_relay_pg_x86 /c_relay_pg_static_musl_x86_64
|
||||
@@ -19,7 +19,7 @@ RELEASE_MODE=false
|
||||
VERSION_INCREMENT_TYPE="patch" # "patch", "minor", or "major"
|
||||
|
||||
show_usage() {
|
||||
echo "C-Relay Increment and Push Script"
|
||||
echo "C-Relay-PG Increment and Push Script"
|
||||
echo ""
|
||||
echo "USAGE:"
|
||||
echo " $0 [OPTIONS] \"commit message\""
|
||||
@@ -330,7 +330,7 @@ build_release_binary() {
|
||||
create_source_tarball() {
|
||||
print_status "Creating source tarball..."
|
||||
|
||||
local tarball_name="c-relay-${NEW_VERSION#v}.tar.gz"
|
||||
local tarball_name="c-relay-pg-${NEW_VERSION#v}.tar.gz"
|
||||
|
||||
# Create tarball excluding build artifacts and git files
|
||||
if tar -czf "$tarball_name" \
|
||||
@@ -365,7 +365,7 @@ upload_release_assets() {
|
||||
fi
|
||||
|
||||
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
|
||||
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
|
||||
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
|
||||
local assets_url="$api_url/releases/$release_id/assets"
|
||||
print_status "Assets URL: $assets_url"
|
||||
|
||||
@@ -427,7 +427,7 @@ create_gitea_release() {
|
||||
fi
|
||||
|
||||
local token=$(cat "$HOME/.gitea_token" | tr -d '\n\r')
|
||||
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay"
|
||||
local api_url="https://git.laantungir.net/api/v1/repos/laantungir/c-relay-pg"
|
||||
|
||||
# Create release
|
||||
print_status "Creating release $NEW_VERSION..."
|
||||
@@ -473,7 +473,7 @@ create_gitea_release() {
|
||||
|
||||
# Main execution
|
||||
main() {
|
||||
print_status "C-Relay Increment and Push Script"
|
||||
print_status "C-Relay-PG Increment and Push Script"
|
||||
|
||||
# Check prerequisites
|
||||
check_git_repo
|
||||
@@ -505,13 +505,13 @@ main() {
|
||||
|
||||
# Build release binary
|
||||
if build_release_binary; then
|
||||
local binary_path="build/c_relay_static_x86_64"
|
||||
local binary_path="build/c_relay_pg_static_x86_64"
|
||||
else
|
||||
print_warning "Binary build failed, continuing with release creation"
|
||||
# Check if binary exists from previous build
|
||||
if [[ -f "build/c_relay_static_x86_64" ]]; then
|
||||
if [[ -f "build/c_relay_pg_static_x86_64" ]]; then
|
||||
print_status "Using existing binary from previous build"
|
||||
binary_path="build/c_relay_static_x86_64"
|
||||
binary_path="build/c_relay_pg_static_x86_64"
|
||||
else
|
||||
binary_path=""
|
||||
fi
|
||||
|
||||
+264
-15
@@ -1,6 +1,6 @@
|
||||
#!/bin/bash
|
||||
|
||||
# C-Relay Build and Restart Script
|
||||
# C-Relay-PG Build and Restart Script
|
||||
# Builds the project first, then stops any running relay and starts a new one in the background
|
||||
|
||||
echo "=== C Nostr Relay Build and Restart Script ==="
|
||||
@@ -13,6 +13,13 @@ ADMIN_KEY=""
|
||||
RELAY_KEY=""
|
||||
PORT_OVERRIDE=""
|
||||
DEBUG_LEVEL="5"
|
||||
DB_BACKEND="postgres"
|
||||
DB_CONNSTRING=""
|
||||
DB_HOST=""
|
||||
DB_PORT=""
|
||||
DB_NAME=""
|
||||
DB_USER=""
|
||||
DB_PASSWORD=""
|
||||
|
||||
# Key validation function
|
||||
validate_hex_key() {
|
||||
@@ -70,6 +77,24 @@ while [[ $# -gt 0 ]]; do
|
||||
;;
|
||||
--test-keys|-t)
|
||||
USE_TEST_KEYS=true
|
||||
# Read keys from .test_keys file
|
||||
if [ -f ".test_keys" ]; then
|
||||
echo "Reading test keys from .test_keys file..."
|
||||
# Source the file to get the variables
|
||||
source .test_keys
|
||||
# Remove any single quotes from the values
|
||||
# Note: -a flag expects ADMIN_PUBKEY (public key), not ADMIN_PRIVKEY
|
||||
ADMIN_KEY=$(echo "$ADMIN_PUBKEY" | tr -d "'")
|
||||
RELAY_KEY=$(echo "$SERVER_PRIVKEY" | tr -d "'")
|
||||
echo "Using admin pubkey from .test_keys: ${ADMIN_KEY:0:16}..."
|
||||
echo "Using relay privkey from .test_keys: ${RELAY_KEY:0:16}..."
|
||||
else
|
||||
echo "ERROR: .test_keys file not found"
|
||||
echo "Please create a .test_keys file with the following format:"
|
||||
echo " ADMIN_PUBKEY='your_admin_public_key_hex'"
|
||||
echo " SERVER_PRIVKEY='your_relay_private_key_hex'"
|
||||
exit 1
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
--debug-level=*)
|
||||
@@ -100,6 +125,84 @@ while [[ $# -gt 0 ]]; do
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-backend)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-backend requires a value (sqlite|postgres)"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_BACKEND="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-backend=*)
|
||||
DB_BACKEND="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--db-connstring)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-connstring requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_CONNSTRING="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-connstring=*)
|
||||
DB_CONNSTRING="${1#*=}"
|
||||
shift
|
||||
;;
|
||||
--db-host)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-host requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_HOST="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-port)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-port requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_PORT="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-name)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-name requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_NAME="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-user)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-user requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_USER="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--db-password)
|
||||
if [ -z "$2" ]; then
|
||||
echo "ERROR: --db-password requires a value"
|
||||
HELP=true
|
||||
shift
|
||||
else
|
||||
DB_PASSWORD="$2"
|
||||
shift 2
|
||||
fi
|
||||
;;
|
||||
--help|-h)
|
||||
HELP=true
|
||||
shift
|
||||
@@ -146,6 +249,98 @@ if [ -n "$DEBUG_LEVEL" ]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
# Validate DB backend
|
||||
if [ "$DB_BACKEND" != "sqlite" ] && [ "$DB_BACKEND" != "postgres" ]; then
|
||||
echo "ERROR: Invalid --db-backend value '$DB_BACKEND'. Use sqlite or postgres."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate DB port if provided
|
||||
if [ -n "$DB_PORT" ]; then
|
||||
if ! [[ "$DB_PORT" =~ ^[0-9]+$ ]] || [ "$DB_PORT" -lt 1 ] || [ "$DB_PORT" -gt 65535 ]; then
|
||||
echo "ERROR: --db-port must be a number between 1 and 65535"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
|
||||
[ -z "$DB_HOST" ] && DB_HOST="localhost"
|
||||
[ -z "$DB_PORT" ] && DB_PORT="5432"
|
||||
[ -z "$DB_NAME" ] && DB_NAME="crelay"
|
||||
[ -z "$DB_USER" ] && DB_USER="crelay"
|
||||
[ -z "$DB_PASSWORD" ] && DB_PASSWORD="crelay"
|
||||
fi
|
||||
|
||||
ensure_postgres_database() {
|
||||
if ! command -v psql >/dev/null 2>&1; then
|
||||
echo "ERROR: psql not found. Install PostgreSQL client tools to continue."
|
||||
return 1
|
||||
fi
|
||||
|
||||
echo "Ensuring PostgreSQL database exists: host=$DB_HOST port=$DB_PORT dbname=$DB_NAME user=$DB_USER"
|
||||
|
||||
if [ -n "$DB_PASSWORD" ]; then
|
||||
export PGPASSWORD="$DB_PASSWORD"
|
||||
fi
|
||||
|
||||
# Prefer local postgres superuser for provisioning when available.
|
||||
if sudo -n -u postgres psql -d postgres -tAc "SELECT 1" >/dev/null 2>&1; then
|
||||
sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname = '$DB_USER'" | grep -q 1 || \
|
||||
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "CREATE ROLE \"$DB_USER\" LOGIN PASSWORD '$DB_PASSWORD';" || return 1
|
||||
|
||||
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "ALTER ROLE \"$DB_USER\" WITH LOGIN PASSWORD '$DB_PASSWORD';" >/dev/null || return 1
|
||||
|
||||
if [ "$PRESERVE_DATABASE" = false ]; then
|
||||
echo "Resetting PostgreSQL database '$DB_NAME' for fresh start..."
|
||||
|
||||
# Ensure no active sessions block DROP DATABASE (common when relay is still running).
|
||||
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c \
|
||||
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '$DB_NAME' AND pid <> pg_backend_pid();" >/dev/null || return 1
|
||||
|
||||
# Prefer force drop when supported; fallback to regular dropdb.
|
||||
if ! sudo -u postgres dropdb --if-exists --force "$DB_NAME" >/dev/null 2>&1; then
|
||||
sudo -u postgres dropdb --if-exists "$DB_NAME" >/dev/null 2>&1 || return 1
|
||||
fi
|
||||
|
||||
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
|
||||
echo "✓ Recreated PostgreSQL database '$DB_NAME' owned by '$DB_USER'"
|
||||
else
|
||||
DB_EXISTS=$(sudo -u postgres psql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
|
||||
if [ "$DB_EXISTS" != "1" ]; then
|
||||
sudo -u postgres createdb -O "$DB_USER" "$DB_NAME" >/dev/null 2>&1 || return 1
|
||||
echo "✓ Created PostgreSQL database '$DB_NAME'"
|
||||
else
|
||||
echo "✓ PostgreSQL database '$DB_NAME' already exists"
|
||||
fi
|
||||
fi
|
||||
|
||||
sudo -u postgres psql -d postgres -v ON_ERROR_STOP=1 -c "GRANT ALL PRIVILEGES ON DATABASE \"$DB_NAME\" TO \"$DB_USER\";" >/dev/null || return 1
|
||||
sudo -u postgres psql -d "$DB_NAME" -v ON_ERROR_STOP=1 -c "ALTER SCHEMA public OWNER TO \"$DB_USER\"; GRANT ALL ON SCHEMA public TO \"$DB_USER\";" >/dev/null || return 1
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Fallback: use configured DB user directly.
|
||||
if ! psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -v ON_ERROR_STOP=1 -tAc "SELECT 1" >/dev/null 2>&1; then
|
||||
echo "ERROR: Cannot connect to PostgreSQL server/database 'postgres' with current settings"
|
||||
echo " host=$DB_HOST port=$DB_PORT user=$DB_USER"
|
||||
return 1
|
||||
fi
|
||||
|
||||
DB_EXISTS=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$DB_NAME'" 2>/dev/null | tr -d '[:space:]')
|
||||
if [ "$DB_EXISTS" != "1" ]; then
|
||||
echo "Database '$DB_NAME' not found. Creating..."
|
||||
if ! createdb -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" "$DB_NAME" >/dev/null 2>&1; then
|
||||
echo "ERROR: Failed to create PostgreSQL database '$DB_NAME'"
|
||||
return 1
|
||||
fi
|
||||
echo "✓ Created PostgreSQL database '$DB_NAME'"
|
||||
else
|
||||
echo "✓ PostgreSQL database '$DB_NAME' already exists"
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Show help
|
||||
if [ "$HELP" = true ]; then
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
@@ -157,6 +352,13 @@ if [ "$HELP" = true ]; then
|
||||
echo " -d, --debug-level <0-5> Set debug level: 0=none, 1=errors, 2=warnings, 3=info, 4=debug, 5=trace"
|
||||
echo " --preserve-database Keep existing database files (don't delete for fresh start)"
|
||||
echo " --test-keys, -t Use deterministic test keys for development (admin: all 'a's, relay: all '1's)"
|
||||
echo " --db-backend <name> Database backend: postgres (default) or sqlite"
|
||||
echo " --db-connstring <str> PostgreSQL libpq connection string"
|
||||
echo " --db-host <host> PostgreSQL host"
|
||||
echo " --db-port <port> PostgreSQL port"
|
||||
echo " --db-name <name> PostgreSQL database name"
|
||||
echo " --db-user <user> PostgreSQL database user"
|
||||
echo " --db-password <pass> PostgreSQL database password"
|
||||
echo " --help, -h Show this help message"
|
||||
echo ""
|
||||
echo "Event-Based Configuration:"
|
||||
@@ -176,6 +378,9 @@ if [ "$HELP" = true ]; then
|
||||
echo " $0 --test-keys # Use test keys for consistent development"
|
||||
echo " $0 -t --preserve-database # Use test keys and preserve database"
|
||||
echo ""
|
||||
echo "Default PostgreSQL connection (when no DB flags provided):"
|
||||
echo " host=localhost port=5432 dbname=crelay user=crelay password=crelay"
|
||||
echo ""
|
||||
echo "Key Format: Keys must be exactly 64 hexadecimal characters (0-9, a-f, A-F)"
|
||||
echo "Default behavior: Deletes existing database files to start fresh with new keys"
|
||||
echo " for development purposes"
|
||||
@@ -199,13 +404,17 @@ fi
|
||||
rm -rf dev-config/ 2>/dev/null
|
||||
rm -f db/c_nostr_relay.db* 2>/dev/null
|
||||
|
||||
if [ "$DB_BACKEND" = "postgres" ] && [ -z "$DB_CONNSTRING" ]; then
|
||||
ensure_postgres_database || exit 1
|
||||
fi
|
||||
|
||||
# Embed web files into C headers before building
|
||||
echo "Embedding web files..."
|
||||
./embed_web_files.sh
|
||||
|
||||
# Build the project - ONLY static build
|
||||
echo "Building project (static binary with SQLite JSON1 extension)..."
|
||||
./build_static.sh
|
||||
echo "Building project (static binary, backend: $DB_BACKEND)..."
|
||||
./build_static.sh --db-backend "$DB_BACKEND"
|
||||
|
||||
# Exit if static build fails - no fallback
|
||||
if [ $? -ne 0 ]; then
|
||||
@@ -224,13 +433,13 @@ fi
|
||||
ARCH=$(uname -m)
|
||||
case "$ARCH" in
|
||||
x86_64)
|
||||
BINARY_PATH="./build/c_relay_static_x86_64"
|
||||
BINARY_PATH="./build/c_relay_pg_static_x86_64"
|
||||
;;
|
||||
aarch64|arm64)
|
||||
BINARY_PATH="./build/c_relay_static_arm64"
|
||||
BINARY_PATH="./build/c_relay_pg_static_arm64"
|
||||
;;
|
||||
*)
|
||||
BINARY_PATH="./build/c_relay_static_$ARCH"
|
||||
BINARY_PATH="./build/c_relay_pg_static_$ARCH"
|
||||
;;
|
||||
esac
|
||||
|
||||
@@ -246,13 +455,26 @@ fi
|
||||
|
||||
echo "Using static binary: $BINARY_PATH"
|
||||
|
||||
# Build the caching_relay binary (from caching/ directory)
|
||||
# This is needed because the caching service launcher forks this binary.
|
||||
if [ -d "caching" ] && [ -f "caching/Makefile" ]; then
|
||||
echo "Building caching_relay binary..."
|
||||
(cd caching && make) > /dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "WARNING: caching_relay build failed. The caching service will not be available."
|
||||
echo "You can build it manually with: cd caching && make"
|
||||
else
|
||||
echo "✓ caching_relay binary built successfully"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "Build successful. Proceeding with relay restart..."
|
||||
|
||||
# Kill existing relay if running - start aggressive immediately
|
||||
echo "Stopping any existing relay servers..."
|
||||
|
||||
# Get all relay processes and kill them immediately with -9
|
||||
RELAY_PIDS=$(pgrep -f "c_relay_" || echo "")
|
||||
RELAY_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
|
||||
if [ -n "$RELAY_PIDS" ]; then
|
||||
echo "Force killing relay processes immediately: $RELAY_PIDS"
|
||||
kill -9 $RELAY_PIDS 2>/dev/null
|
||||
@@ -273,7 +495,7 @@ for attempt in {1..15}; do
|
||||
fuser -k 8888/tcp 2>/dev/null || true
|
||||
|
||||
# Double-check for any remaining relay processes
|
||||
REMAINING_PIDS=$(pgrep -f "c_relay_" || echo "")
|
||||
REMAINING_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
|
||||
if [ -n "$REMAINING_PIDS" ]; then
|
||||
echo "Killing remaining relay processes: $REMAINING_PIDS"
|
||||
kill -9 $REMAINING_PIDS 2>/dev/null || true
|
||||
@@ -291,7 +513,7 @@ for attempt in {1..15}; do
|
||||
done
|
||||
|
||||
# Final safety check - ensure no relay processes remain
|
||||
FINAL_PIDS=$(pgrep -f "c_relay_" || echo "")
|
||||
FINAL_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
|
||||
if [ -n "$FINAL_PIDS" ]; then
|
||||
echo "Final cleanup: killing processes $FINAL_PIDS"
|
||||
kill -9 $FINAL_PIDS 2>/dev/null || true
|
||||
@@ -307,7 +529,7 @@ echo "Database will be initialized automatically on startup if needed"
|
||||
|
||||
# Start relay in background with output redirection
|
||||
echo "Starting relay server..."
|
||||
echo "Debug: Current processes: $(ps aux | grep 'c_relay_' | grep -v grep || echo 'None')"
|
||||
echo "Debug: Current processes: $(ps aux | grep 'c_relay_pg_' | grep -v grep || echo 'None')"
|
||||
|
||||
# Build command line arguments for relay binary
|
||||
RELAY_ARGS=""
|
||||
@@ -332,15 +554,42 @@ if [ -n "$DEBUG_LEVEL" ]; then
|
||||
echo "Using debug level: $DEBUG_LEVEL"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_CONNSTRING" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-connstring '$DB_CONNSTRING'"
|
||||
echo "Using PostgreSQL connection string override"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_HOST" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-host '$DB_HOST'"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_PORT" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-port '$DB_PORT'"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_NAME" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-name '$DB_NAME'"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_USER" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-user '$DB_USER'"
|
||||
fi
|
||||
|
||||
if [ -n "$DB_PASSWORD" ]; then
|
||||
RELAY_ARGS="$RELAY_ARGS --db-password '$DB_PASSWORD'"
|
||||
fi
|
||||
|
||||
# Change to build directory before starting relay so database files are created there
|
||||
cd build
|
||||
# Start relay in background and capture its PID
|
||||
if [ "$USE_TEST_KEYS" = true ]; then
|
||||
echo "Using deterministic test keys for development..."
|
||||
./$(basename $BINARY_PATH) -a 6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3 -r 1111111111111111111111111111111111111111111111111111111111111111 --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
|
||||
echo "Using test keys from .test_keys file..."
|
||||
# shellcheck disable=SC2086
|
||||
eval ./$(basename $BINARY_PATH) -a "$ADMIN_KEY" -r "$RELAY_KEY" $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
|
||||
elif [ -n "$RELAY_ARGS" ]; then
|
||||
echo "Starting relay with custom configuration..."
|
||||
./$(basename $BINARY_PATH) $RELAY_ARGS --debug-level=$DEBUG_LEVEL --strict-port > ../relay.log 2>&1 &
|
||||
# shellcheck disable=SC2086
|
||||
eval ./$(basename $BINARY_PATH) $RELAY_ARGS --strict-port > ../relay.log 2>&1 &
|
||||
else
|
||||
# No command line arguments needed for random key generation
|
||||
echo "Starting relay with random key generation..."
|
||||
@@ -410,8 +659,8 @@ if ps -p "$RELAY_PID" >/dev/null 2>&1; then
|
||||
echo "=== Event-Based Relay Server Running ==="
|
||||
echo "Configuration: Event-based (kind 33334 Nostr events)"
|
||||
echo "Database: Automatically created with relay pubkey naming"
|
||||
echo "To kill relay: pkill -f 'c_relay_'"
|
||||
echo "To check status: ps aux | grep c_relay_"
|
||||
echo "To kill relay: pkill -f 'c_relay_pg_'"
|
||||
echo "To check status: ps aux | grep c_relay_pg_"
|
||||
echo "To view logs: tail -f relay.log"
|
||||
echo "Binary: $BINARY_PATH (zero configuration needed)"
|
||||
echo "Ready for Nostr client connections!"
|
||||
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
#!/bin/bash
|
||||
|
||||
curl -H "Accept: application/nostr+json" http://localhost:8888/
|
||||
curl -H "Accept: application/nostr+json" https://relay.laantungir.net
|
||||
|
||||
+1
-1
Submodule nostr_core_lib updated: a8dc2ed046...98cfd81b01
@@ -12,11 +12,11 @@ After the next crash, analyze it:
|
||||
# List all core dumps (most recent first)
|
||||
sudo coredumpctl list
|
||||
|
||||
# View info about the most recent c-relay crash
|
||||
sudo coredumpctl info c-relay
|
||||
# View info about the most recent c-relay-pg crash
|
||||
sudo coredumpctl info c-relay-pg
|
||||
|
||||
# Load the core dump in gdb for detailed analysis
|
||||
sudo coredumpctl gdb c-relay
|
||||
sudo coredumpctl gdb c-relay-pg
|
||||
|
||||
Inside gdb, run these commands:
|
||||
(gdb) bt full # Full backtrace with all variables
|
||||
@@ -36,8 +36,8 @@ DEBUGGING
|
||||
|
||||
Even simpler: Use this one-liner
|
||||
# Start relay and immediately attach gdb
|
||||
cd /usr/local/bin/c_relay
|
||||
sudo -u c-relay ./c_relay --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay)
|
||||
cd /usr/local/bin/c_relay_pg
|
||||
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 & sleep 2 && sudo gdb -p $(pgrep c_relay_pg)
|
||||
|
||||
Inside gdb, after attaching:
|
||||
|
||||
@@ -48,17 +48,17 @@ Or shorter:
|
||||
|
||||
How to View the Logs
|
||||
Check systemd journal:
|
||||
# View all c-relay logs
|
||||
sudo journalctl -u c-relay
|
||||
# View all c-relay-pg logs
|
||||
sudo journalctl -u c-relay-pg
|
||||
|
||||
# View recent logs (last 50 lines)
|
||||
sudo journalctl -u c-relay -n 50
|
||||
sudo journalctl -u c-relay-pg -n 50
|
||||
|
||||
# Follow logs in real-time
|
||||
sudo journalctl -u c-relay -f
|
||||
sudo journalctl -u c-relay-pg -f
|
||||
|
||||
# View logs since last boot
|
||||
sudo journalctl -u c-relay -b
|
||||
sudo journalctl -u c-relay-pg -b
|
||||
|
||||
Check if service is running:
|
||||
|
||||
@@ -78,9 +78,9 @@ sudo systemctl start rsyslog
|
||||
sudo systemctl status rsyslog
|
||||
|
||||
|
||||
sudo -u c-relay ./c_relay --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
|
||||
sudo -u c-relay-pg ./c_relay_pg --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
|
||||
|
||||
./c_relay_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
|
||||
./c_relay_pg_static_x86_64 -p 7889 --debug-level=5 -r 85d0b37e2ae822966dcadd06b2dc9368cde73865f90ea4d44f8b57d47ef0820a -a 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139
|
||||
|
||||
|
||||
sudo ufw allow 8888/tcp
|
||||
|
||||
Generated
+1
-1
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"name": "c-relay",
|
||||
"name": "c-relay-pg",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
# API Upgrade Implementation Plan — Phases 1, 2, and 5
|
||||
|
||||
Goal: finish the remaining work from `plans/api_upgrade_plan.md`:
|
||||
- **Phase 1 (design fix):** convert the `api-worker` thread from job-queue-driven to **timer + subscriber-check** driven.
|
||||
- **Phase 2:** remove the per-event and per-subscription monitoring triggers so monitoring no longer runs on the event-processing path.
|
||||
- **Phase 5:** add **PostgreSQL `LISTEN`/`NOTIFY`** so the worker wakes on real data changes instead of pure polling.
|
||||
|
||||
Target test relay: **port 7777** (local).
|
||||
|
||||
---
|
||||
|
||||
## Current State (verified in code)
|
||||
|
||||
- `api_worker_main()` ([`src/api.c:172`](src/api.c:172)) blocks on `api_worker_pop_job_blocking()` and reacts to three job types: `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `API_WORK_JOB_STATUS_POST`.
|
||||
- Monitoring hooks `monitoring_on_event_stored()` ([`src/api.c:752`](src/api.c:752)) and `monitoring_on_subscription_change()` ([`src/api.c:759`](src/api.c:759)) are called from [`src/main.c:1169`](src/main.c:1169) and [`src/subscriptions.c:483`](src/subscriptions.c:483), [`src/subscriptions.c:536`](src/subscriptions.c:536). They enqueue jobs; sync fallback runs if the worker is down.
|
||||
- Throttling lives in `monitoring_on_event_stored_sync()` / `monitoring_on_subscription_change_sync()` ([`src/api.c:711`](src/api.c:711), [`src/api.c:731`](src/api.c:731)).
|
||||
- Round-robin selection in `generate_round_robin_monitoring_event()` ([`src/api.c:571`](src/api.c:571)); per-type build/sign/broadcast in `generate_monitoring_event_for_type()` ([`src/api.c:621`](src/api.c:621)).
|
||||
- `has_subscriptions_for_kind(int)` ([`src/subscriptions.c:980`](src/subscriptions.c:980)) checks only subscriptions with an explicit `kinds` filter — it does **not** consider the `no_kind_filter_subs` list, so it alone is insufficient for a "is anyone listening to kind 24567" check.
|
||||
- Worker DB connection is a `PGconn*` opened via `db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)). Schema is embedded in [`src/pg_schema.h`](src/pg_schema.h) (version `"4"`) and mirrored in [`src/pg_schema.sql`](src/pg_schema.sql).
|
||||
- No `LISTEN`/`NOTIFY`/`pg_notify`/`PQnotifies` usage exists anywhere in `src/`.
|
||||
|
||||
---
|
||||
|
||||
## Design
|
||||
|
||||
### Phase 1 + 2 + 5 combined worker loop
|
||||
|
||||
The three phases are coupled, so they are implemented together in one rewrite of `api_worker_main()` and its helpers.
|
||||
|
||||
### CRITICAL BUG FIX (why the web API isn't getting updates today)
|
||||
|
||||
`postgres_db_open_worker_connection()` ([`src/db_ops_postgres.c:226`](src/db_ops_postgres.c:226)) calls
|
||||
`postgres_db_set_thread_connection(conn)` at line 243 — but `g_thread_pg_conn` is declared `__thread`
|
||||
([`src/db_ops_postgres.c:45`](src/db_ops_postgres.c:45)), so the binding is set in the **caller's thread**
|
||||
(lws-main, which calls `start_api_worker`), NOT in the api-worker thread. When `api_worker_main` runs and
|
||||
the monitoring query functions call `postgres_db_active_connection()`, `g_thread_pg_conn` is NULL in that
|
||||
thread, so they fall back to the shared global `g_pg_conn` — concurrent access with the main thread. This
|
||||
causes monitoring queries to fail or interleave, which is why the dashboard stops receiving kind 24567
|
||||
events.
|
||||
|
||||
**Fix:** inside `api_worker_main()`, after opening `worker_db`, call
|
||||
`db_set_thread_connection(worker_db)` so the `__thread` pointer is bound in the api-worker thread itself.
|
||||
Call `db_clear_thread_connection()` before closing the connection on shutdown. This must be done in the
|
||||
new rewrite regardless of the loop design.
|
||||
|
||||
```
|
||||
api-worker thread:
|
||||
1. Open own PG connection.
|
||||
2. db_set_thread_connection(worker_db) // bind __thread pointer in THIS thread
|
||||
3. If PG backend: LISTEN event_stored on this connection.
|
||||
4. Loop while running:
|
||||
a. Determine if anyone is listening to kind 24567
|
||||
(has_subscriptions_for_kind(24567) OR no_kind_filter_subs non-empty).
|
||||
b. If NO subscribers:
|
||||
- condvar_timedwait(throttle_sec) // wakeable for shutdown / STATUS_POST job
|
||||
- on wake: process any pending STATUS_POST job, then continue.
|
||||
// Zero DB work, zero notify polling.
|
||||
c. If subscribers exist:
|
||||
- wait_for_notify_or_timeout(throttle_sec):
|
||||
select() on { PQsocket, self-pipe read fd } with timeout = throttle_sec.
|
||||
- If PQsocket readable: PQconsumeInput + drain PQnotifies.
|
||||
- If self-pipe readable: drain (shutdown or STATUS_POST signal).
|
||||
- If timeout: fall through.
|
||||
- generate_round_robin_monitoring_event() // one d-tag per tick
|
||||
- process any pending STATUS_POST job.
|
||||
4. On shutdown: UNLISTEN, close PG connection.
|
||||
```
|
||||
|
||||
Key properties:
|
||||
- **Zero overhead when no subscribers** — condvar sleep, no `select`, no `LISTEN` polling.
|
||||
- **Reactive when subscribers exist** — wakes within `throttle_sec` of an `event_stored` NOTIFY, but never more than once per `throttle_sec` (rate-limited by the select timeout + the round-robin cadence already in `generate_round_robin_monitoring_event`).
|
||||
- **STATUS_POST preserved** — still enqueued from [`src/websockets.c:3470`](src/websockets.c:3470); the self-pipe wakes the worker's `select` so status posts are not delayed by a pending notify wait.
|
||||
- **SQLite fallback** — `db_worker_poll_notify()` returns "not supported" on SQLite, so the worker falls back to pure timer behavior (condvar timed wait) — identical to the no-PG path.
|
||||
|
||||
### Self-pipe for job/shutdown signalling
|
||||
|
||||
A self-pipe (or `eventfd` on Linux) is added to `api.c`:
|
||||
- `api_worker_enqueue_job(STATUS_POST)` writes one byte to the pipe → wakes `select`.
|
||||
- `stop_api_worker()` sets `g_api_worker_running = 0`, writes a byte, and broadcasts the condvar (covers both the subscriber and no-subscriber wait paths).
|
||||
|
||||
### Subscriber-check helper
|
||||
|
||||
Add `int has_any_subscription_for_kind(int event_kind)` in [`src/subscriptions.c`](src/subscriptions.c) that returns true if `has_subscriptions_for_kind(event_kind)` OR the `no_kind_filter_subs` list is non-empty. Export it in [`src/subscriptions.h`](src/subscriptions.h). The worker calls `has_any_subscription_for_kind(24567)`.
|
||||
|
||||
### DB abstraction for LISTEN/NOTIFY
|
||||
|
||||
Add to [`src/db_ops.h`](src/db_ops.h) / [`src/db_ops_postgres.c`](src/db_ops_postgres.c) / [`src/db_ops_sqlite.c`](src/db_ops_sqlite.c):
|
||||
- `int db_worker_listen(void* conn, const char* channel);` — issues `LISTEN <channel>`.
|
||||
- `int db_worker_poll_notify(void* conn, int timeout_ms);` — returns `1` if a notification was consumed, `0` on timeout, `-1` on error/unsupported. Uses `PQsocket()`, `select()`, `PQconsumeInput()`, `PQnotifies()`.
|
||||
- SQLite stubs return `-1` (unsupported) so the worker falls back to timer mode.
|
||||
|
||||
### Schema change (Phase 5)
|
||||
|
||||
Add to both [`src/pg_schema.sql`](src/pg_schema.sql) and [`src/pg_schema.h`](src/pg_schema.h):
|
||||
```sql
|
||||
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
PERFORM pg_notify('event_stored', json_build_object(
|
||||
'kind', NEW.kind,
|
||||
'pubkey', substring(NEW.pubkey, 1, 8)
|
||||
)::text);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
DROP TRIGGER IF EXISTS trg_notify_event_stored ON events;
|
||||
CREATE TRIGGER trg_notify_event_stored
|
||||
AFTER INSERT ON events
|
||||
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
|
||||
```
|
||||
Bump `EMBEDDED_PG_SCHEMA_VERSION` from `"4"` to `"5"` in [`src/pg_schema.h`](src/pg_schema.h) and the matching `schema_info` version write in [`src/pg_schema.sql`](src/pg_schema.sql). The trigger is idempotent (`DROP TRIGGER IF EXISTS` + `CREATE`), so existing databases upgrade automatically on next startup via `postgres_db_apply_schema()` ([`src/db_ops_postgres.c:134`](src/db_ops_postgres.c:134)).
|
||||
|
||||
### Phase 2 removals
|
||||
|
||||
- Delete the `monitoring_on_event_stored()` call at [`src/main.c:1169`](src/main.c:1169) (and its forward decl at [`src/main.c:459`](src/main.c:459)).
|
||||
- Delete the `monitoring_on_subscription_change()` calls at [`src/subscriptions.c:483`](src/subscriptions.c:483) and [`src/subscriptions.c:536`](src/subscriptions.c:536) (and its forward decl at [`src/subscriptions.c:58`](src/subscriptions.c:58)).
|
||||
- Remove from [`src/api.h`](src/api.h): `monitoring_on_event_stored`, `monitoring_on_subscription_change`.
|
||||
- Remove from [`src/api.c`](src/api.c): `API_WORK_JOB_EVENT_STORED`, `API_WORK_JOB_SUBSCRIPTION_CHANGE`, `monitoring_on_event_stored`, `monitoring_on_subscription_change`, `monitoring_on_event_stored_sync`, `monitoring_on_subscription_change_sync`. Keep `API_WORK_JOB_STATUS_POST`, `generate_round_robin_monitoring_event`, `generate_monitoring_event_for_type`, `generate_event_driven_monitoring`/`generate_subscription_driven_monitoring` (still used by `generate_monitoring_event` legacy path and tests).
|
||||
- The `generate_event_driven_monitoring` / `generate_subscription_driven_monitoring` wrappers become unused by the worker but remain as public helpers for any external/test callers; the worker calls `generate_round_robin_monitoring_event()` directly.
|
||||
|
||||
### Thread layout (unchanged from plan target)
|
||||
|
||||
```
|
||||
c_relay_pg process
|
||||
├── lws-main — WebSocket event loop (no monitoring DB work)
|
||||
├── db-read-1..N — Async REQ/COUNT queries
|
||||
├── event-worker — Async EVENT ingestion (no monitoring hook)
|
||||
├── db-write-1..N — Async sub logging, misc writes
|
||||
└── api-worker — Timer + LISTEN/NOTIFY driven monitoring (own PG conn)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Mermaid: api-worker state machine
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> OpenConn
|
||||
OpenConn --> BindThread: db_set_thread_connection
|
||||
BindThread --> Listen: PG backend
|
||||
BindThread --> Ready: SQLite/no-PG
|
||||
Listen --> Ready
|
||||
Ready --> CheckSubs
|
||||
CheckSubs --> NoSubSleep: no kind-24567 subs
|
||||
CheckSubs --> WaitNotify: subs present
|
||||
NoSubSleep --> CheckSubs: throttle_sec or wake
|
||||
WaitNotify --> Generate: NOTIFY or throttle_sec
|
||||
Generate --> CheckSubs: after round-robin tick
|
||||
WaitNotify --> NoSubSleep: subs dropped + wake
|
||||
Ready --> [*]: stop_api_worker
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Todo List
|
||||
|
||||
1. Add `has_any_subscription_for_kind()` to `subscriptions.c`/`subscriptions.h` (checks kind index + no-kind-filter list).
|
||||
2. Add `db_worker_listen()` and `db_worker_poll_notify()` to `db_ops.h`, `db_ops_postgres.c` (libpq), and `db_ops_sqlite.c` (stub).
|
||||
3. Add `notify_event_stored()` function + `trg_notify_event_stored` trigger to `pg_schema.sql` and `pg_schema.h`; bump schema version to 5.
|
||||
4. Rewrite `api_worker_main()` in `api.c`: bind the worker PG connection to the thread via `db_set_thread_connection()` (fixes the dashboard-not-updating bug), then run the timer + LISTEN/NOTIFY + subscriber-check loop; add self-pipe for STATUS_POST/shutdown wakeups; keep `start_api_worker`/`stop_api_worker`/`api_worker_process_completions`/`api_worker_enqueue_status_post` signatures.
|
||||
5. Remove `API_WORK_JOB_EVENT_STORED`/`API_WORK_JOB_SUBSCRIPTION_CHANGE` job types and the `monitoring_on_event_stored`/`monitoring_on_subscription_change` functions + sync wrappers from `api.c` and `api.h`.
|
||||
6. Remove the `monitoring_on_event_stored()` call from `main.c` (event storage path).
|
||||
7. Remove the `monitoring_on_subscription_change()` calls from `subscriptions.c` (subscription create/close paths).
|
||||
8. Build with `./make_and_restart_relay.sh` on port 7777; fix any compile errors.
|
||||
9. Test: subscribe to kind 24567 on ws://localhost:7777 → confirm monitoring events arrive on throttle cadence (this validates the thread-connection fix).
|
||||
10. Test: publish a kind-1 event → confirm a NOTIFY-driven monitoring event fires within throttle_sec.
|
||||
11. Test: with no kind-24567 subscribers → confirm no monitoring events generated (zero overhead) and event storage latency unaffected.
|
||||
12. Run `tests/quick_error_tests.sh` and `tests/subscribe_all.sh` against port 7777 to confirm no regressions.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- **Self-pipe + condvar mix complexity.** Mitigation: the no-subscriber path uses only the condvar; the subscriber path uses `select` on the self-pipe + PQ socket. The condvar is only used to wake the no-subscriber sleep, so the two wait mechanisms never overlap.
|
||||
- **`LISTEN` on a worker connection that also runs monitoring queries.** `LISTEN` is connection-local and persists; running `SELECT`s on the same connection does not cancel it. `PQconsumeInput` must be called before `PQnotifies`. Mitigation: encapsulate all of this in `db_worker_poll_notify()`.
|
||||
- **Schema trigger on every insert adds per-event overhead.** `pg_notify` is cheap (in-memory queue) and the payload is tiny. The existing `sync_event_tags_from_events` AFTER INSERT trigger already does far more work per row, so this is negligible by comparison.
|
||||
- **SQLite has no LISTEN/NOTIFY.** Mitigation: `db_worker_poll_notify` returns `-1` on SQLite and the worker falls back to pure timer mode — functionally identical to the no-notify path.
|
||||
- **Removing the per-event monitoring hook changes dashboard freshness behavior.** Mitigation: with subscribers present, NOTIFY now drives near-real-time updates (better than before); with no subscribers, nothing is generated (matches the plan's "zero overhead" goal). Dashboard already handles event-driven updates via its kind-24567 subscription.
|
||||
@@ -0,0 +1,224 @@
|
||||
# API Upgrade Plan: Monitoring Thread + PostgreSQL Push
|
||||
|
||||
## Current API Architecture
|
||||
|
||||
### How Monitoring Works Today
|
||||
|
||||
The relay publishes monitoring data as **kind 24567 ephemeral Nostr events**. The admin dashboard is a standard Nostr client that subscribes to these events.
|
||||
|
||||
```
|
||||
Dashboard connects → subscribes to kind 24567 with d-tags
|
||||
Relay detects subscription → starts generating monitoring events
|
||||
Relay queries DB → builds JSON → signs event → broadcasts to subscribers
|
||||
Dashboard receives events → renders stats/charts
|
||||
```
|
||||
|
||||
### Current Monitoring Event Types
|
||||
|
||||
| d-tag | Query Function | What It Contains | Triggered By |
|
||||
|-------|---------------|-----------------|-------------|
|
||||
| `event_kinds` | `query_event_kind_distribution()` | Event count per kind, total events | Event storage |
|
||||
| `time_stats` | `query_time_based_statistics()` | Events in last 24h/7d/30d | Event storage |
|
||||
| `top_pubkeys` | `query_top_pubkeys()` | Top 10 pubkeys by event count | Event storage |
|
||||
| `subscription_details` | `query_subscription_details()` | Active subscriptions list | Subscription changes |
|
||||
| `cpu_metrics` | `query_cpu_metrics()` | PID, memory, CPU, connections | Both triggers |
|
||||
|
||||
### Current Admin Command System
|
||||
|
||||
Two parallel admin interfaces exist:
|
||||
|
||||
**1. Kind 23456 Admin Events (WebSocket)**
|
||||
- Admin sends NIP-44 encrypted commands as kind 23456 events
|
||||
- Relay decrypts, processes, responds with kind 23457 events
|
||||
- Commands: config get/set, auth rules, system commands, SQL queries
|
||||
|
||||
**2. NIP-17 DM Admin (Direct Messages)**
|
||||
- Admin sends gift-wrapped DMs (kind 1059) to relay
|
||||
- Relay unwraps, processes commands, responds via DM
|
||||
- Commands: stats, config, help, SQL queries
|
||||
|
||||
### Current Problems
|
||||
|
||||
1. **All monitoring runs on lws-main thread** — DB queries block the event loop
|
||||
2. **Monitoring is triggered by event storage** — `generate_event_driven_monitoring()` runs synchronously after each event insert
|
||||
3. **No dedicated monitoring thread** — monitoring piggybacks on the relay's main processing
|
||||
4. **Throttle is time-based only** — `kind_24567_reporting_throttle_sec` (default 5s) but queries still run on main thread
|
||||
5. **Dashboard polls via subscription** — no way to get data without the relay actively generating events
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Plan
|
||||
|
||||
### Phase 1: Dedicated API Thread (`api-worker`)
|
||||
|
||||
**What:** Create a new thread that owns all monitoring event generation. It runs on a timer, queries PostgreSQL with its own connection, builds and signs monitoring events, and pushes them into the broadcast system.
|
||||
|
||||
**Thread behavior:**
|
||||
```
|
||||
api-worker thread:
|
||||
1. Open own PG connection
|
||||
2. Loop:
|
||||
a. Sleep for throttle_sec interval
|
||||
b. Check if anyone is subscribed to kind 24567
|
||||
c. If yes: run all monitoring queries
|
||||
d. Build kind 24567 events with results
|
||||
e. Sign events with relay key
|
||||
f. Push completed events to lws-main for broadcast
|
||||
g. Repeat
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
- The api-worker thread does ALL the heavy work (queries + signing)
|
||||
- It only hands off pre-built, signed events to lws-main for broadcast
|
||||
- lws-main never blocks on monitoring queries
|
||||
- The thread sleeps when no one is subscribed (zero overhead)
|
||||
|
||||
**Files to change:**
|
||||
- `src/api.c`: Refactor `generate_monitoring_event_for_type()` to be callable from any thread
|
||||
- `src/websockets.c`: Add monitor thread lifecycle (start/stop), completion queue for broadcast
|
||||
- `src/thread_pool.h`: Add api-worker thread config options
|
||||
|
||||
**New thread in the process:**
|
||||
```
|
||||
c_relay_pg process
|
||||
├── lws-main — WebSocket event loop
|
||||
├── db-read-1..N — Async REQ/COUNT queries
|
||||
├── event-worker — Async EVENT ingestion
|
||||
├── db-write-1..N — Async sub logging, misc writes
|
||||
└── api-worker — NEW: periodic monitoring event generation
|
||||
```
|
||||
|
||||
### Phase 2: Remove Monitoring from Main Thread Path
|
||||
|
||||
**What:** Remove `generate_event_driven_monitoring()` and `generate_subscription_driven_monitoring()` calls from the synchronous event processing path.
|
||||
|
||||
**Currently these are called from:**
|
||||
- After event storage (event-driven monitoring)
|
||||
- After subscription create/close (subscription-driven monitoring)
|
||||
|
||||
**After change:**
|
||||
- The api-worker thread handles all monitoring on its own timer
|
||||
- Event storage and subscription changes no longer trigger monitoring directly
|
||||
- The throttle interval controls how fresh the data is (default 5 seconds)
|
||||
|
||||
**Benefit:** Event processing becomes faster — no monitoring overhead per event.
|
||||
|
||||
### Phase 3: Dashboard Uses Normal Subscriptions for Real-Time Events
|
||||
|
||||
**What:** Instead of the relay generating special monitoring events for "recent events," the dashboard simply subscribes to all events using standard Nostr REQ filters.
|
||||
|
||||
**Current approach:** Dashboard subscribes to kind 24567 monitoring events that contain aggregated stats.
|
||||
|
||||
**New approach for real-time event feed:**
|
||||
```javascript
|
||||
// Dashboard subscribes to ALL events on the relay
|
||||
relayPool.subscribeMany([url], [
|
||||
{ kinds: [1, 3, 5, 7, ...], limit: 50 } // Recent events
|
||||
], {
|
||||
onevent(event) {
|
||||
// Render in real-time event feed
|
||||
addEventToFeed(event);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**What this replaces:** The "Relay Events" page in the dashboard currently shows events from monitoring. Instead, it would show live events as they arrive via normal Nostr subscription.
|
||||
|
||||
**What stays as monitoring events:** Aggregated stats (event_kinds, time_stats, top_pubkeys, cpu_metrics, subscription_details) still need to be generated by the relay because they require SQL aggregation queries that a client can't do.
|
||||
|
||||
### Phase 4: Evaluate Which Monitoring Events Can Be Replaced
|
||||
|
||||
For each current monitoring event type, here's whether it should stay as a relay-generated event or be replaced by dashboard-side logic:
|
||||
|
||||
| d-tag | Current: Relay generates | Could dashboard do it? | Recommendation |
|
||||
|-------|------------------------|----------------------|----------------|
|
||||
| `event_kinds` | SQL: `SELECT kind, COUNT(*) FROM events GROUP BY kind` | No — requires DB aggregation | **Keep as monitoring event** |
|
||||
| `time_stats` | SQL: `SELECT COUNT(*) FROM events WHERE created_at > ?` for 24h/7d/30d | No — requires DB aggregation | **Keep as monitoring event** |
|
||||
| `top_pubkeys` | SQL: `SELECT pubkey, COUNT(*) FROM events GROUP BY pubkey ORDER BY count DESC LIMIT 10` | No — requires DB aggregation | **Keep as monitoring event** |
|
||||
| `subscription_details` | SQL: `SELECT * FROM active_subscriptions_log` | No — requires DB access | **Keep as monitoring event** |
|
||||
| `cpu_metrics` | Reads `/proc/self/stat`, memory info | No — requires server-side access | **Keep as monitoring event** |
|
||||
| Real-time event feed | Currently via monitoring events | **Yes — normal Nostr subscription** | **Replace with REQ subscription** |
|
||||
| Auth rules list | Currently via admin command | No — requires DB access | **Keep as admin command** |
|
||||
| Config values | Currently via admin command | No — requires DB access | **Keep as admin command** |
|
||||
|
||||
**Conclusion:** All 5 monitoring event types should stay as relay-generated events (they require SQL aggregation or server-side data). The only thing that changes is the real-time event feed, which becomes a normal subscription.
|
||||
|
||||
### Phase 5: PostgreSQL LISTEN/NOTIFY for Monitoring Triggers (Future)
|
||||
|
||||
**What:** Instead of the monitor thread polling on a timer, PostgreSQL can push notifications when data changes.
|
||||
|
||||
**Example triggers:**
|
||||
```sql
|
||||
-- Notify when a new event is stored
|
||||
CREATE OR REPLACE FUNCTION notify_event_stored() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
PERFORM pg_notify('event_stored', json_build_object(
|
||||
'kind', NEW.kind,
|
||||
'pubkey', substring(NEW.pubkey, 1, 8)
|
||||
)::text);
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER trg_notify_event_stored
|
||||
AFTER INSERT ON events
|
||||
FOR EACH ROW EXECUTE FUNCTION notify_event_stored();
|
||||
```
|
||||
|
||||
**How the monitor thread would use it:**
|
||||
```
|
||||
api-worker thread:
|
||||
1. LISTEN event_stored
|
||||
2. Wait for notification (blocks efficiently)
|
||||
3. On notification: batch up for throttle_sec, then generate monitoring events
|
||||
4. Repeat
|
||||
```
|
||||
|
||||
**Benefit:** Zero-polling. The monitor thread only wakes up when data actually changes.
|
||||
|
||||
**This is optional and can be added later.** The timer-based approach in Phase 1 works well and is simpler.
|
||||
|
||||
---
|
||||
|
||||
## Updated Thread Layout (After All Phases)
|
||||
|
||||
```
|
||||
c_relay_pg process
|
||||
├── lws-main — WebSocket event loop (never blocks on DB for monitoring)
|
||||
├── db-read-1..4 — Async REQ/COUNT queries
|
||||
├── event-worker — Async EVENT ingestion (validate + store)
|
||||
├── db-write-1..2 — Async sub logging, IP bans, misc writes
|
||||
└── api-worker — Periodic monitoring event generation (own PG conn)
|
||||
```
|
||||
|
||||
**Total: 9 threads, 9 PG connections** (with 4 readers + 2 writers)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
| Phase | Description | Complexity | Impact |
|
||||
|-------|-------------|-----------|--------|
|
||||
| 1 | Dedicated api-worker thread | Medium | High — unblocks main thread |
|
||||
| 2 | Remove monitoring from event processing path | Low | Medium — faster event processing |
|
||||
| 3 | Dashboard uses normal subscriptions for event feed | Low (frontend only) | Medium — simpler, more reliable |
|
||||
| 4 | Evaluate monitoring event types | Analysis only | Confirms architecture |
|
||||
| 5 | PostgreSQL LISTEN/NOTIFY triggers | Medium | Low (optimization) — add later |
|
||||
|
||||
**Recommended start:** Phase 1 + 2 together (they're coupled), then Phase 3 (frontend change), then Phase 5 when needed.
|
||||
|
||||
---
|
||||
|
||||
## Relationship to Thread Pool Plan
|
||||
|
||||
This plan complements `thread_pool_pg_optimization_plan.md`:
|
||||
|
||||
- Thread pool plan handles **reader/writer scaling** for client-facing operations
|
||||
- This plan handles **monitoring/API operations** on a dedicated thread
|
||||
- Both plans share the same PG connection model (one connection per thread)
|
||||
- The api-worker thread is independent from the thread pool — it has its own lifecycle
|
||||
|
||||
The combined target is:
|
||||
```
|
||||
lws-main (1) + readers (4) + event-worker (1) + writers (2) + api-worker (1) = 9 threads
|
||||
```
|
||||
@@ -0,0 +1,240 @@
|
||||
# C-Relay-PG Admin Page Migration Plan
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate the relay admin dashboard from an embedded HTTP-served page (`api/index.html` + `api/index.js`) to a full client-ndk page (`c-relay-pg.html`) built on the client-ndk template. The page will live in the client-ndk project (`~/lt/client-ndk/www/`) alongside other pages and communicate with the relay purely through Nostr protocol subscriptions.
|
||||
|
||||
## Why This Migration
|
||||
|
||||
The current `/api` page is served directly by the relay's `lws-main` thread, which causes:
|
||||
- **Slow page delivery** — HTTP file serving competes with WebSocket event processing on a single thread
|
||||
- **Incomplete page loads** — 10-second `timeout_secs` kills connections when the main thread is busy
|
||||
- **6+ sequential HTTP round-trips** — each asset requires a two-phase header/body delivery cycle
|
||||
- **Unnecessary complexity** — embedded file generation, `lws_set_wsi_user()` overwriting, session data type dispatch
|
||||
|
||||
The admin page already communicates entirely via Nostr protocol:
|
||||
- **Kind 23456** → Admin commands (encrypted, sent by admin)
|
||||
- **Kind 23457** → Admin responses (encrypted, sent by relay)
|
||||
- **Kind 24567** → Monitoring/stats events (ephemeral, broadcast by relay)
|
||||
- **NIP-11** → Relay info (fetched via HTTP, independent of file serving)
|
||||
- **NIP-17** → DM admin commands (gift-wrapped)
|
||||
|
||||
Moving to client-ndk means the relay serves zero HTML — it only speaks Nostr protocol.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph client-ndk project
|
||||
A[c-relay-pg.html] --> B[init-ndk.mjs]
|
||||
B --> C[NDK SharedWorker]
|
||||
end
|
||||
|
||||
subgraph c-relay-pg
|
||||
D[WebSocket Server]
|
||||
E[api-worker thread]
|
||||
end
|
||||
|
||||
C -->|kind 23456 admin cmds| D
|
||||
D -->|kind 23457 responses| C
|
||||
E -->|kind 24567 monitoring| D
|
||||
D -->|kind 24567 events| C
|
||||
C -->|kind 1059 NIP-17 DMs| D
|
||||
```
|
||||
|
||||
## Current Admin Page Sections
|
||||
|
||||
| Section | Data Source | Communication |
|
||||
|---------|-----------|---------------|
|
||||
| Statistics | kind 24567 monitoring events with d-tags: event_kinds, time_stats, top_pubkeys, cpu_metrics | Subscribe to kind 24567 |
|
||||
| Subscriptions | kind 24567 monitoring events with d-tag: subscription_details | Subscribe to kind 24567 |
|
||||
| Configuration | kind 23457 admin responses | Send kind 23456 config_query, receive kind 23457 |
|
||||
| Authorization | kind 23457 admin responses | Send kind 23456 auth rule commands, receive kind 23457 |
|
||||
| IP Bans | kind 23457 admin responses | Send kind 23456 ip_ban_* commands, receive kind 23457 |
|
||||
| Relay Events | Live subscription to all events + kind 0/10050/10002 management | Standard Nostr subscriptions |
|
||||
| DM | NIP-17 gift-wrapped DMs kind 1059 | NIP-17 encrypt/wrap/publish |
|
||||
| SQL Query | kind 23457 admin responses | Send kind 23456 sql_query, receive kind 23457 |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
1. **Full client-ndk page** — Uses `client.css`, `init-ndk.mjs`, `HamburgerMorphing`, `relay-ui.mjs`, `blossom-ui.mjs`, `ai-ui.mjs` — all standard template infrastructure stays intact
|
||||
2. **Sidenav preserved** — All existing sidenav sections (Relay, Blossom, AI) remain. Admin page navigation links are added to `divSideNavBody` as clickable items
|
||||
3. **Page-specific CSS** — Follows client-ndk convention: page styles in inline `<style>` block, using only CSS variables from `client.css`
|
||||
4. **Hardcoded relay URL** — `wss://relay.laantungir.net` for now; future enhancement to support relay selection
|
||||
5. **Auth mode: required** — Admin page requires authentication
|
||||
|
||||
## Migration Steps
|
||||
|
||||
### Step 1: Create `c-relay-pg.html` from Template
|
||||
|
||||
Start from `template.html` in client-ndk. Customize:
|
||||
|
||||
- **Title**: `C-Relay-PG`
|
||||
- **Header text**: `C-Relay-PG Admin`
|
||||
- **Auth mode**: `required` (default)
|
||||
- **`divBody` content**: All 8 admin sections, each in its own container div, all `display:none` except Statistics (default)
|
||||
- **`divSideNavBody`**: Add navigation links for each section:
|
||||
```html
|
||||
<div id="divAdminNav">
|
||||
<div class="adminNavItem active" data-section="statistics">Statistics</div>
|
||||
<div class="adminNavItem" data-section="subscriptions">Subscriptions</div>
|
||||
<div class="adminNavItem" data-section="configuration">Configuration</div>
|
||||
<div class="adminNavItem" data-section="authorization">Authorization</div>
|
||||
<div class="adminNavItem" data-section="ip-bans">IP Bans</div>
|
||||
<div class="adminNavItem" data-section="relay-events">Relay Events</div>
|
||||
<div class="adminNavItem" data-section="dm">DM</div>
|
||||
<div class="adminNavItem" data-section="database">Database Query</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### Step 2: Port Admin Communication Layer
|
||||
|
||||
Replace `SimplePool` + `nostr-lite.js` with NDK equivalents:
|
||||
|
||||
| Current index.js | New c-relay-pg.html |
|
||||
|-----------------|---------------------|
|
||||
| `new SimplePool()` | `subscribe()` from init-ndk.mjs |
|
||||
| `relayPool.subscribeMany()` | `subscribe(filter, opts)` |
|
||||
| `relayPool.publish()` | `publishEvent(event)` |
|
||||
| `window.nostr.signEvent()` | Handled by NDK MessageBasedSigner |
|
||||
| `window.NostrTools.nip42` | Handled by NDK NIP-42 support |
|
||||
| `window.NostrTools.nip44` | `window.NostrTools.nip44` from nostr.bundle.js — same as before |
|
||||
| `nlLite` nostr-login-lite | `initNDKPage()` + `getPubkey()` |
|
||||
|
||||
Key function — `sendAdminCommand(commandArray)`:
|
||||
```javascript
|
||||
async function sendAdminCommand(commandArray) {
|
||||
const conversationKey = window.NostrTools.nip44.v2.utils.getConversationKey(
|
||||
adminPrivateKey, relayPubkey
|
||||
);
|
||||
const encrypted = window.NostrTools.nip44.v2.encrypt(
|
||||
JSON.stringify(commandArray), conversationKey
|
||||
);
|
||||
|
||||
const event = {
|
||||
created_at: Math.floor(Date.now() / 1000),
|
||||
kind: 23456,
|
||||
tags: [['p', relayPubkey]],
|
||||
content: encrypted
|
||||
};
|
||||
|
||||
return await publishEvent(event);
|
||||
}
|
||||
```
|
||||
|
||||
Note: NIP-44 encryption requires the admin's private key. The current page uses `window.nostr.nip44.encrypt()` which delegates to the browser extension. NDK's `publishEvent` handles signing, but encryption must be done before publishing. We'll use `window.nostr.nip44.encrypt()` if available (NIP-07 extension), same as the current page does.
|
||||
|
||||
### Step 3: Port Subscription Setup
|
||||
|
||||
Using NDK's `subscribe()` and `window.addEventListener('ndkEvent', ...)`:
|
||||
|
||||
```javascript
|
||||
// Subscribe to admin responses + monitoring events from relay
|
||||
const adminSub = subscribe(
|
||||
{ kinds: [23457, 24567], authors: [RELAY_PUBKEY] },
|
||||
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
|
||||
);
|
||||
|
||||
// Live event feed
|
||||
const liveSub = subscribe(
|
||||
{ limit: 50 },
|
||||
{ closeOnEose: false, cacheUsage: 'ONLY_RELAY' }
|
||||
);
|
||||
|
||||
// Route events to handlers
|
||||
window.addEventListener('ndkEvent', (e) => {
|
||||
const evt = e.detail;
|
||||
if (evt.kind === 23457) processAdminResponse(evt);
|
||||
else if (evt.kind === 24567) processMonitoringEvent(evt);
|
||||
else addEventToLiveFeed(evt);
|
||||
});
|
||||
```
|
||||
|
||||
### Step 4: Port NIP-44 Encryption
|
||||
|
||||
Admin commands require NIP-44 encryption. The template already loads `nostr.bundle.js` which includes `window.NostrTools.nip44`. Additionally, `window.nostr.nip44.encrypt()` is available from NIP-07 extensions. Use the same approach as the current page.
|
||||
|
||||
### Step 5: Port UI Sections into divBody
|
||||
|
||||
Each section's HTML goes into `divBody` as a container div. Page-specific styles go in the inline `<style>` block following client-ndk conventions:
|
||||
- Use CSS variables only (no hardcoded colors)
|
||||
- Tables use `var(--primary-color)`, `var(--muted-color)`, `var(--accent-color)`
|
||||
- Buttons follow the `var(--button-*)` pattern
|
||||
- Font is always `var(--font-family)`
|
||||
|
||||
Sections to port:
|
||||
1. **Statistics** — DB overview table, event kinds table, time stats table, top pubkeys table, event rate chart area
|
||||
2. **Subscriptions** — Active subscription details table
|
||||
3. **Configuration** — Config key/value table with inline edit, refresh button
|
||||
4. **Authorization** — Auth rules table, whitelist/blacklist inputs, WoT level selector
|
||||
5. **IP Bans** — Ban stats, manual ban form, IP whitelist, ban list table with filters
|
||||
6. **Relay Events** — Live event feed table, kind 0/10050/10002 management forms
|
||||
7. **DM** — NIP-17 message textarea + send button, inbox display
|
||||
8. **SQL Query** — Query dropdown, SQL textarea, execute button, results table
|
||||
|
||||
### Step 6: Page-Specific CSS
|
||||
|
||||
Following client-ndk rules from `client.css`:
|
||||
- Override `#divBody` layout: `flex-direction: column; overflow-y: auto; padding: 20px;`
|
||||
- Admin section containers: hidden by default, shown when nav item clicked
|
||||
- Table styles: use `var(--primary-color)` for headers, `var(--muted-color)` for borders
|
||||
- Button styles: border + transparent background, hover with `var(--accent-color)`
|
||||
- All in inline `<style media="screen">` block in `<head>`
|
||||
|
||||
### Step 7: Relay URL — Hardcoded for Now
|
||||
|
||||
```javascript
|
||||
const RELAY_WS_URL = 'wss://relay.laantungir.net';
|
||||
const RELAY_HTTP_URL = 'https://relay.laantungir.net';
|
||||
```
|
||||
|
||||
NIP-11 fetch to get relay pubkey:
|
||||
```javascript
|
||||
const nip11 = await fetch(RELAY_HTTP_URL, {
|
||||
headers: { 'Accept': 'application/nostr+json' }
|
||||
}).then(r => r.json());
|
||||
const RELAY_PUBKEY = nip11.pubkey;
|
||||
```
|
||||
|
||||
### Step 8: Admin Verification
|
||||
|
||||
Same flow as current page:
|
||||
1. Get relay pubkey from NIP-11
|
||||
2. Send kind 23456 `system_status` command
|
||||
3. If relay responds with kind 23457, user is verified as admin
|
||||
4. Show admin sections; otherwise show access denied
|
||||
|
||||
### Step 9 (Optional, Later): Remove Embedded HTTP Serving from Relay
|
||||
|
||||
After migration is complete and tested:
|
||||
- Remove `handle_embedded_file_request()` from `src/api.c`
|
||||
- Remove `handle_embedded_file_writeable()` from `src/api.c`
|
||||
- Remove embedded file dispatch from `LWS_CALLBACK_HTTP` in `src/websockets.c`
|
||||
- Remove `embed_web_files.sh` and `src/embedded_web_content.c`
|
||||
- Keep NIP-11 serving (lightweight and required by protocol)
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
### New File (in client-ndk project: ~/lt/client-ndk/www/)
|
||||
- `c-relay-pg.html` — Complete admin page, self-contained with inline `<style>` and `<script type="module">`
|
||||
|
||||
### Files Unchanged
|
||||
- All relay C code for kind 23456/23457/24567 processing — stays exactly the same
|
||||
- `src/dm_admin.c` — NIP-17 DM handling stays the same
|
||||
- `src/nip011.c` — NIP-11 stays the same
|
||||
- All client-ndk shared files (client.css, init-ndk.mjs, relay-ui.mjs, etc.) — no modifications needed
|
||||
|
||||
### Files to Modify Later (optional relay cleanup)
|
||||
- `src/api.c` — Remove embedded file serving functions
|
||||
- `src/websockets.c` — Remove embedded file dispatch from HTTP callback
|
||||
- `Makefile` — Remove embedded web content compilation step
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| NIP-44 encryption via window.nostr.nip44 | Same mechanism as current page — no change in encryption approach |
|
||||
| NDK SharedWorker relay connection | Use ONLY_RELAY cache usage to ensure direct relay communication |
|
||||
| Large page size with all sections inline | Sections are display:none by default — no performance impact |
|
||||
| Event rate chart from text_graph.js | Port chart logic inline or skip initially |
|
||||
| Relay pubkey needed before subscriptions | Fetch NIP-11 first in initialization sequence |
|
||||
@@ -0,0 +1,632 @@
|
||||
# c-relay-pg-pg Plan — PostgreSQL Backend + Multi-Instance + Dashboard
|
||||
|
||||
## Overview
|
||||
|
||||
**c-relay-pg-pg** is a new project (separate repository) forked from c-relay-pg after the `db_ops` abstraction layer is complete. It replaces the SQLite backend with PostgreSQL, enabling horizontal scaling, external dashboards, and production-grade deployments.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- The `db_ops.h` / `db_ops.c` abstraction layer must be complete in c-relay-pg first — see [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1.
|
||||
- The fork happens after Phase 1 of that plan is done and tested.
|
||||
|
||||
### Why a Separate Project?
|
||||
|
||||
| | c-relay-pg | c-relay-pg-pg |
|
||||
|---|---------|-----------|
|
||||
| **Database** | SQLite (embedded) | PostgreSQL (client-server) |
|
||||
| **Deployment** | Single binary + DB file | Binary + PostgreSQL server |
|
||||
| **Scaling** | Single instance | Multiple instances + load balancer |
|
||||
| **Dashboard** | Embedded web UI | Separate web server + Grafana |
|
||||
| **Target user** | Personal relay, small community | Medium-large relay, production |
|
||||
| **Complexity** | Minimal | More infrastructure |
|
||||
|
||||
For the full analysis of why PostgreSQL was chosen over MySQL/MariaDB and other databases, see the [database architecture analysis](database_architecture_analysis.md).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph c-relay-pg-pg - Production Deployment
|
||||
LB[nginx - TLS + load balancing] -->|WebSocket| R1[c-relay-pg-pg Instance 1]
|
||||
LB -->|WebSocket| R2[c-relay-pg-pg Instance 2]
|
||||
LB -->|HTTP| DASH[Dashboard]
|
||||
R1 -->|db_ops API| PGBACK[PostgreSQL Backend - db_ops_postgres.c]
|
||||
R2 -->|db_ops API| PGBACK
|
||||
PGBACK -->|libpq| PG[PostgreSQL Server]
|
||||
DASH -->|SQL| PG
|
||||
R1 <-->|LISTEN/NOTIFY| R2
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: PostgreSQL Backend
|
||||
|
||||
### Goal
|
||||
|
||||
Fork c-relay-pg into a new repository called **c-relay-pg-pg**. Replace the SQLite `db_ops` implementation with PostgreSQL using `libpq`. The `db_ops.h` interface stays identical — only the backend changes.
|
||||
|
||||
### New Files
|
||||
|
||||
- `src/db_ops_sqlite.c` — Renamed from `db_ops.c` (the Phase 1 SQLite implementation from c-relay-pg)
|
||||
- `src/db_ops_postgres.c` — New PostgreSQL implementation
|
||||
- `src/db_ops.c` — Thin dispatcher that calls the active backend
|
||||
|
||||
### PostgreSQL Schema
|
||||
|
||||
```sql
|
||||
-- PostgreSQL schema for c-relay-pg-pg
|
||||
-- No event_tags table needed — JSONB + GIN handles everything
|
||||
|
||||
CREATE TABLE events (
|
||||
id TEXT PRIMARY KEY,
|
||||
pubkey TEXT NOT NULL,
|
||||
created_at BIGINT NOT NULL,
|
||||
kind INTEGER NOT NULL,
|
||||
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
|
||||
content TEXT NOT NULL,
|
||||
sig TEXT NOT NULL,
|
||||
tags JSONB NOT NULL DEFAULT '[]',
|
||||
event_json TEXT NOT NULL,
|
||||
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
|
||||
);
|
||||
|
||||
-- Core indexes
|
||||
CREATE INDEX idx_events_pubkey ON events(pubkey);
|
||||
CREATE INDEX idx_events_kind ON events(kind);
|
||||
CREATE INDEX idx_events_created_at ON events(created_at DESC);
|
||||
CREATE INDEX idx_events_kind_created_at ON events(kind, created_at DESC);
|
||||
CREATE INDEX idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
|
||||
CREATE INDEX idx_events_pubkey_kind ON events(pubkey, kind);
|
||||
|
||||
-- THE KEY INDEX: GIN on JSONB tags — replaces the entire event_tags table
|
||||
CREATE INDEX idx_events_tags ON events USING GIN (tags);
|
||||
|
||||
-- Partial index: only non-ephemeral events (what REQ queries actually filter)
|
||||
CREATE INDEX idx_events_non_ephemeral ON events(created_at DESC)
|
||||
WHERE kind < 20000 OR kind >= 30000;
|
||||
|
||||
-- Config table
|
||||
CREATE TABLE config (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL,
|
||||
data_type TEXT NOT NULL CHECK (data_type IN ('string', 'integer', 'boolean', 'json')),
|
||||
description TEXT,
|
||||
category TEXT DEFAULT 'general',
|
||||
requires_restart INTEGER DEFAULT 0,
|
||||
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
|
||||
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
|
||||
);
|
||||
|
||||
-- Auth rules table
|
||||
CREATE TABLE auth_rules (
|
||||
id SERIAL PRIMARY KEY,
|
||||
rule_type TEXT NOT NULL CHECK (rule_type IN ('whitelist', 'blacklist', 'rate_limit', 'auth_required', 'wot_whitelist')),
|
||||
pattern_type TEXT NOT NULL CHECK (pattern_type IN ('pubkey', 'kind', 'ip', 'global')),
|
||||
pattern_value TEXT,
|
||||
active INTEGER NOT NULL DEFAULT 1,
|
||||
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
|
||||
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
|
||||
);
|
||||
|
||||
CREATE INDEX idx_auth_rules_lookup ON auth_rules(rule_type, pattern_type, pattern_value) WHERE active = 1;
|
||||
|
||||
-- Relay private key storage
|
||||
CREATE TABLE relay_seckey (
|
||||
private_key_hex TEXT NOT NULL CHECK (length(private_key_hex) = 64),
|
||||
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
|
||||
);
|
||||
|
||||
-- Subscription logging
|
||||
CREATE TABLE subscriptions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
subscription_id TEXT NOT NULL,
|
||||
wsi_pointer TEXT NOT NULL,
|
||||
client_ip TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL CHECK (event_type IN ('created', 'closed', 'expired', 'disconnected')),
|
||||
filter_json TEXT,
|
||||
events_sent INTEGER DEFAULT 0,
|
||||
created_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
|
||||
ended_at BIGINT,
|
||||
duration INTEGER,
|
||||
UNIQUE(subscription_id, wsi_pointer)
|
||||
);
|
||||
|
||||
-- IP ban persistence
|
||||
CREATE TABLE ip_bans (
|
||||
ip TEXT PRIMARY KEY,
|
||||
failure_count INTEGER NOT NULL DEFAULT 0,
|
||||
ban_count INTEGER NOT NULL DEFAULT 0,
|
||||
banned_until BIGINT NOT NULL DEFAULT 0,
|
||||
first_failure BIGINT NOT NULL DEFAULT 0,
|
||||
has_authed_successfully INTEGER NOT NULL DEFAULT 0,
|
||||
last_success_at BIGINT NOT NULL DEFAULT 0,
|
||||
total_connections INTEGER NOT NULL DEFAULT 0,
|
||||
total_failures INTEGER NOT NULL DEFAULT 0,
|
||||
total_successes INTEGER NOT NULL DEFAULT 0,
|
||||
first_seen BIGINT NOT NULL DEFAULT 0,
|
||||
idle_failure_count INTEGER NOT NULL DEFAULT 0,
|
||||
idle_ban_count INTEGER NOT NULL DEFAULT 0,
|
||||
idle_banned_until BIGINT NOT NULL DEFAULT 0,
|
||||
idle_first_failure BIGINT NOT NULL DEFAULT 0
|
||||
);
|
||||
|
||||
-- Materialized views for dashboard (refreshed periodically)
|
||||
CREATE MATERIALIZED VIEW event_kinds_mv AS
|
||||
SELECT kind, COUNT(*) as count,
|
||||
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
|
||||
FROM events GROUP BY kind;
|
||||
|
||||
CREATE MATERIALIZED VIEW time_stats_mv AS
|
||||
SELECT 'total' as period, COUNT(*) as total_events, COUNT(DISTINCT pubkey) as unique_pubkeys
|
||||
FROM events
|
||||
UNION ALL
|
||||
SELECT '24h', COUNT(*), COUNT(DISTINCT pubkey)
|
||||
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 86400
|
||||
UNION ALL
|
||||
SELECT '7d', COUNT(*), COUNT(DISTINCT pubkey)
|
||||
FROM events WHERE created_at >= EXTRACT(EPOCH FROM NOW())::BIGINT - 604800;
|
||||
|
||||
CREATE MATERIALIZED VIEW top_pubkeys_mv AS
|
||||
SELECT pubkey, COUNT(*) as event_count,
|
||||
ROUND(COUNT(*) * 100.0 / NULLIF((SELECT COUNT(*) FROM events), 0), 2) as percentage
|
||||
FROM events GROUP BY pubkey ORDER BY event_count DESC LIMIT 20;
|
||||
|
||||
-- Unique indexes required for CONCURRENTLY refresh
|
||||
CREATE UNIQUE INDEX idx_event_kinds_mv ON event_kinds_mv(kind);
|
||||
CREATE UNIQUE INDEX idx_time_stats_mv ON time_stats_mv(period);
|
||||
CREATE UNIQUE INDEX idx_top_pubkeys_mv ON top_pubkeys_mv(pubkey);
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### Tag Queries with JSONB
|
||||
|
||||
The biggest win — replacing the `event_tags` subquery with JSONB containment:
|
||||
|
||||
```c
|
||||
// SQLite (current): subquery into event_tags table
|
||||
// AND id IN (SELECT event_id FROM event_tags WHERE tag_name = ? AND tag_value IN (?))
|
||||
|
||||
// PostgreSQL: JSONB containment operator with GIN index
|
||||
// AND tags @> '[["p", "pubkey_hex"]]'
|
||||
|
||||
// For multiple tag values:
|
||||
// AND (tags @> '[["p", "val1"]]' OR tags @> '[["p", "val2"]]')
|
||||
```
|
||||
|
||||
This eliminates the entire `event_tags` table (4 million rows, ~2 GB indexes in the current SQLite schema). The GIN index on JSONB handles everything in ~100–200 MB.
|
||||
|
||||
#### LISTEN/NOTIFY Integration
|
||||
|
||||
For cross-instance event broadcasting:
|
||||
|
||||
```c
|
||||
// In db_ops_postgres.c:
|
||||
int db_notify_new_event(const char* event_id) {
|
||||
char cmd[128];
|
||||
snprintf(cmd, sizeof(cmd), "NOTIFY new_event, '%s'", event_id);
|
||||
PGresult* res = PQexec(g_pg_conn, cmd);
|
||||
PQclear(res);
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Called from the lws event loop every iteration:
|
||||
int db_check_notifications(char* event_id_out, size_t max_len) {
|
||||
PQconsumeInput(g_pg_notify_conn);
|
||||
PGnotify* notify = PQnotifies(g_pg_notify_conn);
|
||||
if (notify) {
|
||||
strncpy(event_id_out, notify->extra, max_len);
|
||||
PQfreemem(notify);
|
||||
return 1; // Got a notification
|
||||
}
|
||||
return 0; // No notifications
|
||||
}
|
||||
```
|
||||
|
||||
#### Connection Management
|
||||
|
||||
```c
|
||||
// db_ops_postgres.c connection pool (simple version)
|
||||
#define DB_POOL_SIZE 4
|
||||
|
||||
static PGconn* g_pg_read_pool[DB_POOL_SIZE]; // For REQ queries
|
||||
static PGconn* g_pg_write_conn; // For EVENT inserts
|
||||
static PGconn* g_pg_notify_conn; // For LISTEN/NOTIFY
|
||||
static pthread_mutex_t g_pool_lock;
|
||||
|
||||
PGconn* db_get_read_connection(void) {
|
||||
pthread_mutex_lock(&g_pool_lock);
|
||||
// Round-robin or find idle connection
|
||||
// ...
|
||||
pthread_mutex_unlock(&g_pool_lock);
|
||||
}
|
||||
```
|
||||
|
||||
### Build System Changes
|
||||
|
||||
```makefile
|
||||
# Makefile additions
|
||||
DB_BACKEND ?= sqlite # Default to sqlite, override with: make DB_BACKEND=postgres
|
||||
|
||||
ifeq ($(DB_BACKEND),postgres)
|
||||
MAIN_SRC += src/db_ops.c src/db_ops_postgres.c
|
||||
LIBS += -lpq
|
||||
CFLAGS += -DDB_BACKEND_POSTGRES
|
||||
else
|
||||
MAIN_SRC += src/db_ops.c src/db_ops_sqlite.c
|
||||
CFLAGS += -DDB_BACKEND_SQLITE
|
||||
endif
|
||||
```
|
||||
|
||||
### Data Migration Tool
|
||||
|
||||
A standalone tool to migrate existing SQLite data to PostgreSQL:
|
||||
|
||||
```bash
|
||||
# migrate_to_postgres.sh
|
||||
# 1. Export events from SQLite
|
||||
sqlite3 old_relay.db ".mode csv" "SELECT id,pubkey,created_at,kind,event_type,content,sig,tags,event_json,first_seen FROM events" > events.csv
|
||||
|
||||
# 2. Import into PostgreSQL
|
||||
psql -d crelay -c "\COPY events FROM 'events.csv' WITH CSV"
|
||||
|
||||
# 3. Migrate config
|
||||
sqlite3 old_relay.db ".mode csv" "SELECT key,value,data_type,description,category,requires_restart FROM config" > config.csv
|
||||
psql -d crelay -c "\COPY config(key,value,data_type,description,category,requires_restart) FROM 'config.csv' WITH CSV"
|
||||
|
||||
# 4. Migrate auth rules
|
||||
sqlite3 old_relay.db ".mode csv" "SELECT rule_type,pattern_type,pattern_value,active FROM auth_rules" > auth_rules.csv
|
||||
psql -d crelay -c "\COPY auth_rules(rule_type,pattern_type,pattern_value,active) FROM 'auth_rules.csv' WITH CSV"
|
||||
|
||||
# 5. Refresh materialized views
|
||||
psql -d crelay -c "REFRESH MATERIALIZED VIEW event_kinds_mv; REFRESH MATERIALIZED VIEW time_stats_mv; REFRESH MATERIALIZED VIEW top_pubkeys_mv;"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Multi-Instance + Dashboard
|
||||
|
||||
### Goal
|
||||
|
||||
Run multiple c-relay-pg-pg instances behind a load balancer with a separate dashboard web server, all sharing the same PostgreSQL database.
|
||||
|
||||
### Components
|
||||
|
||||
1. **nginx config** — WebSocket load balancing with `ip_hash` stickiness
|
||||
2. **systemd template** — `c-relay-pg-pg@.service` for multiple instances
|
||||
3. **LISTEN/NOTIFY integration** — Cross-instance event broadcasting in the `lws_service()` loop
|
||||
4. **PgBouncer** — Connection pooling between c-relay-pg-pg instances and PostgreSQL
|
||||
5. **Dashboard** — Separate web server querying PostgreSQL directly
|
||||
6. **Materialized view refresh** — Background job to refresh analytics views
|
||||
|
||||
### Multi-Instance Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
INTERNET[Internet - Nostr Clients] -->|wss://relay.example.com| NGINX[nginx reverse proxy - TLS termination + load balancing]
|
||||
|
||||
NGINX -->|ws://localhost:8888| R1[c-relay-pg-pg Instance 1 - port 8888]
|
||||
NGINX -->|ws://localhost:8889| R2[c-relay-pg-pg Instance 2 - port 8889]
|
||||
NGINX -->|ws://localhost:8890| R3[c-relay-pg-pg Instance 3 - port 8890]
|
||||
NGINX -->|http://localhost:3000| DASHWEB[Dashboard Web Server]
|
||||
|
||||
R1 -->|libpq connection pool| PGPOOL[PgBouncer - connection pooler]
|
||||
R2 -->|libpq connection pool| PGPOOL
|
||||
R3 -->|libpq connection pool| PGPOOL
|
||||
|
||||
PGPOOL -->|pooled connections| PG[PostgreSQL Primary]
|
||||
DASHWEB -->|read queries| PG
|
||||
|
||||
PG -->|streaming replication| REPLICA[Read Replica - optional]
|
||||
DASHWEB -.->|heavy analytics| REPLICA
|
||||
```
|
||||
|
||||
### nginx Load Balancing Config
|
||||
|
||||
```nginx
|
||||
# nginx.conf - WebSocket load balancing for c-relay-pg-pg
|
||||
upstream relay_backends {
|
||||
# ip_hash ensures a client always hits the same instance
|
||||
# (important for WebSocket session stickiness)
|
||||
ip_hash;
|
||||
|
||||
server 127.0.0.1:8888;
|
||||
server 127.0.0.1:8889;
|
||||
server 127.0.0.1:8890;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name relay.example.com;
|
||||
|
||||
# TLS config...
|
||||
|
||||
location / {
|
||||
proxy_pass http://relay_backends;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_read_timeout 86400; # Keep WebSocket alive for 24h
|
||||
}
|
||||
|
||||
# Dashboard on separate path
|
||||
location /dashboard {
|
||||
proxy_pass http://127.0.0.1:3000;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Session stickiness** (`ip_hash`) ensures that once a client connects to Instance 2, all their subsequent WebSocket frames go to Instance 2. This is important because subscriptions are held in-memory per instance.
|
||||
|
||||
### Starting Multiple Instances
|
||||
|
||||
Each instance is the same binary, just on a different port, all pointing to the same PostgreSQL:
|
||||
|
||||
```bash
|
||||
# Instance 1
|
||||
./build/c_relay_pg_x86 --port 8888 --db-host localhost --db-name crelay &
|
||||
|
||||
# Instance 2
|
||||
./build/c_relay_pg_x86 --port 8889 --db-host localhost --db-name crelay &
|
||||
|
||||
# Instance 3
|
||||
./build/c_relay_pg_x86 --port 8890 --db-host localhost --db-name crelay &
|
||||
```
|
||||
|
||||
Or with systemd template units:
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/c-relay-pg-pg@.service
|
||||
[Unit]
|
||||
Description=C-Relay-PG-PG Nostr Instance %i
|
||||
After=postgresql.service
|
||||
|
||||
[Service]
|
||||
ExecStart=/opt/c-relay-pg-pg/c_relay_pg_x86 --port %i --db-host localhost --db-name crelay
|
||||
Restart=always
|
||||
User=c-relay-pg
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl enable c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
|
||||
systemctl start c-relay-pg-pg@8888 c-relay-pg-pg@8889 c-relay-pg-pg@8890
|
||||
```
|
||||
|
||||
### Cross-Instance Event Broadcasting
|
||||
|
||||
When Instance 1 receives a new EVENT and stores it in PostgreSQL, Instance 2 and Instance 3 need to know about it so they can broadcast to their connected subscribers.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client_A as Client A - connected to Instance 1
|
||||
participant I1 as Instance 1
|
||||
participant PG as PostgreSQL
|
||||
participant I2 as Instance 2
|
||||
participant I3 as Instance 3
|
||||
participant Client_B as Client B - connected to Instance 2
|
||||
participant Client_C as Client C - connected to Instance 3
|
||||
|
||||
Client_A->>I1: EVENT - new kind:1 note
|
||||
I1->>PG: INSERT INTO events...
|
||||
PG-->>I1: OK
|
||||
I1->>I1: broadcast to local subscribers
|
||||
I1->>PG: NOTIFY new_event with event_id
|
||||
|
||||
Note over PG: PostgreSQL delivers notification to all listeners
|
||||
|
||||
PG-->>I2: NOTIFY: new_event event_id
|
||||
PG-->>I3: NOTIFY: new_event event_id
|
||||
|
||||
I2->>PG: SELECT event_json FROM events WHERE id = event_id
|
||||
PG-->>I2: event JSON
|
||||
I2->>I2: match against local subscriptions
|
||||
I2->>Client_B: EVENT message - if subscription matches
|
||||
|
||||
I3->>PG: SELECT event_json FROM events WHERE id = event_id
|
||||
PG-->>I3: event JSON
|
||||
I3->>I3: match against local subscriptions
|
||||
I3->>Client_C: EVENT message - if subscription matches
|
||||
```
|
||||
|
||||
#### PostgreSQL LISTEN/NOTIFY Implementation
|
||||
|
||||
Built into PostgreSQL — no additional infrastructure needed:
|
||||
|
||||
```c
|
||||
// === In the relay's event loop (modified lws_service loop) ===
|
||||
|
||||
// Setup: create a dedicated connection for LISTEN
|
||||
PGconn* notify_conn = PQconnectdb("host=localhost dbname=crelay");
|
||||
PQexec(notify_conn, "LISTEN new_event");
|
||||
int notify_fd = PQsocket(notify_conn); // Get the socket fd for poll()
|
||||
|
||||
// After storing an event:
|
||||
void on_event_stored(PGconn* write_conn, const char* event_id) {
|
||||
char notify_cmd[128];
|
||||
snprintf(notify_cmd, sizeof(notify_cmd),
|
||||
"NOTIFY new_event, '%s'", event_id);
|
||||
PQexec(write_conn, notify_cmd);
|
||||
}
|
||||
|
||||
// In the main event loop (runs every lws_service iteration):
|
||||
void check_cross_instance_events(PGconn* notify_conn) {
|
||||
// Non-blocking check for notifications
|
||||
PQconsumeInput(notify_conn);
|
||||
|
||||
PGnotify* notify;
|
||||
while ((notify = PQnotifies(notify_conn)) != NULL) {
|
||||
// Another instance stored a new event
|
||||
const char* event_id = notify->extra;
|
||||
|
||||
// Fetch the event and check against local subscriptions
|
||||
cJSON* event = db_get_event_by_id(event_id);
|
||||
if (event) {
|
||||
broadcast_event_to_subscriptions(event);
|
||||
cJSON_Delete(event);
|
||||
}
|
||||
|
||||
PQfreemem(notify);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Performance**: LISTEN/NOTIFY adds ~1–5ms latency for cross-instance delivery. For a Nostr relay, this is imperceptible — clients already expect network latency.
|
||||
|
||||
**Payload limit**: NOTIFY payloads are limited to 8000 bytes. For event IDs (64 hex chars), this is fine.
|
||||
|
||||
#### Alternative: Redis Pub/Sub
|
||||
|
||||
If you later need even lower latency or more sophisticated routing:
|
||||
|
||||
```c
|
||||
// Using hiredis (Redis C client)
|
||||
redisContext* redis = redisConnect("127.0.0.1", 6379);
|
||||
|
||||
// After storing event:
|
||||
redisCommand(redis, "PUBLISH new_event %s", event_json);
|
||||
|
||||
// Subscriber (in each instance):
|
||||
redisCommand(redis, "SUBSCRIBE new_event");
|
||||
// Then poll for messages in the event loop
|
||||
```
|
||||
|
||||
Redis adds sub-millisecond pub/sub but requires running a Redis server. For most relays, PostgreSQL LISTEN/NOTIFY is sufficient.
|
||||
|
||||
### Connection Pooling with PgBouncer
|
||||
|
||||
Each relay instance needs multiple database connections. Without pooling, 3 instances × 10 connections = 30 PostgreSQL backend processes. With PgBouncer:
|
||||
|
||||
```ini
|
||||
# /etc/pgbouncer/pgbouncer.ini
|
||||
[databases]
|
||||
crelay = host=127.0.0.1 port=5432 dbname=crelay
|
||||
|
||||
[pgbouncer]
|
||||
listen_port = 6432
|
||||
listen_addr = 127.0.0.1
|
||||
auth_type = md5
|
||||
pool_mode = transaction # Return connection to pool after each transaction
|
||||
max_client_conn = 200 # Total connections from all relay instances
|
||||
default_pool_size = 20 # Actual PostgreSQL connections
|
||||
```
|
||||
|
||||
Relay instances connect to PgBouncer (port 6432) instead of PostgreSQL directly (port 5432). PgBouncer multiplexes 200 client connections onto 20 actual PostgreSQL connections.
|
||||
|
||||
### Zero-Downtime Deployment
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant LB as nginx
|
||||
participant I1 as Instance 1 - v1.0
|
||||
participant I2 as Instance 2 - v1.0
|
||||
participant I3 as Instance 3 - v1.0
|
||||
participant I1_NEW as Instance 1 - v1.1
|
||||
|
||||
Note over LB,I3: Normal operation: 3 instances serving traffic
|
||||
|
||||
LB->>I1: Mark upstream as down
|
||||
Note over I1: Drain: wait for existing connections to close or timeout
|
||||
I1->>I1: Graceful shutdown
|
||||
|
||||
Note over LB: Traffic now goes to I2 and I3 only
|
||||
|
||||
I1_NEW->>I1_NEW: Start with new binary
|
||||
I1_NEW->>LB: Health check passes
|
||||
LB->>I1_NEW: Mark upstream as up
|
||||
|
||||
Note over LB,I1_NEW: Instance 1 now running v1.1, repeat for I2 and I3
|
||||
```
|
||||
|
||||
Rolling deploy script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# rolling_deploy.sh - Zero-downtime deployment
|
||||
|
||||
for port in 8888 8889 8890; do
|
||||
echo "Deploying instance on port $port..."
|
||||
|
||||
# 1. Tell nginx to stop sending new connections
|
||||
sed -i "s/server 127.0.0.1:$port;/server 127.0.0.1:$port down;/" /etc/nginx/nginx.conf
|
||||
nginx -s reload
|
||||
|
||||
# 2. Wait for existing connections to drain (30 seconds)
|
||||
sleep 30
|
||||
|
||||
# 3. Stop old instance
|
||||
systemctl stop c-relay-pg-pg@$port
|
||||
|
||||
# 4. Deploy new binary
|
||||
cp ./build/c_relay_pg_x86 /opt/c-relay-pg-pg/c_relay_pg_x86
|
||||
|
||||
# 5. Start new instance
|
||||
systemctl start c-relay-pg-pg@$port
|
||||
|
||||
# 6. Wait for health check
|
||||
sleep 5
|
||||
|
||||
# 7. Re-enable in nginx
|
||||
sed -i "s/server 127.0.0.1:$port down;/server 127.0.0.1:$port;/" /etc/nginx/nginx.conf
|
||||
nginx -s reload
|
||||
|
||||
echo "Instance on port $port deployed successfully"
|
||||
done
|
||||
```
|
||||
|
||||
### Dashboard Options
|
||||
|
||||
With PostgreSQL, the dashboard can be built with any technology:
|
||||
|
||||
- **Grafana** — Point directly at PostgreSQL, zero custom code
|
||||
- **Custom web app** — React/Vue frontend + any backend (Node, Python, Go) querying PostgreSQL
|
||||
- **Embedded in c-relay-pg-pg** — Keep current approach but queries go through `db_ops` to PostgreSQL (no longer blocks event loop since PostgreSQL handles concurrency)
|
||||
|
||||
### Scaling Scenarios
|
||||
|
||||
| Scenario | Instances | Connections | Events/hour | Setup |
|
||||
|----------|-----------|-------------|-------------|-------|
|
||||
| **Current** | 1 | ~1,200 | 56 | Single process + SQLite |
|
||||
| **Small upgrade** | 2 | ~2,500 | 500 | 2 instances + PostgreSQL |
|
||||
| **Medium relay** | 4 | ~5,000 | 5,000 | 4 instances + PostgreSQL + PgBouncer |
|
||||
| **Large relay** | 8 | ~10,000 | 50,000 | 8 instances + PostgreSQL + read replica |
|
||||
| **Multi-region** | 2–4 per region | ~50,000 | 500,000 | Multiple servers + PostgreSQL replication |
|
||||
|
||||
### Cost Perspective
|
||||
|
||||
Running multiple relay instances with PostgreSQL on a single VPS:
|
||||
- **PostgreSQL**: ~200–500 MB RAM baseline
|
||||
- **PgBouncer**: ~10 MB RAM
|
||||
- **Each relay instance**: ~50–100 MB RAM
|
||||
- **Dashboard web server**: ~50–100 MB RAM
|
||||
- **4 instances + PostgreSQL + PgBouncer + dashboard**: ~1–2 GB total RAM
|
||||
- A **$20/month VPS** with 4 cores and 4 GB RAM handles this easily
|
||||
- A **$40/month VPS** with 8 cores and 8 GB RAM handles 8 instances comfortably
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| PostgreSQL connection failures | `db_ops` returns error codes. Relay logs errors but doesn't crash. Reconnection logic with exponential backoff. |
|
||||
| Performance regression | Benchmark before/after. PostgreSQL should be faster for complex queries, similar for simple ones. |
|
||||
| Data loss during migration | Migration tool is read-only on SQLite. PostgreSQL import is idempotent (INSERT ON CONFLICT). |
|
||||
| SQLite fallback needed | Keep `db_ops_sqlite.c` working. Compile-time flag switches backends. Can always go back. |
|
||||
| LISTEN/NOTIFY message loss | NOTIFY is transactional — if the INSERT commits, the NOTIFY is guaranteed. Missed notifications during reconnect handled by periodic full-sync. |
|
||||
| PgBouncer adds latency | Transaction pooling mode adds <0.1ms. Negligible compared to query time. |
|
||||
|
||||
---
|
||||
|
||||
## Relationship to c-relay-pg
|
||||
|
||||
This project depends on the `db_ops.h` abstraction layer built in [c-relay-pg thread pool plan](c_relay_pg_thread_pool_plan.md) Phase 1. The interface is identical — only the backend implementation changes:
|
||||
|
||||
- **c-relay-pg**: `db_ops.c` → SQLite calls via `sqlite3_*`
|
||||
- **c-relay-pg-pg**: `db_ops_postgres.c` → PostgreSQL calls via `libpq` (`PQexec`, `PQgetvalue`, etc.)
|
||||
|
||||
The `db_ops.h` header file is shared between both projects. Changes to the interface should be coordinated.
|
||||
@@ -0,0 +1,332 @@
|
||||
# c-relay-pg Thread Pool Plan — db_ops Abstraction + SQLite Thread Pool
|
||||
|
||||
## Overview
|
||||
|
||||
This plan covers improvements to **c-relay-pg** (this repo) — the lean, single-binary, embedded-SQLite Nostr relay. Two phases:
|
||||
|
||||
1. **Phase 1: Database Abstraction Layer** — Consolidate all scattered `sqlite3_*` calls into a single `db_ops.h` / `db_ops.c` module. Pure refactor, no behavior change.
|
||||
2. **Phase 2: SQLite Thread Pool** — Add worker threads for concurrent reads, unblocking the `lws_service()` event loop.
|
||||
|
||||
This is the **final form of c-relay-pg**: an embedded-SQLite relay with clean architecture and non-blocking database access. The abstraction layer also enables a future fork to PostgreSQL — see [c-relay-pg-pg plan](c_relay_pg_pg_plan.md).
|
||||
|
||||
### Why This First?
|
||||
|
||||
The current architecture has a fundamental bottleneck documented in the [database architecture analysis](database_architecture_analysis.md): the single-threaded event loop blocks on every database call. A 672ms REQ query freezes every connected client. The thread pool fixes this without changing the database engine or deployment model.
|
||||
|
||||
| Metric | Current | After Thread Pool |
|
||||
|--------|---------|-------------------|
|
||||
| **Event loop blocking** | 0.1–672ms per query | Near-zero |
|
||||
| **Concurrent reads** | 1 | 4–8 |
|
||||
| **Deployment** | Single binary + DB file | Same |
|
||||
| **Database** | SQLite | Same |
|
||||
| **Code risk** | N/A | Low — additive change |
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph c-relay-pg - Final Form
|
||||
RELAY[c-relay-pg binary] -->|db_ops API| DBOPS[db_ops.c - abstraction layer]
|
||||
DBOPS -->|sqlite3 calls| SQLITE[SQLite WAL database]
|
||||
end
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph c-relay-pg with Thread Pool
|
||||
LWS[lws_service event loop] -->|REQ arrives| Q[Thread-safe job queue]
|
||||
LWS -->|EVENT arrives| WQ[Write queue - single writer]
|
||||
Q --> T1[Reader Thread 1 - own sqlite3*]
|
||||
Q --> T2[Reader Thread 2 - own sqlite3*]
|
||||
Q --> T3[Reader Thread 3 - own sqlite3*]
|
||||
Q --> T4[Reader Thread 4 - own sqlite3*]
|
||||
WQ --> TW[Writer Thread - own sqlite3*]
|
||||
T1 -->|results| CB[Callback to lws event loop]
|
||||
T2 -->|results| CB
|
||||
T3 -->|results| CB
|
||||
T4 -->|results| CB
|
||||
TW -->|OK/error| CB
|
||||
CB -->|queue_message| LWS
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Database Abstraction Layer
|
||||
|
||||
### Goal
|
||||
|
||||
Consolidate all 258 scattered `sqlite3_*` calls across 8 files into a single `db_ops.h` / `db_ops.c` module with a backend-agnostic interface. SQLite continues to work — this is a pure refactor.
|
||||
|
||||
### Files That Currently Touch SQLite Directly
|
||||
|
||||
| File | `sqlite3_*` calls | Operations |
|
||||
|------|-------------------|------------|
|
||||
| [`src/main.c`](../src/main.c) | ~80 | Event store/retrieve, tag storage, REQ queries, COUNT queries, DB init, schema migration |
|
||||
| [`src/config.c`](../src/config.c) | ~60 | Config CRUD, auth rules CRUD, WoT sync, relay key storage, startup sequence |
|
||||
| [`src/api.c`](../src/api.c) | ~40 | Stats queries, monitoring views, admin SQL execution, config management |
|
||||
| [`src/dm_admin.c`](../src/dm_admin.c) | ~20 | Auth rule management, WoT whitelist operations |
|
||||
| [`src/websockets.c`](../src/websockets.c) | ~15 | COUNT queries, IP ban stats, connection tracking |
|
||||
| [`src/subscriptions.c`](../src/subscriptions.c) | ~15 | Subscription logging — create/close/disconnect |
|
||||
| [`src/nip009.c`](../src/nip009.c) | ~10 | Event deletion — by ID and by address |
|
||||
| [`src/request_validator.c`](../src/request_validator.c) | ~8 | Blacklist/whitelist checks |
|
||||
| [`src/ip_ban.c`](../src/ip_ban.c) | ~15 | IP ban table CRUD, persistence |
|
||||
|
||||
### Abstraction Layer Design
|
||||
|
||||
New files: `src/db_ops.h` and `src/db_ops.c`
|
||||
|
||||
```c
|
||||
// src/db_ops.h — Database operations abstraction layer
|
||||
#ifndef DB_OPS_H
|
||||
#define DB_OPS_H
|
||||
|
||||
#include "../nostr_core_lib/cjson/cJSON.h"
|
||||
|
||||
// ============================================================
|
||||
// Lifecycle
|
||||
// ============================================================
|
||||
int db_init(const char* connection_string); // SQLite: file path, PG: connection string
|
||||
void db_close(void);
|
||||
int db_is_available(void);
|
||||
|
||||
// ============================================================
|
||||
// Schema / Migration
|
||||
// ============================================================
|
||||
int db_ensure_schema(void);
|
||||
int db_get_schema_version(void);
|
||||
int db_execute_raw(const char* sql); // For schema DDL only
|
||||
|
||||
// ============================================================
|
||||
// Event Operations
|
||||
// ============================================================
|
||||
int db_store_event(cJSON* event);
|
||||
int db_event_exists(const char* event_id);
|
||||
char* db_get_event_json(const char* event_id); // Caller must free
|
||||
int db_delete_event_by_id(const char* event_id, const char* requester_pubkey);
|
||||
int db_delete_events_by_address(const char* pubkey, int kind,
|
||||
const char* d_tag, long before_timestamp);
|
||||
|
||||
// ============================================================
|
||||
// Event Queries - REQ handling
|
||||
// ============================================================
|
||||
typedef struct {
|
||||
int* kinds; int kind_count;
|
||||
char** authors; int author_count;
|
||||
char** ids; int id_count;
|
||||
char** tag_names; // Parallel arrays: tag_names[i] has tag_values[i][]
|
||||
char*** tag_values;
|
||||
int* tag_value_counts;
|
||||
int tag_filter_count;
|
||||
long since; // 0 = not set
|
||||
long until; // 0 = not set
|
||||
int limit; // 0 = default 500
|
||||
char* search; // NIP-50 search term, NULL = not set
|
||||
} db_event_filter_t;
|
||||
|
||||
typedef struct db_result db_result_t;
|
||||
|
||||
db_result_t* db_query_events(const db_event_filter_t* filter);
|
||||
const char* db_result_next_json(db_result_t* result); // Returns event_json, NULL when done
|
||||
int db_result_row_count(db_result_t* result);
|
||||
void db_result_free(db_result_t* result);
|
||||
|
||||
int db_count_events(const db_event_filter_t* filter);
|
||||
|
||||
// ============================================================
|
||||
// Config Operations
|
||||
// ============================================================
|
||||
char* db_get_config(const char* key); // Caller must free, NULL if not found
|
||||
int db_set_config(const char* key, const char* value, const char* data_type,
|
||||
const char* description, const char* category, int requires_restart);
|
||||
int db_update_config(const char* key, const char* value);
|
||||
int db_config_exists(const char* key);
|
||||
int db_get_config_count(void);
|
||||
|
||||
// ============================================================
|
||||
// Auth Rules Operations
|
||||
// ============================================================
|
||||
int db_add_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
|
||||
int db_remove_auth_rule(const char* rule_type, const char* pattern_type, const char* pattern_value);
|
||||
int db_auth_rule_exists(const char* rule_type, const char* pattern_type, const char* pattern_value);
|
||||
int db_clear_auth_rules(const char* rule_type); // NULL = clear all
|
||||
int db_is_blacklisted(const char* pubkey);
|
||||
int db_is_whitelisted(const char* pubkey);
|
||||
int db_has_any_whitelist(void);
|
||||
|
||||
// ============================================================
|
||||
// Relay Key Storage
|
||||
// ============================================================
|
||||
int db_store_relay_private_key(const char* privkey_hex);
|
||||
char* db_get_relay_private_key(void); // Caller must free
|
||||
|
||||
// ============================================================
|
||||
// Subscription Logging
|
||||
// ============================================================
|
||||
int db_log_subscription_created(const char* sub_id, const char* wsi_ptr,
|
||||
const char* client_ip, const char* filter_json);
|
||||
int db_log_subscription_closed(const char* sub_id, const char* client_ip);
|
||||
int db_log_subscription_disconnected(const char* client_ip);
|
||||
int db_update_subscription_events_sent(const char* sub_id, int events_sent);
|
||||
int db_cleanup_orphaned_subscriptions(void);
|
||||
|
||||
// ============================================================
|
||||
// IP Ban Persistence
|
||||
// ============================================================
|
||||
int db_ensure_ip_ban_table(void);
|
||||
int db_load_ip_bans(void* ban_table, int table_size); // Populates in-memory table
|
||||
int db_save_ip_bans(const void* ban_table, int table_size);
|
||||
|
||||
// ============================================================
|
||||
// Analytics / Stats - for dashboard
|
||||
// ============================================================
|
||||
cJSON* db_get_event_kind_distribution(void);
|
||||
cJSON* db_get_time_based_stats(void);
|
||||
cJSON* db_get_top_pubkeys(int limit);
|
||||
cJSON* db_get_subscription_details(void);
|
||||
int db_get_total_event_count(void);
|
||||
|
||||
// ============================================================
|
||||
// Admin SQL Query
|
||||
// ============================================================
|
||||
cJSON* db_execute_admin_query(const char* sql, char* error_msg, size_t error_size);
|
||||
|
||||
#endif // DB_OPS_H
|
||||
```
|
||||
|
||||
### Migration Strategy for Phase 1
|
||||
|
||||
The key principle: **change the interface, not the behavior**. Each function in `db_ops.c` initially just wraps the existing SQLite calls.
|
||||
|
||||
**Step-by-step for each file:**
|
||||
|
||||
1. **Create `db_ops.h` and `db_ops.c`** with the interface above
|
||||
2. **Implement each `db_ops` function** by moving the existing SQLite code from the source files into `db_ops.c`
|
||||
3. **Replace direct `sqlite3_*` calls** in each source file with `db_ops_*` calls
|
||||
4. **Remove `extern sqlite3* g_db`** from each file — only `db_ops.c` knows about `g_db`
|
||||
5. **Remove `#include <sqlite3.h>`** from each file — only `db_ops.c` includes it
|
||||
6. **Update Makefile** to compile `db_ops.c`
|
||||
|
||||
### Order of Migration (by risk, lowest first)
|
||||
|
||||
1. **`src/ip_ban.c`** — Self-contained, simple CRUD. Good warmup.
|
||||
2. **`src/subscriptions.c`** — Logging only, no critical path.
|
||||
3. **`src/nip009.c`** — Event deletion, small file.
|
||||
4. **`src/request_validator.c`** — Auth checks, small file.
|
||||
5. **`src/api.c`** — Stats/monitoring queries. Larger but read-only.
|
||||
6. **`src/dm_admin.c`** — Auth rules + WoT. Medium complexity.
|
||||
7. **`src/config.c`** — Config CRUD. Large but well-structured.
|
||||
8. **`src/websockets.c`** — COUNT queries, minimal DB usage.
|
||||
9. **`src/main.c`** — Event store/retrieve, REQ queries. The big one — do last.
|
||||
|
||||
### Testing Phase 1
|
||||
|
||||
After Phase 1, the relay should behave **identically** to before. Run:
|
||||
- `tests/run_all_tests.sh` — Full test suite
|
||||
- `tests/performance_benchmarks.sh` — Verify no performance regression
|
||||
- Manual testing with production-like traffic
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: SQLite Thread Pool
|
||||
|
||||
### Goal
|
||||
|
||||
Add a pool of N worker threads, each with its own `sqlite3*` connection to the same database file. SQLite WAL mode (already enabled) supports **concurrent readers**. The event loop dispatches database work to the pool and remains responsive.
|
||||
|
||||
### Key Design Points
|
||||
|
||||
1. **Read path**: REQ queries dispatched to thread pool. Each worker opens its own `sqlite3*` connection. SQLite WAL allows unlimited concurrent readers.
|
||||
|
||||
2. **Write path**: EVENT inserts go through a single dedicated writer thread. SQLite only allows one writer at a time anyway — this serializes writes cleanly.
|
||||
|
||||
3. **Result delivery**: Worker threads cannot call `lws_write()` directly (libwebsockets is not thread-safe). Instead, they push results into a per-session message queue and call `lws_cancel_service()` to wake the event loop, which then drains the queue.
|
||||
|
||||
4. **Connection lifecycle**: Each thread opens its own connection with `PRAGMA journal_mode=WAL` and `PRAGMA busy_timeout=5000`.
|
||||
|
||||
### What Changes in the Codebase
|
||||
|
||||
| Component | Current | Thread Pool |
|
||||
|-----------|---------|-------------|
|
||||
| [`g_db`](../src/main.c:49) | Single global connection | One per thread + writer connection |
|
||||
| [`handle_req_message()`](../src/main.c:1101) | Synchronous SQL in callback | Package filter into job, dispatch to pool |
|
||||
| [`store_event()`](../src/main.c:787) | Synchronous INSERT in callback | Dispatch to writer thread |
|
||||
| [`handle_count_message()`](../src/websockets.c:2863) | Synchronous COUNT in callback | Dispatch to pool |
|
||||
| Result delivery | Direct `queue_message()` | Worker pushes to queue + `lws_cancel_service()` |
|
||||
| Config reads | Direct `sqlite3_prepare` on `g_db` | Can stay synchronous with own connection or cache |
|
||||
|
||||
### New Files
|
||||
|
||||
- `src/thread_pool.h` — Thread pool interface
|
||||
- `src/thread_pool.c` — Thread pool implementation (job queue, worker lifecycle, result delivery)
|
||||
|
||||
### Thread Pool Interface (sketch)
|
||||
|
||||
```c
|
||||
// src/thread_pool.h
|
||||
#ifndef THREAD_POOL_H
|
||||
#define THREAD_POOL_H
|
||||
|
||||
typedef enum {
|
||||
JOB_TYPE_REQ_QUERY,
|
||||
JOB_TYPE_COUNT_QUERY,
|
||||
JOB_TYPE_STORE_EVENT,
|
||||
JOB_TYPE_DELETE_EVENT,
|
||||
} job_type_t;
|
||||
|
||||
typedef struct {
|
||||
job_type_t type;
|
||||
void* session; // lws per-session data pointer
|
||||
db_event_filter_t filter; // For REQ/COUNT jobs
|
||||
cJSON* event; // For STORE jobs
|
||||
char event_id[65]; // For DELETE jobs
|
||||
} db_job_t;
|
||||
|
||||
int thread_pool_init(int num_readers, const char* db_path);
|
||||
void thread_pool_shutdown(void);
|
||||
int thread_pool_submit_read(db_job_t* job);
|
||||
int thread_pool_submit_write(db_job_t* job);
|
||||
|
||||
#endif // THREAD_POOL_H
|
||||
```
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| **Concurrent reads** | N readers in parallel (N = thread count, typically 4–8) |
|
||||
| **Write throughput** | Same as current — SQLite serializes writes regardless |
|
||||
| **Event loop latency** | Near-zero — REQ no longer blocks the loop |
|
||||
| **Max theoretical read throughput** | ~4–8x current (limited by disk I/O, not CPU) |
|
||||
| **Memory overhead** | ~50–100 MB per connection (page cache) |
|
||||
| **Latency per query** | Same as current per-query, but no head-of-line blocking |
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Write contention**: SQLite still allows only ONE writer at a time. With WAL, readers don't block writers and writers don't block readers, but two simultaneous writes will serialize. At the current write rate (~56 events/hour), this is a non-issue.
|
||||
- **Database size**: The 2.7 GB index bloat problem remains. Thread pool doesn't fix the schema — it fixes the concurrency.
|
||||
- **Scaling ceiling**: Beyond ~8 reader threads, diminishing returns due to disk I/O contention on a single SQLite file.
|
||||
- **Complexity**: Need a proper job queue, thread lifecycle management, and careful handling of the lws ↔ worker thread boundary.
|
||||
|
||||
---
|
||||
|
||||
## Risk Mitigation
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Phase 1 introduces bugs | Each file migrated independently, tested after each. Full test suite runs after each file. |
|
||||
| Thread pool race conditions | Worker threads only touch their own `sqlite3*` connection. Result delivery uses a mutex-protected queue. `lws_cancel_service()` is documented as thread-safe. |
|
||||
| Performance regression from abstraction | Abstraction layer is thin wrappers — no extra allocations or copies. Benchmark before/after. |
|
||||
| lws thread safety issues | Workers never call `lws_write()` directly. They push to a queue and wake the event loop. This is the documented pattern for libwebsockets multi-threading. |
|
||||
| Rollback needed | Phase 1 is a pure refactor — easy to revert. Phase 2 thread pool is additive — can be disabled with a compile flag. |
|
||||
|
||||
---
|
||||
|
||||
## Relationship to c-relay-pg-pg
|
||||
|
||||
After Phase 1 (abstraction layer) is complete, the codebase is ready to be forked into a separate **c-relay-pg-pg** project that replaces the SQLite backend with PostgreSQL. See [c-relay-pg-pg plan](c_relay_pg_pg_plan.md) for details.
|
||||
|
||||
The `db_ops.h` interface is designed to work with any backend:
|
||||
- **SQLite** (this plan): `db_result_next_json()` copies from `sqlite3_column_text()`
|
||||
- **PostgreSQL** (c-relay-pg-pg): `db_result_next_json()` returns from `PQgetvalue()`
|
||||
- **LMDB** (future possibility): `db_result_next_json()` returns a zero-copy pointer into mmap'd memory
|
||||
@@ -0,0 +1,72 @@
|
||||
# Caching Code Consolidation Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Move all caching_relay source code into a `caching/` directory inside c-relay-pg,
|
||||
and switch the launcher to use PostgreSQL mode (`-p <pg-conn>`) so all config
|
||||
comes from the c-relay-pg config table — no JSON config file needed.
|
||||
|
||||
## Key Finding
|
||||
|
||||
The caching_relay binary **already supports PG mode** via `-p <pg-conn>`. Its
|
||||
`pg_config.c` reads the same config table keys the caching page sets
|
||||
(`caching_root_npubs`, `caching_bootstrap_relays`, `caching_kinds`, etc.), and
|
||||
`pg_inbox.c` writes fetched events to the `caching_event_inbox` table. The PG
|
||||
schema tables already exist in `src/pg_schema.sql`.
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Create `caching/` directory and copy source files
|
||||
|
||||
Copy all `src/*.c` and `src/*.h` from `/home/user/lt/caching_relay/src/` into
|
||||
`caching/src/` in this repo. Also copy the Makefile, build_static.sh,
|
||||
Dockerfile.alpine-musl, and VERSION.
|
||||
|
||||
Files to copy:
|
||||
- src/main.c, src/main.h
|
||||
- src/config.c, src/config.h (legacy JSON config — kept for fallback)
|
||||
- src/pg_config.c, src/pg_config.h (PG config reader)
|
||||
- src/pg_inbox.c, src/pg_inbox.h (PG inbox writer)
|
||||
- src/backfill.c, src/backfill.h
|
||||
- src/follow_graph.c, src/follow_graph.h
|
||||
- src/live_subscriber.c, src/live_subscriber.h
|
||||
- src/relay_sink.c, src/relay_sink.h (legacy WebSocket sink — kept for fallback)
|
||||
- src/relay_discovery.c, src/relay_discovery.h
|
||||
- src/state.c, src/state.h
|
||||
- src/debug.c, src/debug.h
|
||||
- src/jsonc_strip.c, src/jsonc_strip.h
|
||||
- Makefile
|
||||
- VERSION
|
||||
|
||||
### Step 2: Adapt the Makefile
|
||||
|
||||
Update `caching/Makefile` to:
|
||||
- Point `NOSTR_CORE_DIR` at `../nostr_core_lib` (sibling in this repo)
|
||||
- Output binary to `caching/caching_relay` or `build/caching_relay`
|
||||
|
||||
### Step 3: Update the launcher to use PG mode
|
||||
|
||||
Update `src/caching_service_launcher.c`:
|
||||
- Change `caching_service_start_fork()` to pass `-p <pg_conn>` instead of
|
||||
`-c <config_path>`
|
||||
- Update `caching_service_start()` to read `caching_service_pg_conn` from config
|
||||
table instead of `caching_service_config_path`
|
||||
|
||||
### Step 4: Remove config_path from defaults and UI
|
||||
|
||||
- Remove `caching_service_config_path` from `src/default_config_event.h`
|
||||
- Remove the "Caching Service Config Path" field from `api/index.html`
|
||||
- Remove the field from `CACHING_CONFIG_FIELDS` in `api/index.js`
|
||||
- Update `caching_service_binary_path` default to point at the new build
|
||||
location (`./caching/caching_relay` or `./build/caching_relay`)
|
||||
|
||||
### Step 5: Build and test
|
||||
|
||||
- Build the caching_relay binary from the new `caching/` directory
|
||||
- Build c-relay-pg with the updated launcher
|
||||
- Test: set caching config on the page, apply, start service, verify it runs
|
||||
in PG mode reading config from the database
|
||||
|
||||
### Step 6: Push
|
||||
|
||||
Run `./increment_and_push.sh` with a descriptive commit message.
|
||||
@@ -0,0 +1,503 @@
|
||||
# Caching Relay Daemon - Architecture Plan
|
||||
|
||||
> **Status:** Draft v3 - adds NIP-65 outbox model: bootstrap relays for
|
||||
> discovery, kind-10002 per-pubkey relay lists, minimum covering set relay
|
||||
> selection, local-relay-first on subsequent startups.
|
||||
|
||||
## Goal
|
||||
|
||||
A standalone C99 daemon that acts as a "caching relay feeder". It:
|
||||
|
||||
1. Reads a `.jsonc` config file listing one or more **root npubs** (e.g. your own npub), a set of upstream relays, a local relay URL, and configurable event kinds.
|
||||
2. For each root npub, fetches its kind-3 contact list to discover **followed pubkeys**.
|
||||
3. Subscribes **live** to events of the configured kinds from the union of followed pubkeys (root npubs + their follows).
|
||||
4. Performs a **throttled backfill** of historical events per followed pubkey, spread out over time so upstream relays are not hammered.
|
||||
5. Re-publishes every fetched event to the **local relay** via a plain WebSocket `EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg, or any Nostr relay).
|
||||
6. Builds as a **statically linked C99 binary** following the c-relay model, linking against `nostr_core_lib` and `c_utils_lib`.
|
||||
|
||||
The user's Nostr client then points only at the local relay and gets a fast, pre-populated feed without doing any fan-out itself.
|
||||
|
||||
## Design Principles
|
||||
|
||||
- **Relay-agnostic on the sink side.** The daemon is just a Nostr client publishing `EVENT` messages over WebSocket. It does not touch the local relay's database directly. This keeps it decoupled and safe.
|
||||
- **Reuse `nostr_core_lib`.** All Nostr protocol concerns (event validation, kind-3 parsing, relay pool, WebSocket client, NIP-19 npub decoding) come from `nostr_core_lib`. The daemon is orchestration logic only.
|
||||
- **Single statically linked binary.** Built with the same Makefile pattern as c-relay: link `libnostr_core_x64.a` (or arm64) + `libc_utils.a` + system libs (`-lwebsockets -lsqlite3 -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm`).
|
||||
- **C99, `-Wall -Wextra -std=c99 -g -O2`** matching c-relay's `CFLAGS`.
|
||||
- **No config-file framework.** Hand-rolled `.jsonc` parser using cJSON (strip `//` and `/* */` comments before parsing, since cJSON does not natively support JSONC).
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Config
|
||||
CFG[caching_relay.jsonc]
|
||||
end
|
||||
|
||||
subgraph Daemon
|
||||
MAIN[main.c - lifecycle / signals]
|
||||
CFGP[config.c - parse jsonc]
|
||||
FOLLOW[follow_graph.c - root npubs + kind-3 resolution]
|
||||
BACKFILL[backfill.c - throttled historical pull]
|
||||
LIVE[live_subscriber.c - open subscriptions]
|
||||
SINK[relay_sink.c - publish EVENT to local relay]
|
||||
STATE[state.c - in-memory follow set + seen cache]
|
||||
end
|
||||
|
||||
subgraph nostr_core_lib
|
||||
UPOOL[core_relay_pool.c - upstream pool - query/subscribe]
|
||||
SPOOL[core_relay_pool.c - sink pool - publish only]
|
||||
NIP01[nip001.c - validate / parse]
|
||||
NIP19[nip019.c - npub decode]
|
||||
WS[nostr_websocket - client WS]
|
||||
end
|
||||
|
||||
subgraph Upstream
|
||||
R1[wss://relay.damus.io]
|
||||
R2[wss://nos.lol]
|
||||
R3[wss://...]
|
||||
end
|
||||
|
||||
subgraph Local
|
||||
LOCAL[wss://127.0.0.1:8888 - c-relay / c-relay-pg / any]
|
||||
end
|
||||
|
||||
CFG --> CFGP
|
||||
CFGP --> MAIN
|
||||
MAIN --> FOLLOW
|
||||
FOLLOW -->|query kind 3| UPOOL
|
||||
UPOOL --> R1
|
||||
UPOOL --> R2
|
||||
UPOOL --> R3
|
||||
FOLLOW --> STATE
|
||||
MAIN --> LIVE
|
||||
LIVE -->|subscribe authors + kinds| UPOOL
|
||||
LIVE -->|on_event| SINK
|
||||
MAIN --> BACKFILL
|
||||
BACKFILL -->|query per pubkey since T| UPOOL
|
||||
BACKFILL -->|on_event| SINK
|
||||
SINK -->|publish_async| SPOOL
|
||||
SPOOL -->|EVENT json| LOCAL
|
||||
STATE --> LIVE
|
||||
STATE --> BACKFILL
|
||||
CFGP -->|read/write state| CFG
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant D as Daemon
|
||||
participant UP as upstream_pool
|
||||
participant SP as sink_pool
|
||||
participant U as Upstream Relays
|
||||
participant L as Local Relay
|
||||
|
||||
D->>D: parse caching_relay.jsonc + state
|
||||
D->>UP: add_relay x N upstreams
|
||||
D->>SP: add_relay local only
|
||||
D->>UP: query_sync kind=3 authors=root_npubs
|
||||
U-->>UP: kind-3 events
|
||||
UP-->>D: followed pubkeys set
|
||||
D->>D: merge root + follows into author set
|
||||
D->>UP: subscribe live authors=author_set kinds=configured since=now
|
||||
loop live loop - pump upstream_pool_run
|
||||
U-->>UP: EVENT
|
||||
UP-->>D: on_event callback
|
||||
D->>SP: publish_async EVENT
|
||||
SP->>L: EVENT json
|
||||
D->>SP: pump sink_pool_run to flush callbacks
|
||||
end
|
||||
loop backfill - progressive window expansion
|
||||
D->>UP: query_sync authors=pubkey_i kinds since=now-window limit=K
|
||||
U-->>UP: historical events
|
||||
UP-->>D: events
|
||||
D->>SP: publish_async each EVENT
|
||||
D->>D: sleep tick_interval_seconds
|
||||
D->>D: on full round-robin pass - advance window, write state to jsonc
|
||||
end
|
||||
```
|
||||
|
||||
## Config File Format
|
||||
|
||||
`caching_relay.jsonc` (JSONC = JSON with comments). The config file is the
|
||||
**single source of truth** and is also the daemon's persistent state store:
|
||||
the daemon rewrites it to disk whenever the backfill window advances, so a
|
||||
restart resumes exactly where it left off.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
// Root npubs whose follows list we crawl
|
||||
"root_npubs": [
|
||||
"npub1...",
|
||||
"npub1..."
|
||||
],
|
||||
// Upstream relays to pull events from
|
||||
"upstream_relays": [
|
||||
"wss://relay.damus.io",
|
||||
"wss://nos.lol",
|
||||
"wss://relay.nostr.band"
|
||||
],
|
||||
// Local relay to feed events into (publish-only, never queried)
|
||||
"local_relay": "ws://127.0.0.1:8888",
|
||||
// Event kinds to cache
|
||||
"kinds": [1, 3, 6, 10000, 30023],
|
||||
|
||||
// ---- Backfill: progressive window expansion ----
|
||||
"backfill": {
|
||||
"enabled": true,
|
||||
// Window schedule in seconds-from-now, applied in order.
|
||||
// The daemon first pulls everything newer than (now - 24h).
|
||||
// Once complete, it expands to (now - 7d), then (now - 30d), etc.
|
||||
"window_schedule_seconds": [86400, 604800, 2592000, 7776000, 31536000],
|
||||
// Per-pubkey query limit per tick (keeps individual queries light)
|
||||
"events_per_tick": 50,
|
||||
// Delay between pubkey backfill ticks (throttle to be polite)
|
||||
"tick_interval_seconds": 5,
|
||||
// Delay between completing one window and starting the next
|
||||
"window_cooldown_seconds": 60
|
||||
},
|
||||
|
||||
// ---- Live subscription ----
|
||||
"live": {
|
||||
"enabled": true,
|
||||
"resubscribe_interval_seconds": 300
|
||||
},
|
||||
|
||||
// Refresh the follow graph periodically
|
||||
"follow_graph_refresh_seconds": 600,
|
||||
|
||||
// ---- Persistent state (managed by the daemon; do not hand-edit) ----
|
||||
// The daemon writes these back to this file as backfill progresses.
|
||||
// On restart it reads them to avoid re-pulling already-cached history.
|
||||
"state": {
|
||||
// How far back we have fully backfilled, as a unix timestamp.
|
||||
// Starts at 0 / absent on first run. Advances as each window completes.
|
||||
"backfilled_until": 0,
|
||||
// Index into window_schedule_seconds we are currently working on.
|
||||
"current_window_index": 0,
|
||||
// Round-robin cursor: which followed pubkey we backfill next.
|
||||
"backfill_cursor": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**State write-back rules:**
|
||||
- The daemon rewrites the `.jsonc` file (preserving comments is *not* required
|
||||
on rewrite - it may emit plain JSON once it has been modified at runtime;
|
||||
the comments are only for the user's initial authoring convenience).
|
||||
- Write-back happens: (a) when a window completes and `backfilled_until`
|
||||
advances, (b) on graceful shutdown, (c) periodically (e.g. every 60s) so a
|
||||
crash loses at most one minute of cursor progress.
|
||||
- A write-ahead temp file + rename is used so the config is never left
|
||||
half-written.
|
||||
|
||||
## Progressive Window-Expansion Backfill
|
||||
|
||||
Instead of a fixed per-pubkey cursor, the daemon uses a **global window** that
|
||||
expands backwards in time in discrete steps. This is simpler, gives a natural
|
||||
"recent first" experience, and is trivially resumable from the config file.
|
||||
|
||||
### Window schedule
|
||||
|
||||
`backfill.window_schedule_seconds` is an ordered list, e.g.
|
||||
`[86400, 604800, 2592000, 7776000, 31536000]`
|
||||
= 1 day, 1 week, 1 month, 3 months, 1 year.
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. On startup, read `state.backfilled_until` (unix ts) and
|
||||
`state.current_window_index` from the config.
|
||||
2. If `backfilled_until == 0` (first run), set the current target window to
|
||||
`window_schedule[0]` (e.g. 24h). The backfill `since` cutoff is
|
||||
`now - window_schedule[0]`.
|
||||
3. Round-robin through every followed pubkey. For each pubkey, issue
|
||||
`nostr_relay_pool_query_sync` with:
|
||||
- `authors = [pubkey]`
|
||||
- `kinds = configured kinds`
|
||||
- `since = now - current_window_seconds` (i.e. the *full* current window,
|
||||
not an incremental slice - the local relay dedups, so re-overlap is free
|
||||
and simpler than tracking per-pubkey cursors)
|
||||
- `limit = events_per_tick`
|
||||
4. Sleep `tick_interval_seconds` between pubkeys (throttle).
|
||||
5. When one full round-robin pass over all followed pubkeys completes for the
|
||||
current window:
|
||||
- Set `state.backfilled_until = now - current_window_seconds`.
|
||||
- Advance `state.current_window_index += 1`.
|
||||
- Persist the config file (temp + rename).
|
||||
- Sleep `window_cooldown_seconds` before starting the next (wider) window.
|
||||
6. Repeat with the next (wider) window. The `since` cutoff moves further back
|
||||
in time, so each window pulls the *additional* older slice. Because the
|
||||
local relay dedups inserts, events that fall in the overlap with the
|
||||
previous window are simply ignored on insert - no correctness issue.
|
||||
7. After the last (widest) window completes, the daemon switches to
|
||||
**steady-state**: it only keeps the live subscription running and
|
||||
periodically re-runs the widest window to catch any events that migrated
|
||||
into scope (e.g. a followed user backfilling their own old notes to a
|
||||
different relay). `backfilled_until` stays pinned at the oldest window.
|
||||
|
||||
### Restart behavior
|
||||
|
||||
- On restart, the daemon reads `backfilled_until` and `current_window_index`.
|
||||
- It resumes at the *current* window (the one that was in progress when it
|
||||
stopped). Because each window re-pulls `since = now - window_seconds`, a
|
||||
partial window just re-runs from the start of the round-robin - cheap and
|
||||
correct.
|
||||
- `state.backfill_cursor` records which pubkey in the round-robin was next;
|
||||
this is a minor optimization and may be reset to 0 on restart without harm.
|
||||
|
||||
### Throttling summary
|
||||
|
||||
- `tick_interval_seconds` paces individual pubkey queries (e.g. 5s).
|
||||
- `window_cooldown_seconds` paces window-to-window transitions (e.g. 60s).
|
||||
- `events_per_tick` caps each query's result size (e.g. 50).
|
||||
- Per-relay query latency from `nostr_relay_pool_get_relay_query_latency`
|
||||
can be used to prefer fast relays for backfill; slow relays are still used
|
||||
for the live subscription (where completeness matters more than speed).
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
caching_relay/
|
||||
plans/plan.md (this file)
|
||||
Makefile (c-relay-style, static link)
|
||||
build_static.sh (Alpine MUSL static builder, adapted from c-relay)
|
||||
Dockerfile.alpine-musl (adapted from c-relay)
|
||||
src/
|
||||
main.c (lifecycle, signal handling, main loop)
|
||||
main.h
|
||||
config.c / config.h (jsonc parse + validation)
|
||||
follow_graph.c / .h (resolve root npubs -> kind-3 -> followed set)
|
||||
live_subscriber.c / .h (open-ended live subscription on the pool)
|
||||
backfill.c / .h (throttled historical pull worker)
|
||||
relay_sink.c / .h (publish EVENT to local relay via dedicated sink pool)
|
||||
state.c / .h (in-memory author set + seen-event ring buffer; config state read/write)
|
||||
jsonc_strip.c / .h (strip // and /* */ comments before cJSON_Parse)
|
||||
log.c / log.h (colored stderr logging, c-relay style)
|
||||
examples/
|
||||
caching_relay.jsonc (sample config)
|
||||
README.md
|
||||
```
|
||||
|
||||
## Build Model (mirrors c-relay)
|
||||
|
||||
- `Makefile` with:
|
||||
- `CC = gcc`, `CFLAGS = -Wall -Wextra -std=c99 -g -O2`
|
||||
- `INCLUDES = -I. -Isrc -I../nostr_core_lib -I../nostr_core_lib/nostr_core -I../nostr_core_lib/cjson -I../nostr_core_lib/nostr_websocket -I../c_utils_lib/src`
|
||||
- `LIBS = -lwebsockets -lssl -lcrypto -lsecp256k1 -lcurl -lz -ldl -lpthread -lm -L../c_utils_lib -lc_utils`
|
||||
- Note: no `-lsqlite3` - the daemon itself does not use SQLite. (c-relay uses
|
||||
SQLite for its event store, but that is the local relay's concern, not the
|
||||
daemon's.)
|
||||
- `NOSTR_CORE_LIB = ../nostr_core_lib/libnostr_core_x64.a` (or arm64)
|
||||
- Build `nostr_core_lib` with `--nips=1,6,19` (1 basic, 6 keys, 19 npub bech32).
|
||||
NIP-42 is deferred to a later phase (public relays only for now). Kind-3
|
||||
parsing is plain cJSON tag walking in NIP-01, no special NIP-02 module
|
||||
needed.
|
||||
- Single static link line: `$(CC) $(CFLAGS) $(INCLUDES) $(MAIN_SRC) -o $(TARGET) $(NOSTR_CORE_LIB) $(C_UTILS_LIB) $(LIBS)`
|
||||
- `build_static.sh` adapted from c-relay's Alpine MUSL Docker builder to produce a truly static binary.
|
||||
|
||||
## Key nostr_core_lib APIs Used
|
||||
|
||||
- `nostr_relay_pool_create()` / `nostr_relay_pool_add_relay()` / `nostr_relay_pool_destroy()`
|
||||
- `nostr_relay_pool_query_sync()` - one-shot historical/backfill queries (kind-3 fetch, per-pubkey backfill)
|
||||
- `nostr_relay_pool_subscribe()` - long-lived live subscription with `on_event` / `on_eose` callbacks
|
||||
- `nostr_relay_pool_run()` / `nostr_relay_pool_poll()` - event loop driving both live sub and backfill
|
||||
- `nostr_relay_pool_publish_async()` - publish fetched events to the local relay
|
||||
via a **dedicated single-relay sink pool** (separate from the upstream query
|
||||
pool, so the local relay is never included in `query_sync` fan-out)
|
||||
- `nostr_validate_event()` - validate before republishing (defensive; upstream events should already be valid)
|
||||
- NIP-19: `nostr_nip19_decode` (or equivalent) to convert `npub1...` -> hex pubkey for filters
|
||||
- cJSON for filter construction and kind-3 tag parsing (`["p", "<hex>", "<relay>", "<petname>"]`)
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
Keep it simple and single-threaded with a cooperative event loop, matching c-relay's spirit:
|
||||
|
||||
- **Two `nostr_relay_pool_t` instances**, both driven from the main thread:
|
||||
1. `upstream_pool` - holds all `upstream_relays`. Used for `query_sync`
|
||||
(kind-3 fetch, backfill) and the long-lived live `subscribe`.
|
||||
2. `sink_pool` - holds only `local_relay`. Used exclusively for
|
||||
`publish_async` of fetched events. Never queried.
|
||||
- The main loop alternates between:
|
||||
- `nostr_relay_pool_run(upstream_pool, timeout_ms)` to pump live sub events,
|
||||
- a time-sliced backfill step (one pubkey `query_sync` per
|
||||
`tick_interval_seconds`),
|
||||
- `nostr_relay_pool_run(sink_pool, 0)` to flush pending publish callbacks.
|
||||
- Live subscription's `on_event` callback calls `relay_sink_publish()` which
|
||||
enqueues an `publish_async` on the sink pool.
|
||||
- Backfill `query_sync` is synchronous and blocks the upstream loop briefly -
|
||||
acceptable since `events_per_tick` is small and `tick_interval_seconds`
|
||||
provides pacing.
|
||||
- If profiling later shows backfill blocking the live sub too much, backfill
|
||||
can be moved to a second thread with its own upstream pool. Start without
|
||||
that.
|
||||
|
||||
## State Persistence
|
||||
|
||||
- **No SQLite in the daemon.** The daemon's only persistent state is the
|
||||
backfill window cursor, and that lives in the `.jsonc` config file itself
|
||||
under the `state` object (see Config File Format above).
|
||||
- The followed-pubkey set is re-derived from kind-3 on every
|
||||
`follow_graph_refresh_seconds` tick and on startup - it is not persisted.
|
||||
- Event dedup is delegated entirely to the local relay (c-relay's
|
||||
`INSERT OR IGNORE` on event id). The daemon keeps a small in-memory ring
|
||||
buffer of recently-published event ids only to avoid redundant publish
|
||||
*attempts* within a single run; this is not persisted.
|
||||
- Config write-back uses a temp file + atomic rename so the config is never
|
||||
left half-written, even on crash.
|
||||
|
||||
## Signals & Lifecycle
|
||||
|
||||
- `SIGINT` / `SIGTERM` -> graceful shutdown: close subscriptions, destroy pool, close sink WS.
|
||||
- `SIGHUP` -> reload config (re-read jsonc, refresh follow graph, adjust subscriptions).
|
||||
- Logs to stderr with c-relay-style color prefixes.
|
||||
|
||||
## Decisions Resolved
|
||||
|
||||
1. **Sink pool.** Use a *dedicated* single-relay `nostr_relay_pool_t` for the
|
||||
local sink, separate from the upstream pool. The local relay is never
|
||||
queried, only published to. **Confirmed.**
|
||||
- **Update (v3):** On subsequent startups, the local relay IS queried for
|
||||
kind-3 and kind-10002 events (fast local cache lookup). It is still never
|
||||
queried for general event backfill -- only for discovery metadata.
|
||||
2. **NIP-42 auth.** Deferred to a later phase. Phase 1 targets public relays
|
||||
only. **Confirmed.**
|
||||
3. **Kind-3 freshness.** Use the most recent kind-3 per root npub
|
||||
(`nostr_relay_pool_get_event` with `authors=[npub], kinds=[3]`). Re-resolve
|
||||
on `follow_graph_refresh_seconds` interval. **Confirmed.**
|
||||
4. **State persistence.** No SQLite in the daemon. The `.jsonc` config file is
|
||||
the state store; the daemon writes back `state.backfilled_until`,
|
||||
`state.current_window_index`, and `state.backfill_cursor` as backfill
|
||||
progresses. **Confirmed.**
|
||||
5. **Backfill strategy.** Progressive window expansion
|
||||
(24h -> 7d -> 30d -> 90d -> 365d), round-robin per pubkey within each
|
||||
window, full-window re-pull (local relay dedups). **Confirmed.**
|
||||
6. **NIP-65 outbox model.** `upstream_relays` in config are **bootstrap
|
||||
relays** only. The daemon discovers each followed pubkey's outbox relays
|
||||
via kind 10002, then computes the **minimum covering set** of relays
|
||||
(greedy set cover) that covers all followed pubkeys. Bootstrap relays are
|
||||
always included in the final set. Pubkeys with no kind 10002 fall back to
|
||||
bootstrap relays. **Confirmed.**
|
||||
7. **Local-relay-first on subsequent startups.** On startup, if
|
||||
`state.backfilled_until > 0`, the daemon queries the local relay first for
|
||||
kind-3 and kind-10002 events (fast, no network). Falls back to bootstrap
|
||||
relays for any pubkey not found locally. **Confirmed.**
|
||||
8. **Admin kinds.** Root (admin) npubs get a separate kind list (`admin_kinds`)
|
||||
with `["*"]` support for all kinds. Two live subscriptions: follows_sub
|
||||
(non-admin, regular kinds) and admin_sub (admin, admin_kinds). **Confirmed.**
|
||||
|
||||
## NIP-65 Outbox Model Design (Phase 2)
|
||||
|
||||
### New module: `relay_discovery.c`
|
||||
|
||||
Responsibilities:
|
||||
- For each followed pubkey, fetch their most recent kind 10002 (relay list).
|
||||
- First-time: query from bootstrap relays.
|
||||
- Subsequent: query from local relay first, bootstrap fallback.
|
||||
- Parse `r` tags from kind 10002 events: `["r", "<url>"]` or
|
||||
`["r", "<url>", "read"]` / `["r", "<url>", "write"]`.
|
||||
- We care about "read" relays (we are reading events FROM them).
|
||||
- If no read/write marker, assume both.
|
||||
- Build a map: `pubkey -> list of outbox relay URLs`.
|
||||
- Publish all kind-10002 events to the local relay (cache them for next startup).
|
||||
|
||||
### Greedy set cover algorithm
|
||||
|
||||
```
|
||||
Input: pubkey_to_relays map {pubkey -> [relay1, relay2, ...]}
|
||||
Output: minimal set of relay URLs covering all pubkeys
|
||||
|
||||
1. uncovered = set of all pubkeys
|
||||
2. selected = empty set
|
||||
3. Add all bootstrap relays to selected (always included)
|
||||
4. For each bootstrap relay, remove its known pubkeys from uncovered
|
||||
(bootstrap relays cover pubkeys with no 10002)
|
||||
5. While uncovered is not empty:
|
||||
a. Find relay R that covers the most pubkeys in uncovered
|
||||
b. If no relay covers any uncovered pubkey, break (orphaned pubkeys)
|
||||
c. Add R to selected
|
||||
d. Remove all pubkeys covered by R from uncovered
|
||||
6. Return selected
|
||||
```
|
||||
|
||||
### Data structures
|
||||
|
||||
```c
|
||||
/* Per-pubkey outbox relay list */
|
||||
typedef struct {
|
||||
char pubkey[CR_HEX_LEN];
|
||||
char relays[CR_MAX_RELAYS_PER_PUBKEY][CR_URL_LEN];
|
||||
int relay_count;
|
||||
} cr_outbox_entry_t;
|
||||
|
||||
/* Relay-to-pubkeys coverage map (for set cover) */
|
||||
typedef struct {
|
||||
char url[CR_URL_LEN];
|
||||
char pubkeys[CR_MAX_PUBKEYS_PER_RELAY][CR_HEX_LEN];
|
||||
int pubkey_count;
|
||||
} cr_relay_coverage_t;
|
||||
|
||||
/* Result of relay discovery */
|
||||
typedef struct {
|
||||
cr_outbox_entry_t *outboxes; /* per-pubkey relay lists */
|
||||
int outbox_count;
|
||||
char selected_relays[CR_MAX_UPSTREAM][CR_URL_LEN];
|
||||
int selected_count;
|
||||
} cr_relay_map_t;
|
||||
```
|
||||
|
||||
### Startup sequence change
|
||||
|
||||
```
|
||||
OLD:
|
||||
load config -> create pools -> resolve follow graph -> open live sub -> backfill
|
||||
|
||||
NEW:
|
||||
load config -> create pools ->
|
||||
resolve follow graph (local-first on subsequent) ->
|
||||
discover outbox relays (kind 10002, local-first on subsequent) ->
|
||||
compute minimum covering set ->
|
||||
add selected relays to upstream_pool ->
|
||||
log selected relays + coverage ->
|
||||
open live sub -> backfill (per-pubkey from their outbox relays)
|
||||
```
|
||||
|
||||
### Backfill change
|
||||
|
||||
Instead of fanning out each backfill query to ALL upstream relays, backfill
|
||||
queries each pubkey from **that pubkey's specific outbox relays** (or bootstrap
|
||||
relays as fallback). This is more efficient and more polite to relays that
|
||||
don't have that pubkey's events.
|
||||
|
||||
### New config fields
|
||||
|
||||
None required. `upstream_relays` is reinterpreted as bootstrap relays.
|
||||
Optionally a `max_outbox_relays` cap could be added later if the covering set
|
||||
grows too large.
|
||||
|
||||
## Implementation Todo List
|
||||
|
||||
### Phase 0 - Local relay sanity check (DONE)
|
||||
|
||||
0. **Start a local c-relay and verify read/write with `nak`.** DONE.
|
||||
|
||||
### Phase 1 - Daemon implementation (DONE)
|
||||
|
||||
1-12. All implemented, built, and tested. See git history. DONE.
|
||||
|
||||
### Phase 2 - NIP-65 outbox model
|
||||
|
||||
1. Implement `relay_discovery.c/.h`: fetch kind-10002 per pubkey, parse `r`
|
||||
tags, build `pubkey -> relays` map, publish 10002 events to local relay.
|
||||
2. Implement greedy set cover: compute minimum covering relay set from the
|
||||
outbox map, always include bootstrap relays.
|
||||
3. Update `follow_graph.c`: query local relay first for kind-3 on subsequent
|
||||
startups, fall back to bootstrap relays.
|
||||
4. Update `backfill.c`: query each pubkey from their specific outbox relays
|
||||
instead of all upstream relays.
|
||||
5. Update `main.c`: insert relay discovery phase between follow graph
|
||||
resolution and live subscription. Add discovered relays to upstream_pool.
|
||||
Log selected relays and coverage.
|
||||
6. Update `caching_relay_config.jsonc`: update comments to say "bootstrap
|
||||
relays" instead of "upstream relays".
|
||||
7. Build and test NIP-65 outbox model end-to-end with a real npub.
|
||||
8. Update `README.md` if any flowchart details change during implementation.
|
||||
@@ -0,0 +1,543 @@
|
||||
# Caching Relay PostgreSQL Inbox Integration Plan
|
||||
|
||||
> **Status:** Revised and simplified architecture.
|
||||
>
|
||||
> **Decision:** Keep caching as a separate application. The caching application
|
||||
> focuses on polite upstream collection and writes raw event JSON to a minimal
|
||||
> PostgreSQL inbox. c-relay-pg removes small priority-ordered batches and handles
|
||||
> them through relay-owned event processing. The design deliberately accepts a
|
||||
> small crash-loss window between dequeue and canonical storage; live overlap and
|
||||
> backfill retries recover those events naturally.
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Connect the standalone caching application to c-relay-pg without publishing
|
||||
cached events through a loopback WebSocket and without allowing the caching
|
||||
application to write directly to the canonical [`events`](../src/pg_schema.sql:12)
|
||||
table.
|
||||
|
||||
The responsibilities are intentionally narrow:
|
||||
|
||||
- **Caching application:** sensibly and politely acquire the desired events.
|
||||
- **PostgreSQL inbox:** temporarily hold structurally plausible event JSON.
|
||||
- **c-relay-pg:** remain the sole authority for accepting, storing, and
|
||||
broadcasting events.
|
||||
|
||||
This integration is PostgreSQL-only. The relay's normal SQLite operation remains
|
||||
unchanged, but the external caching service is unavailable with that backend.
|
||||
|
||||
## 2. Simplified Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
ROOT[Configured root npubs] --> CACHE[Caching application]
|
||||
UP[Upstream relays] --> CACHE
|
||||
CACHE --> INBOX[PostgreSQL caching event inbox]
|
||||
INBOX --> POLLER[c-relay-pg bounded poller]
|
||||
POLLER --> INGEST[Relay-owned event ingestion]
|
||||
INGEST --> EVENTS[Canonical events table]
|
||||
INGEST --> CLIENTS[Active subscriptions]
|
||||
|
||||
UI[Caching admin page] --> CFG[Shared caching configuration]
|
||||
CFG --> CACHE
|
||||
CACHE --> STATUS[Service heartbeat and progress]
|
||||
STATUS --> UI
|
||||
```
|
||||
|
||||
No loopback sink relay pool is used. No direct insert into the canonical event
|
||||
store is allowed from the caching application.
|
||||
|
||||
## 3. Deliberate Simplicity Rules
|
||||
|
||||
The first implementation will not include:
|
||||
|
||||
- multiple caching workers;
|
||||
- multiple inbox consumers;
|
||||
- row leases or claim expiration;
|
||||
- a queue state machine;
|
||||
- retry rows;
|
||||
- a dead-letter queue;
|
||||
- PostgreSQL `LISTEN`/`NOTIFY`;
|
||||
- persistent relay-health history;
|
||||
- exactly-once delivery;
|
||||
- direct process start/stop through the web page;
|
||||
- upstream NIP-42 authentication;
|
||||
- support for the SQLite backend.
|
||||
|
||||
The design prefers harmless duplicate acquisition over complicated delivery
|
||||
coordination. Event IDs already provide natural deduplication in both the inbox
|
||||
and canonical store.
|
||||
|
||||
## 4. Responsibility Boundaries
|
||||
|
||||
### 4.1 Caching application
|
||||
|
||||
The caching application is responsible for:
|
||||
|
||||
1. Reading caching configuration from PostgreSQL.
|
||||
2. Decoding configured root npubs.
|
||||
3. Resolving the roots' newest kind-3 follow lists.
|
||||
4. Discovering useful read relays from kind 10002.
|
||||
5. Computing a reasonable covering relay set.
|
||||
6. Maintaining live subscriptions for roots and follows.
|
||||
7. Performing polite, resumable historical backfill.
|
||||
8. Inserting received event JSON into the inbox.
|
||||
9. Persisting only the progress needed to resume discovery and backfill.
|
||||
10. Publishing a lightweight heartbeat and status snapshot.
|
||||
|
||||
It is not responsible for:
|
||||
|
||||
- deciding whether an event belongs in the canonical relay database;
|
||||
- applying c-relay-pg publication authorization;
|
||||
- processing NIP-09 deletions;
|
||||
- implementing replacement semantics;
|
||||
- executing relay administrator commands;
|
||||
- broadcasting to connected relay clients.
|
||||
|
||||
The caching callback may perform minimal application-side checks before insert:
|
||||
|
||||
- the callback supplied a JSON object;
|
||||
- an event ID string exists;
|
||||
- the serialized event is below a configured maximum size.
|
||||
|
||||
Cryptographic verification is optional in the fetcher and is not trusted by
|
||||
c-relay-pg. The initial version should omit it unless the existing standalone
|
||||
code already provides it at negligible integration cost.
|
||||
|
||||
### 4.2 PostgreSQL inbox
|
||||
|
||||
The inbox is deliberately dumb. It rejects obvious malformed data, deduplicates
|
||||
events currently waiting in the inbox, and establishes live-over-backfill
|
||||
priority. It does not attempt to verify a Nostr hash or Schnorr signature.
|
||||
|
||||
### 4.3 c-relay-pg
|
||||
|
||||
c-relay-pg is responsible for:
|
||||
|
||||
1. Removing a bounded batch from the inbox.
|
||||
2. Parsing each event JSON object.
|
||||
3. Applying authoritative structural, ID, and signature validation.
|
||||
4. Applying relay-wide event rules such as expiration and PoW where applicable.
|
||||
5. Bypassing client-session NIP-42 requirements because the source is internal.
|
||||
6. Preventing imported kind 23456 events from executing administrator commands.
|
||||
7. Applying duplicate, ephemeral, replaceable, addressable, and NIP-09 behavior.
|
||||
8. Storing accepted events through its normal database abstraction.
|
||||
9. Running post-store work and broadcasting newly accepted events from the main
|
||||
libwebsockets thread.
|
||||
10. Recording simple in-memory and cumulative import counters.
|
||||
|
||||
## 5. Minimal PostgreSQL Schema
|
||||
|
||||
Add the following objects to [`src/pg_schema.sql`](../src/pg_schema.sql) and the
|
||||
embedded PostgreSQL schema header.
|
||||
|
||||
### 5.1 `caching_event_inbox`
|
||||
|
||||
Suggested logical schema:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS caching_event_inbox (
|
||||
queue_id BIGSERIAL PRIMARY KEY,
|
||||
event_id TEXT NOT NULL UNIQUE,
|
||||
event_json JSONB NOT NULL,
|
||||
source_relay TEXT,
|
||||
source_class TEXT NOT NULL DEFAULT 'backfill',
|
||||
priority SMALLINT NOT NULL DEFAULT 1,
|
||||
received_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
|
||||
|
||||
CHECK (jsonb_typeof(event_json) = 'object'),
|
||||
CHECK (jsonb_typeof(event_json->'id') = 'string'),
|
||||
CHECK (length(event_json->>'id') = 64),
|
||||
CHECK (event_id = event_json->>'id'),
|
||||
CHECK (jsonb_typeof(event_json->'pubkey') = 'string'),
|
||||
CHECK (length(event_json->>'pubkey') = 64),
|
||||
CHECK (jsonb_typeof(event_json->'sig') = 'string'),
|
||||
CHECK (length(event_json->>'sig') = 128),
|
||||
CHECK (jsonb_typeof(event_json->'created_at') = 'number'),
|
||||
CHECK (jsonb_typeof(event_json->'kind') = 'number'),
|
||||
CHECK (jsonb_typeof(event_json->'tags') = 'array'),
|
||||
CHECK (jsonb_typeof(event_json->'content') = 'string'),
|
||||
CHECK (source_class IN ('live', 'discovery', 'backfill')),
|
||||
CHECK (priority IN (0, 1))
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_caching_inbox_dequeue
|
||||
ON caching_event_inbox(priority, received_at, queue_id);
|
||||
```
|
||||
|
||||
Priority meanings:
|
||||
|
||||
- `0`: live and discovery metadata;
|
||||
- `1`: historical backfill.
|
||||
|
||||
The caching application uses a parameterized insert with
|
||||
`ON CONFLICT (event_id) DO NOTHING`. It must impose a serialized event-size
|
||||
limit before sending the row. PostgreSQL schema constraints are structural
|
||||
protection, not authoritative Nostr validation.
|
||||
|
||||
### 5.2 `caching_service_state`
|
||||
|
||||
Use one singleton row for inexpensive UI status:
|
||||
|
||||
- service version;
|
||||
- service state: `starting`, `running`, `degraded`, or `stopped`;
|
||||
- applied config generation;
|
||||
- heartbeat timestamp;
|
||||
- followed author count;
|
||||
- selected and connected relay counts;
|
||||
- current backfill window and author cursor;
|
||||
- events fetched and inbox inserts;
|
||||
- last error text and timestamp.
|
||||
|
||||
This is a status snapshot, not an audit log. The caching process overwrites the
|
||||
same row periodically.
|
||||
|
||||
### 5.3 `caching_backfill_progress`
|
||||
|
||||
Persist only enough state for the caching application to resume politely:
|
||||
|
||||
- author pubkey;
|
||||
- window index;
|
||||
- immutable window anchor;
|
||||
- current inclusive `until` cursor;
|
||||
- completion flag;
|
||||
- updated timestamp.
|
||||
|
||||
Use a composite primary key on author and window. Do not store individual fetched
|
||||
event IDs here; inbox and canonical event IDs handle deduplication.
|
||||
|
||||
Timestamp-only Nostr pagination can saturate when many events share one second.
|
||||
For the first version, increase the query limit up to a fixed safety ceiling when
|
||||
a full page ends at one timestamp. If that ceiling remains saturated, leave the
|
||||
author incomplete, log it, and retry later rather than falsely marking it done.
|
||||
|
||||
## 6. Simple Destructive Dequeue
|
||||
|
||||
c-relay-pg uses one PostgreSQL transaction to remove and return a small batch:
|
||||
|
||||
```sql
|
||||
WITH selected AS (
|
||||
SELECT queue_id
|
||||
FROM caching_event_inbox
|
||||
ORDER BY priority ASC, received_at ASC, queue_id ASC
|
||||
LIMIT $1
|
||||
FOR UPDATE
|
||||
)
|
||||
DELETE FROM caching_event_inbox AS inbox
|
||||
USING selected
|
||||
WHERE inbox.queue_id = selected.queue_id
|
||||
RETURNING inbox.event_id,
|
||||
inbox.event_json,
|
||||
inbox.source_relay,
|
||||
inbox.source_class,
|
||||
inbox.received_at;
|
||||
```
|
||||
|
||||
Only one c-relay-pg inbox consumer is supported. `SKIP LOCKED`, leases, and claim
|
||||
owners are therefore unnecessary.
|
||||
|
||||
### 6.1 Accepted trade-off
|
||||
|
||||
There is a small loss window if c-relay-pg commits the delete and crashes before
|
||||
storing the returned events. This is accepted deliberately:
|
||||
|
||||
- live subscriptions reconnect with overlap;
|
||||
- backfill revisits incomplete and steady-state windows;
|
||||
- duplicate reacquisition is safe;
|
||||
- the reduced implementation complexity is worth the rare temporary loss.
|
||||
|
||||
The poller should keep batches small so the loss window and memory footprint are
|
||||
small.
|
||||
|
||||
## 7. c-relay-pg Inbox Consumer
|
||||
|
||||
### 7.1 Polling
|
||||
|
||||
- Compile the consumer only for `DB_BACKEND_POSTGRES`.
|
||||
- Poll a small batch at a configurable interval.
|
||||
- Poll quickly while rows are found and back off to a slower interval when empty.
|
||||
- Always order live/discovery rows before backfill rows.
|
||||
- Allow only one batch in memory at a time.
|
||||
- Stop polling immediately during relay shutdown.
|
||||
|
||||
Initial conservative defaults should be small, for example a few dozen rows per
|
||||
batch and subsecond-to-multisecond active polling with a longer idle interval.
|
||||
Exact defaults should be selected during integration testing rather than encoded
|
||||
as architectural requirements.
|
||||
|
||||
### 7.2 Ingestion behavior
|
||||
|
||||
Do not send dequeued events through a loopback WebSocket. Refactor the existing
|
||||
inbound event handling into a reusable internal ingestion entry point with an
|
||||
explicit source mode:
|
||||
|
||||
- `CLIENT_EVENT`: normal client authentication and response behavior;
|
||||
- `CACHING_INBOX_EVENT`: no client session or NIP-42 requirement, no OK response,
|
||||
and no administrator command execution.
|
||||
|
||||
The shared path should preserve:
|
||||
|
||||
- event ID and signature verification;
|
||||
- event limits;
|
||||
- expiration and PoW behavior;
|
||||
- NIP-09 authorization and deletion behavior;
|
||||
- ephemeral handling;
|
||||
- PostgreSQL replacement semantics in
|
||||
[`postgres_db_insert_event_with_json()`](../src/db_ops_postgres.c:1596);
|
||||
- duplicate detection;
|
||||
- post-store monitoring;
|
||||
- main-thread broadcast.
|
||||
|
||||
Avoid creating a second implementation of these rules solely for inbox events.
|
||||
|
||||
### 7.3 Main-thread broadcast
|
||||
|
||||
Database and cryptographic work may run off the libwebsockets thread, but
|
||||
[`store_event_post_actions()`](../src/main.c:1161) and active subscription
|
||||
broadcast must be queued to the main thread, following the existing asynchronous
|
||||
completion pattern in
|
||||
[`process_async_event_completions()`](../src/websockets.c:725).
|
||||
|
||||
Newly stored events are broadcast. Duplicates and stale replacements are not.
|
||||
Validated ephemeral events are broadcast but not stored.
|
||||
|
||||
### 7.4 Failure behavior
|
||||
|
||||
Because rows have already been removed, malformed or invalid events are simply
|
||||
counted and logged at a rate-limited level. They are not requeued.
|
||||
|
||||
A temporary canonical database failure should stop further inbox polling until
|
||||
the relay database is healthy. The current in-memory batch may be lost under the
|
||||
accepted simple-delivery model.
|
||||
|
||||
## 8. Caching Application Acquisition Strategy
|
||||
|
||||
The external application should spend most of its design effort here.
|
||||
|
||||
### 8.1 Follow graph
|
||||
|
||||
- Decode configured root npubs.
|
||||
- Fetch the newest kind 3 for each root from bootstrap relays.
|
||||
- Include roots themselves in the author set.
|
||||
- Parse valid `p` tags, deduplicate authors, and enforce a configured author cap.
|
||||
- Refresh periodically and replace live subscriptions when the set changes.
|
||||
|
||||
Local-first querying is optional in this architecture. PostgreSQL already holds
|
||||
canonical kind-3 events, so the caching application may query the canonical event
|
||||
store read-only for discovery metadata before contacting upstream relays. It must
|
||||
not write canonical rows.
|
||||
|
||||
### 8.2 NIP-65 relay discovery
|
||||
|
||||
- Fetch newest kind 10002 events in bounded author batches.
|
||||
- Use unmarked or `read` relay tags; ignore `write`-only tags.
|
||||
- Normalize and deduplicate `wss://` URLs.
|
||||
- Enforce per-author and global relay caps.
|
||||
- Use the greedy covering-set logic from the standalone implementation.
|
||||
- Retain configured bootstrap relays as fallback.
|
||||
|
||||
### 8.3 Friendly live subscriptions
|
||||
|
||||
- Separate root and followed-author subscriptions when their kind lists differ.
|
||||
- Batch author lists into reasonable filter sizes.
|
||||
- Reconnect with bounded exponential backoff and jitter.
|
||||
- Use a small timestamp overlap when resubscribing.
|
||||
- Insert callback event JSON into the inbox and continue; do not wait for
|
||||
c-relay-pg processing.
|
||||
|
||||
### 8.4 Friendly backfill
|
||||
|
||||
- Use recent-first progressive windows.
|
||||
- Persist an immutable anchor for each active window.
|
||||
- Query one author at a time from that author's declared outbox relays, with
|
||||
bootstrap fallback.
|
||||
- Use configurable page size and delay between queries.
|
||||
- Apply relay-specific backoff after errors or timeouts.
|
||||
- Keep global and per-relay request rates low.
|
||||
- Advance progress only after the returned page has been inserted into the
|
||||
inbox or identified as already queued.
|
||||
- Periodically repeat a bounded widest-window sweep after initial completion.
|
||||
|
||||
## 9. Shared Configuration
|
||||
|
||||
Keep administrator intent in the existing [`config`](../src/pg_schema.sql:192)
|
||||
table with category `caching` and `requires_restart = 0`.
|
||||
|
||||
Suggested keys:
|
||||
|
||||
- `caching_enabled`;
|
||||
- `caching_config_generation`;
|
||||
- `caching_root_npubs`;
|
||||
- `caching_bootstrap_relays`;
|
||||
- `caching_kinds`;
|
||||
- `caching_admin_kinds`;
|
||||
- `caching_live_enabled`;
|
||||
- `caching_live_resubscribe_seconds`;
|
||||
- `caching_backfill_enabled`;
|
||||
- `caching_backfill_windows`;
|
||||
- `caching_backfill_page_size`;
|
||||
- `caching_backfill_tick_interval_ms`;
|
||||
- `caching_backfill_window_cooldown_seconds`;
|
||||
- `caching_follow_graph_refresh_seconds`;
|
||||
- `caching_relay_discovery_refresh_seconds`;
|
||||
- `caching_max_followed_pubkeys`;
|
||||
- `caching_max_upstream_relays`;
|
||||
- `caching_max_relays_per_pubkey`;
|
||||
- `caching_query_timeout_ms`;
|
||||
- `caching_inbox_batch_size`;
|
||||
- `caching_inbox_active_poll_ms`;
|
||||
- `caching_inbox_idle_poll_ms`;
|
||||
- `caching_max_event_json_bytes`.
|
||||
|
||||
The web UI updates these through the existing encrypted administrator API. After
|
||||
a successful update, increment `caching_config_generation`. The caching
|
||||
application polls the generation and reloads valid changes. No cross-process
|
||||
command queue is needed.
|
||||
|
||||
`caching_enabled = false` tells the external application to close upstream
|
||||
activity. systemd or the container runtime, not the web page, owns the process
|
||||
lifecycle.
|
||||
|
||||
## 10. Admin UI
|
||||
|
||||
### 10.1 Sidenav placement
|
||||
|
||||
In [`api/index.html`](../api/index.html:14), insert **Caching** immediately after
|
||||
**Relay Events** and before **DM**:
|
||||
|
||||
1. Statistics
|
||||
2. Subscriptions
|
||||
3. Configuration
|
||||
4. Authorization
|
||||
5. IP BAN
|
||||
6. Relay Events
|
||||
7. **Caching**
|
||||
8. DM
|
||||
9. Database Query
|
||||
|
||||
Register `cachingSection` in [`switchPage()`](../api/index.js:5349).
|
||||
|
||||
### 10.2 Page layout
|
||||
|
||||
Reuse the existing admin UI classes and divide the page into three blocks.
|
||||
|
||||
**Service status**
|
||||
|
||||
- enabled configuration;
|
||||
- service heartbeat and state;
|
||||
- desired and applied config generation;
|
||||
- followed author count;
|
||||
- selected and connected relay counts;
|
||||
- current backfill window and cursor;
|
||||
- fetched and inbox-inserted counters;
|
||||
- last service error.
|
||||
|
||||
**Relay inbox status**
|
||||
|
||||
- pending live/discovery count;
|
||||
- pending backfill count;
|
||||
- oldest inbox row age;
|
||||
- relay-consumed, accepted, duplicate, invalid, and failed counters;
|
||||
- last successful inbox poll.
|
||||
|
||||
**Configuration**
|
||||
|
||||
- root npubs and bootstrap relays;
|
||||
- normal and root kind lists;
|
||||
- live and backfill toggles;
|
||||
- backfill windows and pacing;
|
||||
- discovery refresh intervals;
|
||||
- author and relay safety caps;
|
||||
- inbox batch and polling settings;
|
||||
- **Apply Configuration** button;
|
||||
- **Reset Backfill Progress** button with confirmation.
|
||||
|
||||
The UI reads service state from the singleton status row and lightweight inbox
|
||||
aggregates. Poll only while the Caching page is visible and use a modest refresh
|
||||
interval.
|
||||
|
||||
## 11. Process and Database Security
|
||||
|
||||
Use separate PostgreSQL roles:
|
||||
|
||||
- **Caching service role:** read caching configuration and permitted canonical
|
||||
discovery metadata; insert/select its own status and progress; insert into the
|
||||
inbox; no insert/update/delete permission on canonical events.
|
||||
- **c-relay-pg role:** normal relay permissions plus dequeue permission on the
|
||||
inbox and read access to caching status.
|
||||
|
||||
Use parameterized SQL throughout. Restrict public upstreams to normalized
|
||||
`wss://` URLs. Apply a database statement timeout to caching queries so the
|
||||
fetcher cannot hold resources indefinitely.
|
||||
|
||||
## 12. Lifecycle
|
||||
|
||||
### 12.1 Caching service
|
||||
|
||||
- Starts and stops independently under systemd or a container runtime.
|
||||
- On startup, loads configuration and progress, updates heartbeat state, then
|
||||
begins discovery/live/backfill.
|
||||
- On shutdown, closes subscriptions, saves current progress, marks status
|
||||
stopped, and disconnects from PostgreSQL.
|
||||
- Its failure does not stop c-relay-pg; cached acquisition merely pauses.
|
||||
|
||||
### 12.2 c-relay-pg
|
||||
|
||||
- Starts the inbox poller only for PostgreSQL builds after database, writer pool,
|
||||
and WebSocket systems are ready.
|
||||
- Stops new polls before shutting down the writer pool or WebSocket context.
|
||||
- Drains already-created main-thread completions before destroying those
|
||||
dependencies.
|
||||
- A missing inbox table should be logged as caching unavailable, not terminate
|
||||
the relay, unless schema migration policy requires otherwise.
|
||||
|
||||
## 13. Implementation Sequence
|
||||
|
||||
1. Add `caching_event_inbox`, `caching_service_state`, and
|
||||
`caching_backfill_progress` to the PostgreSQL schema and embedded schema.
|
||||
2. Add PostgreSQL database abstraction functions for bounded destructive dequeue
|
||||
and lightweight inbox counts.
|
||||
3. Refactor c-relay-pg inbound event processing into a shared source-aware
|
||||
ingestion entry point without changing normal client behavior.
|
||||
4. Add the PostgreSQL-only inbox poller, bounded in-memory batch, and main-thread
|
||||
post-action/broadcast completion path.
|
||||
5. Add caching configuration defaults and validation to c-relay-pg.
|
||||
6. Adapt the standalone caching application to read PostgreSQL configuration,
|
||||
insert raw event JSON into the inbox, and update heartbeat/status.
|
||||
7. Correct the standalone backfill implementation so limited queries paginate
|
||||
and do not falsely complete an author/window.
|
||||
8. Persist simple per-author/window progress and implement polite retry/backoff.
|
||||
9. Add the Caching page after Relay Events and before DM, including service,
|
||||
inbox, configuration, and reset-progress controls.
|
||||
10. Add PostgreSQL role/grant documentation and systemd/container deployment
|
||||
examples for the separate caching process.
|
||||
11. Test live priority, duplicate acquisition, invalid inbox rows, replacement,
|
||||
deletion, ephemeral broadcast, process restarts, destructive-dequeue crash
|
||||
behavior, upstream outages, and graceful shutdown.
|
||||
12. Build and validate c-relay-pg only through
|
||||
[`make_and_restart_relay.sh`](../make_and_restart_relay.sh), using
|
||||
`--preserve-database` where test state must survive.
|
||||
|
||||
## 14. Acceptance Criteria
|
||||
|
||||
- The caching application cannot write canonical event rows.
|
||||
- Received upstream events require only minimal fetcher checks before inbox
|
||||
insertion.
|
||||
- Inbox constraints reject structurally implausible rows and deduplicate pending
|
||||
event IDs.
|
||||
- c-relay-pg removes small batches ordered live/discovery before backfill.
|
||||
- c-relay-pg remains the authoritative ID/signature validator and storage owner.
|
||||
- Imported events never execute relay administrator commands or require a client
|
||||
NIP-42 session.
|
||||
- Newly accepted events are broadcast to active subscriptions from the main
|
||||
libwebsockets thread.
|
||||
- Duplicate and stale replacement events are not rebroadcast.
|
||||
- The caching service behaves politely through bounded filters, paced backfill,
|
||||
targeted outbox queries, and retry backoff.
|
||||
- Backfill progress survives caching-service restarts and does not falsely mark
|
||||
saturated timestamps complete.
|
||||
- A caching-service failure does not stop the relay.
|
||||
- The accepted destructive-dequeue crash window is documented and recoverable
|
||||
through live overlap and repeated backfill.
|
||||
- The Caching UI appears after Relay Events and before DM and distinguishes
|
||||
external service health from relay inbox consumption.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Caching Status "Not Implemented" Fix Plan
|
||||
|
||||
## Root Cause
|
||||
|
||||
The caching page shows "Service status unavailable (caching_status command not
|
||||
implemented on relay)" — but the command **is** implemented at
|
||||
[`src/config.c:4128`](../src/config.c:4128). The real problem is **admin
|
||||
authorization failure**: every admin command from the browser is rejected at
|
||||
publish time with:
|
||||
|
||||
```
|
||||
Unauthorized admin event attempt: invalid admin pubkey
|
||||
```
|
||||
|
||||
This rejection happens at [`src/main.c:2249`](../src/main.c:2249) because the
|
||||
browser's pubkey is not in the relay's `admin_pubkey` config list. The kind
|
||||
23456 event never reaches the command handler, so `caching_status` never
|
||||
executes, and the placeholder text persists.
|
||||
|
||||
### Confirmed admin key
|
||||
|
||||
- Relay admin pubkey (hex): `6a04ab98d9e4774ad806e302dddeb63bea16b5cb5f223ee77478e861bb583eb3`
|
||||
- Relay admin npub: `npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks`
|
||||
- Source: [`.relay.laantungir.net.keys`](../.relay.laantungir.net.keys:1)
|
||||
|
||||
The browser extension (nos2x) was using pubkey `8ff74724...`, which is not an
|
||||
admin on the port 7777 relay.
|
||||
|
||||
## Secondary Issue (latent)
|
||||
|
||||
Even after auth is fixed, the caching status UI would render **empty blocks**
|
||||
due to a schema mismatch between backend and frontend:
|
||||
|
||||
| Frontend expects ([`api/index.js:7046`](../api/index.js:7046)) | Backend emits ([`src/config.c:4135`](../src/config.c:4135)) |
|
||||
|---|---|
|
||||
| `data.service.enabled` | `data.caching_enabled` |
|
||||
| `data.service.running` | *(not emitted)* |
|
||||
| `data.inbox.enabled` | `data.caching_inbox_enabled` |
|
||||
| `data.inbox.queue_depth` | `data.inbox_pending_live` + `data.inbox_pending_backfill` |
|
||||
| *(not expected)* | `data.inbox_total_dequeued`, `data.inbox_total_accepted`, `data.inbox_total_rejected`, `data.inbox_oldest_age_seconds` |
|
||||
|
||||
## Fix Steps
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Step 1: Fix admin auth] --> B[Step 2: Align response schema]
|
||||
B --> C[Step 3: Fix frontend handler]
|
||||
C --> D[Step 4: Improve error messaging]
|
||||
D --> E[Step 5: Test end-to-end]
|
||||
```
|
||||
|
||||
### Step 1 — Fix admin authorization (config, user action)
|
||||
|
||||
Load the admin private key (corresponding to `6a04ab98...` /
|
||||
`npub13lm5wf8dvsdnc2894pkhch9uf8phvw9varrv8zf4sc885hhdmc8q6lx7ks`) into the
|
||||
nos2x browser extension so admin commands authenticate. Use `nak` on the
|
||||
command line to convert/derive the nsec if needed.
|
||||
|
||||
This unblocks **all** admin commands, not just caching.
|
||||
|
||||
### Step 2 — Align backend response schema (code, [`src/config.c:4128`](../src/config.c:4128))
|
||||
|
||||
Restructure the `data` object in the `caching_status` handler to emit nested
|
||||
`service` and `inbox` objects matching the frontend handler:
|
||||
|
||||
```json
|
||||
{
|
||||
"command": "caching_status",
|
||||
"status": "success",
|
||||
"data": {
|
||||
"service": {
|
||||
"enabled": false,
|
||||
"running": false,
|
||||
"connected_relays": 0,
|
||||
"events_cached": 0
|
||||
},
|
||||
"inbox": {
|
||||
"enabled": false,
|
||||
"running": true,
|
||||
"queue_depth": 0,
|
||||
"last_poll": 0,
|
||||
"total_dequeued": 0,
|
||||
"total_accepted": 0,
|
||||
"total_rejected": 0,
|
||||
"total_duplicates": 0,
|
||||
"pending_live": 0,
|
||||
"pending_backfill": 0,
|
||||
"oldest_age_seconds": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Populate `service.running` / `connected_relays` / `events_cached` from the
|
||||
caching service launcher state if available; otherwise emit zeros/defaults.
|
||||
|
||||
### Step 3 — Update frontend handler (code, [`api/index.js:7027`](../api/index.js:7027))
|
||||
|
||||
Update `handleCachingStatusResponse()` to render the inbox poller stats the
|
||||
backend actually produces (dequeued/accepted/rejected/pending/oldest_age), and
|
||||
map `data.inbox.pending_live + pending_backfill` to queue depth.
|
||||
|
||||
### Step 4 — Replace misleading placeholder (code, [`api/index.js:6889`](../api/index.js:6889))
|
||||
|
||||
- Change the pre-send placeholder from "caching_status command not implemented"
|
||||
to a neutral "Loading..." message.
|
||||
- On auth failure, show the actual error (e.g., "Not authorized: pubkey not
|
||||
registered as admin") instead of the generic "unavailable" text.
|
||||
|
||||
### Step 5 — Test end-to-end on port 7777
|
||||
|
||||
Verify the caching page shows real service/inbox status after auth is fixed
|
||||
and the schema is aligned.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,186 @@
|
||||
# Plan: Eliminate g_db from the Main Thread
|
||||
|
||||
## Problem Statement
|
||||
|
||||
All SQLite calls go through `db_ops.c`, but `db_active_connection()` falls back to `g_db` when `g_thread_db` is NULL. Since the main thread never sets `g_thread_db`, every `db_*` call from lws-main executes synchronous SQLite on the main thread. The thread pool workers have their own connections and work correctly — the problem is the **fallback path**.
|
||||
|
||||
Current perf data shows lws-main at **67.6% avg CPU**, dominated by SQLite symbols (`sqlite3BtreeTableMoveto`, `sqlite3VdbeExec`, `pcache1Fetch`).
|
||||
|
||||
## Goal
|
||||
|
||||
Remove `g_db` as a runtime connection used by the main thread. After this change:
|
||||
- The main thread has **no SQLite connection** and cannot execute synchronous queries
|
||||
- All DB work routes through thread pool workers (which have their own connections)
|
||||
- `g_db` is only used during startup/shutdown (before/after the event loop)
|
||||
- Any accidental `db_*` call from lws-main returns an error instead of silently blocking
|
||||
|
||||
## Architecture After Change
|
||||
|
||||
```
|
||||
lws-main thread:
|
||||
- WebSocket protocol handling
|
||||
- Message parsing/routing
|
||||
- Completion queue draining
|
||||
- NO SQLite access during event loop
|
||||
|
||||
db-read-0 thread:
|
||||
- REQ queries
|
||||
- COUNT queries
|
||||
- Config cache miss reads
|
||||
- Auth rule checks
|
||||
- Monitoring/stats queries
|
||||
|
||||
db-write thread:
|
||||
- EVENT inserts
|
||||
- Subscription logging
|
||||
- Config updates
|
||||
- Auth rule modifications
|
||||
|
||||
event-worker thread:
|
||||
- Signature validation
|
||||
- Duplicate check
|
||||
- Store via db-write
|
||||
```
|
||||
|
||||
## Categorized g_db Call Sites
|
||||
|
||||
### Category A: Startup-only — keep using g_db
|
||||
|
||||
These run before the event loop starts. They are fine.
|
||||
|
||||
| Function | File | Purpose |
|
||||
|----------|------|---------|
|
||||
| `db_init()` | db_ops.c:27 | Open database |
|
||||
| `db_exec_sql()` for PRAGMAs | main.c:854-882 | WAL, mmap, cache setup |
|
||||
| `db_table_exists()` | main.c:750 | Schema check |
|
||||
| `db_get_schema_version_dup()` | main.c:754 | Migration check |
|
||||
| `db_exec_sql()` for schema | main.c:842 | Create tables |
|
||||
| `populate_all_config_values_atomic()` | config.c:4290 | First-time config |
|
||||
| `populate_default_config_values()` | config.c:1657 | Default config |
|
||||
| `add_pubkeys_to_config_table()` | config.c:1778 | Pubkey storage |
|
||||
| `apply_cli_overrides_atomic()` | config.c:4211 | CLI overrides |
|
||||
| `db_store_relay_private_key_hex()` | config.c:495 | Key storage |
|
||||
| `db_populate_event_tags_from_existing()` | main.c:1156 | Tag migration |
|
||||
| `cleanup_all_subscriptions_on_startup()` | subscriptions.c:1092 | Orphan cleanup |
|
||||
|
||||
### Category B: Shutdown-only — keep using g_db
|
||||
|
||||
| Function | File | Purpose |
|
||||
|----------|------|---------|
|
||||
| `db_exec_sql()` WAL checkpoint | main.c:900 | Clean shutdown |
|
||||
| `db_close()` | main.c:904 | Close connection |
|
||||
|
||||
### Category C: Runtime hot path — must move off main thread
|
||||
|
||||
These are called during the event loop from lws-main context.
|
||||
|
||||
| Function | File | Called from | Fix strategy |
|
||||
|----------|------|------------|--------------|
|
||||
| `db_get_config_value_dup()` | db_ops.c:777 | config cache miss | Already cached with 5s TTL; pre-warm cache at startup so misses are rare |
|
||||
| `store_event()` for kind 14/1059 | main.c:983 | websockets.c sync path | Route through async event worker |
|
||||
| `store_event_post_actions()` | main.c | completion handler | Already runs on main; its DB calls need routing |
|
||||
| `db_log_subscription_created()` | db_ops.c:145 | subscriptions.c | Fire-and-forget via write queue |
|
||||
| `db_log_subscription_closed()` | db_ops.c:165 | subscriptions.c | Fire-and-forget via write queue |
|
||||
| `db_log_subscription_disconnected()` | db_ops.c:193 | subscriptions.c | Fire-and-forget via write queue |
|
||||
| `db_update_subscription_events_sent()` | db_ops.c:225 | subscriptions.c | Fire-and-forget via write queue |
|
||||
| `db_cleanup_orphaned_subscriptions()` | db_ops.c:244 | subscriptions.c | Move to startup only |
|
||||
| `generate_and_post_status_event()` | websockets.c:3408 | periodic timer | Submit to write worker |
|
||||
| `ip_ban_cleanup()` / `ip_ban_log_stats()` | websockets.c:3196 | periodic timer | Submit to write worker |
|
||||
| `db_get_total_event_count_ll()` | db_ops.c:589 | api.c monitoring | Submit to read worker |
|
||||
| `db_get_event_count_since()` | db_ops.c:606 | api.c monitoring | Submit to read worker |
|
||||
| `db_get_event_kind_distribution_rows()` | db_ops.c:625 | api.c monitoring | Submit to read worker |
|
||||
| `db_get_top_pubkeys_rows()` | db_ops.c:663 | api.c monitoring | Submit to read worker |
|
||||
| `db_get_subscription_details_rows()` | db_ops.c:697 | api.c monitoring | Submit to read worker |
|
||||
| `db_get_all_config_rows()` | db_ops.c:745 | api.c config query | Submit to read worker |
|
||||
| `db_execute_readonly_query_json()` | db_ops.c:442 | api.c SQL query | Submit to read worker |
|
||||
| `db_event_id_exists()` | db_ops.c:1041 | main.c event check | Already moved to event worker |
|
||||
| `db_retrieve_event_by_id()` | db_ops.c:1056 | main.c | Submit to read worker |
|
||||
| `db_get_event_pubkey()` | db_ops.c:262 | NIP-09 deletion | Submit to read worker |
|
||||
| `db_delete_event_by_id()` | db_ops.c:285 | NIP-09 deletion | Submit to write worker |
|
||||
| `db_delete_older_replaceable_events()` | db_ops.c:305 | store_event_core | Already on write worker |
|
||||
| `db_store_config_event()` | db_ops.c:888 | config.c | Submit to write worker |
|
||||
| `db_add_auth_rule()` | db_ops.c:1226 | config.c admin | Submit to write worker |
|
||||
| `db_remove_auth_rule()` | db_ops.c:1242 | config.c admin | Submit to write worker |
|
||||
| `db_delete_wot_whitelist_rules()` | db_ops.c:1258 | config.c WoT | Submit to write worker |
|
||||
| `db_count_wot_whitelist_rules()` | db_ops.c:1266 | config.c | Cache or read worker |
|
||||
| `db_store_event_tags_cjson()` | db_ops.c:1140 | main.c post-actions | Already on write worker path |
|
||||
| `db_get_config_row_count()` | db_ops.c:1121 | config.c diagnostics | Cache or read worker |
|
||||
| `db_set_config_value_full()` | db_ops.c:796 | config.c | Submit to write worker |
|
||||
| `db_update_config_value_only()` | db_ops.c:821 | config.c | Submit to write worker |
|
||||
| `db_upsert_config_value()` | db_ops.c:838 | api.c | Submit to write worker |
|
||||
| `db_exec_sql()` for transactions | config.c | admin events | Submit to write worker |
|
||||
| `db_count_with_sql()` | db_ops.c:415 | config.c admin | Submit to read worker |
|
||||
| `is_config_table_ready()` | config.c:4595 | config.c hybrid | Cache result at startup |
|
||||
| `generate_config_event_from_table()` | config.c:4933 | config.c | Cache or read worker |
|
||||
|
||||
### Category D: Auth rule checks — already use separate connections
|
||||
|
||||
These functions in `db_ops.c` open their own temporary read-only connections:
|
||||
|
||||
| Function | Line | Notes |
|
||||
|----------|------|-------|
|
||||
| `db_is_pubkey_blacklisted()` | 347 | Opens own connection |
|
||||
| `db_is_hash_blacklisted()` | 362 | Opens own connection |
|
||||
| `db_is_pubkey_whitelisted()` | 377 | Opens own connection |
|
||||
| `db_count_active_whitelist_rules()` | 392 | Opens own connection |
|
||||
|
||||
These are already safe — they don't use `g_db`. No change needed.
|
||||
|
||||
## Implementation Strategy
|
||||
|
||||
### Phase 1: Pre-warm caches and eliminate cache-miss DB reads
|
||||
|
||||
1. Pre-warm the config cache at startup by loading all config values into the in-memory cache before the event loop starts
|
||||
2. Pre-warm the NIP-11 cache at startup
|
||||
3. Pre-warm the auth rules fast cache at startup
|
||||
4. This eliminates the most frequent cache-miss `db_get_config_value_dup()` calls
|
||||
|
||||
### Phase 2: Route subscription logging through write worker
|
||||
|
||||
1. Add a `THREAD_POOL_JOB_FIRE_AND_FORGET` job type to thread_pool
|
||||
2. Create async wrappers: `db_log_subscription_created_async()`, etc.
|
||||
3. These submit SQL to the write worker queue and return immediately
|
||||
4. No completion callback needed — fire and forget
|
||||
|
||||
### Phase 3: Route special-kind EVENT store through async worker
|
||||
|
||||
1. Remove the `event_is_async_eligible()` exclusion for kinds 14, 1059, 23456
|
||||
2. For kind 23456: process admin command in completion handler on main thread after store
|
||||
3. For kind 14/1059: process NIP-17 DM in completion handler after store
|
||||
|
||||
### Phase 4: Route periodic timer DB work through workers
|
||||
|
||||
1. `generate_and_post_status_event()` — submit event creation to write worker
|
||||
2. `ip_ban_cleanup()` / `ip_ban_log_stats()` — submit to write worker
|
||||
3. Monitoring queries — submit to read worker with completion callback
|
||||
|
||||
### Phase 5: Null out g_db before event loop
|
||||
|
||||
1. After startup is complete and thread pool is initialized, set `g_db = NULL`
|
||||
2. Change `db_active_connection()` to assert/warn if both `g_thread_db` and `g_db` are NULL
|
||||
3. Any accidental main-thread DB call will now fail loudly instead of silently blocking
|
||||
4. Restore `g_db` briefly for shutdown checkpoint
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Phase 1 — lowest risk, immediate benefit from eliminating cache misses
|
||||
2. Phase 2 — straightforward fire-and-forget pattern
|
||||
3. Phase 3 — moderate complexity, needs careful completion handler design
|
||||
4. Phase 4 — moderate complexity, periodic tasks need async patterns
|
||||
5. Phase 5 — the final enforcement step, only safe after phases 1-4
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Phase 1**: Very low risk — just pre-warming existing caches
|
||||
- **Phase 2**: Low risk — subscription logging is non-critical, fire-and-forget is safe
|
||||
- **Phase 3**: Medium risk — admin/DM event processing has complex state; needs careful testing
|
||||
- **Phase 4**: Medium risk — periodic tasks have side effects that need to complete
|
||||
- **Phase 5**: High risk if done prematurely — must verify ALL runtime paths are covered first
|
||||
|
||||
## Expected Impact
|
||||
|
||||
After all phases:
|
||||
- lws-main CPU should drop from ~67% to near 0% SQLite overhead
|
||||
- All SQLite work happens on db-read and db-write threads
|
||||
- Main thread only does WebSocket I/O, JSON parsing, and completion queue draining
|
||||
- Thread model becomes: lws-main = pure I/O, workers = all DB
|
||||
@@ -0,0 +1,579 @@
|
||||
# Event JSON Storage & Database Migration Plan
|
||||
|
||||
**Goal:** Store full event JSON in database for 2,500x faster retrieval + implement proper database migration system
|
||||
|
||||
---
|
||||
|
||||
## Decision: Fresh Start vs Migration
|
||||
|
||||
### Option A: Fresh Start (Recommended for This Change)
|
||||
|
||||
**Pros:**
|
||||
- ✅ Clean implementation (no migration complexity)
|
||||
- ✅ Fast deployment (no data conversion)
|
||||
- ✅ No risk of migration bugs
|
||||
- ✅ Opportunity to fix any schema issues
|
||||
- ✅ Smaller database (no legacy data)
|
||||
|
||||
**Cons:**
|
||||
- ❌ Lose existing events
|
||||
- ❌ Relay starts "empty"
|
||||
- ❌ Historical data lost
|
||||
|
||||
**Recommendation:** **Fresh start for this change** because:
|
||||
1. Your relay is still in development/testing phase
|
||||
2. The schema change is fundamental (affects every event)
|
||||
3. Migration would require reconstructing JSON for every existing event (expensive)
|
||||
4. You've been doing fresh starts anyway
|
||||
|
||||
### Option B: Implement Migration System
|
||||
|
||||
**Pros:**
|
||||
- ✅ Preserve existing events
|
||||
- ✅ No data loss
|
||||
- ✅ Professional approach
|
||||
- ✅ Reusable for future changes
|
||||
|
||||
**Cons:**
|
||||
- ❌ Complex implementation
|
||||
- ❌ Slow migration (reconstruct JSON for all events)
|
||||
- ❌ Risk of bugs during migration
|
||||
- ❌ Requires careful testing
|
||||
|
||||
**Recommendation:** **Implement migration system for FUTURE changes**, but start fresh for this one.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Schema Change
|
||||
|
||||
### New Schema (v11)
|
||||
|
||||
```sql
|
||||
CREATE TABLE events (
|
||||
id TEXT PRIMARY KEY,
|
||||
pubkey TEXT NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
kind INTEGER NOT NULL,
|
||||
event_type TEXT NOT NULL CHECK (event_type IN ('regular', 'replaceable', 'ephemeral', 'addressable')),
|
||||
content TEXT NOT NULL,
|
||||
sig TEXT NOT NULL,
|
||||
tags JSON NOT NULL DEFAULT '[]',
|
||||
event_json TEXT NOT NULL, -- NEW: Full event as JSON string
|
||||
first_seen INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))
|
||||
);
|
||||
|
||||
-- Keep all existing indexes (they query the columns, not event_json)
|
||||
CREATE INDEX idx_events_pubkey ON events(pubkey);
|
||||
CREATE INDEX idx_events_kind ON events(kind);
|
||||
CREATE INDEX idx_events_created_at ON events(created_at DESC);
|
||||
CREATE INDEX idx_events_kind_created_at ON events(kind, created_at DESC);
|
||||
CREATE INDEX idx_events_pubkey_created_at ON events(pubkey, created_at DESC);
|
||||
```
|
||||
|
||||
### Why Keep Both Columns AND event_json?
|
||||
|
||||
**Columns (id, pubkey, kind, etc.):**
|
||||
- Used for **querying** (WHERE clauses, indexes)
|
||||
- Fast filtering and sorting
|
||||
- Required for SQL operations
|
||||
|
||||
**event_json:**
|
||||
- Used for **retrieval** (SELECT results)
|
||||
- Pre-serialized, ready to send
|
||||
- Eliminates JSON reconstruction
|
||||
|
||||
**This is a common pattern** in high-performance systems (denormalization for read performance).
|
||||
|
||||
---
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Schema Update (v11)
|
||||
|
||||
**File:** `src/sql_schema.h`
|
||||
|
||||
```c
|
||||
#define EMBEDDED_SCHEMA_VERSION "11"
|
||||
|
||||
// In schema SQL:
|
||||
"CREATE TABLE events (\n\
|
||||
id TEXT PRIMARY KEY,\n\
|
||||
pubkey TEXT NOT NULL,\n\
|
||||
created_at INTEGER NOT NULL,\n\
|
||||
kind INTEGER NOT NULL,\n\
|
||||
event_type TEXT NOT NULL,\n\
|
||||
content TEXT NOT NULL,\n\
|
||||
sig TEXT NOT NULL,\n\
|
||||
tags JSON NOT NULL DEFAULT '[]',\n\
|
||||
event_json TEXT NOT NULL,\n\ -- NEW COLUMN
|
||||
first_seen INTEGER NOT NULL DEFAULT (strftime('%s', 'now'))\n\
|
||||
);\n\
|
||||
```
|
||||
|
||||
### Phase 2: Update store_event() Function
|
||||
|
||||
**File:** `src/main.c` (lines 660-773)
|
||||
|
||||
**Current:**
|
||||
```c
|
||||
int store_event(cJSON* event) {
|
||||
// Extract fields
|
||||
cJSON* id = cJSON_GetObjectItem(event, "id");
|
||||
// ... extract other fields ...
|
||||
|
||||
// INSERT with individual columns
|
||||
const char* sql = "INSERT INTO events (id, pubkey, ...) VALUES (?, ?, ...)";
|
||||
}
|
||||
```
|
||||
|
||||
**New:**
|
||||
```c
|
||||
int store_event(cJSON* event) {
|
||||
// Serialize event to JSON string ONCE
|
||||
char* event_json = cJSON_PrintUnformatted(event);
|
||||
if (!event_json) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Extract fields for indexed columns
|
||||
cJSON* id = cJSON_GetObjectItem(event, "id");
|
||||
// ... extract other fields ...
|
||||
|
||||
// INSERT with columns + event_json
|
||||
const char* sql = "INSERT INTO events (id, pubkey, ..., event_json) VALUES (?, ?, ..., ?)";
|
||||
|
||||
// ... bind parameters ...
|
||||
sqlite3_bind_text(stmt, 9, event_json, -1, SQLITE_TRANSIENT);
|
||||
|
||||
// ... execute ...
|
||||
free(event_json);
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Update handle_req_message() Function
|
||||
|
||||
**File:** `src/main.c` (lines 1302-1361)
|
||||
|
||||
**Current:**
|
||||
```c
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
// Build event JSON from 7 columns
|
||||
cJSON* event = cJSON_CreateObject();
|
||||
cJSON_AddStringToObject(event, "id", (char*)sqlite3_column_text(stmt, 0));
|
||||
// ... 6 more fields ...
|
||||
cJSON* tags = cJSON_Parse(tags_json); // Parse tags
|
||||
cJSON_AddItemToObject(event, "tags", tags);
|
||||
|
||||
// Create EVENT message
|
||||
cJSON* event_msg = cJSON_CreateArray();
|
||||
cJSON_AddItemToArray(event_msg, cJSON_CreateString("EVENT"));
|
||||
cJSON_AddItemToArray(event_msg, cJSON_CreateString(sub_id));
|
||||
cJSON_AddItemToArray(event_msg, event);
|
||||
|
||||
char* msg_str = cJSON_Print(event_msg);
|
||||
queue_message(wsi, pss, msg_str, msg_len, LWS_WRITE_TEXT);
|
||||
}
|
||||
```
|
||||
|
||||
**New:**
|
||||
```c
|
||||
// Update SQL to select event_json
|
||||
const char* sql = "SELECT event_json FROM events WHERE ...";
|
||||
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
const char* event_json = (char*)sqlite3_column_text(stmt, 0);
|
||||
|
||||
// Build EVENT message with pre-serialized event
|
||||
// Format: ["EVENT","sub_id",{...event_json...}]
|
||||
size_t msg_len = 12 + strlen(sub_id) + strlen(event_json); // ["EVENT","",""]
|
||||
char* msg_str = malloc(msg_len + 1);
|
||||
snprintf(msg_str, msg_len + 1, "[\"EVENT\",\"%s\",%s]", sub_id, event_json);
|
||||
|
||||
queue_message(wsi, pss, msg_str, strlen(msg_str), LWS_WRITE_TEXT);
|
||||
free(msg_str);
|
||||
}
|
||||
```
|
||||
|
||||
**Speedup:** 366 × (cJSON operations) eliminated!
|
||||
|
||||
---
|
||||
|
||||
## Database Migration System Design
|
||||
|
||||
### For Future Schema Changes
|
||||
|
||||
**File:** `src/migrations.c` (new file)
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
int from_version;
|
||||
int to_version;
|
||||
const char* description;
|
||||
int (*migrate_func)(sqlite3* db);
|
||||
} migration_t;
|
||||
|
||||
// Migration from v10 to v11: Add event_json column
|
||||
int migrate_v10_to_v11(sqlite3* db) {
|
||||
// Step 1: Add column
|
||||
const char* add_column_sql =
|
||||
"ALTER TABLE events ADD COLUMN event_json TEXT";
|
||||
|
||||
if (sqlite3_exec(db, add_column_sql, NULL, NULL, NULL) != SQLITE_OK) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Step 2: Populate event_json for existing events
|
||||
const char* select_sql =
|
||||
"SELECT id, pubkey, created_at, kind, content, sig, tags FROM events";
|
||||
|
||||
sqlite3_stmt* stmt;
|
||||
if (sqlite3_prepare_v2(db, select_sql, -1, &stmt, NULL) != SQLITE_OK) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
// Reconstruct JSON
|
||||
cJSON* event = cJSON_CreateObject();
|
||||
cJSON_AddStringToObject(event, "id", (char*)sqlite3_column_text(stmt, 0));
|
||||
// ... add other fields ...
|
||||
|
||||
char* event_json = cJSON_PrintUnformatted(event);
|
||||
|
||||
// Update row
|
||||
const char* update_sql = "UPDATE events SET event_json = ? WHERE id = ?";
|
||||
sqlite3_stmt* update_stmt;
|
||||
sqlite3_prepare_v2(db, update_sql, -1, &update_stmt, NULL);
|
||||
sqlite3_bind_text(update_stmt, 1, event_json, -1, SQLITE_TRANSIENT);
|
||||
sqlite3_bind_text(update_stmt, 2, (char*)sqlite3_column_text(stmt, 0), -1, SQLITE_STATIC);
|
||||
sqlite3_step(update_stmt);
|
||||
sqlite3_finalize(update_stmt);
|
||||
|
||||
free(event_json);
|
||||
cJSON_Delete(event);
|
||||
}
|
||||
|
||||
sqlite3_finalize(stmt);
|
||||
|
||||
// Step 3: Make column NOT NULL
|
||||
// (SQLite doesn't support ALTER COLUMN, so we'd need to recreate table)
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Migration registry
|
||||
static migration_t migrations[] = {
|
||||
{10, 11, "Add event_json column for fast retrieval", migrate_v10_to_v11},
|
||||
// Future migrations go here
|
||||
};
|
||||
|
||||
int run_migrations(sqlite3* db, int current_version, int target_version) {
|
||||
for (int i = 0; i < sizeof(migrations) / sizeof(migration_t); i++) {
|
||||
if (migrations[i].from_version >= current_version &&
|
||||
migrations[i].to_version <= target_version) {
|
||||
|
||||
printf("Running migration: %s\n", migrations[i].description);
|
||||
|
||||
if (migrations[i].migrate_func(db) != 0) {
|
||||
fprintf(stderr, "Migration failed: %s\n", migrations[i].description);
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Update schema version
|
||||
char update_version_sql[256];
|
||||
snprintf(update_version_sql, sizeof(update_version_sql),
|
||||
"PRAGMA user_version = %d", migrations[i].to_version);
|
||||
sqlite3_exec(db, update_version_sql, NULL, NULL, NULL);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Recommendation: Hybrid Approach
|
||||
|
||||
### For This Change (v10 → v11): Fresh Start
|
||||
|
||||
**Rationale:**
|
||||
1. Your relay is still in development
|
||||
2. Migration would be slow (reconstruct JSON for all events)
|
||||
3. You've been doing fresh starts anyway
|
||||
4. Clean slate for performance testing
|
||||
|
||||
**Steps:**
|
||||
1. Update schema to v11 with event_json column
|
||||
2. Update store_event() to populate event_json
|
||||
3. Update handle_req_message() to use event_json
|
||||
4. Deploy with fresh database
|
||||
5. Test performance improvement
|
||||
|
||||
### For Future Changes: Use Migration System
|
||||
|
||||
**Rationale:**
|
||||
1. Once relay is in production, data preservation matters
|
||||
2. Migration system is reusable
|
||||
3. Professional approach for production relay
|
||||
|
||||
**Steps:**
|
||||
1. Create `src/migrations.c` and `src/migrations.h`
|
||||
2. Implement migration framework
|
||||
3. Add migration functions for each schema change
|
||||
4. Test migrations thoroughly before deployment
|
||||
|
||||
---
|
||||
|
||||
## Migration System Features
|
||||
|
||||
### Core Features
|
||||
|
||||
1. **Version Detection**
|
||||
- Read current schema version from database
|
||||
- Compare with embedded schema version
|
||||
- Determine which migrations to run
|
||||
|
||||
2. **Migration Chain**
|
||||
- Run migrations in sequence (v8 → v9 → v10 → v11)
|
||||
- Skip already-applied migrations
|
||||
- Stop on first failure
|
||||
|
||||
3. **Backup Before Migration**
|
||||
- Automatic database backup before migration
|
||||
- Rollback capability if migration fails
|
||||
- Backup retention policy
|
||||
|
||||
4. **Progress Reporting**
|
||||
- Log migration progress
|
||||
- Show estimated time remaining
|
||||
- Report success/failure
|
||||
|
||||
### Safety Features
|
||||
|
||||
1. **Transaction Wrapping**
|
||||
```c
|
||||
sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, NULL);
|
||||
int result = migrate_v10_to_v11(db);
|
||||
if (result == 0) {
|
||||
sqlite3_exec(db, "COMMIT", NULL, NULL, NULL);
|
||||
} else {
|
||||
sqlite3_exec(db, "ROLLBACK", NULL, NULL, NULL);
|
||||
}
|
||||
```
|
||||
|
||||
2. **Validation After Migration**
|
||||
- Verify row counts match
|
||||
- Check data integrity
|
||||
- Validate indexes created
|
||||
|
||||
3. **Dry-Run Mode**
|
||||
- Test migration without committing
|
||||
- Report what would be changed
|
||||
- Estimate migration time
|
||||
|
||||
---
|
||||
|
||||
## Implementation Timeline
|
||||
|
||||
### Immediate (Today): Fresh Start with event_json
|
||||
|
||||
**Changes:**
|
||||
1. Update schema to v11 (add event_json column)
|
||||
2. Update store_event() to populate event_json
|
||||
3. Update handle_req_message() to use event_json
|
||||
4. Deploy with fresh database
|
||||
|
||||
**Effort:** 4 hours
|
||||
**Impact:** 2,500x faster event retrieval
|
||||
|
||||
### This Week: Build Migration Framework
|
||||
|
||||
**Changes:**
|
||||
1. Create src/migrations.c and src/migrations.h
|
||||
2. Implement migration runner
|
||||
3. Add backup/rollback capability
|
||||
4. Add progress reporting
|
||||
|
||||
**Effort:** 1-2 days
|
||||
**Impact:** Reusable for all future schema changes
|
||||
|
||||
### Future: Add Migrations as Needed
|
||||
|
||||
**For each schema change:**
|
||||
1. Write migration function
|
||||
2. Add to migrations array
|
||||
3. Test thoroughly
|
||||
4. Deploy with automatic migration
|
||||
|
||||
---
|
||||
|
||||
## Code Structure
|
||||
|
||||
### File Organization
|
||||
|
||||
```
|
||||
src/
|
||||
├── migrations.c # NEW: Migration system
|
||||
├── migrations.h # NEW: Migration API
|
||||
├── sql_schema.h # Schema definition (v11)
|
||||
├── main.c # Updated store_event() and handle_req_message()
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Migration API
|
||||
|
||||
```c
|
||||
// migrations.h
|
||||
int init_migration_system(sqlite3* db);
|
||||
int run_pending_migrations(sqlite3* db);
|
||||
int backup_database(const char* db_path, char* backup_path, size_t backup_path_size);
|
||||
int rollback_migration(sqlite3* db, const char* backup_path);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### For Fresh Start (v11)
|
||||
|
||||
1. **Local testing:**
|
||||
- Build with new schema
|
||||
- Post test events
|
||||
- Query events and measure performance
|
||||
- Verify event_json is populated correctly
|
||||
|
||||
2. **Performance testing:**
|
||||
- Query 366 events
|
||||
- Measure time (should be <10ms instead of 18s)
|
||||
- Check CPU usage (should be <20%)
|
||||
|
||||
3. **Production deployment:**
|
||||
- Stop relay
|
||||
- Delete old database
|
||||
- Start relay with v11 schema
|
||||
- Monitor performance
|
||||
|
||||
### For Migration System (Future)
|
||||
|
||||
1. **Unit tests:**
|
||||
- Test each migration function
|
||||
- Test rollback capability
|
||||
- Test error handling
|
||||
|
||||
2. **Integration tests:**
|
||||
- Create database with old schema
|
||||
- Run migration
|
||||
- Verify data integrity
|
||||
- Test rollback
|
||||
|
||||
3. **Performance tests:**
|
||||
- Measure migration time for large databases
|
||||
- Test with 10K, 100K, 1M events
|
||||
- Optimize slow migrations
|
||||
|
||||
---
|
||||
|
||||
## Migration Complexity Analysis
|
||||
|
||||
### For v10 → v11 Migration
|
||||
|
||||
**If we were to migrate existing data:**
|
||||
|
||||
```sql
|
||||
-- Step 1: Add column (fast)
|
||||
ALTER TABLE events ADD COLUMN event_json TEXT;
|
||||
|
||||
-- Step 2: Populate event_json (SLOW!)
|
||||
-- For each of N events:
|
||||
-- 1. SELECT 7 columns
|
||||
-- 2. Reconstruct JSON (cJSON operations)
|
||||
-- 3. Serialize to string (cJSON_Print)
|
||||
-- 4. UPDATE event_json column
|
||||
-- 5. Free memory
|
||||
|
||||
-- Estimated time:
|
||||
-- - 1000 events: ~10 seconds
|
||||
-- - 10000 events: ~100 seconds
|
||||
-- - 100000 events: ~1000 seconds (16 minutes)
|
||||
```
|
||||
|
||||
**Conclusion:** Migration is expensive for this change. Fresh start is better.
|
||||
|
||||
---
|
||||
|
||||
## Future Migration Examples
|
||||
|
||||
### Easy Migrations (Fast)
|
||||
|
||||
**Adding an index:**
|
||||
```c
|
||||
int migrate_add_index(sqlite3* db) {
|
||||
return sqlite3_exec(db,
|
||||
"CREATE INDEX idx_new ON events(new_column)",
|
||||
NULL, NULL, NULL);
|
||||
}
|
||||
```
|
||||
|
||||
**Adding a column with default:**
|
||||
```c
|
||||
int migrate_add_column(sqlite3* db) {
|
||||
return sqlite3_exec(db,
|
||||
"ALTER TABLE events ADD COLUMN new_col TEXT DEFAULT ''",
|
||||
NULL, NULL, NULL);
|
||||
}
|
||||
```
|
||||
|
||||
### Hard Migrations (Slow)
|
||||
|
||||
**Changing column type:**
|
||||
- Requires table recreation
|
||||
- Copy all data
|
||||
- Recreate indexes
|
||||
- Can take minutes for large databases
|
||||
|
||||
**Populating computed columns:**
|
||||
- Requires row-by-row processing
|
||||
- Can take minutes for large databases
|
||||
|
||||
---
|
||||
|
||||
## Recommendation Summary
|
||||
|
||||
### For This Change (event_json)
|
||||
|
||||
**Do:** Fresh start with v11 schema
|
||||
- Fast deployment
|
||||
- Clean implementation
|
||||
- Immediate performance benefit
|
||||
- No migration complexity
|
||||
|
||||
**Don't:** Migrate existing data
|
||||
- Too slow (reconstruct JSON for all events)
|
||||
- Too complex (first migration)
|
||||
- Not worth it (relay still in development)
|
||||
|
||||
### For Future Changes
|
||||
|
||||
**Do:** Implement migration system
|
||||
- Professional approach
|
||||
- Data preservation
|
||||
- Reusable framework
|
||||
- Required for production relay
|
||||
|
||||
**Timeline:**
|
||||
- **Today:** Deploy v11 with fresh start
|
||||
- **This week:** Build migration framework
|
||||
- **Future:** Use migrations for all schema changes
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Update schema to v11 (add event_json column)
|
||||
2. ✅ Update store_event() to populate event_json
|
||||
3. ✅ Update handle_req_message() to use event_json
|
||||
4. ✅ Test locally with 366-event query
|
||||
5. ✅ Deploy to production with fresh database
|
||||
6. ✅ Measure performance improvement
|
||||
7. ⏳ Build migration system for future use
|
||||
|
||||
**Expected result:** 366-event retrieval time drops from 18s to <10ms (2,500x speedup)
|
||||
@@ -0,0 +1,312 @@
|
||||
# Event Tags Denormalization Plan
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The relay is at 99% CPU with 1,178 WebSocket connections and ~120 new REQ subscriptions per minute. The root cause is that every tag-filtered REQ query (e.g., `#g`, `#e`, `#p`) uses `json_each(json(tags))` with `json_extract()` — a correlated subquery that parses the JSON tags column for every candidate row. With geohash-based subscriptions from a location app generating constant connection churn, this creates unsustainable CPU load.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `EXPLAIN QUERY PLAN` shows: `CORRELATED SCALAR SUBQUERY → SCAN json_each VIRTUAL TABLE`
|
||||
- 8,077 subscription creates/hour but only 442 active at any time (connection churn)
|
||||
- Only 56 events stored/hour — the load is entirely from **read queries**, not writes
|
||||
- 14,819 events match the kind filter; each query parses JSON tags on candidate rows
|
||||
|
||||
## Solution: Denormalized `event_tags` Table
|
||||
|
||||
Replace `json_each()` queries with indexed lookups on a separate `event_tags` table. This is the standard approach used by strfry, nostream, and other production Nostr relays.
|
||||
|
||||
### Performance Impact
|
||||
|
||||
| Metric | Before | After |
|
||||
|--------|--------|-------|
|
||||
| Tag query type | `json_each()` correlated subquery | Indexed JOIN/subquery |
|
||||
| Complexity per query | O(rows × tags_per_event) JSON parsing | O(log n) B-tree lookup |
|
||||
| Expected CPU reduction | 99% | <10% for same traffic |
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Changes |
|
||||
|------|---------|
|
||||
| `src/sql_schema.h` | Add `event_tags` table, indexes, cascade triggers; bump to schema v12 |
|
||||
| `src/main.c` | Add `store_event_tags()`, update `handle_req_message()` tag filter SQL, move config reads out of row loop, add startup tag population |
|
||||
| `src/nip009.c` | Add `DELETE FROM event_tags` alongside event deletions |
|
||||
| `src/dm_admin.c` | Downgrade NIP-17 decryption failure log level |
|
||||
| `src/websockets.c` | Downgrade NIP-17 error log to DEBUG at lines 780 and 1488 |
|
||||
|
||||
## Detailed Changes
|
||||
|
||||
### 1. Schema: `src/sql_schema.h`
|
||||
|
||||
Add the `event_tags` table after the events table definition. Bump schema version to 12.
|
||||
|
||||
```sql
|
||||
-- Denormalized event tags for fast indexed lookups
|
||||
-- Replaces json_each(json(tags)) queries which cause full JSON parsing per row
|
||||
CREATE TABLE event_tags (
|
||||
event_id TEXT NOT NULL,
|
||||
tag_name TEXT NOT NULL,
|
||||
tag_value TEXT NOT NULL,
|
||||
tag_index INTEGER NOT NULL DEFAULT 0,
|
||||
FOREIGN KEY (event_id) REFERENCES events(id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Primary lookup index: find events by tag name + value
|
||||
CREATE INDEX idx_event_tags_lookup ON event_tags(tag_name, tag_value);
|
||||
|
||||
-- Reverse lookup: find all tags for an event (for cleanup)
|
||||
CREATE INDEX idx_event_tags_event ON event_tags(event_id);
|
||||
|
||||
-- Composite index for common query pattern: tag + kind (via join)
|
||||
CREATE INDEX idx_event_tags_value_name ON event_tags(tag_value, tag_name);
|
||||
```
|
||||
|
||||
**Key design decisions:**
|
||||
- `ON DELETE CASCADE` handles cleanup when events are deleted (NIP-09, replaceable events)
|
||||
- `tag_index` stores the position within the tags array (useful for ordered tag access)
|
||||
- Only the first two elements of each tag are indexed (`tag_name` = `$[0]`, `tag_value` = `$[1]`) — this covers all standard Nostr tag filters (`#e`, `#p`, `#t`, `#g`, `#d`, etc.)
|
||||
- `PRAGMA foreign_keys = ON` is already in the schema, so CASCADE will work
|
||||
|
||||
### 2. Store Tags: `src/main.c` — New `store_event_tags()` Function
|
||||
|
||||
Add a new function called after successful event INSERT in `store_event()`:
|
||||
|
||||
```c
|
||||
// Insert denormalized tags into event_tags table for fast indexed lookups
|
||||
int store_event_tags(const char* event_id, cJSON* tags) {
|
||||
if (!g_db || !event_id || !tags || !cJSON_IsArray(tags)) {
|
||||
return 0; // Not an error if no tags
|
||||
}
|
||||
|
||||
const char* sql = "INSERT INTO event_tags (event_id, tag_name, tag_value, tag_index) VALUES (?, ?, ?, ?)";
|
||||
sqlite3_stmt* stmt;
|
||||
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
|
||||
if (rc != SQLITE_OK) {
|
||||
DEBUG_ERROR("Failed to prepare event_tags insert: %s", sqlite3_errmsg(g_db));
|
||||
return -1;
|
||||
}
|
||||
|
||||
int tag_index = 0;
|
||||
cJSON* tag = NULL;
|
||||
cJSON_ArrayForEach(tag, tags) {
|
||||
if (cJSON_IsArray(tag) && cJSON_GetArraySize(tag) >= 2) {
|
||||
cJSON* name = cJSON_GetArrayItem(tag, 0);
|
||||
cJSON* value = cJSON_GetArrayItem(tag, 1);
|
||||
|
||||
if (cJSON_IsString(name) && cJSON_IsString(value)) {
|
||||
sqlite3_reset(stmt);
|
||||
sqlite3_bind_text(stmt, 1, event_id, -1, SQLITE_STATIC);
|
||||
sqlite3_bind_text(stmt, 2, cJSON_GetStringValue(name), -1, SQLITE_STATIC);
|
||||
sqlite3_bind_text(stmt, 3, cJSON_GetStringValue(value), -1, SQLITE_STATIC);
|
||||
sqlite3_bind_int(stmt, 4, tag_index);
|
||||
|
||||
rc = sqlite3_step(stmt);
|
||||
if (rc != SQLITE_DONE) {
|
||||
DEBUG_ERROR("Failed to insert event tag: %s", sqlite3_errmsg(g_db));
|
||||
}
|
||||
}
|
||||
}
|
||||
tag_index++;
|
||||
}
|
||||
|
||||
sqlite3_finalize(stmt);
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
**Call site** — in `store_event()` at line ~784, after `monitoring_on_event_stored()`:
|
||||
|
||||
```c
|
||||
// After successful event storage, insert denormalized tags
|
||||
store_event_tags(cJSON_GetStringValue(id), tags);
|
||||
```
|
||||
|
||||
### 3. Update REQ Query Builder: `src/main.c` — `handle_req_message()`
|
||||
|
||||
Replace the `json_each()` tag filter at lines 1243-1270 with an `event_tags` subquery:
|
||||
|
||||
**Before** (current code at line 1244):
|
||||
```c
|
||||
snprintf(sql_ptr, remaining,
|
||||
" AND EXISTS (SELECT 1 FROM json_each(json(tags)) "
|
||||
"WHERE json_extract(value, '$[0]') = ? "
|
||||
"AND json_extract(value, '$[1]') IN (");
|
||||
```
|
||||
|
||||
**After** (new code):
|
||||
```c
|
||||
snprintf(sql_ptr, remaining,
|
||||
" AND id IN (SELECT event_id FROM event_tags "
|
||||
"WHERE tag_name = ? AND tag_value IN (");
|
||||
```
|
||||
|
||||
The rest of the parameter binding code (lines 1248-1270) stays the same — the bind params are identical (`tag_name` then `tag_value(s)`). Only the SQL template changes.
|
||||
|
||||
The closing parenthesis changes from `"))` to `"))` — same syntax, just different semantics.
|
||||
|
||||
### 4. NIP-09 Delete Cascade: `src/nip009.c`
|
||||
|
||||
The `ON DELETE CASCADE` foreign key handles this automatically. When `DELETE FROM events WHERE id = ?` executes, SQLite will automatically delete matching rows from `event_tags`. **No code changes needed in nip009.c** as long as `PRAGMA foreign_keys = ON` is set (it already is in the schema).
|
||||
|
||||
However, verify the replaceable/addressable event triggers in the schema also cascade properly. The existing triggers at schema lines 97-119 use `DELETE FROM events WHERE ...` which will trigger the CASCADE.
|
||||
|
||||
### 5. Performance Fix: Move Config Reads Out of Row Loop
|
||||
|
||||
In `handle_req_message()` at lines 1400-1401, `get_config_bool()` is called **inside the `while (sqlite3_step())` loop** — meaning it executes a SQLite query for every row returned. Move these before the loop:
|
||||
|
||||
**Before** (inside loop):
|
||||
```c
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
// ...
|
||||
int expiration_enabled = get_config_bool("expiration_enabled", 1); // SQLite query per row!
|
||||
int filter_responses = get_config_bool("expiration_filter", 1); // SQLite query per row!
|
||||
```
|
||||
|
||||
**After** (before loop):
|
||||
```c
|
||||
int expiration_enabled = get_config_bool("expiration_enabled", 1);
|
||||
int filter_responses = get_config_bool("expiration_filter", 1);
|
||||
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
// ... use cached values
|
||||
```
|
||||
|
||||
### 6. Log Level Fixes
|
||||
|
||||
#### NIP-17 Decryption Failure — `src/websockets.c` lines 780 and 1488
|
||||
|
||||
Change from `DEBUG_ERROR` to `DEBUG_INFO`:
|
||||
```c
|
||||
// Before:
|
||||
DEBUG_ERROR("NIP-17 admin message processing failed");
|
||||
// After:
|
||||
DEBUG_INFO("NIP-17 admin message processing failed");
|
||||
```
|
||||
|
||||
This is expected behavior when non-admin gift wraps arrive — not an error condition.
|
||||
|
||||
#### Duplicate Event INSERT — `src/main.c` line 741
|
||||
|
||||
The `DEBUG_ERROR` at line 741 fires before the CONSTRAINT check at line 746. Suppress it for constraint violations:
|
||||
|
||||
```c
|
||||
// Before (line 738-741):
|
||||
if (rc != SQLITE_DONE) {
|
||||
const char* err_msg = sqlite3_errmsg(g_db);
|
||||
int extended_errcode = sqlite3_extended_errcode(g_db);
|
||||
DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
|
||||
}
|
||||
|
||||
// After:
|
||||
if (rc != SQLITE_DONE) {
|
||||
const char* err_msg = sqlite3_errmsg(g_db);
|
||||
int extended_errcode = sqlite3_extended_errcode(g_db);
|
||||
if (rc != SQLITE_CONSTRAINT) {
|
||||
DEBUG_ERROR("INSERT failed: rc=%d, extended_errcode=%d, msg=%s", rc, extended_errcode, err_msg);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Startup Tag Population
|
||||
|
||||
Since you said no migration code is needed (fresh deploy), the `event_tags` table will start empty and populate as new events arrive. Existing events won't have tags in the lookup table.
|
||||
|
||||
However, to avoid a period where old events don't appear in tag-filtered queries, add a one-time population function that runs at startup:
|
||||
|
||||
```c
|
||||
// Populate event_tags from existing events (run once at startup)
|
||||
int populate_event_tags_from_existing(void) {
|
||||
if (!g_db) return -1;
|
||||
|
||||
// Check if event_tags is already populated
|
||||
sqlite3_stmt* check_stmt;
|
||||
sqlite3_prepare_v2(g_db, "SELECT COUNT(*) FROM event_tags", -1, &check_stmt, NULL);
|
||||
if (sqlite3_step(check_stmt) == SQLITE_ROW && sqlite3_column_int(check_stmt, 0) > 0) {
|
||||
sqlite3_finalize(check_stmt);
|
||||
DEBUG_INFO("event_tags already populated, skipping");
|
||||
return 0;
|
||||
}
|
||||
sqlite3_finalize(check_stmt);
|
||||
|
||||
DEBUG_INFO("Populating event_tags from existing events...");
|
||||
|
||||
const char* sql = "SELECT id, tags FROM events WHERE tags != '[]'";
|
||||
sqlite3_stmt* stmt;
|
||||
int rc = sqlite3_prepare_v2(g_db, sql, -1, &stmt, NULL);
|
||||
if (rc != SQLITE_OK) return -1;
|
||||
|
||||
// Use a transaction for bulk insert performance
|
||||
sqlite3_exec(g_db, "BEGIN TRANSACTION", NULL, NULL, NULL);
|
||||
|
||||
int event_count = 0;
|
||||
while (sqlite3_step(stmt) == SQLITE_ROW) {
|
||||
const char* event_id = (const char*)sqlite3_column_text(stmt, 0);
|
||||
const char* tags_json = (const char*)sqlite3_column_text(stmt, 1);
|
||||
|
||||
if (event_id && tags_json) {
|
||||
cJSON* tags = cJSON_Parse(tags_json);
|
||||
if (tags) {
|
||||
store_event_tags(event_id, tags);
|
||||
cJSON_Delete(tags);
|
||||
event_count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sqlite3_finalize(stmt);
|
||||
sqlite3_exec(g_db, "COMMIT", NULL, NULL, NULL);
|
||||
|
||||
DEBUG_INFO("Populated event_tags for %d events", event_count);
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
Call this during startup, after database initialization but before accepting connections.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph "Current: O(n) per query"
|
||||
A1["REQ with #g filter"] --> B1["SQL: SELECT ... WHERE EXISTS"]
|
||||
B1 --> C1["json_each(json(tags))"]
|
||||
C1 --> D1["json_extract per row"]
|
||||
D1 --> E1["🔴 Full JSON parse per candidate row"]
|
||||
end
|
||||
|
||||
subgraph "New: O(log n) per query"
|
||||
A2["REQ with #g filter"] --> B2["SQL: SELECT ... WHERE id IN"]
|
||||
B2 --> C2["event_tags table"]
|
||||
C2 --> D2["idx_event_tags_lookup"]
|
||||
D2 --> E2["🟢 B-tree index lookup"]
|
||||
end
|
||||
|
||||
subgraph "Write Path (unchanged speed)"
|
||||
F["EVENT arrives"] --> G["store_event()"]
|
||||
G --> H["INSERT INTO events"]
|
||||
H --> I["store_event_tags()"]
|
||||
I --> J["INSERT INTO event_tags\n(one row per tag)"]
|
||||
end
|
||||
|
||||
subgraph "Delete Path (automatic)"
|
||||
K["NIP-09 DELETE"] --> L["DELETE FROM events"]
|
||||
L --> M["ON DELETE CASCADE"]
|
||||
M --> N["event_tags rows auto-deleted"]
|
||||
end
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. Run existing test suite: `tests/run_all_tests.sh`
|
||||
2. Specifically run NIP tests that use tag filters: `tests/run_nip_tests.sh`
|
||||
3. Verify tag-filtered REQ queries return correct results
|
||||
4. Verify NIP-09 deletion cascades to event_tags
|
||||
5. Verify replaceable event triggers cascade to event_tags
|
||||
6. Monitor CPU usage after deployment with same traffic pattern
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
- **Low risk**: The `event_tags` table is additive — it doesn't change the events table structure
|
||||
- **Low risk**: `ON DELETE CASCADE` is a well-tested SQLite feature
|
||||
- **Low risk**: The SQL change in `handle_req_message()` is a drop-in replacement (same bind params)
|
||||
- **Medium risk**: Startup population on a large database could take a few seconds — but with only 55K events, this should complete in under 1 second
|
||||
- **Write overhead**: Each event INSERT now also inserts ~5 rows into event_tags — negligible at 56 events/hour
|
||||
@@ -0,0 +1,585 @@
|
||||
# Plan: Ban Idle and Early-Disconnect Connections
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The relay needs to defend against spam connections that:
|
||||
1. Connect via WebSocket and sit idle doing nothing (the original problem)
|
||||
2. Connect and immediately disconnect without subscribing or posting (the new requirement)
|
||||
|
||||
Both patterns indicate bot/scanner behavior that should result in IP bans.
|
||||
|
||||
## Goal
|
||||
|
||||
Implement a system that:
|
||||
- Bans IPs that connect but never send a REQ or EVENT (regardless of how the connection ends)
|
||||
- Works **independent of NIP-42 authentication** (no auth required)
|
||||
- Allows legitimate users to connect and use the relay normally
|
||||
- Uses existing IP ban infrastructure
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Client connects] --> B{IP banned?}
|
||||
B -->|Yes| C[Reject immediately]
|
||||
B -->|No| D[Set idle timeout timer]
|
||||
D --> E{Client sends REQ or EVENT?}
|
||||
E -->|Yes| F[Mark session ACTIVE<br>Cancel idle timer]
|
||||
E -->|No| G{Connection closes?}
|
||||
G --> H{Session was ACTIVE?}
|
||||
H -->|Yes| I[Clean close - no ban]
|
||||
H -->|No| J["ip_ban_record_failure()<br>→ may trigger ban"]
|
||||
F --> K[Normal operation]
|
||||
K --> L[Connection closes]
|
||||
L --> I
|
||||
```
|
||||
|
||||
## Key Insight
|
||||
|
||||
The distinction between "idle timeout" and "early disconnect" is minimal:
|
||||
- **Idle timeout**: Connection closed by server because client never became ACTIVE
|
||||
- **Early disconnect**: Connection closed by client because client never became ACTIVE
|
||||
|
||||
Both result in the same check: `if (!session_was_active) ban_ip()`
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### 1. Add Session Activity Tracking
|
||||
|
||||
**File: `src/websockets.h`**
|
||||
|
||||
Add to `struct per_session_data`:
|
||||
|
||||
```c
|
||||
// Session activity tracking for idle connection banning
|
||||
int session_active; // 1 if client sent REQ or EVENT, 0 otherwise
|
||||
int idle_timeout_sec; // Timeout value for this session (copied from config)
|
||||
```
|
||||
|
||||
### 2. Set Idle Timeout on All Connections
|
||||
|
||||
**File: `src/websockets.c` - `LWS_CALLBACK_ESTABLISHED`**
|
||||
|
||||
Currently (lines 561-572):
|
||||
```c
|
||||
// Only set timeout if auth is required
|
||||
if (pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) {
|
||||
int auth_timeout = get_config_int("nip42_auth_timeout_sec", 10);
|
||||
if (auth_timeout > 0) {
|
||||
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, auth_timeout);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change to:
|
||||
```c
|
||||
// Initialize session activity tracking
|
||||
pss->session_active = 0;
|
||||
pss->idle_timeout_sec = get_config_int("idle_connection_timeout_sec", 30);
|
||||
|
||||
// Set idle timeout for ALL connections (not just auth-required)
|
||||
// This catches bots that connect and do nothing
|
||||
if (pss->idle_timeout_sec > 0) {
|
||||
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, pss->idle_timeout_sec);
|
||||
DEBUG_TRACE("Idle timeout set: %d seconds for connection from %s",
|
||||
pss->idle_timeout_sec, pss->client_ip);
|
||||
}
|
||||
|
||||
// Also set auth timeout if auth is required (separate concern)
|
||||
if (pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) {
|
||||
int auth_timeout = get_config_int("nip42_auth_timeout_sec", 10);
|
||||
// Use the shorter of the two timeouts
|
||||
int effective_timeout = (pss->idle_timeout_sec > 0 && pss->idle_timeout_sec < auth_timeout)
|
||||
? pss->idle_timeout_sec
|
||||
: auth_timeout;
|
||||
if (effective_timeout > 0) {
|
||||
lws_set_timeout(wsi, PENDING_TIMEOUT_AWAITING_PING, effective_timeout);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Mark Session as Active on Valid Activity
|
||||
|
||||
**File: `src/websockets.c` - `LWS_CALLBACK_RECEIVE`**
|
||||
|
||||
When processing REQ message (around line 1030):
|
||||
```c
|
||||
// Before handling REQ, mark session as active
|
||||
pthread_mutex_lock(&pss->session_lock);
|
||||
pss->session_active = 1;
|
||||
pthread_mutex_unlock(&pss->session_lock);
|
||||
|
||||
// Cancel idle timeout - this is legitimate usage
|
||||
lws_set_timeout(wsi, NO_PENDING_TIMEOUT, 0);
|
||||
```
|
||||
|
||||
When processing EVENT message (around line 1380):
|
||||
```c
|
||||
// Before handling EVENT, mark session as active
|
||||
pthread_mutex_lock(&pss->session_lock);
|
||||
pss->session_active = 1;
|
||||
pthread_mutex_unlock(&pss->session_lock);
|
||||
|
||||
// Cancel idle timeout - this is legitimate usage
|
||||
lws_set_timeout(wsi, NO_PENDING_TIMEOUT, 0);
|
||||
```
|
||||
|
||||
**Note**: We should also consider AUTH as "activity" since the client is engaging with the protocol:
|
||||
```c
|
||||
// In handle_nip42_auth_signed_event (nip042.c)
|
||||
// After successful auth, also mark session active
|
||||
pthread_mutex_lock(&pss->session_lock);
|
||||
pss->session_active = 1;
|
||||
pthread_mutex_unlock(&pss->session_lock);
|
||||
```
|
||||
|
||||
### 3.5. Separate Rate Limiting for Idle Connections
|
||||
|
||||
Per user feedback, idle/early-disconnect failures have **separate rate limiting** from auth failures. This requires:
|
||||
|
||||
**New Config Keys:**
|
||||
|
||||
| Config Key | Type | Default | Range | Description |
|
||||
|------------|------|---------|-------|-------------|
|
||||
| `idle_ban_threshold` | int | 3 | 1-100 | Idle failures before ban |
|
||||
| `idle_ban_window_sec` | int | 60 | 10-3600 | Window for counting idle failures |
|
||||
| `idle_ban_duration_sec` | int | 300 | 60-86400 | Initial idle ban duration |
|
||||
|
||||
**Add to `ip_ban_entry_t` in `src/ip_ban.c`:**
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
int state;
|
||||
char ip[46];
|
||||
|
||||
// Auth failure tracking (existing)
|
||||
int failure_count;
|
||||
time_t first_failure;
|
||||
time_t banned_until;
|
||||
int ban_count;
|
||||
|
||||
// NEW: Idle failure tracking (separate)
|
||||
int idle_failure_count;
|
||||
time_t idle_first_failure;
|
||||
time_t idle_banned_until;
|
||||
int idle_ban_count;
|
||||
|
||||
// Other existing fields...
|
||||
int has_authed_successfully;
|
||||
time_t last_success_at;
|
||||
int total_connections;
|
||||
int total_failures;
|
||||
int total_successes;
|
||||
time_t first_seen;
|
||||
} ip_ban_entry_t;
|
||||
```
|
||||
|
||||
**New function in `src/ip_ban.c`:**
|
||||
|
||||
```c
|
||||
// Record an idle/early-disconnect failure for an IP
|
||||
void ip_ban_record_idle_failure(const char* ip) {
|
||||
if (!ip || !g_initialized) return;
|
||||
if (!get_config_bool("idle_ban_enabled", 1)) return;
|
||||
|
||||
int threshold = get_config_int("idle_ban_threshold", 3);
|
||||
int window_sec = get_config_int("idle_ban_window_sec", 60);
|
||||
int ban_duration = get_config_int("idle_ban_duration_sec", 300);
|
||||
|
||||
pthread_mutex_lock(&g_ban_mutex);
|
||||
ip_ban_entry_t* entry = get_or_create_entry(ip);
|
||||
if (!entry) {
|
||||
pthread_mutex_unlock(&g_ban_mutex);
|
||||
return;
|
||||
}
|
||||
|
||||
time_t now = time(NULL);
|
||||
|
||||
// Reset window if expired
|
||||
if (entry->idle_first_failure > 0 && (now - entry->idle_first_failure) > window_sec) {
|
||||
entry->idle_failure_count = 0;
|
||||
entry->idle_first_failure = now;
|
||||
}
|
||||
if (entry->idle_first_failure == 0) {
|
||||
entry->idle_first_failure = now;
|
||||
}
|
||||
|
||||
entry->idle_failure_count++;
|
||||
entry->total_failures++;
|
||||
|
||||
DEBUG_TRACE("IP %s idle failure count: %d/%d", ip, entry->idle_failure_count, threshold);
|
||||
|
||||
if (entry->idle_failure_count >= threshold) {
|
||||
int duration = ban_duration;
|
||||
for (int i = 0; i < entry->idle_ban_count && duration < 86400; i++) {
|
||||
duration *= 2;
|
||||
}
|
||||
if (duration > 86400) duration = 86400;
|
||||
|
||||
entry->idle_banned_until = now + duration;
|
||||
entry->idle_ban_count++;
|
||||
entry->idle_failure_count = 0;
|
||||
entry->idle_first_failure = 0;
|
||||
|
||||
DEBUG_WARN("IP %s banned for %d seconds (idle ban #%d) after %d idle failures",
|
||||
ip, duration, entry->idle_ban_count, threshold);
|
||||
}
|
||||
|
||||
pthread_mutex_unlock(&g_ban_mutex);
|
||||
}
|
||||
```
|
||||
|
||||
**Update `ip_ban_is_banned()` to check BOTH ban types:**
|
||||
|
||||
```c
|
||||
int ip_ban_is_banned(const char* ip) {
|
||||
if (!ip || !g_initialized) return 0;
|
||||
|
||||
pthread_mutex_lock(&g_ban_mutex);
|
||||
int idx = find_slot(ip);
|
||||
if (idx < 0 || g_ban_table[idx].state == IP_BAN_EMPTY) {
|
||||
pthread_mutex_unlock(&g_ban_mutex);
|
||||
return 0;
|
||||
}
|
||||
|
||||
ip_ban_entry_t* entry = &g_ban_table[idx];
|
||||
time_t now = time(NULL);
|
||||
int banned = 0;
|
||||
|
||||
// Check auth ban (if enabled)
|
||||
if (get_config_bool("auth_fail_ban_enabled", 1) &&
|
||||
entry->banned_until > 0 && now < entry->banned_until) {
|
||||
banned = 1;
|
||||
}
|
||||
|
||||
// Check idle ban (if enabled)
|
||||
if (get_config_bool("idle_ban_enabled", 1) &&
|
||||
entry->idle_banned_until > 0 && now < entry->idle_banned_until) {
|
||||
banned = 1;
|
||||
}
|
||||
|
||||
// Clear expired bans
|
||||
if (!banned) {
|
||||
if (entry->banned_until > 0 && now >= entry->banned_until) {
|
||||
entry->banned_until = 0;
|
||||
entry->failure_count = 0;
|
||||
entry->first_failure = 0;
|
||||
}
|
||||
if (entry->idle_banned_until > 0 && now >= entry->idle_banned_until) {
|
||||
entry->idle_banned_until = 0;
|
||||
entry->idle_failure_count = 0;
|
||||
entry->idle_first_failure = 0;
|
||||
}
|
||||
}
|
||||
|
||||
pthread_mutex_unlock(&g_ban_mutex);
|
||||
return banned;
|
||||
}
|
||||
```
|
||||
|
||||
**Update persistence:**
|
||||
- `ip_ban_load_from_db()`: Load idle_* fields from new columns
|
||||
- `ip_ban_save_to_db()`: Save idle_* fields to new columns
|
||||
- Add migration SQL to create new columns if they don't exist
|
||||
|
||||
**Add to `src/ip_ban.h`:**
|
||||
|
||||
```c
|
||||
// Record an idle/early-disconnect failure for an IP
|
||||
void ip_ban_record_idle_failure(const char* ip);
|
||||
```
|
||||
|
||||
### 4. Record Failure on Inactive Connection Close
|
||||
|
||||
**File: `src/websockets.c` - `LWS_CALLBACK_CLOSED`**
|
||||
|
||||
Currently (lines 2121-2128):
|
||||
```c
|
||||
// Record auth failure if connection closed while unauthenticated and auth was required
|
||||
if (!pss->authenticated &&
|
||||
(pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) &&
|
||||
pss->auth_challenge_sent &&
|
||||
strlen(pss->client_ip) > 0) {
|
||||
ip_ban_record_failure(pss->client_ip);
|
||||
}
|
||||
```
|
||||
|
||||
Change to:
|
||||
```c
|
||||
// Record failure if connection closed without ever becoming active
|
||||
// This catches:
|
||||
// 1. Idle connections that timed out (never sent REQ/EVENT/AUTH)
|
||||
// 2. Early disconnects (client closed before sending REQ/EVENT/AUTH)
|
||||
if (!pss->session_active && strlen(pss->client_ip) > 0) {
|
||||
// Use separate idle failure recording (has its own threshold/duration)
|
||||
ip_ban_record_idle_failure(pss->client_ip);
|
||||
DEBUG_LOG("Recording idle/early-disconnect failure for IP %s (connected %ld seconds)",
|
||||
pss->client_ip,
|
||||
time(NULL) - pss->connection_established);
|
||||
}
|
||||
// Legacy: record auth failure if auth was required but not completed
|
||||
else if (!pss->authenticated &&
|
||||
(pss->nip42_auth_required_events || pss->nip42_auth_required_subscriptions) &&
|
||||
pss->auth_challenge_sent &&
|
||||
strlen(pss->client_ip) > 0) {
|
||||
ip_ban_record_failure(pss->client_ip);
|
||||
}
|
||||
```
|
||||
|
||||
### 5. Add Configuration Keys
|
||||
|
||||
**File: `src/config.c` - Add validation for new keys**
|
||||
|
||||
In the config validation section, add:
|
||||
|
||||
```c
|
||||
// Idle connection ban settings
|
||||
if (strcmp(key, "idle_connection_timeout_sec") == 0) {
|
||||
if (!is_valid_positive_integer(value)) {
|
||||
snprintf(error_msg, error_size, "invalid idle_connection_timeout_sec '%s' (must be positive integer)", value);
|
||||
return -1;
|
||||
}
|
||||
int timeout = atoi(value);
|
||||
if (timeout < 5 || timeout > 300) {
|
||||
snprintf(error_msg, error_size, "idle_connection_timeout_sec must be between 5 and 300 seconds");
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (strcmp(key, "idle_ban_enabled") == 0) {
|
||||
if (!is_valid_boolean(value)) {
|
||||
snprintf(error_msg, error_size, "invalid boolean value '%s' for idle_ban_enabled", value);
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (strcmp(key, "idle_ban_threshold") == 0) {
|
||||
if (!is_valid_positive_integer(value)) {
|
||||
snprintf(error_msg, error_size, "invalid idle_ban_threshold '%s' (must be positive integer)", value);
|
||||
return -1;
|
||||
}
|
||||
int threshold = atoi(value);
|
||||
if (threshold < 1 || threshold > 100) {
|
||||
snprintf(error_msg, error_size, "idle_ban_threshold must be between 1 and 100");
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (strcmp(key, "idle_ban_window_sec") == 0) {
|
||||
if (!is_valid_positive_integer(value)) {
|
||||
snprintf(error_msg, error_size, "invalid idle_ban_window_sec '%s' (must be positive integer)", value);
|
||||
return -1;
|
||||
}
|
||||
int window = atoi(value);
|
||||
if (window < 10 || window > 3600) {
|
||||
snprintf(error_msg, error_size, "idle_ban_window_sec must be between 10 and 3600");
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (strcmp(key, "idle_ban_duration_sec") == 0) {
|
||||
if (!is_valid_positive_integer(value)) {
|
||||
snprintf(error_msg, error_size, "invalid idle_ban_duration_sec '%s' (must be positive integer)", value);
|
||||
return -1;
|
||||
}
|
||||
int duration = atoi(value);
|
||||
if (duration < 60 || duration > 86400) {
|
||||
snprintf(error_msg, error_size, "idle_ban_duration_sec must be between 60 and 86400");
|
||||
return -1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
Also update the config introspection/documentation functions to describe these keys.
|
||||
|
||||
### 6. Add API Documentation
|
||||
|
||||
**File: `src/api.c` - Add to config metadata**
|
||||
|
||||
```c
|
||||
{"idle_connection_timeout_sec", "int", 5, 300},
|
||||
{"idle_ban_enabled", "bool", 0, 1},
|
||||
{"idle_ban_threshold", "int", 1, 100},
|
||||
{"idle_ban_window_sec", "int", 10, 3600},
|
||||
{"idle_ban_duration_sec", "int", 60, 86400},
|
||||
```
|
||||
|
||||
Add descriptions in the config query handler:
|
||||
```c
|
||||
} else if (strcmp(key, "idle_connection_timeout_sec") == 0) {
|
||||
description = "Seconds before an idle connection (no REQ/EVENT) is closed and IP flagged. 0 to disable.";
|
||||
} else if (strcmp(key, "idle_ban_enabled") == 0) {
|
||||
description = "Whether to ban IPs that repeatedly connect without sending REQ or EVENT.";
|
||||
} else if (strcmp(key, "idle_ban_threshold") == 0) {
|
||||
description = "Number of idle/early-disconnect failures before banning an IP.";
|
||||
} else if (strcmp(key, "idle_ban_window_sec") == 0) {
|
||||
description = "Time window in seconds for counting idle failures.";
|
||||
} else if (strcmp(key, "idle_ban_duration_sec") == 0) {
|
||||
description = "Initial ban duration in seconds for idle failures (doubles each time).";
|
||||
}
|
||||
```
|
||||
|
||||
### 7. Update ip_ban Persistence and Cleanup
|
||||
|
||||
**File: `src/ip_ban.c` - Update `ip_ban_load_from_db()`**
|
||||
|
||||
Add migration SQL to create new columns if they don't exist, then load them:
|
||||
```sql
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0;
|
||||
```
|
||||
|
||||
**File: `src/ip_ban.c` - Update `ip_ban_save_to_db()`**
|
||||
|
||||
Add the idle_* fields to the INSERT/REPLACE statement.
|
||||
|
||||
**File: `src/ip_ban.c` - Update `ip_ban_cleanup()`**
|
||||
|
||||
Add cleanup logic for idle ban entries (same pattern as auth ban cleanup).
|
||||
|
||||
**File: `src/ip_ban.c` - Update `ip_ban_log_stats()`**
|
||||
|
||||
Add idle ban count to the periodic log summary:
|
||||
```c
|
||||
DEBUG_WARN("IP BAN SUMMARY: %d auth-banned, %d idle-banned, %d tracked, %d trusted",
|
||||
auth_banned_count, idle_banned_count, tracked_count, trusted_count);
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### Idle Connection Settings
|
||||
|
||||
| Config Key | Type | Default | Range | Description |
|
||||
|------------|------|---------|-------|-------------|
|
||||
| `idle_connection_timeout_sec` | int | 30 | 5-300 | Seconds to wait for REQ/EVENT before closing connection. 0 = disable idle timeout. |
|
||||
| `idle_ban_enabled` | bool | true | true/false | Whether to ban IPs with repeated idle/early-disconnect failures. |
|
||||
| `idle_ban_threshold` | int | 3 | 1-100 | Idle failures before ban. |
|
||||
| `idle_ban_window_sec` | int | 60 | 10-3600 | Window for counting idle failures. |
|
||||
| `idle_ban_duration_sec` | int | 300 | 60-86400 | Initial idle ban duration. |
|
||||
|
||||
### Auth Failure Settings (Existing)
|
||||
|
||||
| Config Key | Type | Default | Range | Description |
|
||||
|------------|------|---------|-------|-------------|
|
||||
| `auth_fail_ban_enabled` | bool | true | true/false | Whether to ban IPs with failed auth. |
|
||||
| `auth_fail_ban_threshold` | int | 3 | 1-100 | Auth failures before ban. |
|
||||
| `auth_fail_window_sec` | int | 60 | 10-3600 | Window for counting auth failures. |
|
||||
| `auth_fail_ban_duration_sec` | int | 300 | 60-86400 | Initial auth ban duration. |
|
||||
|
||||
## Behavior Matrix
|
||||
|
||||
| Scenario | idle_timeout_sec | idle_ban_enabled | Result |
|
||||
|----------|------------------|------------------|--------|
|
||||
| Connect → do nothing → timeout | >0 | true | Connection closed, IP failure recorded, may be banned |
|
||||
| Connect → do nothing → timeout | >0 | false | Connection closed, no ban |
|
||||
| Connect → immediate disconnect | any | true | IP failure recorded, may be banned |
|
||||
| Connect → immediate disconnect | any | false | No ban |
|
||||
| Connect → send REQ | any | any | Session active, no ban |
|
||||
| Connect → send EVENT | any | any | Session active, no ban |
|
||||
| Connect → send AUTH (NIP-42) | any | any | Session active, no ban |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Idle timeout test**: Connect via wscat, wait 30 seconds, verify disconnect and ban table entry
|
||||
2. **Early disconnect test**: Connect via wscat, immediately Ctrl-C, verify ban table entry
|
||||
3. **Active session test**: Connect, send REQ, disconnect immediately, verify NO ban
|
||||
4. **Config disable test**: Set `idle_ban_enabled=false`, repeat test 1, verify no ban
|
||||
5. **Timeout config test**: Set `idle_connection_timeout_sec=5`, verify faster disconnect
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Change Type |
|
||||
|------|-------------|
|
||||
| `src/websockets.h` | Add `session_active` and `idle_timeout_sec` fields to `per_session_data` |
|
||||
| `src/websockets.c` ~565 | Set idle timeout on ALL connections in `LWS_CALLBACK_ESTABLISHED` |
|
||||
| `src/websockets.c` ~1030 | Mark `session_active=1` and cancel idle timer on REQ |
|
||||
| `src/websockets.c` ~1380 | Mark `session_active=1` and cancel idle timer on EVENT |
|
||||
| `src/websockets.c` ~2121 | Call `ip_ban_record_idle_failure()` for inactive sessions in `LWS_CALLBACK_CLOSED` |
|
||||
| `src/nip042.c` ~130 | Mark `session_active=1` on successful AUTH |
|
||||
| `src/ip_ban.h` | Add `ip_ban_record_idle_failure()` declaration |
|
||||
| `src/ip_ban.c` | Add idle_* fields to `ip_ban_entry_t`, new `ip_ban_record_idle_failure()` function, update `ip_ban_is_banned()` to check both ban types, update persistence and cleanup |
|
||||
| `src/config.c` ~987+ | Add validation for 5 new idle_* config keys |
|
||||
| `src/api.c` ~664 | Add config metadata entries for idle_* keys |
|
||||
|
||||
## Deployment on Existing Server
|
||||
|
||||
### Config Keys: No Manual SQL Required
|
||||
|
||||
All config keys use [`get_config_int()`](src/config.c:300) and [`get_config_bool()`](src/config.c:313) which return **hardcoded defaults** when a key doesn't exist in the config table. The new code will work immediately with these defaults:
|
||||
|
||||
- `idle_connection_timeout_sec` → defaults to 30
|
||||
- `idle_ban_enabled` → defaults to true
|
||||
- `idle_ban_threshold` → defaults to 3
|
||||
- `idle_ban_window_sec` → defaults to 60
|
||||
- `idle_ban_duration_sec` → defaults to 300
|
||||
|
||||
If you want to **override** any defaults, you can insert them into the config table. From the same directory as the `.db` file:
|
||||
|
||||
```bash
|
||||
# Find your database file
|
||||
DB_FILE=$(ls *.db | head -1)
|
||||
|
||||
# Optional: Insert custom values (only if you want non-default settings)
|
||||
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_connection_timeout_sec', '30');"
|
||||
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_enabled', 'true');"
|
||||
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_threshold', '3');"
|
||||
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_window_sec', '60');"
|
||||
sqlite3 "$DB_FILE" "INSERT OR REPLACE INTO config (key, value) VALUES ('idle_ban_duration_sec', '300');"
|
||||
```
|
||||
|
||||
Or change them via the admin API/web UI after deployment.
|
||||
|
||||
### ip_bans Table: Automatic Migration
|
||||
|
||||
The `ip_bans` table needs 4 new columns for idle tracking. The updated [`ip_ban_load_from_db()`](src/ip_ban.c:93) will run `ALTER TABLE` statements automatically on startup. These are safe because SQLite's `ALTER TABLE ADD COLUMN` is a no-op if the column already exists (we'll use error-tolerant execution).
|
||||
|
||||
If you prefer to run the migration manually before deploying:
|
||||
|
||||
```bash
|
||||
DB_FILE=$(ls *.db | head -1)
|
||||
|
||||
sqlite3 "$DB_FILE" <<'SQL'
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_failure_count INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_ban_count INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_banned_until INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE ip_bans ADD COLUMN idle_first_failure INTEGER NOT NULL DEFAULT 0;
|
||||
SQL
|
||||
```
|
||||
|
||||
**Note:** SQLite will error on `ALTER TABLE ADD COLUMN` if the column already exists, but the code will handle this gracefully (ignore the error). Running it manually is optional — the relay will do it on startup.
|
||||
|
||||
### Deployment Steps
|
||||
|
||||
1. Build and deploy with `deploy_lt.sh` as usual
|
||||
2. The relay starts, `ip_ban_load_from_db()` auto-migrates the `ip_bans` table
|
||||
3. Config keys use defaults immediately — no manual SQL needed
|
||||
4. Optionally tune settings via admin API or direct SQL
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- Default `idle_connection_timeout_sec=30` provides immediate protection
|
||||
- Default `idle_ban_enabled=true` maintains current spam protection behavior
|
||||
- Setting `idle_connection_timeout_sec=0` restores old behavior (no idle timeout)
|
||||
- Setting `idle_ban_enabled=false` allows connections without banning (for debugging)
|
||||
- Existing auth-based banning continues to work independently with its own thresholds
|
||||
- Database migration adds new columns with defaults — existing ban data preserved
|
||||
|
||||
## Summary
|
||||
|
||||
This plan implements a unified "session activity" tracking system with **separate rate limiting** for idle vs auth failures:
|
||||
|
||||
1. **Catches idle connections** via `lws_set_timeout()` on ALL connections (not just auth-required)
|
||||
2. **Catches early disconnects** via the same `!session_active` check on close
|
||||
3. **Respects legitimate users** who actually use the relay (REQ/EVENT/AUTH marks session active)
|
||||
4. **Separate idle ban tracking** — idle failures have their own threshold, window, and duration independent of auth failures
|
||||
5. **Extends existing ban infrastructure** — adds idle_* fields to `ip_ban_entry_t`, unified `ip_ban_is_banned()` checks both ban types
|
||||
6. **Fully configurable** — 5 new config keys for idle banning, all tunable via admin events
|
||||
|
||||
The key insight is that there's no meaningful difference between "idle timeout" and "early disconnect" — both indicate a client that connected but never actually used the relay. The same `!session_active` check handles both cases, and the separate rate limiting ensures idle bots don't interfere with auth failure tracking for legitimate clients who fail NIP-42.
|
||||
@@ -0,0 +1,194 @@
|
||||
# Main Thread CPU Offload Plan
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The main `c_relay_pg` thread consumes ~56% CPU while the DB worker threads (`db-read-1..4`, `db-write`) sit near 0%. All protocol handling, JSON parsing, event validation, subscription matching, and message queueing happens synchronously inside `nostr_relay_callback()` on the main libwebsockets event-loop thread.
|
||||
|
||||
## Current Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
LWS[lws_service - main thread] --> CB[nostr_relay_callback]
|
||||
CB --> PARSE[JSON parse - cJSON_Parse]
|
||||
CB --> VALIDATE[Signature verify - nostr_validate_unified_request]
|
||||
CB --> STORE[store_event - builds payload]
|
||||
STORE --> SYNC_WRITE[thread_pool_execute_store_event_sync]
|
||||
SYNC_WRITE --> |blocks main thread| DB_WRITE[db-write thread]
|
||||
DB_WRITE --> |signal| SYNC_WRITE
|
||||
CB --> REQ[handle_req_message - builds SQL]
|
||||
REQ --> SYNC_READ[thread_pool_execute_req_sync]
|
||||
SYNC_READ --> |blocks main thread| DB_READ[db-read-N thread]
|
||||
DB_READ --> |signal| SYNC_READ
|
||||
CB --> BROADCAST[broadcast_event_to_subscriptions]
|
||||
BROADCAST --> QUEUE[queue_message per subscriber]
|
||||
CB --> WRITEABLE[LWS_CALLBACK_SERVER_WRITEABLE]
|
||||
WRITEABLE --> DRAIN[process_message_queue - lws_write]
|
||||
```
|
||||
|
||||
### Why the main thread is hot
|
||||
|
||||
| Work item | Where | Cost |
|
||||
|-----------|-------|------|
|
||||
| JSON parsing | `cJSON_Parse` in `LWS_CALLBACK_RECEIVE` | Medium - per message |
|
||||
| Signature verification | `nostr_validate_unified_request` - ed25519 crypto | **High** - per EVENT |
|
||||
| SQL query building | `handle_req_message` filter-to-SQL loop | Low-Medium |
|
||||
| Sync DB wait | `thread_pool_execute_*_sync` - pthread_cond_wait | Blocks but yields CPU |
|
||||
| Result iteration + expiration check | Row loop in `handle_req_message` with `cJSON_Parse` per row | Medium-High for large result sets |
|
||||
| Subscription matching | `broadcast_event_to_subscriptions` - filter matching | Medium - scales with subscriber count |
|
||||
| Message serialization + queueing | `snprintf` + `queue_message_take_ownership` per subscriber | Medium |
|
||||
| Config lookups | `get_config_int/bool` called repeatedly in hot paths | Low but frequent |
|
||||
|
||||
### Key constraint: libwebsockets is single-threaded
|
||||
|
||||
libwebsockets requires that **all `lws_write`, `lws_callback_on_writable`, and `lws_close_reason` calls happen from the service thread** (the thread running `lws_service`). This means we cannot directly write to WebSocket connections from worker threads. However, we **can** do computation on worker threads and post results back to the main thread for I/O.
|
||||
|
||||
## Offload Strategy
|
||||
|
||||
### Phase 1: Async EVENT Processing (Highest Impact)
|
||||
|
||||
Convert EVENT handling from synchronous to async. Currently the main thread does: parse → validate → store → broadcast, all blocking. Instead:
|
||||
|
||||
1. **Main thread**: Parse JSON (fast), extract event ID for dedup check, then submit a job to a new "event processing" worker thread
|
||||
2. **Worker thread**: Signature verification (expensive crypto), store_event (already goes to DB thread), prepare broadcast payload
|
||||
3. **Main thread callback**: Receive result via `wake_loop_cb` + `lws_cancel_service`, send OK response and broadcast to subscribers
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
LWS[lws_service - main thread] --> RECEIVE[LWS_CALLBACK_RECEIVE]
|
||||
RECEIVE --> PARSE[JSON parse + dedup check]
|
||||
PARSE --> SUBMIT[Submit to event-worker queue]
|
||||
SUBMIT --> LWS
|
||||
|
||||
WORKER[event-worker thread] --> VERIFY[Signature verification]
|
||||
VERIFY --> STORE_DB[store_event via DB pool]
|
||||
STORE_DB --> PREP[Prepare broadcast payload]
|
||||
PREP --> RESULT_Q[Push result to completion queue]
|
||||
RESULT_Q --> WAKE[lws_cancel_service]
|
||||
|
||||
LWS2[lws_service wakes] --> POLL[Poll completion queue]
|
||||
POLL --> OK[Send OK response via queue_message]
|
||||
POLL --> BCAST[broadcast_event_to_subscriptions]
|
||||
```
|
||||
|
||||
**What this offloads**: ed25519 signature verification (~the most expensive per-event operation), event classification, tag serialization, and the synchronous DB store wait.
|
||||
|
||||
**What stays on main thread**: JSON parse (needed to extract event ID for dedup), OK response writing, broadcast fan-out (requires lws access).
|
||||
|
||||
### Phase 2: Async REQ Query Execution (Medium Impact)
|
||||
|
||||
Convert REQ handling from sync to async:
|
||||
|
||||
1. **Main thread**: Parse filters, build SQL, create subscription, submit query job
|
||||
2. **DB reader thread**: Execute query (already happens, but currently blocks main thread via `_sync`)
|
||||
3. **Main thread callback**: Iterate results, queue EVENT messages, send EOSE
|
||||
|
||||
This is simpler than Phase 1 because the thread pool already supports async submission via `thread_pool_submit_read` with a `result_cb`. The `_sync` wrappers just add a condvar wait on top. We need to:
|
||||
- Use `thread_pool_submit_read` directly instead of `thread_pool_execute_req_sync`
|
||||
- Store pending REQ context (sub_id, wsi, pss) so the callback can complete the work
|
||||
- In the result callback, push results to a completion queue and call `lws_cancel_service`
|
||||
- On the main thread, drain the completion queue and send EVENT + EOSE messages
|
||||
|
||||
### Phase 3: Reduce Per-Row Overhead in REQ Results (Low-Medium Impact)
|
||||
|
||||
Currently each row from a REQ query gets `cJSON_Parse` just to check NIP-40 expiration. This is wasteful:
|
||||
|
||||
- Option A: Add an `expiration` column to the events table so expiration filtering can be done in SQL
|
||||
- Option B: Store expiration timestamp in a fast-parse format (extract during INSERT, store as integer column)
|
||||
- Option C: Use string search on the raw JSON for the expiration tag instead of full parse
|
||||
|
||||
### Phase 4: Config Value Caching (Low Impact, Easy Win)
|
||||
|
||||
`get_config_int` and `get_config_bool` are called on every message in hot paths. These do SQLite queries. Cache config values in memory with a TTL or invalidation signal, so the main loop only refreshes them periodically (already done for `debug_level` every 60s — extend to all hot-path config values).
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
| Phase | Impact | Risk | Complexity |
|
||||
|-------|--------|------|------------|
|
||||
| Phase 1: Async EVENT | **High** - removes crypto from main thread | Medium - async state management | Medium-High |
|
||||
| Phase 2: Async REQ | **Medium** - unblocks main thread during queries | Low - infrastructure exists | Medium |
|
||||
| Phase 3: Expiration optimization | **Low-Medium** - reduces per-row parse cost | Low | Low |
|
||||
| Phase 4: Config caching | **Low** - reduces DB round-trips | Very Low | Low |
|
||||
|
||||
## Detailed Design: Phase 1 (Async EVENT Processing)
|
||||
|
||||
### New Components
|
||||
|
||||
#### Completion Queue (`src/completion_queue.h/.c`)
|
||||
A thread-safe FIFO queue for posting results from worker threads back to the main thread:
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
int type; // COMPLETION_TYPE_EVENT_RESULT, COMPLETION_TYPE_REQ_RESULT, etc.
|
||||
void* data; // Type-specific result data
|
||||
struct lws* wsi; // Target WebSocket connection
|
||||
void* pss; // Per-session data
|
||||
} completion_item_t;
|
||||
|
||||
int completion_queue_init(void);
|
||||
int completion_queue_push(completion_item_t* item);
|
||||
completion_item_t* completion_queue_pop(void); // Non-blocking
|
||||
void completion_queue_shutdown(void);
|
||||
```
|
||||
|
||||
#### Event Worker Thread
|
||||
A dedicated pthread that processes EVENT validation/storage:
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
cJSON* event; // Parsed event JSON - ownership transferred
|
||||
cJSON* full_message; // Full message JSON for context
|
||||
struct lws* wsi;
|
||||
void* pss;
|
||||
char sub_id[64]; // For response routing
|
||||
} event_work_item_t;
|
||||
```
|
||||
|
||||
#### Main Loop Integration
|
||||
Add a completion queue drain step to the main event loop. After `lws_service` returns (either from timeout or `lws_cancel_service` wake), check the completion queue:
|
||||
|
||||
```c
|
||||
while (g_server_running && !g_shutdown_flag) {
|
||||
int result = lws_service(ws_context, 1000);
|
||||
|
||||
// NEW: Drain completion queue
|
||||
completion_item_t* item;
|
||||
while ((item = completion_queue_pop()) != NULL) {
|
||||
process_completion(item); // Send OK, broadcast, etc.
|
||||
free(item);
|
||||
}
|
||||
|
||||
// ... existing periodic checks ...
|
||||
}
|
||||
```
|
||||
|
||||
### Changes to Existing Code
|
||||
|
||||
1. **`nostr_relay_callback` EVENT path**: After JSON parse and dedup check, instead of calling `nostr_validate_unified_request` + `store_event` + `broadcast_event_to_subscriptions` synchronously, submit an `event_work_item_t` to the event worker queue and return 0 immediately.
|
||||
|
||||
2. **`store_event`**: No changes needed — it already uses `thread_pool_execute_store_event_sync` which will run on the DB writer thread. The event worker thread will call it.
|
||||
|
||||
3. **`broadcast_event_to_subscriptions`**: No changes needed — it will be called from the main thread when processing the completion item, which is correct since it calls `queue_message_take_ownership` (requires lws thread).
|
||||
|
||||
### Thread Safety Considerations
|
||||
|
||||
- The `cJSON* event` object must be fully owned by the worker thread during processing. The main thread must not access it after submission.
|
||||
- The `wsi` and `pss` pointers could become invalid if the client disconnects while the event is being processed. The completion handler must validate that the connection is still alive before sending the OK response.
|
||||
- A generation counter or epoch on `pss` can detect stale references.
|
||||
|
||||
## What Cannot Be Offloaded
|
||||
|
||||
- **`lws_write` / `queue_message`**: Must happen on the lws service thread
|
||||
- **`lws_callback_on_writable`**: Must happen on the lws service thread
|
||||
- **`lws_close_reason`**: Must happen on the lws service thread
|
||||
- **Subscription list iteration for broadcast**: Accesses `lws_wsi_user` which is lws-internal
|
||||
|
||||
These are fundamental libwebsockets constraints. The pattern is always: do computation off-thread, post result to completion queue, wake main thread, do I/O on main thread.
|
||||
|
||||
## Expected Impact
|
||||
|
||||
With Phase 1 alone, the main thread would no longer perform:
|
||||
- ed25519 signature verification (~100-500μs per event depending on CPU)
|
||||
- Synchronous DB store wait (~50-200μs per event)
|
||||
- Event classification, tag serialization, JSON serialization for storage
|
||||
|
||||
This should reduce main-thread CPU by roughly 30-50% for EVENT-heavy workloads, shifting that work to the event worker thread and DB threads.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user