laantungir/c-relay-pg
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v2.1.23 - Caching service: descriptive error reporting, retry limit, inbox poller fix, config reload fix, PHP admin page

Browse Source
This commit is contained in:
Laan Tungir
2026-07-28 16:37:08 -04:00
parent 23c37abe3a
commit a766f08738
56 changed files with 5900 additions and 679 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitmodules
-3
View File
@@ -1,6 +1,3 @@
[submodule "nostr_core_lib"]
path = nostr_core_lib
url = https://git.laantungir.net/laantungir/nostr_core_lib.git
[submodule "c_utils_lib"]
path = c_utils_lib
url = ssh://git@git.laantungir.net:2222/laantungir/c_utils_lib.git
Dockerfile.alpine-musl
+3 -2
View File
@@ -91,12 +91,13 @@ RUN cd c_utils_lib && \
# Build nostr_core_lib with required NIPs (cached unless nostr_core_lib changes)
# Disable fortification in build.sh to prevent __*_chk symbol issues
# NIPs: 001(Basic), 006(Keys), 013(PoW), 017(DMs), 019(Bech32), 044(Encryption), 059(Gift Wrap - required by NIP-17)
# NIPs: 001(Basic), 004(Encryption), 006(Keys), 013(PoW), 017(DMs), 019(Bech32), 044(Encryption), 059(Gift Wrap - required by NIP-17)
# NIP-04 is required by nostr_signer.c (NIP-46 bunker signer uses NIP-04 encryption)
RUN cd nostr_core_lib && \
chmod +x build.sh && \
sed -i 's/CFLAGS="-Wall -Wextra -std=c99 -fPIC -O2"/CFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -Wall -Wextra -std=c99 -fPIC -O2"/' build.sh && \
rm -f *.o *.a 2>/dev/null || true && \
./build.sh --nips=1,6,13,17,19,44,59
./build.sh --nips=1,4,6,13,17,19,44,59
# Copy c-relay-pg source files LAST (only this layer rebuilds on source changes)
COPY src/ /build/src/
admin/README.md
+133
View File
@@ -0,0 +1,133 @@
# C-Relay-PG PHP Admin Page
A traditional PHP admin interface for the caching service that connects
directly to PostgreSQL, bypassing the NIP-44 64KB encryption limit of the
Nostr admin API. Handles thousands of followed pubkeys via server-side
pagination.
## Quick Start
### 1. Install PHP + PHP-FPM + PostgreSQL extension
```bash
# Debian/Ubuntu
sudo apt install php-fpm php-pgsql
# RHEL/Fedora/Alma
sudo dnf install php-fpm php-pgsql
```
### 2. Copy admin files to the server
```bash
sudo mkdir -p /opt/c-relay-pg/admin
sudo cp -r admin/* /opt/c-relay-pg/admin/
```
### 3. Configure database credentials
Edit `/opt/c-relay-pg/admin/lib/config.php` and set the PostgreSQL
connection parameters to match your relay's `--db-*` flags:
```php
return [
'db_host' => 'localhost',
'db_port' => '5432',
'db_name' => 'crelay',
'db_user' => 'crelay',
'db_password' => 'crelay',
// ...
];
```
### 4. Set up HTTP Basic Auth
```bash
sudo htpasswd -c /opt/c-relay-pg/admin/.htpasswd admin
# Enter a password when prompted
```
### 5. Add nginx location block
Add this to your existing nginx server block (the one that proxies to
the relay on port 8888):
```nginx
# PHP admin page for caching service
location /admin/ {
alias /opt/c-relay-pg/admin/;
index index.php;
auth_basic "Relay Admin";
auth_basic_user_file /opt/c-relay-pg/admin/.htpasswd;
location ~ \.php$ {
fastcgi_pass unix:/run/php/php-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $request_filename;
}
# Deny access to the lib/ directory (contains DB credentials)
location ^~ /admin/lib/ {
deny all;
}
}
```
> **Note:** The `fastcgi_pass` socket path varies by distro:
> - Debian/Ubuntu: `unix:/run/php/php8.2-fpm.sock`
> - RHEL/Fedora: `unix:/run/php-fpm/www.sock`
>
> Check with: `ls /run/php/` or `ls /run/php-fpm/`
### 6. Reload nginx
```bash
sudo nginx -t && sudo systemctl reload nginx
```
### 7. Visit the admin page
```
https://relay.yourdomain.com/admin/
```
Enter the HTTP Basic Auth credentials you set in step 4.
## Pages
| Page | URL | Description |
|------|-----|-------------|
| Dashboard | `/admin/` | Service state, backfill progress, error summary, active target |
| Follows | `/admin/follows.php` | Paginated followed-pubkey table with names, event counts, relay status |
| Relays | `/admin/relays.php` | Per (author, relay) backfill progress with descriptive error statuses |
| Config | `/admin/config-edit.php` | Read/edit caching config values; bumps config generation on save |
| Inbox | `/admin/inbox.php` | Queue depth monitor for `caching_event_inbox` |
## Security
- **HTTP Basic Auth** protects all pages (nginx `auth_basic`)
- **`lib/` directory denied** via nginx (contains DB credentials)
- **PDO prepared statements** everywhere (no SQL injection)
- **HTTPS** via existing nginx SSL config
- **No CORS needed** — same origin as the relay
## How It Works
The PHP pages query PostgreSQL directly using PDO with the same
`crelay` database user the relay uses. The dashboard auto-refreshes
every 10 seconds via AJAX polling of `api/status.php`. Paginated tables
(follows, relays) use server-side pagination with `LIMIT`/`OFFSET`.
The config editor writes to the `config` table and bumps
`caching_config_generation`, which the caching service detects and
hot-reloads — no relay restart needed.
## Upgrading to Real-Time (Future)
If you later need true push updates (sub-second), you can add a small
Node.js WebSocket server alongside the PHP pages that uses PostgreSQL
`LISTEN/NOTIFY` to push updates to connected browsers. The PHP pages
and JSON endpoints are reusable — the WebSocket server would just
replace the polling `fetch()` calls in the JavaScript.
admin/api/status.php
+69
View File
@@ -0,0 +1,69 @@
<?php
/**
* C-Relay-PG Admin — JSON status endpoint for dashboard polling.
*
* Returns all dashboard stats in a single JSON object.
* Called by index.php's JavaScript every 10 seconds.
*/
require_once __DIR__ . '/../lib/helpers.php';
$pdo = db();
// Service state
$state = $pdo->query("SELECT * FROM caching_service_state WHERE id = 1")->fetch();
// Active backfill target (table may not exist yet on fresh start)
$active = ['active_pubkey' => '', 'active_relay' => ''];
try {
$active = $pdo->query("SELECT active_pubkey, active_relay FROM caching_backfill_active WHERE id = 1")->fetch() ?: $active;
} catch (PDOException $e) { /* table doesn't exist yet */ }
// Error relay summary (consecutive_errors column may not exist yet on fresh start)
$error_stats = ['auto_completed' => 0, 'error_count' => 0, 'timeout_count' => 0];
try {
$error_stats = $pdo->query("
SELECT
COUNT(*) FILTER (WHERE consecutive_errors >= 3) AS auto_completed,
COUNT(*) FILTER (WHERE last_status LIKE 'error%') AS error_count,
COUNT(*) FILTER (WHERE last_status = 'timeout') AS timeout_count
FROM caching_backfill_relay_progress
")->fetch() ?: $error_stats;
} catch (PDOException $e) { /* column/table doesn't exist yet */ }
// Inbox stats
$inbox = $pdo->query("
SELECT
COUNT(*) AS pending,
COUNT(*) FILTER (WHERE source_class = 'live') AS live,
COUNT(*) FILTER (WHERE source_class = 'backfill') AS backfill
FROM caching_event_inbox
")->fetch();
// Event count
$events = $pdo->query("SELECT COUNT(*) AS total FROM events")->fetch();
// Heartbeat age
$hb_age = intval($state['heartbeat_at'] ?? 0);
$hb_ago = $hb_age > 0 ? time() - $hb_age : 0;
json_response([
'service_state' => $state['service_state'] ?? 'unknown',
'heartbeat_ago' => $hb_ago > 0 ? $hb_ago . 's ago' : 'never',
'config_generation' => intval($state['config_generation'] ?? 0),
'followed_count' => intval($state['followed_author_count'] ?? 0),
'selected_relays' => intval($state['selected_relay_count'] ?? 0),
'connected_relays' => intval($state['connected_relay_count'] ?? 0),
'bf_complete' => intval($state['backfill_authors_complete'] ?? 0),
'bf_total' => intval($state['backfill_authors_total'] ?? 0),
'events_fetched' => intval($state['events_fetched'] ?? 0),
'inbox_inserts' => intval($state['inbox_inserts'] ?? 0),
'db_events' => intval($events['total'] ?? 0),
'inbox_pending' => intval($inbox['pending'] ?? 0),
'inbox_live' => intval($inbox['live'] ?? 0),
'inbox_backfill' => intval($inbox['backfill'] ?? 0),
'error_count' => intval($error_stats['error_count'] ?? 0),
'timeout_count' => intval($error_stats['timeout_count'] ?? 0),
'auto_completed' => intval($error_stats['auto_completed'] ?? 0),
'active_pubkey' => $active['active_pubkey'] ?? '',
'active_relay' => $active['active_relay'] ?? '',
]);
admin/assets/style.css
+419
View File
@@ -0,0 +1,419 @@
/*
* C-Relay-PG Admin — PHP caching service admin pages.
* Styled to match the original api/index.css Nostr admin interface.
* Courier New monospace, black/white/red color scheme, dark mode by default.
*/
:root {
/* Core Variables — matches api/index.css light mode (default) */
--primary-color: #000000;
--secondary-color: #ffffff;
--accent-color: #ff0000;
--muted-color: #dddddd;
--border-color: var(--muted-color);
--font-family: "Courier New", Courier, monospace;
--border-radius: 5px;
--border-width: 1px;
}
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: var(--font-family);
background-color: var(--secondary-color);
color: var(--primary-color);
padding: 0;
max-width: 1200px;
margin: 0 auto;
}
a {
color: var(--primary-color);
text-decoration: none;
}
a:hover { color: var(--accent-color); }
/* ================================
SIDE NAVIGATION (matches original)
================================ */
.side-nav {
position: fixed;
top: 0;
left: -300px;
width: 280px;
height: 100vh;
background: var(--secondary-color);
border-right: var(--border-width) solid var(--border-color);
z-index: 1000;
transition: left 0.3s ease;
overflow-y: auto;
padding-top: 80px;
}
.side-nav.open { left: 0; }
.side-nav-overlay {
position: fixed;
top: 0; left: 0; width: 100%; height: 100%;
background: rgba(0, 0, 0, 0.5);
z-index: 999;
display: none;
}
.side-nav-overlay.show { display: block; }
.nav-menu { list-style: none; padding: 0; margin: 0; }
.nav-menu li { border-bottom: var(--border-width) solid var(--muted-color); }
.nav-menu li:last-child { border-bottom: none; }
.nav-item {
display: block;
padding: 15px 20px;
color: var(--primary-color);
text-decoration: none;
font-family: var(--font-family);
font-size: 16px;
font-weight: bold;
transition: all 0.2s ease;
cursor: pointer;
border: 2px solid var(--secondary-color);
background: none;
width: 100%;
text-align: left;
}
.nav-item:hover {
background: var(--muted-color);
color: var(--accent-color);
}
.nav-item.active {
text-decoration: underline;
padding-left: 16px;
}
.nav-footer {
position: absolute;
bottom: 20px; left: 0; right: 0;
padding: 0 20px;
}
.nav-footer-btn {
display: block;
width: 100%;
padding: 12px 20px;
margin-bottom: 8px;
color: var(--primary-color);
border: 1px solid var(--border-color);
border-radius: 4px;
font-family: var(--font-family);
font-size: 14px;
font-weight: bold;
cursor: pointer;
transition: all 0.2s ease;
background: none;
}
.nav-footer-btn:hover {
background: var(--muted-color);
border-color: var(--accent-color);
}
/* ================================
HEADER (matches original)
================================ */
.main-header {
background-color: var(--secondary-color);
padding: 15px 20px;
z-index: 100;
max-width: 1200px;
margin: 0 auto;
}
.header-content {
display: flex;
justify-content: space-between;
align-items: center;
position: relative;
}
.header-title {
margin: 0;
font-size: 24px;
font-weight: bolder;
color: var(--primary-color);
cursor: pointer;
transition: all 0.2s ease;
}
.header-title:hover { opacity: 0.8; }
.header-title .relay-letter { display: inline-block; }
.menu-btn {
background: none;
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
color: var(--primary-color);
font-family: var(--font-family);
font-size: 20px;
padding: 5px 12px;
cursor: pointer;
width: auto;
margin: 0;
}
.menu-btn:hover { border-color: var(--accent-color); }
/* ================================
SECTIONS (matches original)
================================ */
.section {
background: var(--secondary-color);
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
padding: 20px;
margin-bottom: 20px;
margin-left: 5px;
margin-right: 5px;
}
.section-header {
display: flex;
justify-content: center;
align-items: center;
padding-bottom: 15px;
font-size: 16px;
font-weight: normal;
font-family: var(--font-family);
color: var(--primary-color);
}
/* ================================
TABLES (matches .config-table)
================================ */
.config-table {
border: 1px solid var(--border-color);
border-radius: var(--border-radius);
width: 100%;
border-collapse: separate;
border-spacing: 0;
margin: 10px 0;
overflow: hidden;
}
.config-table th,
.config-table td {
border: 0.1px solid var(--muted-color);
padding: 4px 8px;
text-align: left;
font-family: var(--font-family);
font-size: 10px;
}
.config-table th {
font-weight: bold;
height: 24px;
line-height: 24px;
}
.config-table tbody tr:hover {
background-color: rgba(0, 0, 0, 0.05);
}
.config-table-container {
overflow-x: auto;
max-width: 100%;
}
/* ================================
INPUTS / BUTTONS (matches original)
================================ */
input, textarea, select {
width: 100%;
padding: 8px;
background: var(--secondary-color);
color: var(--primary-color);
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
font-family: var(--font-family);
font-size: 14px;
box-sizing: border-box;
transition: all 0.2s ease;
}
input:focus, textarea:focus, select:focus {
border-color: var(--accent-color);
outline: none;
}
button {
width: 100%;
padding: 8px;
background: var(--secondary-color);
color: var(--primary-color);
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
font-family: var(--font-family);
font-size: 14px;
cursor: pointer;
margin: 5px 0;
font-weight: bold;
transition: all 0.2s ease;
}
button:hover { border-color: var(--accent-color); }
button:active {
background: var(--accent-color);
color: var(--secondary-color);
}
/* ================================
STAT CARDS (dashboard)
================================ */
.cards {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 10px;
margin-bottom: 20px;
}
.card {
background: var(--secondary-color);
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
padding: 12px;
}
.card .label {
color: var(--muted-color);
font-size: 10px;
text-transform: uppercase;
letter-spacing: 0.5px;
}
.card .value {
font-size: 20px;
font-weight: bold;
margin-top: 4px;
}
.card .sub {
color: var(--muted-color);
font-size: 10px;
margin-top: 4px;
}
/* ================================
STATUS BADGES
================================ */
.badge {
display: inline-block;
padding: 2px 6px;
border-radius: 3px;
font-size: 10px;
font-weight: bold;
border: var(--border-width) solid var(--border-color);
}
.badge-success { border-color: #4caf50; color: #4caf50; }
.badge-error { border-color: var(--accent-color); color: var(--accent-color); }
.badge-warning { border-color: #ff9800; color: #ff9800; }
.badge-muted { border-color: var(--muted-color); color: var(--muted-color); }
/* ================================
STATUS INDICATORS
================================ */
.status-working { color: var(--accent-color); }
.status-complete { color: #4caf50; }
.status-error { color: var(--accent-color); }
/* ================================
PAGINATION
================================ */
.pagination {
display: flex;
justify-content: space-between;
align-items: center;
padding: 12px 0;
gap: 16px;
}
.pagination a {
padding: 6px 14px;
background: var(--secondary-color);
border: var(--border-width) solid var(--border-color);
border-radius: var(--border-radius);
color: var(--primary-color);
font-size: 14px;
width: auto;
}
.pagination a:hover {
border-color: var(--accent-color);
color: var(--accent-color);
text-decoration: none;
}
.pagination span { color: var(--muted-color); font-size: 12px; }
/* ================================
FILTERS BAR
================================ */
.filters {
display: flex;
gap: 10px;
margin-bottom: 16px;
flex-wrap: wrap;
align-items: center;
}
.filters select, .filters input {
width: auto;
min-width: 120px;
}
.filters input[type="text"] { min-width: 250px; }
.filters button {
width: auto;
margin: 0;
min-width: 80px;
}
/* ================================
CONFIG EDITOR FORM
================================ */
.config-form .row {
display: grid;
grid-template-columns: 250px 1fr;
gap: 12px;
padding: 12px 0;
border-bottom: 0.1px solid var(--muted-color);
align-items: start;
}
.config-form .row label {
font-weight: bold;
color: var(--primary-color);
font-size: 12px;
}
.config-form .row .hint {
color: var(--muted-color);
font-size: 10px;
margin-top: 4px;
}
.config-form button {
width: auto;
margin-top: 16px;
padding: 10px 24px;
}
/* ================================
MISC
================================ */
.pubkey {
font-family: var(--font-family);
font-size: 10px;
color: var(--muted-color);
}
.npub-link {
font-family: var(--font-family);
font-size: 10px;
color: var(--primary-color);
text-decoration: none;
}
.npub-link:hover { color: var(--accent-color); }
h2 {
font-weight: normal;
text-align: center;
font-size: 16px;
font-family: var(--font-family);
color: var(--primary-color);
margin-bottom: 16px;
}
/* ================================
RESPONSIVE
================================ */
@media (max-width: 768px) {
.cards { grid-template-columns: 1fr 1fr; }
.config-form .row { grid-template-columns: 1fr; }
.filters { flex-direction: column; align-items: stretch; }
.filters select, .filters input, .filters button { width: 100%; }
}
admin/config-edit.php
+99
View File
@@ -0,0 +1,99 @@
<?php
/**
* C-Relay-PG Admin — Caching Config Editor.
* On save, bumps caching_config_generation so the caching service hot-reloads.
*/
require_once __DIR__ . '/lib/helpers.php';
$pdo = db();
$message = '';
$error = '';
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
$fields = [
'caching_root_npubs', 'caching_bootstrap_relays', 'caching_kinds',
'caching_admin_kinds', 'caching_live_enabled', 'caching_backfill_enabled',
'caching_backfill_page_size', 'caching_backfill_tick_interval_ms',
'caching_follow_graph_refresh_seconds', 'caching_relay_discovery_refresh_seconds',
'caching_max_followed_pubkeys', 'caching_max_upstream_relays',
'caching_max_relays_per_pubkey', 'caching_query_timeout_ms',
'caching_live_resubscribe_seconds',
];
try {
$pdo->beginTransaction();
foreach ($fields as $key) {
$val = $_POST[$key] ?? null;
if ($val === null) continue;
$pdo->prepare("UPDATE config SET value = ? WHERE key = ?")->execute([$val, $key]);
}
$gen = $pdo->query("SELECT value FROM config WHERE key = 'caching_config_generation'")->fetchColumn();
$new_gen = intval($gen) + 1;
$pdo->prepare("UPDATE config SET value = ? WHERE key = 'caching_config_generation'")->execute([$new_gen]);
$pdo->commit();
$message = "Configuration saved. Config generation bumped to $new_gen. The caching service will reload automatically.";
} catch (Exception $e) {
$pdo->rollBack();
$error = 'Failed to save: ' . $e->getMessage();
}
}
$config_keys = [
'caching_root_npubs' => 'Root npubs (comma-separated npub... values)',
'caching_bootstrap_relays' => 'Bootstrap relays (comma-separated wss://... URLs)',
'caching_kinds' => 'Kinds to cache (comma-separated integers)',
'caching_admin_kinds' => 'Admin kinds (* for all, or comma-separated)',
'caching_live_enabled' => 'Live subscription enabled (true/false)',
'caching_backfill_enabled' => 'Backfill enabled (true/false)',
'caching_backfill_page_size' => 'Backfill page size (events per tick)',
'caching_backfill_tick_interval_ms' => 'Backfill tick interval (ms)',
'caching_follow_graph_refresh_seconds' => 'Follow graph refresh interval (seconds)',
'caching_relay_discovery_refresh_seconds' => 'Relay discovery refresh (seconds)',
'caching_max_followed_pubkeys' => 'Max followed pubkeys',
'caching_max_upstream_relays' => 'Max upstream relays',
'caching_max_relays_per_pubkey' => 'Max relays per pubkey',
'caching_query_timeout_ms' => 'Query timeout (ms)',
'caching_live_resubscribe_seconds' => 'Live resubscribe interval (seconds)',
];
$values = [];
foreach (array_keys($config_keys) as $key) {
$stmt = $pdo->prepare("SELECT value FROM config WHERE key = ?");
$stmt->execute([$key]);
$values[$key] = $stmt->fetchColumn() ?: '';
}
admin_header('config', 'C-Relay-PG Admin — Config');
?>
<div class="section">
<div class="section-header">CACHING CONFIGURATION</div>
<?php if ($message): ?>
<p class="status-complete" style="padding:10px;border:1px solid #4caf50;border-radius:5px;margin-bottom:16px">✓ <?= e($message) ?></p>
<?php endif; ?>
<?php if ($error): ?>
<p class="status-error" style="padding:10px;border:1px solid var(--accent-color);border-radius:5px;margin-bottom:16px">✗ <?= e($error) ?></p>
<?php endif; ?>
<form class="config-form" method="post">
<?php foreach ($config_keys as $key => $label): ?>
<div class="row">
<div>
<label for="<?= e($key) ?>"><?= e($label) ?></label>
<div class="hint"><?= e($key) ?></div>
</div>
<div>
<input type="text" id="<?= e($key) ?>" name="<?= e($key) ?>" value="<?= e($values[$key]) ?>">
</div>
</div>
<?php endforeach; ?>
<button type="submit">Save & Apply (bumps config generation)</button>
</form>
<p style="margin-top:16px;color:var(--muted-color);font-size:10px">
Saving bumps <code>caching_config_generation</code> so the caching service
detects the change and hot-reloads. No relay restart needed.
</p>
</div>
<?php admin_footer(); ?>
admin/follows.php
+153
View File
@@ -0,0 +1,153 @@
<?php
/**
* C-Relay-PG Admin — Followed Pubkeys (paginated).
* Replaces the NIP-44-encrypted caching_follows_status admin command.
*/
require_once __DIR__ . '/lib/helpers.php';
$pdo = db();
$page = current_page();
$per = cfg('follows_per_page', 50);
$offset = ($page - 1) * $per;
$filter = query_param('filter', 'all');
$search = query_param('search', '');
$sort = query_param('sort', 'events');
$where = [];
$params = [];
if ($filter === 'incomplete') {
$where[] = 'fp.backfill_complete = false';
} elseif ($filter === 'root') {
$where[] = 'fp.is_root = true';
} elseif ($filter === 'error') {
$where[] = "EXISTS (SELECT 1 FROM caching_backfill_relay_progress rp WHERE rp.author_pubkey = fp.pubkey AND rp.last_status LIKE 'error%')";
}
if ($search) {
$where[] = '(fp.pubkey ILIKE ? OR e.content::json->>\'name\' ILIKE ? OR e.content::json->>\'display_name\' ILIKE ?)';
$params[] = "%$search%"; $params[] = "%$search%"; $params[] = "%$search%";
}
$where_sql = $where ? 'WHERE ' . implode(' AND ', $where) : '';
$sort_map = [
'events' => 'fp.events_fetched DESC',
'name' => "COALESCE(e.content::json->>'display_name', e.content::json->>'name') ASC NULLS LAST",
'recent' => 'fp.last_seen DESC',
'complete' => 'fp.backfill_complete ASC, fp.events_fetched DESC',
];
$sort_sql = $sort_map[$sort] ?? $sort_map['events'];
$stmt = $pdo->prepare("SELECT COUNT(*) FROM caching_followed_pubkeys fp LEFT JOIN LATERAL (SELECT content FROM events WHERE pubkey = fp.pubkey AND kind = 0 ORDER BY created_at DESC LIMIT 1) e ON true $where_sql");
$stmt->execute($params);
$total = intval($stmt->fetchColumn());
$sql = "
SELECT fp.pubkey, fp.is_root, fp.backfill_complete, fp.events_fetched,
fp.first_seen, fp.last_seen,
e.content::json->>'name' AS name,
e.content::json->>'display_name' AS display_name,
e.content::json->>'picture' AS picture,
e.content::json->>'nip05' AS nip05,
(SELECT COUNT(*) FROM events WHERE pubkey = fp.pubkey) AS total_events
FROM caching_followed_pubkeys fp
LEFT JOIN LATERAL (SELECT content FROM events WHERE pubkey = fp.pubkey AND kind = 0 ORDER BY created_at DESC LIMIT 1) e ON true
$where_sql
ORDER BY $sort_sql
LIMIT $per OFFSET $offset
";
$stmt = $pdo->prepare($sql);
$stmt->execute($params);
$follows = $stmt->fetchAll();
admin_header('follows', 'C-Relay-PG Admin — Follows');
?>
<div class="section">
<div class="section-header">FOLLOWED PUBKEYS (<?= number_format($total) ?> total)</div>
<form class="filters" method="get">
<select name="filter" onchange="this.form.submit()">
<option value="all" <?= $filter==='all'?'selected':'' ?>>All</option>
<option value="incomplete" <?= $filter==='incomplete'?'selected':'' ?>>Incomplete only</option>
<option value="root" <?= $filter==='root'?'selected':'' ?>>Root only</option>
<option value="error" <?= $filter==='error'?'selected':'' ?>>Has error relays</option>
</select>
<select name="sort" onchange="this.form.submit()">
<option value="events" <?= $sort==='events'?'selected':'' ?>>Sort: Events fetched</option>
<option value="name" <?= $sort==='name'?'selected':'' ?>>Sort: Name</option>
<option value="recent" <?= $sort==='recent'?'selected':'' ?>>Sort: Recently seen</option>
<option value="complete" <?= $sort==='complete'?'selected':'' ?>>Sort: Backfill status</option>
</select>
<input type="text" name="search" value="<?= e($search) ?>" placeholder="Search name or pubkey…">
<button type="submit">Search</button>
<?php if ($filter !== 'all' || $search): ?>
<a href="follows.php" style="font-size:12px">Clear</a>
<?php endif; ?>
</form>
<div class="config-table-container">
<table class="config-table">
<thead>
<tr>
<th>Name</th>
<th>npub</th>
<th>Root?</th>
<th>Events in DB</th>
<th>Backfill</th>
<th>Last Seen</th>
<th>Relays</th>
</tr>
</thead>
<tbody>
<?php foreach ($follows as $f):
$npub = hex_to_npub($f['pubkey']);
$name = $f['display_name'] ?: $f['name'];
$rc = ['total' => 0, 'incomplete' => 0, 'errors' => 0];
try {
$relay_count = $pdo->prepare("SELECT COUNT(*) AS total, COUNT(*) FILTER (WHERE complete = false) AS incomplete, COUNT(*) FILTER (WHERE last_status LIKE 'error%') AS errors FROM caching_backfill_relay_progress WHERE author_pubkey = ?");
$relay_count->execute([$f['pubkey']]);
$rc = $relay_count->fetch() ?: $rc;
} catch (PDOException $ex) {}
?>
<tr>
<td>
<?php if ($name): ?>
<strong><?= e($name) ?></strong>
<?php if ($f['nip05']): ?>
<br><span style="color:var(--muted-color)">✓ <?= e($f['nip05']) ?></span>
<?php endif; ?>
<?php else: ?>
<span style="color:var(--muted-color);font-style:italic">unknown</span>
<?php endif; ?>
</td>
<td><a href="https://njump.me/<?= e($npub) ?>" target="_blank" class="npub-link"><?= e(trunc($npub, 22)) ?></a></td>
<td><?= $f['is_root'] ? '<span class="badge badge-warning">root</span>' : '' ?></td>
<td><?= number_format(intval($f['total_events'])) ?></td>
<td><?= $f['backfill_complete']
? '<span class="badge badge-success">✓ complete</span>'
: '<span class="badge badge-warning">in progress</span>' ?></td>
<td><?= time_ago(intval($f['last_seen'])) ?></td>
<td>
<?php if (intval($rc['total']) > 0): ?>
<?= intval($rc['total']) ?> relays (<?= intval($rc['incomplete']) ?> incomplete
<?php if (intval($rc['errors']) > 0): ?>
, <span class="status-error"><?= intval($rc['errors']) ?> errors</span>
<?php endif; ?>)
<br><a href="relays.php?pubkey=<?= e($f['pubkey']) ?>" style="font-size:10px">View relay details →</a>
<?php else: ?>
<span style="color:var(--muted-color)">—</span>
<?php endif; ?>
</td>
</tr>
<?php endforeach; ?>
<?php if (empty($follows)): ?>
<tr><td colspan="7" style="text-align:center;color:var(--muted-color);padding:20px">No followed pubkeys found.</td></tr>
<?php endif; ?>
</tbody>
</table>
</div>
<?= pagination($page, $per, $total, 'follows.php' . ($filter !== 'all' ? '?filter=' . e($filter) : '') . ($search ? '&search=' . e($search) : '') . ($sort !== 'events' ? '&sort=' . e($sort) : '')) ?>
</div>
<?php admin_footer(); ?>
admin/inbox.php
+111
View File
@@ -0,0 +1,111 @@
<?php
/**
* C-Relay-PG Admin — Inbox Queue Monitor.
*/
require_once __DIR__ . '/lib/helpers.php';
$pdo = db();
$stats = $pdo->query("
SELECT
COUNT(*) AS pending,
COUNT(*) FILTER (WHERE source_class = 'live') AS live,
COUNT(*) FILTER (WHERE source_class = 'backfill') AS backfill,
COUNT(*) FILTER (WHERE source_class = 'discovery') AS discovery,
COUNT(*) FILTER (WHERE priority = 0) AS high_priority,
COALESCE(EXTRACT(EPOCH FROM NOW())::BIGINT - MIN(received_at), 0) AS oldest_age
FROM caching_event_inbox
")->fetch() ?: ['pending' => 0, 'live' => 0, 'backfill' => 0, 'discovery' => 0, 'high_priority' => 0, 'oldest_age' => 0];
$recent = $pdo->query("
SELECT event_id, event_json->>'pubkey' AS pubkey, event_json->>'kind' AS kind,
source_relay, source_class, priority, received_at
FROM caching_event_inbox
ORDER BY received_at DESC
LIMIT 20
")->fetchAll();
admin_header('inbox', 'C-Relay-PG Admin — Inbox');
?>
<div class="section">
<div class="section-header">INBOX QUEUE MONITOR</div>
<div class="cards">
<div class="card">
<div class="label">Pending Events</div>
<div class="value" id="inbox-pending"><?= number_format(intval($stats['pending'])) ?></div>
<div class="sub" id="inbox-detail"><?= intval($stats['live']) ?> live, <?= intval($stats['backfill']) ?> backfill, <?= intval($stats['discovery']) ?> discovery</div>
</div>
<div class="card">
<div class="label">High Priority</div>
<div class="value"><?= number_format(intval($stats['high_priority'])) ?></div>
<div class="sub">priority = 0 (live events)</div>
</div>
<div class="card">
<div class="label">Oldest Pending</div>
<div class="value" id="inbox-oldest"><?= intval($stats['oldest_age']) ?>s</div>
<div class="sub">age of oldest event in queue</div>
</div>
</div>
<?php if (intval($stats['oldest_age']) > 300): ?>
<p class="status-error" style="padding:10px;border:1px solid var(--accent-color);border-radius:5px;margin-bottom:16px">
⚠ Oldest pending event is <?= intval($stats['oldest_age']) ?>s old — the inbox poller may not be keeping up.
Check that <code>caching_inbox_enabled = true</code> in the config table and the relay is running.
</p>
<?php endif; ?>
</div>
<div class="section">
<div class="section-header">RECENT EVENTS IN QUEUE (last 20)</div>
<div class="config-table-container">
<table class="config-table">
<thead>
<tr>
<th>Event ID</th>
<th>Pubkey</th>
<th>Kind</th>
<th>Source Relay</th>
<th>Class</th>
<th>Priority</th>
<th>Received</th>
</tr>
</thead>
<tbody>
<?php foreach ($recent as $r): ?>
<tr>
<td class="pubkey"><?= e(trunc($r['event_id'], 16)) ?></td>
<td class="pubkey"><?= e(trunc($r['pubkey'], 16)) ?></td>
<td><?= e($r['kind']) ?></td>
<td><?= e($r['source_relay'] ?? '—') ?></td>
<td><span class="badge badge-muted"><?= e($r['source_class']) ?></span></td>
<td><?= intval($r['priority']) ?></td>
<td><?= time_ago(intval($r['received_at'])) ?></td>
</tr>
<?php endforeach; ?>
<?php if (empty($recent)): ?>
<tr><td colspan="7" style="text-align:center;color:var(--muted-color);padding:20px">Inbox is empty — all events have been dequeued.</td></tr>
<?php endif; ?>
</tbody>
</table>
</div>
</div>
<script>
async function refreshInbox() {
try {
const res = await fetch('api/status.php');
if (!res.ok) return;
const d = await res.json();
console.log('[inbox] refreshed at', new Date().toLocaleTimeString(), '— pending:', d.inbox_pending);
const p = document.getElementById('inbox-pending');
const det = document.getElementById('inbox-detail');
if (p) p.textContent = (d.inbox_pending ?? 0).toLocaleString();
if (det) det.textContent = (d.inbox_live ?? 0) + ' live, ' + (d.inbox_backfill ?? 0) + ' backfill';
} catch (e) {}
}
setInterval(refreshInbox, <?= cfg('refresh_ms', 10000) ?>);
</script>
<?php admin_footer(); ?>
admin/index.php
+158
View File
@@ -0,0 +1,158 @@
<?php
/**
* C-Relay-PG Admin — Dashboard.
* Auto-refreshes every 10 seconds via AJAX polling of api/status.php.
*/
require_once __DIR__ . '/lib/helpers.php';
$pdo = db();
// Service state
$state = $pdo->query("SELECT * FROM caching_service_state WHERE id = 1")->fetch() ?: [];
// Active backfill target (table may not exist yet on fresh start)
$active = ['active_pubkey' => '', 'active_relay' => ''];
try {
$active = $pdo->query("SELECT active_pubkey, active_relay FROM caching_backfill_active WHERE id = 1")->fetch() ?: $active;
} catch (PDOException $e) {}
// Error relay summary
$error_stats = ['auto_completed' => 0, 'error_count' => 0, 'timeout_count' => 0, 'incomplete' => 0, 'total' => 0];
try {
$error_stats = $pdo->query("
SELECT
COUNT(*) FILTER (WHERE consecutive_errors >= 3) AS auto_completed,
COUNT(*) FILTER (WHERE last_status LIKE 'error%') AS error_count,
COUNT(*) FILTER (WHERE last_status = 'timeout') AS timeout_count,
COUNT(*) FILTER (WHERE complete = false) AS incomplete,
COUNT(*) AS total
FROM caching_backfill_relay_progress
")->fetch() ?: $error_stats;
} catch (PDOException $e) {}
// Inbox stats
$inbox = $pdo->query("
SELECT
COUNT(*) AS pending,
COUNT(*) FILTER (WHERE source_class = 'live') AS live,
COUNT(*) FILTER (WHERE source_class = 'backfill') AS backfill,
COALESCE(EXTRACT(EPOCH FROM NOW())::BIGINT - MIN(received_at), 0) AS oldest_age
FROM caching_event_inbox
")->fetch() ?: ['pending' => 0, 'live' => 0, 'backfill' => 0, 'oldest_age' => 0];
// Event counts
$event_stats = $pdo->query("SELECT COUNT(*) AS total, COUNT(DISTINCT pubkey) AS authors FROM events")->fetch() ?: ['total' => 0, 'authors' => 0];
$kind_stats = $pdo->query("SELECT kind, COUNT(*) AS cnt FROM events GROUP BY kind ORDER BY cnt DESC LIMIT 10")->fetchAll();
admin_header('dashboard', 'C-Relay-PG Admin — Dashboard');
?>
<div class="section">
<div class="section-header">CACHING SERVICE STATUS</div>
<div class="cards" id="dashboard-cards">
<div class="card">
<div class="label">Service State</div>
<div class="value" id="card-state"><?= e($state['service_state'] ?? 'unknown') ?></div>
<div class="sub" id="card-heartbeat">Heartbeat: <?= time_ago(intval($state['heartbeat_at'] ?? 0)) ?></div>
</div>
<div class="card">
<div class="label">Followed Authors</div>
<div class="value" id="card-follows"><?= intval($state['followed_author_count'] ?? 0) ?></div>
<div class="sub" id="card-relays"><?= intval($state['selected_relay_count'] ?? 0) ?> relays, <?= intval($state['connected_relay_count'] ?? 0) ?> connected</div>
</div>
<div class="card">
<div class="label">Backfill Progress</div>
<div class="value" id="card-backfill"><?= intval($state['backfill_authors_complete'] ?? 0) ?> / <?= intval($state['backfill_authors_total'] ?? 0) ?></div>
<div class="sub" id="card-backfill-pct"><?= ($state['backfill_authors_total'] ?? 0) > 0 ? round(intval($state['backfill_authors_complete'] ?? 0) / intval($state['backfill_authors_total']) * 100) : 0 ?>% complete</div>
</div>
<div class="card">
<div class="label">Events Fetched</div>
<div class="value" id="card-events"><?= number_format(intval($state['events_fetched'] ?? 0)) ?></div>
<div class="sub" id="card-inserts"><?= number_format(intval($state['inbox_inserts'] ?? 0)) ?> inbox inserts</div>
</div>
<div class="card">
<div class="label">Events in DB</div>
<div class="value" id="card-db-events"><?= number_format(intval($event_stats['total'] ?? 0)) ?></div>
<div class="sub"><?= number_format(intval($event_stats['authors'] ?? 0)) ?> authors</div>
</div>
<div class="card">
<div class="label">Inbox Pending</div>
<div class="value" id="card-inbox"><?= number_format(intval($inbox['pending'] ?? 0)) ?></div>
<div class="sub" id="card-inbox-detail"><?= intval($inbox['live'] ?? 0) ?> live, <?= intval($inbox['backfill'] ?? 0) ?> backfill</div>
</div>
<div class="card">
<div class="label">Error Relays</div>
<div class="value status-error" id="card-errors"><?= intval($error_stats['error_count'] ?? 0) ?></div>
<div class="sub" id="card-errors-detail"><?= intval($error_stats['auto_completed'] ?? 0) ?> auto-completed</div>
</div>
<div class="card">
<div class="label">Config Generation</div>
<div class="value" id="card-gen"><?= intval($state['config_generation'] ?? 0) ?></div>
<div class="sub">Updated <?= time_ago(intval($state['updated_at'] ?? 0)) ?></div>
</div>
</div>
</div>
<div class="section">
<div class="section-header">ACTIVE BACKFILL TARGET</div>
<div id="active-target" style="text-align:center;padding:10px">
<?php if (!empty($active['active_pubkey'])): ?>
<p class="status-working">⚡ Working on: <span class="pubkey"><?= e(trunc($active['active_pubkey'], 16)) ?></span> @ <?= e($active['active_relay'] ?? '') ?></p>
<?php else: ?>
<p class="status-complete">✓ Caching complete — no active backfill target</p>
<?php endif; ?>
</div>
</div>
<div class="section">
<div class="section-header">TOP EVENT KINDS</div>
<div class="config-table-container">
<table class="config-table">
<thead><tr><th>Kind</th><th>Count</th></tr></thead>
<tbody id="kind-table">
<?php foreach ($kind_stats as $ks): ?>
<tr><td><?= intval($ks['kind']) ?></td><td><?= number_format(intval($ks['cnt'])) ?></td></tr>
<?php endforeach; ?>
</tbody>
</table>
</div>
</div>
<script>
const REFRESH_MS = <?= cfg('refresh_ms', 10000) ?>;
async function refreshDashboard() {
console.log('[dashboard] polling api/status.php at', new Date().toLocaleTimeString());
try {
const res = await fetch('api/status.php');
if (!res.ok) { console.warn('[dashboard] status.php returned', res.status); return; }
const d = await res.json();
console.log('[dashboard] refreshed at', new Date().toLocaleTimeString(), '— state:', d.service_state, 'follows:', d.followed_count, 'events:', d.db_events);
const set = (id, val) => { const el = document.getElementById(id); if (el) el.textContent = val; };
set('card-state', d.service_state || 'unknown');
set('card-heartbeat', 'Heartbeat: ' + (d.heartbeat_ago || 'never'));
set('card-follows', d.followed_count ?? 0);
set('card-relays', (d.selected_relays ?? 0) + ' relays, ' + (d.connected_relays ?? 0) + ' connected');
set('card-backfill', (d.bf_complete ?? 0) + ' / ' + (d.bf_total ?? 0));
set('card-backfill-pct', (d.bf_total > 0 ? Math.round(d.bf_complete / d.bf_total * 100) : 0) + '% complete');
set('card-events', (d.events_fetched ?? 0).toLocaleString());
set('card-inserts', (d.inbox_inserts ?? 0).toLocaleString() + ' inbox inserts');
set('card-db-events', (d.db_events ?? 0).toLocaleString());
set('card-inbox', (d.inbox_pending ?? 0).toLocaleString());
set('card-inbox-detail', (d.inbox_live ?? 0) + ' live, ' + (d.inbox_backfill ?? 0) + ' backfill');
set('card-errors', d.error_count ?? 0);
set('card-errors-detail', (d.auto_completed ?? 0) + ' auto-completed');
set('card-gen', d.config_generation ?? 0);
const atEl = document.getElementById('active-target');
if (d.active_pubkey && d.active_relay) {
atEl.innerHTML = '<p class="status-working">⚡ Working on: <span class="pubkey">' + d.active_pubkey.substring(0,16) + '…</span> @ ' + d.active_relay + '</p>';
} else if (d.active_pubkey) {
atEl.innerHTML = '<p class="status-working">⚡ Working on: <span class="pubkey">' + d.active_pubkey.substring(0,16) + '…</span></p>';
} else {
atEl.innerHTML = '<p class="status-complete">✓ Caching complete — no active backfill target</p>';
}
} catch (e) { console.error('[dashboard] refresh error:', e); }
}
setInterval(refreshDashboard, REFRESH_MS);
console.log('[dashboard] auto-refresh started, interval:', REFRESH_MS, 'ms');
</script>
<?php admin_footer(); ?>
admin/lib/config.php
+27
View File
@@ -0,0 +1,27 @@
<?php
/**
* C-Relay-PG Admin — Database connection configuration.
*
* This file is NOT web-accessible (denied by nginx location block).
* It contains the PostgreSQL connection string used by all admin pages.
*
* Edit these values to match your relay's PostgreSQL credentials.
*/
// PostgreSQL connection parameters.
// These should match the --db-* flags passed to the c-relay-pg binary
// or the connection string passed to caching_relay -p "...".
return [
'db_host' => 'localhost',
'db_port' => '5432',
'db_name' => 'crelay',
'db_user' => 'crelay',
'db_password' => 'crelay',
// Page sizes for paginated tables.
'follows_per_page' => 50,
'relays_per_page' => 50,
// Auto-refresh interval for dashboard polling (milliseconds).
'refresh_ms' => 10000,
];
admin/lib/db.php
+44
View File
@@ -0,0 +1,44 @@
<?php
/**
* C-Relay-PG Admin — PDO database connection helper.
*
* Provides a singleton PDO connection to the relay's PostgreSQL database.
* All admin pages use db() to get the connection.
*/
require_once __DIR__ . '/config.php';
function db(): PDO {
static $pdo = null;
if ($pdo === null) {
$cfg = require __DIR__ . '/config.php';
$dsn = sprintf(
'pgsql:host=%s;port=%d;dbname=%s;user=%s;password=%s',
$cfg['db_host'],
$cfg['db_port'],
$cfg['db_name'],
$cfg['db_user'],
$cfg['db_password']
);
try {
$pdo = new PDO($dsn, null, null, [
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
PDO::ATTR_EMULATE_PREPARES => false,
]);
} catch (PDOException $e) {
http_response_code(500);
die('Database connection failed: ' . htmlspecialchars($e->getMessage()));
}
}
return $pdo;
}
/** Get a config value from the admin config file. */
function cfg(string $key, $default = null) {
static $cfg = null;
if ($cfg === null) {
$cfg = require __DIR__ . '/config.php';
}
return $cfg[$key] ?? $default;
}
admin/lib/helpers.php
+191
View File
@@ -0,0 +1,191 @@
<?php
/**
* C-Relay-PG Admin — Shared helper functions.
*/
require_once __DIR__ . '/db.php';
/** HTML-escape a string. */
function e(?string $s): string {
return htmlspecialchars($s ?? '', ENT_QUOTES, 'UTF-8');
}
/** Format a Unix timestamp as a human-readable relative time. */
function time_ago(int $ts): string {
if ($ts <= 0) return 'never';
$diff = time() - $ts;
if ($diff < 60) return $diff . 's ago';
if ($diff < 3600) return floor($diff / 60) . 'm ago';
if ($diff < 86400) return floor($diff / 3600) . 'h ago';
return floor($diff / 86400) . 'd ago';
}
/** Format a Unix timestamp as a date/time string. */
function fmt_date(int $ts): string {
if ($ts <= 0) return '-';
return date('Y-m-d H:i:s', $ts);
}
/**
* Convert a 64-char hex pubkey to npub (bech32).
* Uses the standard bech32 encoding (NIP-19).
*/
function hex_to_npub(string $hex): string {
if (!preg_match('/^[0-9a-fA-F]{64}$/', $hex)) return $hex;
$hex = strtolower($hex);
// Bech32 constants
$charset = 'qpzry9x8gf2tvdw0s3jn54khce6mua7l';
$data = [];
for ($i = 0; $i < strlen($hex); $i += 2) {
$data[] = intval(substr($hex, $i, 2), 16);
}
// Convert 8-bit groups to 5-bit groups
$conv = [];
$buffer = 0;
$bits = 0;
foreach ($data as $byte) {
$buffer = ($buffer << 8) | $byte;
$bits += 8;
while ($bits >= 5) {
$conv[] = ($buffer >> ($bits - 5)) & 31;
$bits -= 5;
}
}
if ($bits > 0) {
$conv[] = ($buffer << (5 - $bits)) & 31;
}
// Compute checksum — HRP values are positions of each char in the charset.
$values = array_merge(
array_map(function($c) {
return strpos('qpzry9x8gf2tvdw0s3jn54khce6mua7l', $c);
}, str_split('npub')),
$conv
);
$gen = [0x3b6a57b2, 0x26508e6d, 0x1ea119fa, 0x3d4233dd, 0x2a1462b3];
$chk = 1;
foreach ($values as $v) {
$top = $chk >> 25;
$chk = (($chk & 0x1ffffff) << 5) ^ $v;
for ($i = 0; $i < 5; $i++) {
if (($top >> $i) & 1) $chk ^= $gen[$i];
}
}
$chk ^= 1;
$checksum = [];
for ($i = 0; $i < 6; $i++) {
$checksum[] = ($chk >> (5 * (5 - $i))) & 31;
}
$result = 'npub1';
foreach (array_merge($conv, $checksum) as $v) {
$result .= $charset[$v];
}
return $result;
}
/** Truncate a string to $len chars with ellipsis. */
function trunc(string $s, int $len = 16): string {
if (strlen($s) <= $len) return $s;
return substr($s, 0, $len) . '…';
}
/** Build pagination links HTML. */
function pagination(int $page, int $per_page, int $total, string $base_url): string {
$total_pages = max(1, ceil($total / $per_page));
if ($total_pages <= 1) return '';
$html = '<div class="pagination">';
if ($page > 1) {
$html .= '<a href="' . e($base_url) . '?page=' . ($page - 1) . '">← Prev</a>';
}
$html .= '<span>Page ' . $page . ' of ' . $total_pages . ' (' . $total . ' total)</span>';
if ($page < $total_pages) {
$html .= '<a href="' . e($base_url) . '?page=' . ($page + 1) . '">Next →</a>';
}
$html .= '</div>';
return $html;
}
/** Get the current page number from the query string. */
function current_page(): int {
$p = intval($_GET['page'] ?? 1);
return max(1, $p);
}
/** JSON response helper for AJAX endpoints. */
function json_response($data): void {
header('Content-Type: application/json');
echo json_encode($data, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);
exit;
}
/** Get a query parameter with a default. */
function query_param(string $key, $default = null) {
return $_GET[$key] ?? $default;
}
/**
* Render the shared HTML header with side navigation.
* Pass the active page name to highlight the current nav item.
*/
function admin_header(string $active = '', string $title = 'C-Relay-PG Admin'): void {
$pages = [
'dashboard' => ['index.php', 'Dashboard'],
'follows' => ['follows.php', 'Follows'],
'relays' => ['relays.php', 'Relays'],
'config' => ['config-edit.php', 'Config'],
'inbox' => ['inbox.php', 'Inbox'],
];
?>
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title><?= e($title) ?></title>
<link rel="stylesheet" href="assets/style.css">
</head>
<body>
<!-- Side Navigation -->
<nav class="side-nav" id="side-nav">
<ul class="nav-menu">
<?php foreach ($pages as $key => [$url, $label]): ?>
<li><a class="nav-item<?= $active === $key ? ' active' : '' ?>" href="<?= e($url) ?>"><?= e($label) ?></a></li>
<?php endforeach; ?>
</ul>
</nav>
<div class="side-nav-overlay" id="side-nav-overlay" onclick="closeNav()"></div>
<!-- Header -->
<div class="main-header">
<div class="header-content">
<button class="menu-btn" onclick="openNav()">☰</button>
<div class="header-title" onclick="location.href='index.php'">
<span class="relay-letter">R</span><span class="relay-letter">E</span><span class="relay-letter">L</span><span class="relay-letter">A</span><span class="relay-letter">Y</span>
</div>
<div style="width:40px"></div>
</div>
</div>
<script>
function openNav() {
document.getElementById('side-nav').classList.add('open');
document.getElementById('side-nav-overlay').classList.add('show');
}
function closeNav() {
document.getElementById('side-nav').classList.remove('open');
document.getElementById('side-nav-overlay').classList.remove('show');
}
</script>
<?php
}
/** Render the shared HTML footer. */
function admin_footer(): void {
?>
</body>
</html>
<?php
}
admin/relays.php
+132
View File
@@ -0,0 +1,132 @@
<?php
/**
* C-Relay-PG Admin — Per-Relay Backfill Progress.
*/
require_once __DIR__ . '/lib/helpers.php';
$pdo = db();
$page = current_page();
$per = cfg('relays_per_page', 50);
$offset = ($page - 1) * $per;
$filter = query_param('filter', 'all');
$pubkey = query_param('pubkey', '');
$where = [];
$params = [];
if ($filter === 'incomplete') {
$where[] = 'rp.complete = false';
} elseif ($filter === 'error') {
$where[] = "rp.last_status LIKE 'error%'";
} elseif ($filter === 'complete') {
$where[] = 'rp.complete = true';
}
if ($pubkey) {
$where[] = 'rp.author_pubkey = ?';
$params[] = $pubkey;
}
$where_sql = $where ? 'WHERE ' . implode(' AND ', $where) : '';
$total = 0;
$relays = [];
try {
$stmt = $pdo->prepare("SELECT COUNT(*) FROM caching_backfill_relay_progress rp $where_sql");
$stmt->execute($params);
$total = intval($stmt->fetchColumn());
$sql = "
SELECT rp.author_pubkey, rp.relay_url, rp.until_cursor, rp.complete,
rp.events_fetched,
COALESCE(rp.consecutive_errors, 0) AS consecutive_errors,
rp.last_status, rp.updated_at,
e.content::json->>'name' AS name,
e.content::json->>'display_name' AS display_name
FROM caching_backfill_relay_progress rp
LEFT JOIN LATERAL (SELECT content FROM events WHERE pubkey = rp.author_pubkey AND kind = 0 ORDER BY created_at DESC LIMIT 1) e ON true
$where_sql
ORDER BY rp.complete ASC, COALESCE(rp.consecutive_errors, 0) DESC, rp.updated_at DESC
LIMIT $per OFFSET $offset
";
$stmt = $pdo->prepare($sql);
$stmt->execute($params);
$relays = $stmt->fetchAll();
} catch (PDOException $ex) {}
function status_badge(string $status): string {
if ($status === 'eose') return '<span class="badge badge-success">eose</span>';
if ($status === 'timeout') return '<span class="badge badge-warning">timeout</span>';
if (str_starts_with($status, 'error')) return '<span class="badge badge-error">' . e($status) . '</span>';
if (str_starts_with($status, 'NOTICE')) return '<span class="badge badge-warning">' . e($status) . '</span>';
if (str_starts_with($status, 'CLOSED')) return '<span class="badge badge-warning">' . e($status) . '</span>';
if ($status === '') return '<span class="badge badge-muted">—</span>';
return '<span class="badge badge-muted">' . e($status) . '</span>';
}
admin_header('relays', 'C-Relay-PG Admin — Relay Progress');
?>
<div class="section">
<div class="section-header">PER-RELAY BACKFILL PROGRESS (<?= number_format($total) ?> total)</div>
<form class="filters" method="get">
<select name="filter" onchange="this.form.submit()">
<option value="all" <?= $filter==='all'?'selected':'' ?>>All</option>
<option value="incomplete" <?= $filter==='incomplete'?'selected':'' ?>>Incomplete only</option>
<option value="error" <?= $filter==='error'?'selected':'' ?>>Errors only</option>
<option value="complete" <?= $filter==='complete'?'selected':'' ?>>Complete only</option>
</select>
<?php if ($pubkey): ?>
<input type="hidden" name="pubkey" value="<?= e($pubkey) ?>">
<span style="font-size:10px;color:var(--muted-color)">Filtered by: <span class="pubkey"><?= e(trunc($pubkey, 16)) ?></span></span>
<a href="relays.php?filter=<?= e($filter) ?>" style="font-size:10px">Clear</a>
<?php endif; ?>
</form>
<div class="config-table-container">
<table class="config-table">
<thead>
<tr>
<th>Author</th>
<th>Relay URL</th>
<th>Complete</th>
<th>Events</th>
<th>Errors</th>
<th>Last Status</th>
<th>Updated</th>
</tr>
</thead>
<tbody>
<?php foreach ($relays as $r):
$name = $r['display_name'] ?: $r['name'];
$npub = hex_to_npub($r['author_pubkey']);
?>
<tr <?= intval($r['consecutive_errors']) >= 3 ? 'style="background:rgba(255,0,0,0.05)"' : '' ?>>
<td>
<?php if ($name): ?>
<strong><?= e($name) ?></strong>
<?php else: ?>
<span style="color:var(--muted-color);font-style:italic">unknown</span>
<?php endif; ?>
<br><a href="follows.php?search=<?= e($r['author_pubkey']) ?>" class="npub-link"><?= e(trunc($npub, 20)) ?></a>
</td>
<td><?= e($r['relay_url']) ?></td>
<td><?= $r['complete'] ? '<span class="badge badge-success">✓</span>' : '<span class="badge badge-warning">…</span>' ?></td>
<td><?= number_format(intval($r['events_fetched'])) ?></td>
<td><?= intval($r['consecutive_errors']) > 0
? '<span class="status-error">' . intval($r['consecutive_errors']) . '</span>'
: '0' ?></td>
<td><?= status_badge($r['last_status']) ?></td>
<td><?= time_ago(intval($r['updated_at'])) ?></td>
</tr>
<?php endforeach; ?>
<?php if (empty($relays)): ?>
<tr><td colspan="7" style="text-align:center;color:var(--muted-color);padding:20px">No relay progress rows found.</td></tr>
<?php endif; ?>
</tbody>
</table>
</div>
<?= pagination($page, $per, $total, 'relays.php' . ($filter !== 'all' ? '?filter=' . e($filter) : '') . ($pubkey ? '&pubkey=' . e($pubkey) : '')) ?>
</div>
<?php admin_footer(); ?>
api/index.html
+24 -2
View File
@@ -204,6 +204,7 @@
<thead>
<tr>
<th>Rank</th>
<th>Name</th>
<th>Pubkey</th>
<th>Event Count</th>
<th>Percentage</th>
@@ -211,7 +212,7 @@
</thead>
<tbody id="stats-pubkeys-table-body">
<tr>
<td colspan="4" style="text-align: center; font-style: italic;">No data loaded</td>
<td colspan="5" style="text-align: center; font-style: italic;">No data loaded</td>
</tr>
</tbody>
</table>
@@ -676,7 +677,7 @@ wss://laantungir.net/relay</textarea>
<div class="form-group">
<label for="caching-inbox-batch-size">Inbox Batch Size:</label>
<input type="number" id="caching-inbox-batch-size" placeholder="25">
<input type="number" id="caching-inbox-batch-size" placeholder="150">
</div>
<div class="form-group">
@@ -715,6 +716,27 @@ wss://laantungir.net/relay</textarea>
</div>
<div id="caching-service-control-status" class="status-message"></div>
</div>
<div class="input-group">
<h3>Followed Pubkeys</h3>
<p>Pubkeys being cached with their profile names, event counts, outbox relays, and backfill status.</p>
<div id="caching-follows-status" class="status-message"></div>
<table id="caching-follows-table" style="width: 100%; border-collapse: collapse; font-size: 13px;">
<thead>
<tr style="border-bottom: 2px solid var(--border-color, #444); text-align: left;">
<th style="padding: 6px 8px;">Name</th>
<th style="padding: 6px 8px;">npub</th>
<th style="padding: 6px 8px;">Root?</th>
<th style="padding: 6px 8px;">Events in DB</th>
<th style="padding: 6px 8px;">Backfill</th>
<th style="padding: 6px 8px; width: 40%;">Relays (status, events fetched)</th>
</tr>
</thead>
<tbody id="caching-follows-table-body">
<tr><td colspan="6" style="text-align: center; font-style: italic; padding: 12px;">Loading follows status...</td></tr>
</tbody>
</table>
</div>
</div>
<!-- SQL QUERY Section -->
api/index.js
+239 -8
View File
@@ -2092,6 +2092,11 @@ function handleSystemCommandResponse(responseData) {
}
}
// Handle caching follows status response
if (responseData.command === 'caching_follows_status') {
handleCachingFollowsResponse(responseData);
}
if (typeof logTestEvent === 'function') {
logTestEvent('RECV', `System command response: ${responseData.command} - ${responseData.status}`, 'SYSTEM_CMD');
}
@@ -4849,7 +4854,7 @@ function populateStatsPubkeys(data) {
if (data.top_pubkeys.length === 0) {
const row = document.createElement('tr');
row.innerHTML = '<td colspan="4" style="text-align: center; font-style: italic;">No pubkey data</td>';
row.innerHTML = '<td colspan="5" style="text-align: center; font-style: italic;">No pubkey data</td>';
tableBody.appendChild(row);
return;
}
@@ -4868,8 +4873,12 @@ function populateStatsPubkeys(data) {
} catch (error) {
console.log('Failed to encode pubkey to npub:', error.message);
}
// Name from kind-0 metadata (enriched by backend)
const name = pubkey.name || '';
const nameDisplay = name ? escapeHtml(name) : '<span style="color: var(--text-muted, #888); font-style: italic;">unknown</span>';
row.innerHTML = `
<td>${index + 1}</td>
<td>${nameDisplay}</td>
<td style="font-family: 'Courier New', monospace; font-size: 12px; word-break: break-all;">${npubLink}</td>
<td>${pubkey.event_count}</td>
<td>${pubkey.percentage}%</td>
@@ -4887,7 +4896,7 @@ function populateStatsPubkeysFromMonitoring(pubkeysData, totalEvents) {
if (pubkeysData.length === 0) {
const row = document.createElement('tr');
row.innerHTML = '<td colspan="4" style="text-align: center; font-style: italic;">No pubkey data</td>';
row.innerHTML = '<td colspan="5" style="text-align: center; font-style: italic;">No pubkey data</td>';
tableBody.appendChild(row);
return;
}
@@ -4910,8 +4919,13 @@ function populateStatsPubkeysFromMonitoring(pubkeysData, totalEvents) {
// Calculate percentage using totalEvents parameter
const percentage = totalEvents > 0 ? ((pubkey.event_count / totalEvents) * 100).toFixed(1) : '0.0';
// Name from kind-0 metadata (enriched by backend)
const name = pubkey.name || '';
const nameDisplay = name ? escapeHtml(name) : '<span style="color: var(--text-muted, #888); font-style: italic;">unknown</span>';
row.innerHTML = `
<td>${index + 1}</td>
<td>${nameDisplay}</td>
<td style="font-family: 'Courier New', monospace; font-size: 12px; word-break: break-all;">${npubLink}</td>
<td>${pubkey.event_count}</td>
<td>${percentage}%</td>
@@ -5446,6 +5460,9 @@ function switchPage(pageName) {
fetchCachingStatus().catch(error => {
console.log('Failed to refresh caching status on page switch: ' + error.message);
});
fetchCachingFollowsStatus().catch(error => {
console.log('Failed to refresh caching follows status on page switch: ' + error.message);
});
// Start periodic refresh every 10 seconds while the caching page is visible
if (cachingRefreshInterval) {
clearInterval(cachingRefreshInterval);
@@ -5455,6 +5472,9 @@ function switchPage(pageName) {
fetchCachingStatus().catch(error => {
console.log('Periodic caching status refresh failed: ' + error.message);
});
fetchCachingFollowsStatus().catch(error => {
console.log('Periodic caching follows refresh failed: ' + error.message);
});
}
}, 10000);
} else {
@@ -6882,7 +6902,14 @@ async function fetchCachingStatus() {
waited += 200;
}
// Populate form fields from currentConfig
// Populate form fields from currentConfig.
// IMPORTANT: skip fields the user is actively editing so the periodic
// auto-refresh (every 10s via cachingRefreshInterval) does not clobber
// in-progress edits before "Apply configuration" is pressed. A field is
// considered "being edited" if it currently has focus OR it has been
// modified by the user since the last populate (dirty). Dirty state is
// tracked by the input/change listeners installed in
// initCachingEventListeners() via markCachingFieldDirty().
CACHING_CONFIG_FIELDS.forEach(field => {
const value = getCachingConfigValue(field.key);
const el = document.getElementById(field.field);
@@ -6892,6 +6919,15 @@ async function fetchCachingStatus() {
return;
}
// Don't stomp on a field the user is currently focused on.
if (document.activeElement === el) {
return;
}
// Don't stomp on a field the user has edited but not yet applied.
if (el.dataset.cachingDirty === '1') {
return;
}
if (field.type === 'textarea-list') {
// Config stores comma-separated; display as newline-separated
el.value = String(value).split(',').map(s => s.trim()).filter(s => s.length > 0).join('\n');
@@ -6965,20 +7001,28 @@ async function applyCachingConfig() {
// Convert newlines back to comma-separated for storage
value = value.split('\n').map(s => s.trim()).filter(s => s.length > 0).join(',');
} else if (field.type === 'integer') {
// Normalize empty to 0
// Trim whitespace; leave empty for the skip check below.
// Do NOT coerce empty to '0' — the backend validates numeric
// ranges (e.g. caching_inbox_batch_size 1-500) and rejects 0,
// and config_update is atomic, so a single rejected field
// rolls back the entire batch (including unrelated boolean
// toggles like caching_enabled). Empty here means the form
// field was not populated, so we skip it and preserve the
// existing stored value.
value = String(value).trim();
if (value === '') value = '0';
} else {
value = String(value).trim();
}
// Skip empty values for optional string/textarea-list fields.
// Skip empty values for optional string/textarea-list/integer fields.
// The backend (update_config_in_table) rejects empty strings to
// prevent accidental data loss, and config_update is atomic —
// one rejected field rolls back the entire batch. Fields like
// caching_root_npubs and caching_bootstrap_relays are optional
// and may legitimately be empty until configured.
if (value === '' && (field.type === 'textarea-list' || field.type === 'string')) {
// and may legitimately be empty until configured. Integer fields
// are also skipped when empty so their existing DB value is
// preserved instead of being overwritten with an out-of-range 0.
if (value === '' && (field.type === 'textarea-list' || field.type === 'string' || field.type === 'integer')) {
return;
}
@@ -7022,6 +7066,10 @@ async function applyCachingConfig() {
}
log('Caching configuration applied successfully', 'INFO');
// Clear dirty flags so the upcoming refresh can repopulate fields from
// the freshly-applied DB values without skipping user-edited fields.
clearCachingFieldsDirty();
// Refresh status to reflect applied values
setTimeout(() => fetchCachingStatus().catch(() => {}), 1500);
@@ -7101,12 +7149,18 @@ function handleCachingStatusResponse(responseData) {
const running = pick(service.running, data.running);
const connectedRelays = pick(service.connected_relays, data.connected_relays);
const eventsCached = pick(service.events_cached, data.events_cached);
// Per-author until-cursor drain backfill progress.
const bfComplete = pick(service.backfill_authors_complete, data.backfill_authors_complete);
const bfTotal = pick(service.backfill_authors_total, data.backfill_authors_total);
let html = '<ul style="list-style:none;padding:0;margin:0;">';
if (enabled !== null) html += `<li><strong>Enabled:</strong> ${escapeHtml(String(enabled))}</li>`;
if (running !== null) html += `<li><strong>Running:</strong> ${escapeHtml(String(running))}</li>`;
if (connectedRelays !== null) html += `<li><strong>Connected Relays:</strong> ${escapeHtml(String(connectedRelays))}</li>`;
if (eventsCached !== null) html += `<li><strong>Events Cached:</strong> ${escapeHtml(String(eventsCached))}</li>`;
if (bfComplete !== null && bfTotal !== null) {
html += `<li><strong>Backfill:</strong> ${escapeHtml(String(bfComplete))}/${escapeHtml(String(bfTotal))} authors complete</li>`;
}
html += '</ul>';
serviceStatusEl.innerHTML = html;
}
@@ -7149,6 +7203,156 @@ function handleCachingStatusResponse(responseData) {
}
}
// Fetch caching follows status (per-follow detail with usernames, event counts, relays)
async function fetchCachingFollowsStatus() {
try {
if (!isLoggedIn || !userPubkey) {
throw new Error('Must be logged in to fetch caching follows status');
}
await sendAdminCommand(['system_command', 'caching_follows_status']);
} catch (e) {
console.log('caching_follows_status command failed: ' + e.message);
const tbody = document.getElementById('caching-follows-table-body');
if (tbody) {
tbody.innerHTML = `<tr><td colspan="7" style="text-align: center; color: red; padding: 8px;">Failed to load: ${escapeHtml(e.message || 'Unknown error')}</td></tr>`;
}
}
}
// Handle caching_follows_status response and render the follows table
function handleCachingFollowsResponse(responseData) {
const tbody = document.getElementById('caching-follows-table-body');
const statusEl = document.getElementById('caching-follows-status');
if (!tbody) return;
if (responseData.status !== 'success') {
tbody.innerHTML = `<tr><td colspan="7" style="text-align: center; color: red; padding: 8px;">Error: ${escapeHtml(responseData.message || 'Unknown error')}</td></tr>`;
return;
}
const follows = responseData.follows || [];
const totalFollows = responseData.total_follows || follows.length;
const totalEvents = responseData.total_events_in_db || 0;
// Active backfill target (written by the caching_relay process to PG).
const activePk = responseData.active_backfill_pubkey || '';
const activeRelay = responseData.active_backfill_relay || '';
if (statusEl) {
let html = `<span style="color: var(--text-muted, #888);">${totalFollows} follows, ${totalEvents} total events in DB</span>`;
if (activePk && activeRelay) {
html += `<br><span style="color: var(--accent-color, #ff0);">⚡ Working on: ${escapeHtml(activeRelay)}</span>`;
} else if (activePk) {
html += `<br><span style="color: var(--accent-color, #ff0);">⚡ Working on: ${escapeHtml(activePk.substring(0,12))}...</span>`;
} else {
// No active backfill target — check if all follows are complete.
const allComplete = follows.length > 0 && follows.every(f => f.backfill_complete);
if (allComplete) {
html += `<br><span style="color: green;">✓ Caching complete</span>`;
}
}
statusEl.innerHTML = html;
}
if (follows.length === 0) {
tbody.innerHTML = '<tr><td colspan="6" style="text-align: center; font-style: italic; padding: 12px;">No followed pubkeys found.</td></tr>';
return;
}
// Sort by total_events_in_db descending (most active first)
follows.sort((a, b) => (b.total_events_in_db || 0) - (a.total_events_in_db || 0));
tbody.innerHTML = '';
follows.forEach((follow, index) => {
const row = document.createElement('tr');
row.style.borderBottom = '1px solid var(--border-color, #333)';
// Highlight the row if this is the currently-active backfill target.
const isActiveAuthor = activePk && follow.pubkey === activePk;
if (isActiveAuthor) {
row.style.borderLeft = '4px solid red';
row.style.borderRight = '4px solid red';
row.style.borderTop = '2px solid red';
row.style.borderBottom = '2px solid red';
}
// Name column
const name = follow.name || '';
const nameDisplay = name ? escapeHtml(name) : '<span style="color: var(--text-muted, #888); font-style: italic;">unknown</span>';
// npub column (convert hex to npub)
let npubLink = follow.pubkey ? follow.pubkey.substring(0, 12) + '...' : '-';
try {
if (follow.pubkey && follow.pubkey.length === 64 && /^[0-9a-fA-F]+$/.test(follow.pubkey)) {
const npub = window.NostrTools.nip19.npubEncode(follow.pubkey);
npubLink = `<a href="https://njump.me/${npub}" target="_blank" class="npub-link" style="font-family: 'Courier New', monospace; font-size: 11px;">${npub.substring(0, 20)}...</a>`;
}
} catch (e) { /* keep short hex */ }
// Root badge
const rootBadge = follow.is_root
? '<span style="color: var(--accent-color, #ff0); font-weight: bold;">★ Root</span>'
: '<span style="color: var(--text-muted, #888);">—</span>';
// Events in DB
const totalInDb = follow.total_events_in_db || 0;
// Backfill status
const relayProgress = follow.relay_progress || [];
const doneRelays = relayProgress.filter(r => r.complete).length;
const totalRelays = relayProgress.length;
const bfStatus = follow.backfill_complete
? `<span style="color: green;">✓ Complete</span>`
: `<span style="color: orange;">${doneRelays}/${totalRelays} relays</span>`;
// Outbox relays with per-relay event counts from relay_progress.
// Normalize URLs by stripping trailing slashes for dedup.
const normUrl = u => u ? u.replace(/\/+$/, '') : u;
const relayProgressMap = {};
(follow.relay_progress || []).forEach(rp => {
relayProgressMap[normUrl(rp.relay_url)] = rp;
});
const outboxRelays = (follow.outbox_relays || []).map(normUrl);
// Merge outbox relays with relay_progress relays, deduplicating by
// normalized URL. Relays in relay_progress that aren't in outbox_relays
// are bootstrap relays that were queried but aren't outbox relays.
const allRelays = [...new Set([...outboxRelays, ...Object.keys(relayProgressMap)])];
const outboxDisplay = allRelays.length > 0
? allRelays.map(r => {
const short = escapeHtml(r.replace('wss://', '').replace('https://', '').replace('ws://', ''));
const rp = relayProgressMap[r];
// Highlight if this is the relay currently being queried.
const isActiveRelay = isActiveAuthor && activeRelay && r === normUrl(activeRelay);
const activePrefix = isActiveRelay ? '⚡ ' : '';
if (rp) {
const evCount = rp.events_fetched || 0;
const done = rp.complete ? '✓' : '⏳';
const status = rp.last_status || '';
const statusDisplay = status ? ` <span style="color: var(--text-muted, #888); font-size: 10px;">[${escapeHtml(status)}]</span>` : '';
const color = isActiveRelay
? 'color: red; font-weight: bold;'
: (evCount > 0 ? 'color: green;' : 'color: var(--text-muted, #888);');
return `<span style="${color}">${activePrefix}${done} ${short} (${evCount})${statusDisplay}</span>`;
}
const color = isActiveRelay
? 'color: red; font-weight: bold;'
: 'color: var(--text-muted, #888);';
return `<span style="${color}">${activePrefix}${short}</span>`;
}).join('<br>')
: '<span style="color: var(--text-muted, #888); font-style: italic;">none</span>';
row.innerHTML = `
<td style="padding: 6px 8px;">${nameDisplay}</td>
<td style="padding: 6px 8px;">${npubLink}</td>
<td style="padding: 6px 8px; text-align: center;">${rootBadge}</td>
<td style="padding: 6px 8px; text-align: right;">${totalInDb}</td>
<td style="padding: 6px 8px;">${bfStatus}</td>
<td style="padding: 6px 8px; font-size: 11px;">${outboxDisplay}</td>
`;
tbody.appendChild(row);
});
}
// Start the external caching service
async function startCachingService() {
const statusDiv = document.getElementById('caching-service-control-status');
@@ -7199,6 +7403,21 @@ async function stopCachingService() {
}
}
// Mark a caching config field as dirty (user-modified) so the periodic
// fetchCachingStatus() refresh does not overwrite in-progress edits.
function markCachingFieldDirty() {
this.dataset.cachingDirty = '1';
}
// Clear the dirty flag on all caching config fields. Called after a successful
// apply so subsequent refreshes can repopulate fields from the DB again.
function clearCachingFieldsDirty() {
CACHING_CONFIG_FIELDS.forEach(field => {
const el = document.getElementById(field.field);
if (el) delete el.dataset.cachingDirty;
});
}
// Initialize Caching page event listeners
function initCachingEventListeners() {
const applyBtn = document.getElementById('caching-apply-btn');
@@ -7219,6 +7438,18 @@ function initCachingEventListeners() {
stopServiceBtn.addEventListener('click', stopCachingService);
}
// Track user edits to caching config fields so the 10s auto-refresh
// (fetchCachingStatus -> populate form) does not clobber in-progress
// edits before "Apply configuration" is pressed. Both 'input' (fires
// on every keystroke) and 'change' (fires on blur/commit for selects
// and number inputs) are tracked.
CACHING_CONFIG_FIELDS.forEach(field => {
const el = document.getElementById(field.field);
if (!el) return;
el.addEventListener('input', markCachingFieldDirty);
el.addEventListener('change', markCachingFieldDirty);
});
console.log('Caching page event listeners initialized');
}
caching/Makefile
+1 -1
View File
@@ -6,7 +6,7 @@ SHELL := /bin/bash
CC = gcc
CFLAGS = -Wall -Wextra -std=c99 -g -O2
# nostr_core_lib is a sibling of c-relay-pg (one dir up from caching/).
# nostr_core_lib is inside the c-relay-pg project root (one dir up from caching/).
NOSTR_CORE_DIR = ../nostr_core_lib
INCLUDES = -I. -Isrc -I$(NOSTR_CORE_DIR) -I$(NOSTR_CORE_DIR)/nostr_core \
caching/build_static.sh
+2 -3
View File
@@ -83,9 +83,8 @@ else
fi
echo ""
# nostr_core_lib is a sibling directory; copy it into the build context.
# Docker cannot follow symlinks outside the context, so we do a real copy
# (excluding .git and build artifacts) and clean up afterwards.
# nostr_core_lib is inside the c-relay-pg project root (one dir up from caching/).
# Copy it into the Docker build context (excluding .git and build artifacts).
NOSTR_CORE_DIR="$SCRIPT_DIR/../nostr_core_lib"
if [ ! -d "$NOSTR_CORE_DIR" ]; then
echo "ERROR: nostr_core_lib not found at $NOSTR_CORE_DIR"
caching/src/backfill.c
+299 -300
View File
@@ -1,10 +1,16 @@
/*
* caching_relay - progressive window-expansion backfill
* caching_relay - relay-by-relay per-author until-cursor drain backfill.
*
* Uses until-based pagination so authors with more events than the page size
* are fully drained across multiple ticks instead of being skipped. In
* PostgreSQL mode, per-author/window progress is persisted to the
* caching_backfill_progress table so the service can resume after restart.
* Each (author, relay) pair has its own persistent until_cursor stored in
* the caching_backfill_relay_progress table. Each backfill tick picks the
* next author that still has at least one incomplete relay (round-robin),
* then queries each incomplete relay one at a time with
* since=0 & until=cursor & limit=page_size
* publishes the page, and advances that relay's cursor to
* oldest_event_created_at - 1. A relay is marked complete when a page
* returns fewer than page_size events (the relay is drained for this
* author). An author is marked complete in caching_followed_pubkeys when
* all their relay progress rows are complete.
*/
#define _GNU_SOURCE
#include "backfill.h"
@@ -18,6 +24,84 @@
#include <stdlib.h>
#include <time.h>
/* ------------------------------------------------------------------ */
/* Global access for status reporting */
/* ------------------------------------------------------------------ */
/* Singleton pointer set in cr_backfill_init, used by the accessor
* functions below so the main loop / status reporter can see what
* the backfill is currently working on. */
static cr_backfill_t *g_bf = NULL;
const char *cr_backfill_active_pubkey(void) {
return g_bf ? g_bf->active_pubkey : "";
}
const char *cr_backfill_active_relay(void) {
return g_bf ? g_bf->active_relay : "";
}
int cr_backfill_active_got_eose(void) {
return g_bf ? g_bf->active_relay_got_eose : 0;
}
/* ------------------------------------------------------------------ */
/* EOSE callback for synchronous_query_relays_with_progress */
/* ------------------------------------------------------------------ */
/* Context passed to the query callback. Records whether the relay sent
* EOSE (we got everything) or timed out (partial result). */
typedef struct {
int got_eose; /* 1 if EOSE received, 0 if timed out/errored */
int events_count; /* events reported by callback */
char last_status[256]; /* last status string from callback (may include message) */
} backfill_query_ctx_t;
static void backfill_query_callback(const char *relay_url, const char *status,
const char *event_id, int events_received,
int total_relays, int completed_relays,
void *user_data) {
(void)relay_url; (void)total_relays; (void)completed_relays;
backfill_query_ctx_t *ctx = (backfill_query_ctx_t *)user_data;
if (!ctx || !status) return;
/* Ignore the synthetic "all_complete" status from core_relays.c — it
* is sent AFTER the real EOSE/timeout and would overwrite the real
* status with a misleading "all_complete" string, and clobber
* events_count with total_unique_events. The real per-relay status
* (eose/timeout/error) is what we need for completion decisions. */
if (strcmp(status, "all_complete") == 0) {
return;
}
/* For NOTICE/CLOSED/error, event_id carries the descriptive message
* content (relay's own text for NOTICE/CLOSED, or our transport-level
* diagnosis for error). Include it in the status for display.
* Priority: relay's own response (NOTICE/CLOSED) > our error > bare
* status string. */
if ((strcmp(status, "NOTICE") == 0 || strcmp(status, "CLOSED") == 0) &&
event_id && event_id[0] != '\0') {
snprintf(ctx->last_status, sizeof(ctx->last_status), "%s: %s", status, event_id);
} else if (strcmp(status, "error") == 0) {
if (event_id && event_id[0] != '\0') {
snprintf(ctx->last_status, sizeof(ctx->last_status), "error: %s", event_id);
} else {
snprintf(ctx->last_status, sizeof(ctx->last_status), "error");
}
} else {
snprintf(ctx->last_status, sizeof(ctx->last_status), "%s", status);
}
if (strcmp(status, "eose") == 0) {
ctx->got_eose = 1;
} else if (strcmp(status, "timeout") == 0) {
ctx->got_eose = 0;
} else if (strcmp(status, "error") == 0) {
ctx->got_eose = 0;
}
/* NOTICE/CLOSED don't change got_eose — the relay might still send
* events or EOSE after a NOTICE. */
ctx->events_count = events_received;
}
/* ------------------------------------------------------------------ */
/* Helpers */
/* ------------------------------------------------------------------ */
@@ -50,16 +134,6 @@ static int find_oldest_created_at(cJSON **events, int count, long *out_oldest) {
return found;
}
/* Count how many events in the array have created_at == ts. */
static int count_at_timestamp(cJSON **events, int count, long ts) {
int n = 0;
for (int i = 0; i < count; i++) {
cJSON *ca = cJSON_GetObjectItem(events[i], "created_at");
if (ca && cJSON_IsNumber(ca) && (long)ca->valuedouble == ts) n++;
}
return n;
}
/* Build the kinds array for a given pubkey (admin vs regular).
* Returns NULL if no kinds filter should be applied (admin_all_kinds). */
static cJSON *build_kinds(cr_config_t *cfg, const char *pk) {
@@ -89,146 +163,90 @@ static int resolve_relays(nostr_relay_pool_t *upstream,
*out_urls = NULL;
*out_n = 0;
/* Collect outbox relays for this author (from kind-10002). */
int outbox_count = 0;
const cr_outbox_entry_t *oe = NULL;
if (relay_map) {
const cr_outbox_entry_t *oe = cr_relay_map_get_outbox(relay_map, pk);
if (oe && oe->relay_count > 0) {
const char **urls = malloc(oe->relay_count * sizeof(char *));
if (!urls) return -1;
for (int j = 0; j < oe->relay_count; j++) urls[j] = oe->relays[j];
*out_urls = urls;
*out_n = oe->relay_count;
return 0;
oe = cr_relay_map_get_outbox(relay_map, pk);
if (oe) outbox_count = oe->relay_count;
}
/* Also collect all upstream (bootstrap) relays. We query BOTH the
* author's outbox relays AND the bootstrap relays to maximize coverage.
* Different relays may have different subsets of the author's events. */
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int upstream_count = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
/* Merge outbox + upstream, deduplicating by URL. */
int max_total = outbox_count + upstream_count;
if (max_total <= 0) {
free(listed);
free(statuses);
return 0;
}
const char **urls = malloc(max_total * sizeof(char *));
if (!urls) { free(listed); free(statuses); return -1; }
int n = 0;
/* Add outbox relays first. */
if (oe) {
for (int j = 0; j < oe->relay_count && n < max_total; j++) {
urls[n++] = oe->relays[j];
}
}
/* Fallback: all upstream relays. */
char **listed = NULL;
nostr_pool_relay_status_t *statuses = NULL;
int n = nostr_relay_pool_list_relays(upstream, &listed, &statuses);
if (n > 0) {
const char **urls = malloc(n * sizeof(char *));
if (!urls) { free(listed); free(statuses); return -1; }
for (int j = 0; j < n; j++) urls[j] = listed[j];
*out_urls = urls;
*out_n = n;
/* Add upstream relays, skipping duplicates. */
for (int j = 0; j < upstream_count && n < max_total; j++) {
int dup = 0;
for (int k = 0; k < n; k++) {
if (strcmp(urls[k], listed[j]) == 0) { dup = 1; break; }
}
if (!dup) {
urls[n++] = listed[j];
}
}
free(listed);
free(statuses);
*out_urls = urls;
*out_n = n;
return 0;
}
/* Issue a query_sync for one author with the given since/until/limit.
* Returns the events array (caller frees each event + the array) or NULL.
* Sets *out_count. */
static cJSON **query_author(nostr_relay_pool_t *upstream,
const cr_relay_map_t *relay_map,
cr_config_t *cfg, const char *pk,
long since, long until, int limit,
int *out_count) {
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(pk));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = build_kinds(cfg, pk);
if (kinds) cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)since));
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)until));
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)limit));
const char **urls = NULL;
int n = 0;
resolve_relays(upstream, relay_map, pk, &urls, &n);
cJSON **events = nostr_relay_pool_query_sync(upstream, urls, n, filter,
out_count, 15000);
free(urls);
cJSON_Delete(filter);
return events;
}
/* ------------------------------------------------------------------ */
/* Window advancement */
/* ------------------------------------------------------------------ */
static void advance_window(cr_backfill_t *bf, cr_config_t *cfg) {
/* Record that we've backfilled up to (now - current_window). */
cfg->state.backfilled_until = (long)time(NULL) - bf->current_window_s;
cfg->state.current_window_index = bf->window_index + 1;
cfg->state.backfill_cursor = 0;
if (!pg_mode_active()) {
cr_config_save_state(cfg);
}
DEBUG_INFO("backfill: window %d complete (%ld events). backfilled_until=%ld",
bf->window_index, bf->events_this_window, cfg->state.backfilled_until);
bf->window_index++;
if (bf->window_index >= cfg->backfill.window_count) {
DEBUG_INFO("backfill: all windows complete, entering steady-state");
bf->in_progress = 0;
return;
}
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
bf->cursor = 0;
bf->until_cursor = 0;
bf->author_complete = 0;
bf->blocked = 0;
bf->blocked_until_ts = 0;
bf->events_this_window = 0;
bf->window_started = time(NULL);
DEBUG_INFO("backfill: advancing to window %d (%ld seconds)",
bf->window_index, bf->current_window_s);
}
/* Persist the current cursor state. In PG mode we rely on the per-author
* progress table; in legacy mode we write the .jsonc state. */
static void persist_cursor(cr_backfill_t *bf, cr_config_t *cfg) {
cfg->state.backfill_cursor = bf->cursor;
if (!pg_mode_active()) {
cr_config_save_state(cfg);
}
}
/* Save progress for the current author to PG (no-op outside PG mode). */
static void pg_save_current(cr_backfill_t *bf, cr_config_t *cfg,
const char *pk, int complete) {
(void)cfg; /* window anchor is derived from bf + time(), cfg not needed */
if (!pg_mode_active() || !pk) return;
long anchor = (long)time(NULL) - bf->current_window_s;
pg_inbox_save_backfill_progress(pk, bf->window_index, anchor,
bf->until_cursor, complete);
}
/* ------------------------------------------------------------------ */
/* Init */
/* ------------------------------------------------------------------ */
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg) {
(void)cfg;
memset(bf, 0, sizeof(*bf));
bf->window_index = cfg->state.current_window_index;
if (bf->window_index >= cfg->backfill.window_count) {
/* Already done; steady state. */
bf->in_progress = 0;
DEBUG_INFO("backfill: already complete (window index %d >= %d), steady-state",
bf->window_index, cfg->backfill.window_count);
g_bf = bf; /* Set singleton for status accessors */
/* In PG mode, check whether any incomplete authors remain in the DB.
* If so, backfill is in progress; otherwise we're in steady state.
* Outside PG mode there is no per-author table, so we just start. */
if (pg_mode_active()) {
int complete = 0, total = 0;
if (pg_inbox_count_backfill_progress(&complete, &total) == 0) {
bf->in_progress = (total > 0 && complete < total) ? 1 : 0;
DEBUG_INFO("backfill: init complete=%d/%d -> in_progress=%d",
complete, total, bf->in_progress);
return;
}
/* Count query failed - assume in progress so we keep trying. */
bf->in_progress = 1;
DEBUG_WARN("backfill: init could not read progress counts, "
"defaulting to in_progress=1");
return;
}
bf->current_window_s = cfg->backfill.window_schedule_seconds[bf->window_index];
bf->cursor = cfg->state.backfill_cursor;
bf->until_cursor = 0;
bf->author_complete = 0;
bf->blocked = 0;
bf->blocked_until_ts = 0;
bf->in_progress = 1;
bf->window_started = time(NULL);
/* In PG mode, per-author restore is deferred to the first tick (the
* followed set is not resolved yet at init time). The tick loop skips
* already-complete authors via the progress table. */
DEBUG_INFO("backfill: starting at window %d (%ld seconds), cursor %d",
bf->window_index, bf->current_window_s, bf->cursor);
/* Legacy (non-PG) mode: no per-author table; just start. */
bf->in_progress = 1;
DEBUG_INFO("backfill: init (legacy mode) in_progress=1");
}
/* ------------------------------------------------------------------ */
@@ -236,196 +254,177 @@ void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg) {
/* ------------------------------------------------------------------ */
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed,
cr_sink_t *sink, const cr_relay_map_t *relay_map) {
(void)followed; /* followed set is tracked in the DB now */
(void)relay_map; /* relays are tracked per-author in the DB */
if (!cfg->backfill.enabled) return -2;
if (!bf->in_progress) return -2;
if (followed->count == 0) return 0;
/* Throttle: only one query per tick_interval. */
/* Throttle: one query per tick_interval. */
time_t now = time(NULL);
if (bf->last_tick && (now - bf->last_tick) < cfg->backfill.tick_interval_seconds) {
if (bf->last_tick && (now - bf->last_tick) < cfg->backfill.tick_interval_seconds)
return 0;
}
/* In PG mode, skip past authors already marked complete for this window
* (resume case). Restore the until_cursor for the first incomplete author. */
if (pg_mode_active()) {
while (bf->cursor < followed->count) {
const char *pk = followed->items[bf->cursor];
long uc = 0;
int complete = 0, found = 0;
if (pg_inbox_load_backfill_progress(pk, bf->window_index,
&uc, &complete, &found) == 0
&& found && complete) {
DEBUG_TRACE("backfill: author[%d] %s already complete, skipping",
bf->cursor, pk);
bf->cursor++;
cfg->state.backfill_cursor = bf->cursor;
continue;
}
if (found && uc > 0 && bf->until_cursor == 0) {
bf->until_cursor = uc;
DEBUG_TRACE("backfill: resumed author[%d] %s at until=%ld",
bf->cursor, pk, bf->until_cursor);
}
break;
}
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
return 1;
}
/* Pick the next author that has at least one incomplete relay progress
* row (round-robin). */
char pk[CR_HEX_LEN];
if (pg_inbox_pick_next_author_with_incomplete_relays(pk, sizeof(pk),
&bf->author_round_cursor) != 0) {
/* No incomplete authors - steady state. */
bf->in_progress = 0;
DEBUG_INFO("backfill: no authors with incomplete relays, steady-state");
return -2;
}
const char *pk = followed->items[bf->cursor];
/* Determine the until cursor for this query. First query for an author:
* until = now. Subsequent queries: until = oldest event created_at - 1
* from the previous page (set at the end of the previous tick). */
long until = bf->until_cursor;
if (until == 0) {
until = (long)now;
}
long since = (long)now - bf->current_window_s;
int page_size = cfg->backfill.events_per_tick;
if (page_size < 1) page_size = 50;
int max_cap = CR_BACKFILL_MAX_PAGE_CAP;
if (max_cap < page_size) max_cap = page_size;
if (page_size < 1) page_size = 500;
bf->last_tick = now;
DEBUG_TRACE("backfill: querying %s (window %ld, since %ld, until %ld, limit %d)",
pk, bf->current_window_s, since, until, page_size);
int ev_count = 0;
cJSON **events = query_author(upstream, relay_map, cfg, pk,
since, until, page_size, &ev_count);
/* Saturation handling: if we got exactly page_size events, the page may
* be truncated. Bump the limit up to max_cap and retry once. */
int limit_used = page_size;
if (events && ev_count == page_size && page_size < max_cap) {
for (int k = 0; k < ev_count; k++) cJSON_Delete(events[k]);
free(events);
DEBUG_TRACE("backfill: saturated page for %s, retrying with limit %d",
pk, max_cap);
events = query_author(upstream, relay_map, cfg, pk,
since, until, max_cap, &ev_count);
limit_used = max_cap;
}
/* If still saturated at the cap, we cannot safely paginate past the
* oldest timestamp (the relay may have more events at that exact ts).
* Mark the author blocked at that timestamp and move on; it will be
* retried in a later window sweep. */
if (events && ev_count == max_cap && max_cap > page_size) {
long oldest = 0;
find_oldest_created_at(events, ev_count, &oldest);
int at_oldest = count_at_timestamp(events, ev_count, oldest);
DEBUG_WARN("backfill: author %s saturated at limit %d (oldest ts %ld, "
"%d events at that ts) - marking blocked, will retry next window",
pk, max_cap, oldest, at_oldest);
/* Publish what we have. */
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
for (int k = 0; k < ev_count; k++) {
cr_sink_publish(sink, events[k]);
cJSON_Delete(events[k]);
bf->events_this_window++;
bf->events_total++;
}
free(events);
bf->blocked = 1;
bf->blocked_until_ts = oldest;
bf->until_cursor = oldest; /* resume point for next window */
pg_save_current(bf, cfg, pk, 0); /* not complete - retry next window */
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
/* Get the incomplete relays for this author. */
cJSON *relays = pg_inbox_get_incomplete_relays(pk);
if (!relays) {
DEBUG_WARN("backfill: no incomplete relays returned for %s", pk);
/* Author has no relay rows - mark complete so we don't loop. */
pg_inbox_check_and_mark_author_complete(pk);
return 1;
}
/* Normal page processing. Capture the oldest timestamp BEFORE publishing
* so we can paginate if saturated. */
long oldest_ts = 0;
int have_oldest = 0;
if (events && ev_count > 0) {
have_oldest = find_oldest_created_at(events, ev_count, &oldest_ts);
}
int relay_count = cJSON_GetArraySize(relays);
DEBUG_TRACE("backfill: author %s has %d incomplete relay(s)", pk, relay_count);
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
/* Query each incomplete relay one at a time. */
for (int r = 0; r < relay_count; r++) {
cJSON *entry = cJSON_GetArrayItem(relays, r);
if (!entry) continue;
cJSON *url_node = cJSON_GetObjectItem(entry, "relay_url");
cJSON *cur_node = cJSON_GetObjectItem(entry, "until_cursor");
if (!url_node || !cJSON_IsString(url_node)) continue;
const char *relay_url = cJSON_GetStringValue(url_node);
long until_cursor = (cur_node && cJSON_IsNumber(cur_node))
? (long)cur_node->valuedouble : 0;
if (until_cursor == 0) until_cursor = (long)now;
/* Build the filter for this single relay. */
cJSON *filter = cJSON_CreateObject();
cJSON *authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(pk));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON *kinds = build_kinds(cfg, pk);
if (kinds) cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber(0.0));
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)until_cursor));
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)page_size));
DEBUG_TRACE("backfill: %s @ %s (until=%ld, limit=%d)",
pk, relay_url, until_cursor, page_size);
/* Record the active target for UI status display (both
* in-memory for local accessors and in PG for the relay). */
strncpy(bf->active_pubkey, pk, sizeof(bf->active_pubkey) - 1);
bf->active_pubkey[sizeof(bf->active_pubkey) - 1] = '\0';
strncpy(bf->active_relay, relay_url, sizeof(bf->active_relay) - 1);
bf->active_relay[sizeof(bf->active_relay) - 1] = '\0';
bf->active_relay_got_eose = 0;
pg_inbox_set_active_target(pk, relay_url);
/* Query this single relay using synchronous_query_relays_with_progress
* which gives us an EOSE/timeout callback — the relay's own signal
* of whether it sent everything or we timed out. */
backfill_query_ctx_t qctx = {0, 0};
const char *one_url = relay_url;
int ev_count = 0;
cJSON **events = synchronous_query_relays_with_progress(
&one_url, 1, filter, RELAY_QUERY_ALL_RESULTS,
&ev_count, 30, /* timeout in seconds */
backfill_query_callback, &qctx,
0, NULL /* nip42 disabled, no private key */);
cJSON_Delete(filter);
bf->active_relay_got_eose = qctx.got_eose;
if (!events || ev_count == 0) {
/* No events from this relay - it is drained for this author.
* (ev_count == 0 with EOSE means truly empty; without EOSE it
* means a connection failure — keep incomplete to retry.
* The pg_inbox_update_relay_progress SQL will auto-mark
* complete once consecutive_errors reaches 3.) */
int mark_complete = qctx.got_eose ? 1 : 0;
pg_inbox_update_relay_progress(pk, relay_url, until_cursor,
mark_complete, 0, qctx.last_status);
free(events);
if (!qctx.got_eose) {
/* Error or timeout with no events — surface the
* descriptive status at WARN so operators can see why. */
DEBUG_WARN("backfill: %s @ %s failed: %s (will auto-complete after 3 consecutive errors)",
pk, relay_url, qctx.last_status);
} else {
DEBUG_LOG("backfill: %s @ %s complete (no events, eose)",
pk, relay_url);
}
continue;
}
/* Find oldest timestamp, then publish every event. */
long oldest_ts = 0;
find_oldest_created_at(events, ev_count, &oldest_ts);
if (events && ev_count > 0) {
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
for (int k = 0; k < ev_count; k++) {
cr_sink_publish(sink, events[k]);
cJSON_Delete(events[k]);
bf->events_this_window++;
bf->events_total++;
}
free(events);
bf->events_total += ev_count;
DEBUG_LOG("backfill: %s @ %s -> %d events (until=%ld, oldest=%ld, eose=%d)",
pk, relay_url, ev_count, until_cursor, oldest_ts, qctx.got_eose);
/* Completion decision based on EOSE status:
* - ev_count >= page_size: saturated, advance cursor, keep incomplete
* - ev_count < page_size AND got EOSE: relay is truly drained
* - ev_count < page_size AND no EOSE (timeout): partial result,
* advance cursor but keep incomplete to retry next tick
* - Guard: if cursor didn't advance (oldest_ts >= until_cursor),
* mark complete to avoid infinite loop on identical timestamps. */
long next_cursor = (oldest_ts > 0) ? oldest_ts - 1 : 0;
int cursor_advanced = (oldest_ts > 0 && oldest_ts < until_cursor);
if (ev_count >= page_size) {
pg_inbox_update_relay_progress(pk, relay_url, next_cursor, 0, ev_count, qctx.last_status);
} else if (qctx.got_eose) {
/* Relay sent EOSE with < page_size events — truly drained. */
pg_inbox_update_relay_progress(pk, relay_url, next_cursor, 1, ev_count, qctx.last_status);
} else {
/* Timed out before EOSE — partial result. Keep incomplete
* to retry, unless the cursor didn't advance (stall guard).
* Timeouts DO count toward the consecutive_errors limit
* (a relay that consistently times out is effectively dead
* for backfill purposes), so the SQL will auto-mark
* complete after 3 consecutive timeouts/errors. */
int mark_complete = cursor_advanced ? 0 : 1;
pg_inbox_update_relay_progress(pk, relay_url, next_cursor,
mark_complete, ev_count, qctx.last_status);
DEBUG_WARN("backfill: %s @ %s partial: %s (%d events, will auto-complete after 3 consecutive timeouts/errors)",
pk, relay_url, qctx.last_status, ev_count);
}
}
DEBUG_LOG("backfill: pubkey[%d/%d] %s -> %d events (until=%ld)",
bf->cursor, followed->count, pk, ev_count, until);
/* Clear active target — done with this author. */
bf->active_relay[0] = '\0';
pg_inbox_set_active_target(pk, "");
int saturated = (ev_count == limit_used) && (ev_count > 0);
cJSON_Delete(relays);
if (ev_count == 0) {
/* No more events for this author in the window. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else if (!saturated) {
/* Last page for this author. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else {
/* Saturated: paginate. Set until = oldest - 1 and keep the same
* author for the next tick. The `until` parameter is inclusive, so
* subtracting 1 avoids re-fetching the oldest event. */
if (have_oldest) {
bf->until_cursor = oldest_ts - 1;
if (bf->until_cursor < since) {
/* Walked back past the window start; author is done. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
} else {
pg_save_current(bf, cfg, pk, 0);
}
} else {
/* Could not determine oldest (no valid created_at); mark done. */
bf->author_complete = 1;
bf->until_cursor = 0;
pg_save_current(bf, cfg, pk, 1);
bf->cursor++;
persist_cursor(bf, cfg);
if (bf->cursor >= followed->count) {
advance_window(bf, cfg);
}
}
/* After querying all incomplete relays for this author, check whether
* the author is now fully complete. */
int crc = pg_inbox_check_and_mark_author_complete(pk);
if (crc == 1) {
DEBUG_LOG("backfill: author %s fully complete (all relays drained)", pk);
}
return 1;
caching/src/backfill.h
+33 -40
View File
@@ -1,17 +1,13 @@
/*
* caching_relay - progressive window-expansion backfill.
* caching_relay - per-author until-cursor drain backfill.
*
* Round-robin through the followed-pubkey set, issuing query_sync per pubkey
* for the current window (since = now - window_seconds). When a full pass
* completes, advance the window and persist state to the config file.
*
* Pagination: each author is queried with an `until` cursor that walks
* backwards in time. If a query returns exactly `page_size` events (saturated),
* the limit is bumped up to a configurable cap and retried; if still saturated
* the author is marked "blocked" at that timestamp and retried in a later
* window sweep. Otherwise the until cursor advances to
* (oldest_event_created_at - 1) and the same author is queried again on the
* next tick until no more events are returned.
* Each followed author has a single persistent cursor (starting at `now`)
* stored in the caching_followed_pubkeys table. Each backfill tick picks
* the next incomplete author (round-robin), queries
* since=0 & until=cursor & limit=page_size
* publishes the page, and sets cursor = oldest_event_created_at - 1.
* Repeat until a page returns fewer than page_size events (beginning of
* the author's history reached), then mark the author complete.
*/
#ifndef CACHING_RELAY_BACKFILL_H
#define CACHING_RELAY_BACKFILL_H
@@ -22,44 +18,41 @@
#include "relay_discovery.h"
#include "../nostr_core_lib/nostr_core/nostr_core.h"
/* Maximum limit we will bump a saturated page to before marking an author
* blocked at a timestamp. Keeps a single query bounded. */
#define CR_BACKFILL_MAX_PAGE_CAP 1000
typedef struct {
int window_index;
long current_window_s;
int cursor; /* current index into followed set */
long until_cursor; /* current until timestamp for the current author */
int in_progress; /* 1 if a window pass is in progress */
long events_this_window; /* events pulled in current window */
long events_total; /* events pulled across all windows */
time_t last_tick; /* last time a pubkey was queried */
time_t window_started; /* when current window pass started */
int author_complete; /* 1 when current author is done, advance cursor */
int blocked; /* 1 when current author is blocked at a saturated timestamp */
long blocked_until_ts; /* timestamp where the author is blocked */
int in_progress; /* 1 if any incomplete authors remain */
long events_total; /* events pulled across all ticks (diagnostic) */
time_t last_tick; /* last time an author was queried (throttle) */
int author_round_cursor;/* round-robin cursor for author selection */
/* Current backfill target (for UI status display). */
char active_pubkey[65]; /* pubkey currently being backfilled (or "") */
char active_relay[256]; /* relay URL currently being queried (or "") */
int active_relay_got_eose; /* 1 if last query got EOSE, 0 if timed out */
} cr_backfill_t;
/* Initialize backfill state from the persisted config state.
* In PostgreSQL mode (pg_inbox initialized), restores per-author progress
* for the current window from the caching_backfill_progress table. */
/* Global access to the current backfill target for status reporting.
* Returns pointers to static storage in the singleton cr_backfill_t.
* Safe to call from the main loop; not thread-safe but backfill is
* single-threaded. */
const char *cr_backfill_active_pubkey(void);
const char *cr_backfill_active_relay(void);
int cr_backfill_active_got_eose(void);
/* Initialize backfill state. Checks the caching_followed_pubkeys table for
* any incomplete authors; sets in_progress accordingly. */
void cr_backfill_init(cr_backfill_t *bf, cr_config_t *cfg);
/* Perform one backfill tick. This queries ONE followed pubkey (the one at
* cfg->state.backfill_cursor) and publishes results to the sink. Uses
* until-based pagination so authors with more events than the page size are
* fully drained across multiple ticks. When a full round-robin pass completes,
* advances the window and saves state.
/* Perform one backfill tick. Picks the next incomplete author from the DB
* (round-robin), queries one page of history, publishes it to the sink, and
* updates the per-author cursor in the DB.
*
* Returns:
* 1 if a tick was performed (a pubkey was queried)
* 1 if a tick was performed (an author was queried)
* 0 if throttled (tick_interval not elapsed) - caller should pump pools
* -1 on hard error
* -2 if backfill is complete (all windows done) - caller should go steady-state
* -2 if backfill is complete (no incomplete authors) - caller goes steady-state
*/
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
nostr_relay_pool_t *upstream, cr_pubkey_set_t *followed,
nostr_relay_pool_t *upstream,
cr_pubkey_set_t *followed,
cr_sink_t *sink, const cr_relay_map_t *relay_map);
#endif /* CACHING_RELAY_BACKFILL_H */
caching/src/config.c
+3 -28
View File
@@ -146,15 +146,10 @@ int cr_config_load(cr_config_t *cfg, const char *path) {
cJSON *bf = cJSON_GetObjectItem(root, "backfill");
if (bf) {
cfg->backfill.enabled = cJSON_IsTrue(cJSON_GetObjectItem(bf, "enabled"));
copy_long_array(cJSON_GetObjectItem(bf, "window_schedule_seconds"),
cfg->backfill.window_schedule_seconds, CR_MAX_WINDOWS,
&cfg->backfill.window_count);
cJSON *ept = cJSON_GetObjectItem(bf, "events_per_tick");
if (ept) cfg->backfill.events_per_tick = (int)cJSON_GetNumberValue(ept);
cJSON *tis = cJSON_GetObjectItem(bf, "tick_interval_seconds");
if (tis) cfg->backfill.tick_interval_seconds = (int)cJSON_GetNumberValue(tis);
cJSON *wcs = cJSON_GetObjectItem(bf, "window_cooldown_seconds");
if (wcs) cfg->backfill.window_cooldown_seconds = (int)cJSON_GetNumberValue(wcs);
}
/* live */
@@ -174,25 +169,13 @@ int cr_config_load(cr_config_t *cfg, const char *path) {
if (st) {
cJSON *bu = cJSON_GetObjectItem(st, "backfilled_until");
if (bu) cfg->state.backfilled_until = (long)cJSON_GetNumberValue(bu);
cJSON *cwi = cJSON_GetObjectItem(st, "current_window_index");
if (cwi) cfg->state.current_window_index = (int)cJSON_GetNumberValue(cwi);
cJSON *bc = cJSON_GetObjectItem(st, "backfill_cursor");
if (bc) cfg->state.backfill_cursor = (int)cJSON_GetNumberValue(bc);
}
cJSON_Delete(root);
/* Defaults if missing. */
if (cfg->backfill.window_count == 0) {
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
for (int i = 0; i < n; i++) cfg->backfill.window_schedule_seconds[i] = def_windows[i];
cfg->backfill.window_count = n;
}
if (cfg->backfill.events_per_tick == 0) cfg->backfill.events_per_tick = 50;
if (cfg->backfill.events_per_tick == 0) cfg->backfill.events_per_tick = 500;
if (cfg->backfill.tick_interval_seconds == 0) cfg->backfill.tick_interval_seconds = 5;
if (cfg->backfill.window_cooldown_seconds == 0) cfg->backfill.window_cooldown_seconds = 60;
if (cfg->live.resubscribe_interval_seconds == 0) cfg->live.resubscribe_interval_seconds = 300;
if (cfg->follow_graph_refresh_seconds == 0) cfg->follow_graph_refresh_seconds = 600;
@@ -253,13 +236,8 @@ int cr_config_save_state(cr_config_t *cfg) {
cJSON *bf = cJSON_CreateObject();
cJSON_AddItemToObject(bf, "enabled", cJSON_CreateBool(cfg->backfill.enabled));
cJSON *ws = cJSON_CreateArray();
for (int i = 0; i < cfg->backfill.window_count; i++)
cJSON_AddItemToArray(ws, cJSON_CreateNumber(cfg->backfill.window_schedule_seconds[i]));
cJSON_AddItemToObject(bf, "window_schedule_seconds", ws);
cJSON_AddItemToObject(bf, "events_per_tick", cJSON_CreateNumber(cfg->backfill.events_per_tick));
cJSON_AddItemToObject(bf, "tick_interval_seconds", cJSON_CreateNumber(cfg->backfill.tick_interval_seconds));
cJSON_AddItemToObject(bf, "window_cooldown_seconds", cJSON_CreateNumber(cfg->backfill.window_cooldown_seconds));
cJSON_AddItemToObject(root, "backfill", bf);
cJSON *lv = cJSON_CreateObject();
@@ -272,8 +250,6 @@ int cr_config_save_state(cr_config_t *cfg) {
cJSON *st = cJSON_CreateObject();
cJSON_AddItemToObject(st, "backfilled_until", cJSON_CreateNumber(cfg->state.backfilled_until));
cJSON_AddItemToObject(st, "current_window_index", cJSON_CreateNumber(cfg->state.current_window_index));
cJSON_AddItemToObject(st, "backfill_cursor", cJSON_CreateNumber(cfg->state.backfill_cursor));
cJSON_AddItemToObject(root, "state", st);
char *json = cJSON_Print(root);
@@ -304,9 +280,8 @@ int cr_config_save_state(cr_config_t *cfg) {
DEBUG_ERROR("config save: rename failed: %s", strerror(errno));
return -1;
}
DEBUG_LOG("config state saved: backfilled_until=%ld window=%d cursor=%d",
cfg->state.backfilled_until, cfg->state.current_window_index,
cfg->state.backfill_cursor);
DEBUG_LOG("config state saved: backfilled_until=%ld",
cfg->state.backfilled_until);
return 0;
}
caching/src/config.h
+6 -8
View File
@@ -11,18 +11,14 @@
#define CR_MAX_ROOT_NPUBS 16
#define CR_MAX_UPSTREAM 32
#define CR_MAX_KINDS 32
#define CR_MAX_WINDOWS 16
#define CR_NPUB_LEN 64 /* npub1... bech32, generous */
#define CR_URL_LEN 256
#define CR_HEX_PUBKEY_LEN 65 /* 64 hex chars + NUL */
typedef struct {
int enabled;
long window_schedule_seconds[CR_MAX_WINDOWS];
int window_count;
int events_per_tick;
int events_per_tick; /* per-query page size (default 500) */
int tick_interval_seconds;
int window_cooldown_seconds;
} cr_backfill_config_t;
typedef struct {
@@ -30,10 +26,12 @@ typedef struct {
int resubscribe_interval_seconds;
} cr_live_config_t;
/* Persistent state. The per-author until-cursor drain model stores all
* backfill progress in the caching_followed_pubkeys table, so the only
* state kept here is the legacy-mode first-time flag (used to drive relay
* discovery caching behavior). */
typedef struct {
long backfilled_until; /* unix ts; 0 = nothing backfilled yet */
int current_window_index; /* index into window_schedule */
int backfill_cursor; /* round-robin cursor over followed pubkeys */
long backfilled_until; /* legacy: 0 = first-time startup */
} cr_state_t;
typedef struct {
caching/src/live_subscriber.c
+20 -2
View File
@@ -12,8 +12,9 @@
/* Context passed as user_data to the subscription callbacks. */
typedef struct {
cr_sink_t *sink;
cr_live_t *live;
cr_sink_t *sink;
cr_live_t *live;
cr_config_t *cfg;
} cr_live_ctx_t;
static cr_live_ctx_t g_live_ctx;
@@ -23,6 +24,22 @@ static void live_on_event(cJSON *event, const char *relay_url, void *user_data)
(void)relay_url;
ctx->live->events_received++;
cr_sink_publish(ctx->sink, event);
/* Detect admin kind-3 (contact list) changes from a root pubkey and
* signal the main loop to refresh the follow graph immediately. */
if (ctx->cfg) {
cJSON *kind = cJSON_GetObjectItem(event, "kind");
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
if (kind && cJSON_IsNumber(kind) && kind->valuedouble == 3.0 &&
pubkey && cJSON_IsString(pubkey)) {
if (cr_follow_is_root(ctx->cfg, cJSON_GetStringValue(pubkey))) {
ctx->live->follow_graph_changed = 1;
DEBUG_INFO("live: root kind-3 detected from %s, "
"signaling follow graph refresh",
cJSON_GetStringValue(pubkey));
}
}
}
}
static void live_on_eose(cJSON **events, int event_count, void *user_data) {
@@ -102,6 +119,7 @@ static int open_subs(cr_live_t *live, cr_config_t *cfg, nostr_relay_pool_t *upst
cr_pubkey_set_t *followed, cr_sink_t *sink) {
g_live_ctx.sink = sink;
g_live_ctx.live = live;
g_live_ctx.cfg = cfg;
int admin_count = 0;
int follows_count = 0;
caching/src/live_subscriber.h
+1
View File
@@ -21,6 +21,7 @@ typedef struct {
nostr_pool_subscription_t *admin_sub; /* admin pubkeys, admin_kinds */
long events_received;
time_t last_resubscribe;
int follow_graph_changed; /* set when a root npub publishes a kind-3 */
} cr_live_t;
/* Open the live subscription(s). Returns 0 on success. */
caching/src/main.c
+184 -18
View File
@@ -50,6 +50,65 @@ static void nostr_log_forwarder(int level, const char *component,
}
}
/* Initialize per-relay backfill progress rows for every followed pubkey.
* For each author, merges its discovered outbox relays (from relay_map)
* with the bootstrap (upstream) relays from cfg, then inserts one
* caching_backfill_relay_progress row per relay (ON CONFLICT DO NOTHING).
* Safe to call repeatedly - existing rows are left untouched. */
static void init_relay_progress_for_all(const cr_config_t *cfg,
const cr_relay_map_t *relay_map,
const cr_pubkey_set_t *followed) {
if (!followed || followed->count <= 0) return;
/* Bootstrap relay URLs from cfg->upstream_relays. */
const char **bootstrap = NULL;
int bootstrap_n = cfg->upstream_count;
if (bootstrap_n > 0) {
bootstrap = malloc(bootstrap_n * sizeof(char *));
if (!bootstrap) return;
for (int i = 0; i < bootstrap_n; i++)
bootstrap[i] = cfg->upstream_relays[i];
}
for (int i = 0; i < followed->count; i++) {
const char *pk = followed->items[i];
if (!pk || !*pk) continue;
/* Collect this author's outbox relays. */
const cr_outbox_entry_t *oe = NULL;
int outbox_n = 0;
if (relay_map) {
oe = cr_relay_map_get_outbox(relay_map, pk);
if (oe) outbox_n = oe->relay_count;
}
int total = outbox_n + bootstrap_n;
if (total <= 0) continue;
const char **urls = malloc(total * sizeof(char *));
if (!urls) continue;
int n = 0;
if (oe) {
for (int j = 0; j < oe->relay_count && n < total; j++)
urls[n++] = oe->relays[j];
}
for (int j = 0; j < bootstrap_n && n < total; j++) {
int dup = 0;
for (int k = 0; k < n; k++) {
if (strcmp(urls[k], bootstrap[j]) == 0) { dup = 1; break; }
}
if (!dup) urls[n++] = bootstrap[j];
}
if (n > 0) {
pg_inbox_init_relay_progress_for_author(pk, urls, n);
}
free(urls);
}
free(bootstrap);
}
volatile sig_atomic_t g_shutdown = 0;
static volatile sig_atomic_t g_reload = 0;
@@ -175,10 +234,10 @@ int main(int argc, char **argv) {
if (restart) {
DEBUG_INFO("RESTART: resetting state to first-time startup");
cfg.state.backfilled_until = 0;
cfg.state.current_window_index = 0;
cfg.state.backfill_cursor = 0;
if (pg_conn) {
/* Clear per-author backfill progress so the next run starts fresh. */
/* Reset per-author backfill progress (caching_followed_pubkeys)
* so the next run starts fresh. The followed set itself is
* preserved; only cursor/completion state is reset. */
pg_inbox_reset_backfill_progress();
} else {
cr_config_save_state(&cfg);
@@ -242,6 +301,20 @@ int main(int argc, char **argv) {
return 1;
}
/* Sync the followed set to the DB so the backfill tick can iterate
* over it (per-author until-cursor drain). Also run the one-time
* migration from the old caching_backfill_progress table. */
if (pg_conn) {
pg_inbox_migrate_backfill_progress();
const char **root_pks = malloc(cfg.root_npub_count * sizeof(char *));
for (int i = 0; i < cfg.root_npub_count; i++)
root_pks[i] = cfg.root_hex[i];
pg_inbox_sync_followed_pubkeys((const char **)followed.items,
followed.count, root_pks,
cfg.root_npub_count);
free(root_pks);
}
if (g_shutdown) goto shutdown;
/* Discover outbox relays (NIP-65) and compute minimum covering set. */
@@ -287,6 +360,13 @@ int main(int argc, char **argv) {
free(statuses);
}
/* Create per-relay backfill progress rows for every followed pubkey
* (outbox relays + bootstrap relays). Existing rows are left untouched
* so cursor/completion state is preserved across restarts. */
if (pg_conn) {
init_relay_progress_for_all(&cfg, &relay_map, &followed);
}
/* Open live subscription. */
cr_live_t live;
if (cfg.live.enabled) {
@@ -307,10 +387,11 @@ int main(int argc, char **argv) {
/* Initial status heartbeat in PostgreSQL mode. */
if (pg_conn) {
int bf_complete = 0, bf_total = 0;
pg_inbox_count_backfill_progress(&bf_complete, &bf_total);
pg_inbox_update_status("starting", config_generation, (long)time(NULL),
followed.count, relay_map.selected_count, 0,
cfg.state.current_window_index,
cfg.state.backfill_cursor, 0, 0, NULL, 0);
bf_complete, bf_total, 0, 0, NULL, 0);
}
DEBUG_INFO("entering main loop");
@@ -322,7 +403,8 @@ int main(int argc, char **argv) {
cr_config_t newcfg;
int reload_ok = 0;
if (pg_conn) {
if (pg_config_load(&newcfg) == 0) reload_ok = 1;
if (pg_config_load(&newcfg) == 0 &&
cr_follow_decode_roots(&newcfg) == 0) reload_ok = 1;
} else if (config_path) {
if (cr_config_load(&newcfg, config_path) == 0) reload_ok = 1;
}
@@ -332,6 +414,9 @@ int main(int argc, char **argv) {
cr_config_free(&cfg);
cfg = newcfg;
DEBUG_INFO("config reloaded");
/* Force an immediate follow-graph refresh so any new
* root npubs are picked up right away. */
last_follow_refresh = 0;
} else {
DEBUG_ERROR("config reload failed, keeping old config");
}
@@ -346,14 +431,29 @@ int main(int argc, char **argv) {
config_generation, new_gen);
cr_config_t newcfg;
if (pg_config_load(&newcfg) == 0) {
newcfg.state = cfg.state;
cr_config_free(&cfg);
cfg = newcfg;
config_generation = new_gen;
DEBUG_INFO("config reloaded from PostgreSQL");
/* Resubscribe live with new config. */
if (cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
/* Decode root npubs to hex — pg_config_load fills
* root_npubs[] but NOT root_hex[] / root_hex_ready.
* Without this, cr_follow_is_root() returns 0 for
* everything and the new root's follows are never
* resolved. */
if (cr_follow_decode_roots(&newcfg) != 0) {
DEBUG_ERROR("config reload: failed to decode root npubs, keeping old config");
cr_config_free(&newcfg);
} else {
newcfg.state = cfg.state;
cr_config_free(&cfg);
cfg = newcfg;
config_generation = new_gen;
DEBUG_INFO("config reloaded from PostgreSQL");
/* Force an immediate follow-graph refresh so the
* new root npub's follows are picked up right
* away (instead of waiting up to
* follow_graph_refresh_seconds). */
last_follow_refresh = 0;
/* Resubscribe live with new config. */
if (cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
}
} else {
DEBUG_ERROR("PostgreSQL config reload failed, keeping old config");
@@ -373,6 +473,49 @@ int main(int argc, char **argv) {
int brc = cr_backfill_tick(&bf, &cfg, upstream, &followed, &sink, &relay_map);
(void)brc;
/* Immediate follow-graph refresh when a root npub publishes a new
* kind-3 contact list (detected by the live subscriber). */
if (cfg.live.enabled && live.follow_graph_changed) {
live.follow_graph_changed = 0;
DEBUG_INFO("live: admin kind-3 detected, refreshing follow graph immediately");
cr_pubkey_set_t new_followed;
cr_pubkey_set_init(&new_followed);
if (cr_follow_resolve(&cfg, upstream, &new_followed) == 0) {
int changed = (new_followed.count != followed.count);
if (!changed) {
for (int i = 0; i < followed.count; i++) {
if (!cr_pubkey_set_contains(&new_followed, followed.items[i])) {
changed = 1; break;
}
}
}
cr_pubkey_set_free(&followed);
followed = new_followed;
if (pg_conn) {
const char **root_pks = malloc(cfg.root_npub_count * sizeof(char *));
for (int i = 0; i < cfg.root_npub_count; i++)
root_pks[i] = cfg.root_hex[i];
pg_inbox_sync_followed_pubkeys((const char **)followed.items,
followed.count, root_pks,
cfg.root_npub_count);
free(root_pks);
/* Create relay progress rows for newly added follows.
* Existing rows are preserved (ON CONFLICT DO NOTHING). */
init_relay_progress_for_all(&cfg, &relay_map, &followed);
}
if (changed && cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
/* Reset the backfill in_progress flag in case we had entered
* steady-state; new follows need to be drained. */
bf.in_progress = 1;
last_follow_refresh = time(NULL);
} else {
DEBUG_WARN("immediate follow graph refresh failed");
cr_pubkey_set_free(&new_followed);
}
}
/* Periodic follow-graph refresh. */
time_t now = time(NULL);
if (cfg.follow_graph_refresh_seconds > 0 &&
@@ -392,9 +535,30 @@ int main(int argc, char **argv) {
}
cr_pubkey_set_free(&followed);
followed = new_followed;
/* Sync the refreshed followed set to the DB so newly added
* follows get backfilled and dropped follows stop. Existing
* cursor/completion state is preserved (upsert only touches
* is_root and last_seen). */
if (pg_conn) {
const char **root_pks = malloc(cfg.root_npub_count * sizeof(char *));
for (int i = 0; i < cfg.root_npub_count; i++)
root_pks[i] = cfg.root_hex[i];
pg_inbox_sync_followed_pubkeys((const char **)followed.items,
followed.count, root_pks,
cfg.root_npub_count);
free(root_pks);
/* Create relay progress rows for newly added follows.
* Existing rows are preserved (ON CONFLICT DO NOTHING). */
init_relay_progress_for_all(&cfg, &relay_map, &followed);
}
if (changed && cfg.live.enabled) {
cr_live_resubscribe(&live, &cfg, upstream, &followed, &sink);
}
/* Reset the backfill in_progress flag in case we had entered
* steady-state; new follows need to be drained. The backfill
* tick will quickly re-enter steady-state if there are no
* incomplete authors, so this is cheap. */
bf.in_progress = 1;
} else {
DEBUG_WARN("follow graph refresh failed");
cr_pubkey_set_free(&new_followed);
@@ -429,10 +593,11 @@ int main(int argc, char **argv) {
free(listed);
free(statuses);
}
int bf_complete = 0, bf_total = 0;
pg_inbox_count_backfill_progress(&bf_complete, &bf_total);
pg_inbox_update_status("running", config_generation, (long)now,
followed.count, relay_map.selected_count,
connected, cfg.state.current_window_index,
cfg.state.backfill_cursor,
connected, bf_complete, bf_total,
events_fetched, inbox_inserts, NULL, 0);
last_status_heartbeat = now;
}
@@ -444,10 +609,11 @@ int main(int argc, char **argv) {
cr_live_close(&live);
if (!pg_conn) cr_config_save_state(&cfg);
if (pg_conn) {
int bf_complete = 0, bf_total = 0;
pg_inbox_count_backfill_progress(&bf_complete, &bf_total);
pg_inbox_update_status("stopped", config_generation, (long)time(NULL),
followed.count, relay_map.selected_count, 0,
cfg.state.current_window_index,
cfg.state.backfill_cursor,
bf_complete, bf_total,
live.events_received + bf.events_total,
sink.published_ok, NULL, 0);
}
caching/src/pg_config.c
+5 -23
View File
@@ -166,49 +166,31 @@ int pg_config_load(cr_config_t *cfg) {
/* backfill */
cfg->backfill.enabled = get_bool("caching_backfill_enabled", 1);
char *windows = get_key("caching_backfill_windows");
split_csv_long(windows, cfg->backfill.window_schedule_seconds,
CR_MAX_WINDOWS, &cfg->backfill.window_count);
free(windows);
cfg->backfill.events_per_tick =
get_int("caching_backfill_page_size", 50);
get_int("caching_backfill_page_size", 500);
/* tick interval is stored in ms in the config table; convert to seconds. */
int tick_ms = get_int("caching_backfill_tick_interval_ms", 5000);
cfg->backfill.tick_interval_seconds = (tick_ms + 999) / 1000;
if (cfg->backfill.tick_interval_seconds < 1)
cfg->backfill.tick_interval_seconds = 1;
cfg->backfill.window_cooldown_seconds =
get_int("caching_backfill_window_cooldown_seconds", 60);
/* follow graph refresh */
cfg->follow_graph_refresh_seconds =
get_int("caching_follow_graph_refresh_seconds", 600);
/* Defaults if missing (mirror cr_config_load). */
if (cfg->backfill.window_count == 0) {
static const long def_windows[] = {86400, 604800, 2592000, 7776000, 31536000};
int n = (int)(sizeof(def_windows) / sizeof(def_windows[0]));
if (n > CR_MAX_WINDOWS) n = CR_MAX_WINDOWS;
for (int i = 0; i < n; i++)
cfg->backfill.window_schedule_seconds[i] = def_windows[i];
cfg->backfill.window_count = n;
}
/* Defaults if missing. */
if (cfg->backfill.events_per_tick == 0)
cfg->backfill.events_per_tick = 50;
cfg->backfill.events_per_tick = 500;
if (cfg->backfill.tick_interval_seconds == 0)
cfg->backfill.tick_interval_seconds = 5;
if (cfg->backfill.window_cooldown_seconds == 0)
cfg->backfill.window_cooldown_seconds = 60;
if (cfg->live.resubscribe_interval_seconds == 0)
cfg->live.resubscribe_interval_seconds = 300;
if (cfg->follow_graph_refresh_seconds == 0)
cfg->follow_graph_refresh_seconds = 600;
/* State: initialize to defaults for this phase. Phase 7 will read
* caching_backfill_progress / caching_service_state. */
/* State: per-author backfill progress lives in caching_followed_pubkeys.
* backfilled_until is only used as a first-time flag in legacy mode. */
cfg->state.backfilled_until = 0;
cfg->state.current_window_index = 0;
cfg->state.backfill_cursor = 0;
/* Validation. */
if (cfg->root_npub_count == 0) {
caching/src/pg_inbox.c
+803 -189
View File
File diff suppressed because it is too large Load Diff
caching/src/pg_inbox.h
+92 -28
View File
@@ -38,8 +38,8 @@ int pg_inbox_update_status(const char *service_state,
int followed_author_count,
int selected_relay_count,
int connected_relay_count,
int current_window_index,
int backfill_cursor,
int backfill_authors_complete,
int backfill_authors_total,
long events_fetched,
long inbox_inserts,
const char *last_error,
@@ -54,35 +54,99 @@ char *pg_inbox_get_config_value(const char *key);
long pg_inbox_get_config_generation(void);
/* ------------------------------------------------------------------ */
/* Backfill progress persistence */
/* Per-author until-cursor drain backfill model */
/* ------------------------------------------------------------------ */
/* Save backfill progress for a specific author+window (UPSERT).
* Returns 0 on success, -1 on error. */
int pg_inbox_save_backfill_progress(const char *author_pubkey,
int window_index,
long window_anchor,
long until_cursor,
int complete);
/* Load backfill progress for a specific author+window.
* Returns 0 on success (found), -1 on error.
* Sets *out_until_cursor and *out_complete. If not found, sets *out_found = 0
* and returns 0. */
int pg_inbox_load_backfill_progress(const char *author_pubkey,
int window_index,
long *out_until_cursor,
int *out_complete,
int *out_found);
/* Load all incomplete backfill progress rows for a given window.
* Returns a cJSON array of objects with "author_pubkey", "until_cursor" and
* "complete" fields. Caller must cJSON_Delete the result.
* Returns NULL on error or empty. */
cJSON *pg_inbox_load_incomplete_progress(int window_index);
/* Reset all backfill progress (delete all rows).
/* Reset all backfill progress: set until_cursor=0 and backfill_complete=FALSE
* for all rows in caching_followed_pubkeys (the followed set itself is
* preserved). Used by the "Reset Backfill" API button and --restart.
* Returns 0 on success, -1 on error. */
int pg_inbox_reset_backfill_progress(void);
/* Upsert all followed pubkeys into caching_followed_pubkeys.
* For each pubkey: INSERT if new (until_cursor=0, backfill_complete=FALSE),
* or UPDATE is_root and last_seen if existing (until_cursor and
* backfill_complete are NOT touched). root_pubkeys marks which entries are
* root follows. Returns 0 on success, -1 on error. */
int pg_inbox_sync_followed_pubkeys(const char **pubkeys, int count,
const char **root_pubkeys, int root_count);
/* Pick the next incomplete author for backfill, round-robin.
* On success fills out_pk, out_until_cursor and out_is_root, advances
* *round_cursor, and returns 0. Returns -1 if no incomplete authors or
* on error. */
int pg_inbox_pick_next_backfill_author(char *out_pk, int pk_len,
long *out_until_cursor,
int *out_is_root,
int *round_cursor);
/* Update the cursor and completion state for an author.
* events_this_page is added to the running events_fetched counter.
* Returns 0 on success, -1 on error. */
int pg_inbox_update_backfill_progress(const char *pk, long until_cursor,
int complete, int events_this_page);
/* One-time migration from the old caching_backfill_progress table to the
* new caching_followed_pubkeys table. Only runs if the new table is empty
* and the old table has rows. Returns 0 on success (including no-op),
* -1 on error. */
int pg_inbox_migrate_backfill_progress(void);
/* Count complete/total authors for status reporting.
* Returns 0 on success, -1 on error. */
int pg_inbox_count_backfill_progress(int *out_complete, int *out_total);
/* ------------------------------------------------------------------ */
/* Per-relay backfill progress (relay-by-relay drain model) */
/* ------------------------------------------------------------------ */
/* Initialize relay progress rows for an author. Creates one row per relay
* URL with until_cursor=0, complete=FALSE. Skips rows that already exist
* (ON CONFLICT DO NOTHING). Returns 0 on success, -1 on error. */
int pg_inbox_init_relay_progress_for_author(const char *pk,
const char **relay_urls,
int relay_count);
/* Pick the next author that has at least one incomplete relay progress row.
* Round-robin via *round_cursor. On success fills out_pk and returns 0.
* Returns -1 if no incomplete authors or on error. */
int pg_inbox_pick_next_author_with_incomplete_relays(char *out_pk, int pk_len,
int *round_cursor);
/* Get incomplete relays for an author. Returns a cJSON array of
* {relay_url, until_cursor} objects (caller frees with cJSON_Delete).
* Returns NULL on error or if no incomplete relays. */
cJSON *pg_inbox_get_incomplete_relays(const char *pk);
/* Update relay progress: advance cursor and/or mark complete.
* events_this_page is added to the running events_fetched counter.
* last_status is a short string like "eose", "timeout", "error", or NULL
* to leave unchanged. Returns 0 on success, -1 on error. */
int pg_inbox_update_relay_progress(const char *pk, const char *relay_url,
long until_cursor, int complete,
int events_this_page,
const char *last_status);
/* Check if all relays for an author are complete. If so, mark the author
* as backfill_complete=TRUE in caching_followed_pubkeys.
* Returns 1 if marked complete, 0 if still incomplete, -1 on error. */
int pg_inbox_check_and_mark_author_complete(const char *pk);
/* ------------------------------------------------------------------ */
/* Active backfill target (for UI status display) */
/* ------------------------------------------------------------------ */
/* Set the currently-active backfill target (pubkey + relay being queried).
* Either parameter may be NULL/empty to clear. Creates the
* caching_backfill_active table if needed (single-row singleton).
* Returns 0 on success, -1 on error. */
int pg_inbox_set_active_target(const char *pubkey, const char *relay_url);
/* Read the active backfill target. Fills out_pubkey/out_relay with
* the current values (or empty strings if none). Buffers must be
* at least 65 and 256 bytes respectively.
* Returns 0 on success, -1 on error. */
int pg_inbox_get_active_target(char *out_pubkey, int pk_len,
char *out_relay, int relay_len);
#endif /* CACHING_RELAY_PG_INBOX_H */
deploy_lt.sh
+37
View File
@@ -8,8 +8,10 @@ set -euo pipefail
# Configuration
REMOTE_HOST="ubuntu@laantungir.net"
LOCAL_BINARY="build/c_relay_pg_static_x86_64"
LOCAL_CACHING_BINARY="build/caching_relay"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
REMOTE_CACHING_BINARY_PATH="/opt/c-relay-pg/caching_relay"
SERVICE_NAME="c-relay-pg"
LOCAL_SERVICE_FILE="systemd/c-relay.service"
@@ -35,6 +37,15 @@ echo "==> Uploading artifacts to $REMOTE_HOST"
scp "$LOCAL_BINARY" "$REMOTE_HOST:/tmp/c_relay_pg.tmp"
scp "$LOCAL_SERVICE_FILE" "$REMOTE_HOST:/tmp/c-relay-pg.service"
scp "$LOCAL_PG_SETUP_SCRIPT" "$REMOTE_HOST:/tmp/setup_postgres_18.sh"
# Upload the caching_relay binary if it was built locally. The caching
# service is NOT started by this deploy (the systemd unit does not pass
# --start-caching), but the binary is placed in the relay's WorkingDirectory
# so it is ready for when caching is enabled via event-based config.
if [ -f "$LOCAL_CACHING_BINARY" ]; then
scp "$LOCAL_CACHING_BINARY" "$REMOTE_HOST:/tmp/caching_relay.tmp"
else
echo "WARNING: caching_relay binary not found locally ($LOCAL_CACHING_BINARY) — skipping caching binary upload"
fi
echo "==> Running remote install/configuration"
ssh "$REMOTE_HOST" 'bash -s' <<'EOF'
@@ -44,6 +55,7 @@ SERVICE_NAME="c-relay-pg"
RELAY_USER="c-relay-pg"
REMOTE_BINARY_DIR="/usr/local/bin/c_relay_pg"
REMOTE_BINARY_PATH="/usr/local/bin/c_relay_pg/c_relay_pg"
REMOTE_CACHING_BINARY_PATH="/opt/c-relay-pg/caching_relay"
echo "[remote] Ensuring service user exists"
if ! id "$RELAY_USER" >/dev/null 2>&1; then
@@ -80,11 +92,36 @@ sudo mv /tmp/c_relay_pg.tmp "$REMOTE_BINARY_PATH"
sudo chown "$RELAY_USER:$RELAY_USER" "$REMOTE_BINARY_PATH"
sudo chmod +x "$REMOTE_BINARY_PATH"
echo "[remote] Installing caching_relay binary (if uploaded)"
if [ -f /tmp/caching_relay.tmp ]; then
sudo mv /tmp/caching_relay.tmp "$REMOTE_CACHING_BINARY_PATH"
sudo chown "$RELAY_USER:$RELAY_USER" "$REMOTE_CACHING_BINARY_PATH"
sudo chmod +x "$REMOTE_CACHING_BINARY_PATH"
echo " -> caching_relay installed at $REMOTE_CACHING_BINARY_PATH (not started; enable via config when ready)"
else
echo " -> no caching_relay binary uploaded, skipping"
fi
echo "[remote] Installing systemd unit"
sudo mv /tmp/c-relay-pg.service /etc/systemd/system/c-relay-pg.service
sudo chown root:root /etc/systemd/system/c-relay-pg.service
sudo chmod 644 /etc/systemd/system/c-relay-pg.service
echo "[remote] Killing stale caching_relay processes (safety)"
# The caching service is forked by the relay with setsid(), so it detaches
# from the relay's process group and survives when the relay is killed.
# Without this, every restart orphans the previous caching_relay child and
# a new relay forks another, leading to many stale processes with dead PG
# connections spamming errors. Kill them here so the new relay starts clean.
CACHING_PIDS=$(pgrep -f "caching_relay" || echo "")
if [ -n "$CACHING_PIDS" ]; then
echo " -> killing stale caching_relay PIDs: $CACHING_PIDS"
kill -9 $CACHING_PIDS 2>/dev/null || true
sleep 1
else
echo " -> no stale caching_relay processes found"
fi
echo "[remote] Reloading and restarting service"
sudo systemctl daemon-reload
sudo systemctl enable "$SERVICE_NAME"
examples/deployment/nginx-proxy/nginx.conf
+28
View File
@@ -155,6 +155,34 @@ http {
log_not_found off;
}
# PHP admin page for caching service (direct PostgreSQL access).
# Bypasses the NIP-44 64KB encryption limit of the Nostr admin API.
# Requires: php-fpm + php-pgsql installed and configured.
location /admin/ {
alias /opt/c-relay-pg/admin/;
index index.php;
# HTTP Basic Auth
auth_basic "Relay Admin";
auth_basic_user_file /opt/c-relay-pg/admin/.htpasswd;
# Pass .php files to PHP-FPM
location ~ \.php$ {
# Adjust socket path for your distro:
# Debian/Ubuntu: unix:/run/php/php8.2-fpm.sock
# RHEL/Fedora: unix:/run/php-fpm/www.sock
fastcgi_pass unix:/run/php/php8.2-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $request_filename;
}
# Deny access to the lib/ directory (contains DB credentials)
location ^~ /admin/lib/ {
deny all;
}
}
# Optional: Metrics endpoint (if implemented)
location /metrics {
proxy_pass http://c_relay_pg_backend/metrics;
make_and_restart_relay.sh
+26 -2
View File
@@ -494,6 +494,21 @@ else
echo "No existing relay processes found"
fi
# Kill any stale caching_relay processes from previous runs.
# The caching service is forked by the relay with setsid(), so it detaches
# from the relay's process group and survives when the relay is killed -9.
# Without this, every restart orphans the previous caching_relay child and
# a new relay forks another, leading to many stale processes with dead PG
# connections spamming errors. Kill them here so the new relay starts fresh.
CACHING_PIDS=$(pgrep -f "caching_relay" || echo "")
if [ -n "$CACHING_PIDS" ]; then
echo "Killing stale caching_relay processes: $CACHING_PIDS"
kill -9 $CACHING_PIDS 2>/dev/null || true
sleep 1
else
echo "No existing caching_relay processes found"
fi
# Ensure port 8888 is completely free with retry loop
echo "Ensuring port 8888 is available..."
for attempt in {1..15}; do
@@ -524,14 +539,21 @@ for attempt in {1..15}; do
fi
done
# Final safety check - ensure no relay processes remain
# Final safety check - ensure no relay or caching processes remain
FINAL_PIDS=$(pgrep -f "c_relay_pg_" || echo "")
if [ -n "$FINAL_PIDS" ]; then
echo "Final cleanup: killing processes $FINAL_PIDS"
echo "Final cleanup: killing relay processes $FINAL_PIDS"
kill -9 $FINAL_PIDS 2>/dev/null || true
sleep 1
fi
FINAL_CACHING_PIDS=$(pgrep -f "caching_relay" || echo "")
if [ -n "$FINAL_CACHING_PIDS" ]; then
echo "Final cleanup: killing stale caching_relay processes $FINAL_CACHING_PIDS"
kill -9 $FINAL_CACHING_PIDS 2>/dev/null || true
sleep 1
fi
# Clean up PID file
rm -f relay.pid
@@ -682,8 +704,10 @@ if ps -p "$RELAY_PID" >/dev/null 2>&1; then
echo "Configuration: Event-based (kind 33334 Nostr events)"
echo "Database: Automatically created with relay pubkey naming"
echo "To kill relay: pkill -f 'c_relay_pg_'"
echo "To kill caching service: pkill -f 'caching_relay'"
echo "To check status: ps aux | grep c_relay_pg_"
echo "To view logs: tail -f relay.log"
echo "To view caching logs: tail -f build/caching_relay.log"
echo "Binary: $BINARY_PATH (zero configuration needed)"
echo "Ready for Nostr client connections!"
else
nostr_core_lib
-1
Submodule nostr_core_lib deleted from 61581afcc9
plans/backfill_redesign_plan.md
+368
View File
@@ -0,0 +1,368 @@
# Backfill Redesign: Per-Author Until-Cursor Drain (Option B)
## Problem
The current window-expansion backfill (`caching/src/backfill.c`) has four
compounding structural defects that make cache coverage unacceptably poor
(~10-28% of upstream availability even for a 7-pubkey follow set):
1. **Overlapping windows re-fetch the same events.** Window N+1's time range
fully contains window N's range. The only dedup is a 4096-entry in-memory
ring + inbox `ON CONFLICT DO NOTHING`, which overflows for high-volume
authors, wasting query budget on duplicates.
2. **Broken saturation detection.** When a relay caps at 500 events, the code
retries with `limit=5000`, still gets 500, compares `500 == 5000` (false),
and incorrectly concludes the page was not saturated — marking the author
complete without paginating.
3. **`until`-cursor walk-back never triggers** because it's gated on the
broken saturation check.
4. **Window 4 (full history) was skipped for 5/7 pubkeys** and the loop won't
retry once `current_window_index` advances past the last window.
## Solution
Replace the window-expansion model with a **per-author `until`-cursor drain**:
each followed author has a single persistent cursor (starting at `now`). Each
backfill tick queries `since=0&until=cursor&limit=500` for one author,
publishes the page, and sets `cursor = oldest_event_created_at - 1`. Repeat
until a page returns fewer than the relay's cap (beginning of history reached).
The live subscriber stays as-is (it's already correct).
## Architecture
```mermaid
flowchart TD
A[cr_follow_resolve] --> B[caching_followed_pubkeys table]
B --> C[Backfill tick: pick next incomplete author]
C --> D[Query since=0 and until=cursor and limit=500]
D --> E{ev_count == relay_cap?}
E -- yes --> F[Publish page, set cursor = oldest - 1]
F --> D
E -- no < relay_cap --> G[Publish page, mark backfill_complete = true]
G --> H{More incomplete authors?}
H -- yes --> C
H -- no --> I[Steady state: live sub only]
J[Live subscriber] --> K[caching_event_inbox]
K --> L[c-relay-pg inbox poller]
L --> M[events table]
```
## Changes
### 1. New table: `caching_followed_pubkeys`
Replaces both the ephemeral in-memory `cr_pubkey_set_t` (for persistence) and
the window-coupled `caching_backfill_progress` (for cursor state).
**Schema** (add to `src/pg_schema.sql`):
```sql
CREATE TABLE IF NOT EXISTS caching_followed_pubkeys (
pubkey TEXT PRIMARY KEY,
is_root BOOLEAN NOT NULL DEFAULT FALSE,
until_cursor BIGINT NOT NULL DEFAULT 0,
backfill_complete BOOLEAN NOT NULL DEFAULT FALSE,
events_fetched INTEGER NOT NULL DEFAULT 0,
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
last_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX IF NOT EXISTS idx_caching_followed_incomplete
ON caching_followed_pubkeys(backfill_complete, last_seen)
WHERE backfill_complete = FALSE;
```
- `until_cursor`: the `until` timestamp for the next backfill query. `0` means
"not started yet" (treat as `now` on first query). After each page, set to
`oldest_event_created_at - 1`.
- `backfill_complete`: set true when a page returns fewer than the relay cap
(we've reached the beginning of the author's history).
- `is_root`: true for root/admin npubs (used to apply `admin_kinds`).
- `events_fetched`: cumulative count for diagnostics.
- `last_seen`: updated each follow-graph refresh (prune stale follows).
**Migration**: On startup, if `caching_followed_pubkeys` is empty but
`caching_backfill_progress` has rows, migrate: for each distinct
`author_pubkey` in the old table, insert a row with `until_cursor = 0`,
`backfill_complete = FALSE` (re-backfill from scratch — the old data was
incomplete anyway). Drop `caching_backfill_progress` after migration.
### 2. Follow graph resolver: persist to `caching_followed_pubkeys`
**File**: `caching/src/follow_graph.c` + `caching/src/pg_inbox.c`
After `cr_follow_resolve()` builds the in-memory set, sync it to the table:
```c
/* For each pubkey in the followed set: */
/* INSERT INTO caching_followed_pubkeys (pubkey, is_root, last_seen)
* VALUES ($1, $2, now)
* ON CONFLICT (pubkey) DO UPDATE SET
* is_root = EXCLUDED.is_root,
* last_seen = EXCLUDED.last_seen;
*
* Mark follows not seen this refresh as stale (for pruning / diagnostics).
* Do NOT touch until_cursor or backfill_complete on existing rows. */
```
New function: `pg_inbox_sync_followed_pubkeys(cr_pubkey_set_t *followed,
cr_config_t *cfg)` — upserts all followed pubkeys, updates `last_seen`, sets
`is_root` based on `cr_follow_is_root()`.
New function: `pg_inbox_prune_stale_follows(long threshold_seconds)` —
optionally deletes follows not seen in `threshold_seconds` (e.g. 24h). This is
off by default; stale follows just stop getting backfilled.
The in-memory `cr_pubkey_set_t followed` stays as the runtime working set
(used by the live subscriber). The table is the durable source of truth for
backfill iteration.
### 3. Rewrite `cr_backfill_tick` — per-author until-cursor drain
**File**: `caching/src/backfill.c`
Replace the entire window-expansion logic. New `cr_backfill_tick`:
```c
int cr_backfill_tick(cr_backfill_t *bf, cr_config_t *cfg,
nostr_relay_pool_t *upstream,
cr_sink_t *sink, const cr_relay_map_t *relay_map) {
if (!cfg->backfill.enabled) return -2;
/* Throttle: one query per tick_interval. */
time_t now = time(NULL);
if (bf->last_tick && (now - bf->last_tick) < cfg->backfill.tick_interval_seconds)
return 0;
bf->last_tick = now;
/* Pick the next incomplete author from the DB (round-robin via
* last_seen ordering, or a cursor we persist in bf). */
char pk[CR_HEX_LEN];
long until_cursor = 0;
int is_root = 0;
if (pg_inbox_pick_next_backfill_author(pk, &until_cursor, &is_root,
&bf->author_round_cursor) != 0) {
/* No incomplete authors — steady state. */
bf->in_progress = 0;
return -2;
}
/* First-time: cursor 0 means start from now. */
if (until_cursor == 0) until_cursor = (long)now;
int page_size = 500; /* matches typical relay cap */
int ev_count = 0;
cJSON **events = query_author(upstream, relay_map, cfg, pk,
0 /* since=0 full history */,
until_cursor, page_size, &ev_count);
if (!events || ev_count == 0) {
/* No events at all (or no more) — author is complete. */
pg_inbox_update_backfill_progress(pk, 0, 1 /* complete */, 0);
free(events);
return 1;
}
/* Find oldest timestamp BEFORE publishing. */
long oldest_ts = 0;
find_oldest_created_at(events, ev_count, &oldest_ts);
/* Publish all events. */
cr_sink_set_source_class(sink, CR_SINK_CLASS_BACKFILL);
for (int k = 0; k < ev_count; k++) {
cr_sink_publish(sink, events[k]);
cJSON_Delete(events[k]);
}
free(events);
bf->events_total += ev_count;
/* Saturation check: if we got exactly page_size, the page may be
* truncated. Set cursor = oldest - 1 and keep going next tick.
* If we got fewer than page_size, we've reached the beginning. */
if (ev_count >= page_size) {
/* Saturated — more history likely exists. Advance cursor. */
long next_cursor = (oldest_ts > 0) ? oldest_ts - 1 : 0;
pg_inbox_update_backfill_progress(pk, next_cursor, 0 /* not complete */,
ev_count);
} else {
/* Not saturated — beginning of history reached. */
pg_inbox_update_backfill_progress(pk, until_cursor, 1 /* complete */,
ev_count);
}
return 1;
}
```
Key differences from the old code:
- **No windows.** Single `since=0` query per author, always full history.
- **Saturation check compares against `page_size` (500)**, not `max_cap`
(5000). This fixes Defect 2 — if the relay returns 500, we paginate.
- **Cursor persisted per-author** in `caching_followed_pubkeys.until_cursor`,
not in a window-coupled table. This fixes Defect 4 — no window to skip.
- **No overlapping queries.** Each event is fetched exactly once as the cursor
walks backwards. This fixes Defect 1.
- **`pg_inbox_pick_next_backfill_author`** selects the next
`backfill_complete = FALSE` author, round-robin, so all authors get fair
backfill time.
### 4. New PG inbox functions
**File**: `caching/src/pg_inbox.c` + `caching/src/pg_inbox.h`
```c
/* Sync the followed set to the DB (upsert + update last_seen).
* Does NOT touch until_cursor or backfill_complete on existing rows. */
int pg_inbox_sync_followed_pubkeys(const cr_pubkey_set_t *followed,
const cr_config_t *cfg);
/* Pick the next incomplete author for backfill (round-robin).
* Sets out_pk, out_until_cursor, out_is_root.
* Returns 0 on success, -1 if no incomplete authors remain. */
int pg_inbox_pick_next_backfill_author(char *out_pk,
long *out_until_cursor,
int *out_is_root,
int *round_cursor);
/* Update backfill progress for an author. */
int pg_inbox_update_backfill_progress(const char *pk,
long until_cursor,
int complete,
int events_this_page);
/* Migrate old caching_backfill_progress rows to caching_followed_pubkeys.
* Called once on startup if the new table is empty and the old one has rows. */
int pg_inbox_migrate_backfill_progress(void);
/* Reset all backfill progress (set until_cursor=0, backfill_complete=FALSE
* for all followed pubkeys). Used by the "Reset Backfill" API button. */
int pg_inbox_reset_backfill_progress(void);
```
### 5. Simplify `cr_backfill_t` and `cr_config_t` state
**File**: `caching/src/backfill.h`, `caching/src/config.h`
Remove from `cr_backfill_t`:
- `window_index`, `current_window_s`, `events_this_window`, `window_started`
- `author_complete`, `blocked`, `blocked_until_ts`
Keep:
- `cursor` (now: round-robin index into followed set, for fair scheduling)
- `until_cursor` (transient, per-tick — not persisted here, read from DB)
- `in_progress` (1 if any incomplete authors remain)
- `events_total` (cumulative diagnostic)
- `last_tick` (throttle)
Remove from `cr_config_t` state:
- `state.backfilled_until` (no longer meaningful)
- `state.current_window_index` (no windows)
Remove from `cr_config_t` backfill config:
- `backfill.window_schedule_seconds[]`, `backfill.window_count`
- `backfill.window_cooldown_seconds`
- `backfill.events_per_tick` → rename to `backfill.page_size` (default 500)
Keep:
- `backfill.enabled`
- `backfill.page_size` (was `events_per_tick`)
- `backfill.tick_interval_seconds`
### 6. Update `caching_service_state` table
**File**: `src/pg_schema.sql`
The `current_window_index` and `backfill_cursor` columns are no longer
meaningful. Replace with:
```sql
ALTER TABLE caching_service_state
DROP COLUMN IF EXISTS current_window_index,
DROP COLUMN IF EXISTS backfill_cursor,
ADD COLUMN IF NOT EXISTS backfill_authors_complete INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS backfill_authors_total INTEGER NOT NULL DEFAULT 0;
```
Update `pg_inbox_update_status()` to report
`backfill_authors_complete / backfill_authors_total` instead of
`current_window_index / backfill_cursor`. The API status panel shows this as a
progress bar (e.g. "Backfill: 3/7 authors complete").
### 7. Update main loop
**File**: `caching/src/main.c`
- Remove `advance_window` calls (no windows).
- After `cr_follow_resolve()`, call `pg_inbox_sync_followed_pubkeys()`.
- On startup, call `pg_inbox_migrate_backfill_progress()` if needed.
- `cr_backfill_init` just sets `in_progress = 1` (check if any incomplete
authors exist in the DB).
- The follow-graph refresh at [`main.c:378`](caching/src/main.c:378) now also
calls `pg_inbox_sync_followed_pubkeys()` so new follows get backfilled and
unfollowed pubkeys stop getting backfilled.
### 8. Update API status display
**File**: `api/index.js`, `src/config.c` (caching_status handler)
- Show "Backfill: X/Y authors complete" instead of "Window N, cursor M".
- The "Reset Backfill Progress" button now calls
`pg_inbox_reset_backfill_progress()` which sets
`until_cursor=0, backfill_complete=FALSE` for all rows in
`caching_followed_pubkeys`.
### 9. Config changes
**File**: `src/pg_schema.sql` (default config inserts), `caching/src/pg_config.c`
Remove config keys:
- `caching_backfill_windows` (no window schedule)
- `caching_backfill_window_cooldown_seconds` (no windows)
Add config key:
- `caching_backfill_page_size` already exists (value 500) — repurpose as the
per-query limit (was already used this way, just rename in docs).
Keep:
- `caching_backfill_enabled`
- `caching_backfill_tick_interval_ms` (throttle between queries)
- `caching_backfill_page_size` (per-query limit, default 500)
## Implementation Order
1. **Schema**: Add `caching_followed_pubkeys` table + alter
`caching_service_state` in `src/pg_schema.sql`.
2. **PG inbox functions**: Add `pg_inbox_sync_followed_pubkeys`,
`pg_inbox_pick_next_backfill_author`, `pg_inbox_update_backfill_progress`,
`pg_inbox_migrate_backfill_progress`, `pg_inbox_reset_backfill_progress`
in `caching/src/pg_inbox.c` + `.h`.
3. **Follow graph sync**: Call `pg_inbox_sync_followed_pubkeys()` after
`cr_follow_resolve()` in `caching/src/main.c`.
4. **Rewrite backfill**: Replace `cr_backfill_tick` in
`caching/src/backfill.c`. Simplify `cr_backfill_t` in `backfill.h`.
5. **Config cleanup**: Remove window config from `caching/src/pg_config.c`
and `caching/src/config.c`. Update defaults in `src/pg_schema.sql`.
6. **State reporting**: Update `pg_inbox_update_status()` in
`caching/src/pg_inbox.c` to report authors complete/total.
7. **API status**: Update `api/index.js` and `src/config.c` status handler
to show backfill author progress.
8. **Migration**: Implement `pg_inbox_migrate_backfill_progress()` and call
on startup.
9. **Test**: Rebuild, restart with `--reset-backfill`, verify 100% coverage
for all 7 followed pubkeys via `nak req` comparison against upstream.
## Risk and Mitigation
- **Relays that don't honor `until`**: If a relay ignores the `until` filter,
the cursor won't advance and we'll re-fetch the same page. Mitigation:
detect when `oldest_ts` doesn't decrease between consecutive pages for the
same author and mark the author complete (or skip to a different relay).
- **High-volume authors take many ticks**: At 500 events/tick and 5s
tick_interval, a 10,000-event author takes ~100s. This is acceptable — the
live subscriber keeps you current, and the backfill makes steady progress.
If faster backfill is needed later, Option C (parallel workers) can be added
on top of this design.
- **Follow graph churn**: If follows change frequently, the sync upsert is
O(N) per refresh (every 10 min). Fine for personal-scale (hundreds of
follows). For thousands, batch the upsert.
plans/backfill_relay_by_relay_plan.md
+307
View File
@@ -0,0 +1,307 @@
# Backfill v2: Relay-by-Relay Per-Author Until-Cursor Drain
## Overview
Refine the per-author until-cursor backfill to query each relay **one at a
time** with **per-relay cursor tracking**, and pick up new follows
**immediately** when the admin's kind-3 contact list changes.
## Problem with the current implementation
The current backfill (`caching/src/backfill.c`) calls `nostr_relay_pool_query_sync`
with **all relay URLs at once**. This function:
1. Sends the same REQ to all connected relays
2. Collects and deduplicates events from all relays into one merged set
3. Waits until ALL relays send EOSE, or the timeout (now 30s) expires
4. Returns the merged set — but **does not tell the caller whether it exited
via all-EOSE or timeout**
If one relay is slow (e.g. nos.lol streaming 500+ events), the timeout can
expire before that relay sends EOSE. The result is a partial event set, but
the backfill code can't distinguish "partial due to timeout" from "complete
because the author only has 225 events." This causes premature completion
marking.
Additionally, the current design has a single `until_cursor` per author in
`caching_followed_pubkeys`, which doesn't track per-relay progress. And new
follows are only discovered on the 10-minute follow-graph refresh cycle.
## Solution
### 1. Per-relay cursor tracking
Add a new table to track the backfill cursor for each author on each relay
independently:
```sql
CREATE TABLE IF NOT EXISTS caching_backfill_relay_progress (
author_pubkey TEXT NOT NULL,
relay_url TEXT NOT NULL,
until_cursor BIGINT NOT NULL DEFAULT 0,
complete BOOLEAN NOT NULL DEFAULT FALSE,
events_fetched INTEGER NOT NULL DEFAULT 0,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
PRIMARY KEY (author_pubkey, relay_url)
);
CREATE INDEX IF NOT EXISTS idx_caching_relay_progress_incomplete
ON caching_backfill_relay_progress(author_pubkey)
WHERE complete = FALSE;
```
- `until_cursor`: the `until` timestamp for the next query to this specific
relay for this author. `0` means "not started" (treat as `now`).
- `complete`: set true when this relay returns < page_size events for the
current cursor range (this relay is drained for this author).
- `events_fetched`: cumulative count for diagnostics.
The existing `caching_followed_pubkeys.backfill_complete` becomes a derived
value: an author is complete when all their relay progress rows are complete
(or when they have no relay progress rows at all — no relays to query).
### 2. Relay-by-relay querying
Instead of one `query_sync` call with N relays, make N `query_sync` calls
with 1 relay each. Each call waits for that single relay's EOSE (or timeout).
No merge/dedup needed — publish every event directly to the inbox; the
database handles deduplication (`ON CONFLICT DO NOTHING`).
### 3. Immediate new-follow detection
When the admin publishes a new kind-3 contact list, the live subscriber's
`on_event` callback receives it in real-time. We detect this and trigger an
immediate follow-graph refresh, which:
- Resolves the new follow set
- Syncs new pubkeys to `caching_followed_pubkeys`
- Creates relay progress rows for new follows (cursor=0, complete=false)
- The backfill loop picks them up on the next tick
## Algorithm
### Per backfill tick:
```
1. Pick next author that has at least one incomplete relay progress row
(round-robin from caching_followed_pubkeys WHERE backfill_complete=FALSE)
2. Get incomplete relays for this author:
SELECT relay_url, until_cursor FROM caching_backfill_relay_progress
WHERE author_pubkey = $1 AND complete = FALSE
3. For each incomplete relay (one at a time):
a. cursor = until_cursor (0 = now)
b. query_sync(pool, &relay_url, 1, filter, &ev_count, 30000)
filter = {authors: [pk], kinds: [configured], since: 0,
until: cursor, limit: 500}
c. Publish every event directly to inbox (cr_sink_publish)
d. Find oldest_ts in this batch
e. If ev_count == 0:
- Mark this relay complete for this author
(UPDATE caching_backfill_relay_progress SET complete=TRUE)
f. If ev_count >= page_size (saturated):
- Advance this relay's cursor: cursor = oldest_ts - 1
(UPDATE caching_backfill_relay_progress SET until_cursor = oldest-1)
g. If ev_count < page_size but > 0:
- This relay is drained for this cursor range
- Advance cursor to oldest_ts - 1 AND mark complete
(the next query at this lower cursor would return 0 anyway,
but we mark complete to avoid an extra round-trip)
4. After querying all incomplete relays for this author:
Check if ALL relays for this author are now complete:
SELECT COUNT(*) WHERE author_pubkey=$1 AND complete=FALSE
If 0: mark author complete in caching_followed_pubkeys
```
### Why per-relay cursors are better
- **Relay A has 500+ events, relay B has 50**: Relay A gets paginated across
multiple ticks (cursor walks back 500 at a time). Relay B is drained in one
query and marked complete. The author stays incomplete until relay A is
also drained. With a single shared cursor, we'd advance the cursor based on
the oldest across both relays, potentially skipping events on relay A.
- **Resume after restart**: each relay's cursor is persisted independently.
On restart, we resume each relay exactly where it left off.
- **Relay goes offline**: if a relay is unreachable, its progress row stays
incomplete. The author stays incomplete. When the relay comes back, we
resume querying it. No events are lost.
### Completion criteria
An author is marked `backfill_complete = TRUE` in `caching_followed_pubkeys`
when ALL their relay progress rows in `caching_backfill_relay_progress` are
marked `complete = TRUE`. This means every known relay has been queried down
to the beginning of the author's history.
### New follow detection
In `caching/src/live_subscriber.c`, the `live_on_event` callback currently
just publishes events. We add a check:
```c
static void live_on_event(cJSON *event, const char *relay_url, void *user_data) {
cr_live_ctx_t *ctx = (cr_live_ctx_t *)user_data;
ctx->live->events_received++;
cr_sink_publish(ctx->sink, event);
/* Detect admin kind-3 (contact list) changes → trigger follow refresh */
cJSON *kind = cJSON_GetObjectItem(event, "kind");
cJSON *pubkey = cJSON_GetObjectItem(event, "pubkey");
if (kind && cJSON_IsNumber(kind) && kind->valuedouble == 3 &&
pubkey && cJSON_IsString(pubkey)) {
if (cr_follow_is_root(ctx->cfg, cJSON_GetStringValue(pubkey))) {
ctx->live->follow_graph_changed = 1; /* signal to main loop */
}
}
}
```
The main loop checks this flag each iteration:
```c
if (live.follow_graph_changed) {
live.follow_graph_changed = 0;
/* Immediate follow-graph refresh */
cr_pubkey_set_t new_followed;
cr_pubkey_set_init(&new_followed);
if (cr_follow_resolve(&cfg, upstream, &new_followed) == 0) {
/* Sync to DB — new follows get relay progress rows automatically */
pg_inbox_sync_followed_pubkeys(...);
/* Create relay progress rows for new follows */
pg_inbox_init_relay_progress_for_new_follows(&new_followed, &relay_map);
/* Resubscribe live with new follow set */
cr_live_resubscribe(&live, &cfg, upstream, &new_followed, &sink);
cr_pubkey_set_free(&followed);
followed = new_followed;
}
}
```
### Relay progress row creation
When a new follow is discovered, we need to create relay progress rows for
it. The relays to query are determined by the relay discovery phase (NIP-65
outbox relays + bootstrap relays). New function:
```c
int pg_inbox_init_relay_progress_for_author(const char *pk,
const char **relay_urls,
int relay_count);
```
For each relay URL, insert a row with `until_cursor=0, complete=FALSE` (if
not already present). This is called:
- After follow-graph resolution for new follows
- After relay discovery for newly discovered outbox relays
## Database schema changes
### New table: `caching_backfill_relay_progress`
(see SQL above)
### `caching_followed_pubkeys` — unchanged
The `until_cursor` and `backfill_complete` columns remain. `until_cursor`
becomes less important (per-relay cursors are the source of truth), but we
keep it for backward compatibility and as a quick "has this author started"
flag. `backfill_complete` is updated when all relay progress rows are
complete.
## New PG inbox functions
### `caching/src/pg_inbox.c` + `.h`
```c
/* Initialize relay progress rows for an author. Creates one row per relay
* with until_cursor=0, complete=FALSE. Skips rows that already exist. */
int pg_inbox_init_relay_progress_for_author(const char *pk,
const char **relay_urls,
int relay_count);
/* Pick the next author that has at least one incomplete relay progress row.
* Round-robin via round_cursor. Returns 0 on success, -1 if none. */
int pg_inbox_pick_next_author_with_incomplete_relays(char *out_pk, int pk_len,
int *round_cursor);
/* Get incomplete relays for an author. Returns a JSON array of
* {relay_url, until_cursor} objects. Caller frees. */
cJSON *pg_inbox_get_incomplete_relays(const char *pk);
/* Update relay progress: advance cursor and/or mark complete. */
int pg_inbox_update_relay_progress(const char *pk, const char *relay_url,
long until_cursor, int complete,
int events_this_page);
/* Check if all relays for an author are complete. Returns 1 if all complete,
* 0 if some incomplete, -1 on error. */
int pg_inbox_check_author_relays_complete(const char *pk);
/* Mark author complete in caching_followed_pubkeys if all relay progress
* rows are complete. Called after each relay update. */
int pg_inbox_check_and_mark_author_complete(const char *pk);
```
## Changes
### 1. `src/pg_schema.sql` + `src/pg_schema.h`
Add the `caching_backfill_relay_progress` table and index.
### 2. `caching/src/pg_inbox.c` + `.h`
Add the new relay progress functions listed above.
### 3. `caching/src/backfill.c` — rewrite `cr_backfill_tick`
Replace the single `query_author` call with a relay-by-relay loop using
per-relay cursors from the database.
### 4. `caching/src/live_subscriber.c` — detect admin kind-3 changes
Add `follow_graph_changed` flag to `cr_live_t`. Set it in `live_on_event`
when a root npub publishes a kind-3.
### 5. `caching/src/live_subscriber.h` — add flag to struct
Add `int follow_graph_changed;` to `cr_live_t`.
### 6. `caching/src/main.c` — handle follow_graph_changed signal
Check `live.follow_graph_changed` each main loop iteration. If set, trigger
immediate follow-graph refresh + relay progress initialization for new
follows.
### 7. `caching/src/main.c` — create relay progress rows after relay discovery
After `cr_relay_discovery_run()`, call `pg_inbox_init_relay_progress_for_author`
for each followed pubkey with its discovered outbox relays + bootstrap relays.
## Implementation order
1. Schema: add `caching_backfill_relay_progress` table
2. PG inbox functions: relay progress CRUD
3. Backfill rewrite: relay-by-relay loop with per-relay cursors
4. Live subscriber: kind-3 detection + `follow_graph_changed` flag
5. Main loop: handle `follow_graph_changed`, create relay progress rows
6. Build and test
## Testing
1. Rebuild: `cd caching && make`
2. Reset backfill: restart relay with `--reset-backfill`
3. Monitor relay progress:
```sql
SELECT author_pubkey, relay_url, until_cursor, complete, events_fetched
FROM caching_backfill_relay_progress
ORDER BY author_pubkey, relay_url;
```
4. Monitor author completion:
```sql
SELECT pubkey, backfill_complete
FROM caching_followed_pubkeys ORDER BY pubkey;
```
5. Compare coverage: `nak req -k 1 -a <pk> ws://localhost:7777` vs upstream
6. Test new-follow detection: follow a new pubkey from the admin account,
verify it appears in `caching_followed_pubkeys` and gets backfilled within
seconds (not 10 minutes)
plans/caching_error_reporting_retry_limit_plan.md
+338
View File
@@ -0,0 +1,338 @@
# Caching Service: Descriptive Errors + Retry Limit
## Problem Summary
Two issues with the caching service backfill:
1. **Errors recorded as just "Error"** — When a relay fails, the
`caching_backfill_relay_progress.last_status` column stores the literal
string `"error"` with no detail. Investigation with `nak` showed the
relays currently flagged as `error` actually fail for **three different
reasons** that we conflate:
- TCP/connect timeout (e.g. `wss://relay.orly.dev`,
`wss://relay.snort.social` — "connection took too long")
- Query timeout (relay connected, REQ sent, but no EOSE within 30s —
e.g. `wss://nostr.wine`, `wss://nostr.land` which actually work fine
via `nak` but our 30s timeout is too aggressive)
- Connection refused / TLS handshake failure
Root cause: [`nostr_core_lib/nostr_core/core_relays.c:140-147`](../nostr_core_lib/nostr_core/core_relays.c:140)
invokes the progress callback with `status="error"`, `message=NULL`
for **all** failure paths (connect failure, REQ send failure). The
underlying [`nostr_ws_connect()`](../nostr_core_lib/nostr_websocket/nostr_websocket_openssl.c:146)
returns `NULL` with no error string surfaced. The caching callback in
[`caching/src/backfill.c:88`](../caching/src/backfill.c:88) just stores
`"error"` verbatim.
2. **No retry limit — infinite retry loop** —
[`caching/src/backfill.c:342-354`](../caching/src/backfill.c:342) keeps
`complete=false` when a relay errors with no events, so the next tick
[`pg_inbox_pick_next_author_with_incomplete_relays`](../caching/src/pg_inbox.c)
picks the same author+relay again. The schema
`caching_backfill_relay_progress` has no `attempt_count` /
`consecutive_errors` / `last_attempt_at` column, so a permanently-dead
relay (e.g. `relay.orly.dev`) is retried **every tick forever**. The DB
confirms this — error rows have `updated_at` bumped repeatedly within
the same minute.
## Goals
1. Surface **descriptive error messages** from the relay (or our end) all
the way to the `last_status` column so an operator can see *why* a
relay is failing.
2. Stop retrying a (author, relay) pair after **3 consecutive errors**,
marking it `complete=true` so backfill moves on. Log a `WARN`.
3. The existing **"Reset Backfill Progress"** button (UI +
`caching_reset_progress` admin command) must also clear the new error
counters so a manual re-run starts fresh. (Duplicates are deduped by
the inbox, so re-running is safe — this matches the user's mental
model: "once we cache everyone's back events we never have to do it
again, but if a relay was down we can re-run".)
## Architecture
```mermaid
flowchart LR
A[backfill_tick] --> B[nostr_ws_connect]
B -->|fail| C[core_relays callback]
C -->|status=error, msg=descriptive| D[backfill_query_callback]
D --> E[pg_inbox_update_relay_progress]
E --> F{consecutive_errors >= 3?}
F -->|yes| G[mark complete=true, WARN log]
F -->|no| H[keep incomplete, retry next tick]
I[Reset Backfill button] --> J[caching_reset_progress]
J --> K[clear consecutive_errors=0, complete=false, until_cursor=0]
```
## Error Message Priority Order
The `last_status` column should reflect the **most descriptive** failure
reason available, in this priority order:
1. **Relay's own response text** (highest priority) — when a relay sends
a `NOTICE` or `CLOSED` message with content like
`"auth-required: subscribers only"`, `"private relay"`,
`"rate-limited: please slow down"`, etc. This **already works today**:
[`core_relays.c:326-342`](../nostr_core_lib/nostr_core/core_relays.c:326)
captures the message and
[`backfill.c:75-82`](../caching/src/backfill.c:75) formats it as
`"NOTICE: <msg>"` / `"CLOSED: <msg>"` into `last_status`. No change
needed for this path.
**DB verification (2026-07-28):** Querying the live DB shows only
`eose` (49 rows) and `error` (6 rows) — **zero NOTICE/CLOSED rows**.
So the relay-response path is wired but not currently firing for the
followed set. The 6 `error` rows are all connection/timeout failures
(confirmed by `nak`: `relay.orly.dev` and `relay.snort.social`
genuinely time out at TCP; the others work via `nak` but hit our
30s query timeout). This means **the descriptive-transport-error
work in Steps 1-3 is what will actually populate `last_status` for
the errors we see today**. The NOTICE/CLOSED path remains as a
ready fallback for when a relay does respond with a restriction
message.
2. **Our connection/transport error** (this plan, Steps 1-3) — when the
relay never responds at all (TCP timeout, TLS handshake failure,
connection refused), surface a descriptive string like
`"error: connect failed: Connection refused"` instead of just
`"error"`.
3. **Generic fallback** — `"error"` / `"timeout"` only if no more
specific information is available.
So after this change, an operator querying the DB will see:
- `NOTICE: auth-required: please authenticate` (relay spoke, we listened)
- `CLOSED: rate-limited: too many requests` (relay spoke, we listened)
- `error: connect failed: Connection timed out` (relay never responded)
- `error: tls handshake failed` (relay never responded)
- `timeout` (relay connected but didn't send EOSE in 30s)
The relay's own words are always preferred over our transport-level
diagnosis.
## Implementation Plan
### Step 1 — Surface descriptive errors from `nostr_ws_connect`
**File:** [`nostr_core_lib/nostr_websocket/nostr_websocket_openssl.c`](../nostr_core_lib/nostr_websocket/nostr_websocket_openssl.c)
**File:** [`nostr_core_lib/nostr_websocket/nostr_websocket_tls.h`](../nostr_core_lib/nostr_websocket/nostr_websocket_tls.h)
Add a new variant that fills an error buffer:
```c
/* Connect and on failure write a short descriptive error into err_buf
* (at most err_buf_len-1 bytes, always null-terminated). err_buf may be
* NULL. Returns client handle or NULL. */
nostr_ws_client_t* nostr_ws_connect_with_error(const char* url,
char* err_buf,
size_t err_buf_len);
```
Implementation reuses the existing `nostr_ws_connect` body but, on each
early-return failure path, writes a descriptive string:
- URL parse failure → `"invalid url"`
- TCP connect failure → `"connect failed: <errno string>"`
- TLS handshake failure → `"tls handshake failed"`
- WebSocket handshake failure → `"ws handshake failed"`
- Memory allocation failure → `"out of memory"`
Keep `nostr_ws_connect()` as a thin wrapper that calls the new function
with `err_buf=NULL` (no behaviour change for existing callers).
### Step 2 — Pass descriptive error through `core_relays.c`
**File:** [`nostr_core_lib/nostr_core/core_relays.c`](../nostr_core_lib/nostr_core/core_relays.c)
In `synchronous_query_relays_with_progress` (lines 125-148), replace:
```c
relays[i].client = nostr_ws_connect(relays[i].url);
```
with:
```c
char connect_err[128] = {0};
relays[i].client = nostr_ws_connect_with_error(relays[i].url,
connect_err, sizeof(connect_err));
```
Then on the error paths, pass the descriptive string as the `message`
(4th) argument to the callback instead of `NULL`:
```c
if (callback) {
callback(relays[i].url, "error",
connect_err[0] ? connect_err : "connect failed",
0, relay_count, 0, user_data);
}
```
Also distinguish the REQ-send-failure path (line 138-142) with message
`"req send failed"`.
For the **timeout** path (line 184-189), the callback already sends
`status="timeout"` with `message=NULL` — leave that as-is (the caching
callback already records `"timeout"` distinctly, which is the right
semantic).
### Step 3 — Capture descriptive error in caching backfill callback
**File:** [`caching/src/backfill.c`](../caching/src/backfill.c)
Update `backfill_query_callback` (lines 58-94) so that when
`status == "error"`, the `event_id` parameter (which now carries the
descriptive message from Step 2) is included in `last_status`, the same
way NOTICE/CLOSED already are:
```c
if (strcmp(status, "error") == 0) {
ctx->got_eose = 0;
if (event_id && event_id[0] != '\0') {
snprintf(ctx->last_status, sizeof(ctx->last_status),
"error: %s", event_id);
} else {
snprintf(ctx->last_status, sizeof(ctx->last_status), "error");
}
}
```
This makes `last_status` store e.g. `"error: connect failed: Connection refused"`
or `"error: tls handshake failed"` instead of just `"error"`.
### Step 4 — Add error counter column to the schema
**File:** [`caching/src/pg_inbox.c`](../caching/src/pg_inbox.c)
In `pg_inbox_update_relay_progress` (around line 866, where
`last_status` is already ensured via `ALTER TABLE ... ADD COLUMN IF NOT
EXISTS`), add the same idempotent ALTER for the new column:
```c
"ALTER TABLE caching_backfill_relay_progress "
"ADD COLUMN IF NOT EXISTS consecutive_errors INTEGER NOT NULL DEFAULT 0"
```
Extend the UPDATE statement to bump `consecutive_errors` on error and
reset it to 0 on success (eose/timeout-with-events). The cleanest
approach: add a new parameter `int is_error` to
`pg_inbox_update_relay_progress` (or infer from `last_status` starting
with `"error"` — simpler, no signature change). We'll infer from
`last_status`:
```sql
UPDATE caching_backfill_relay_progress
SET until_cursor = $3,
complete = $4 OR (consecutive_errors >= 3),
events_fetched = events_fetched + $5,
last_status = CASE WHEN $6 = '' THEN last_status ELSE $6 END,
consecutive_errors = CASE
WHEN $6 LIKE 'error%' THEN consecutive_errors + 1
ELSE 0
END,
updated_at = EXTRACT(EPOCH FROM NOW())::BIGINT
WHERE author_pubkey = $1 AND relay_url = $2
```
The `complete = $4 OR (consecutive_errors >= 3)` clause auto-marks
complete once the threshold is hit, **without** requiring the caller to
know the count. This keeps the backfill.c logic simple.
> Note: `consecutive_errors >= 3` in the SET refers to the **pre-update**
> value (the value before this tick's increment). To mark complete on
> the *same* tick that the 3rd error is recorded, we want
> `consecutive_errors + 1 >= 3` when `$6 LIKE 'error%'`. So the clause
> becomes:
> ```sql
> complete = $4 OR ($6 LIKE 'error%' AND consecutive_errors + 1 >= 3)
> ```
### Step 5 — Log a WARN when a relay is auto-completed by error limit
**File:** [`caching/src/backfill.c`](../caching/src/backfill.c)
After each `pg_inbox_update_relay_progress` call in `cr_backfill_tick`,
check the returned status. The simplest approach: after the loop over
incomplete relays, re-query `pg_inbox_get_incomplete_relays` is not
needed — instead, in the per-relay branch where we currently log
`"complete"` vs `"incomplete (will retry)"` (lines 350-353 and 381-391),
detect the auto-complete case:
```c
int will_auto_complete = (strncmp(qctx.last_status, "error", 5) == 0);
/* The SQL will mark complete if consecutive_errors+1 >= 3. We can't
* see the count from here without an extra query, so log at WARN
* whenever an error path leaves the row incomplete — and at WARN
* when the row would be auto-completed. Simplest: just log WARN
* for every error result with the descriptive status. */
if (will_auto_complete) {
DEBUG_WARN("backfill: %s @ %s error: %s (will auto-complete after 3 consecutive)",
pk, relay_url, qctx.last_status);
}
```
Optionally, add a `pg_inbox_get_relay_error_count(pk, relay_url)` helper
to log the exact count. This is a nice-to-have, not required for the
fix.
### Step 6 — Reset error counters in "Reset Backfill"
**File:** [`caching/src/pg_inbox.c`](../caching/src/pg_inbox.c) — `pg_inbox_reset_backfill_progress`
The existing UPDATE on `caching_followed_pubkeys` is fine, but we also
need to reset the per-relay table. The relay-side admin command in
[`src/config.c:4411`](../src/config.c:4411) already does
`DELETE FROM caching_backfill_relay_progress` — which clears the new
column too (since the rows are gone). So **no change needed** here:
when the user clicks "Reset Backfill Progress", the relay progress
rows are deleted, and `cr_backfill_tick` will re-create them via
`pg_inbox_init_relay_progress_for_author` with
`consecutive_errors=0` (column default).
Verify this by re-reading
[`caching/src/pg_inbox.c`](../caching/src/pg_inbox.c)
`pg_inbox_init_relay_progress_for_author` — the INSERT must not
explicitly set `consecutive_errors` so the DEFAULT 0 applies. (If it
uses an explicit column list, add `consecutive_errors` omission
check.)
### Step 7 — Build and verify
1. Build nostr_core_lib: `cd nostr_core_lib && make`
2. Build caching: `cd caching && make`
3. Build relay: `./make_and_restart_relay.sh --start-caching --reset-backfill`
4. Watch `build/caching_relay.log` for `WARN` lines mentioning
`auto-complete after 3 consecutive`.
5. Query the DB:
```sql
SELECT author_pubkey, relay_url, complete, consecutive_errors, last_status
FROM caching_backfill_relay_progress
WHERE last_status LIKE 'error%'
ORDER BY consecutive_errors DESC;
```
Expect `last_status` values like `error: connect failed: ...` and
rows with `consecutive_errors >= 3` marked `complete=true`.
6. Manually verify a known-dead relay (`wss://relay.orly.dev`) hits the
limit and stops being retried (no more `updated_at` bumps on
subsequent ticks).
7. Click "Reset Backfill Progress" in the UI → confirm
`caching_backfill_relay_progress` is emptied and backfill restarts
from scratch.
## Files Touched
| File | Change |
|------|--------|
| `nostr_core_lib/nostr_websocket/nostr_websocket_tls.h` | Add `nostr_ws_connect_with_error()` decl |
| `nostr_core_lib/nostr_websocket/nostr_websocket_openssl.c` | Implement `nostr_ws_connect_with_error()`, make `nostr_ws_connect()` a wrapper |
| `nostr_core_lib/nostr_core/core_relays.c` | Use new connect variant, pass descriptive msg in error callback |
| `caching/src/backfill.c` | Capture descriptive msg in `last_status` for error status; WARN log on error |
| `caching/src/pg_inbox.c` | Add `consecutive_errors` column (idempotent ALTER); extend UPDATE to auto-complete at 3 |
## Out of Scope (deliberately)
- No exponential backoff — the user explicitly chose "3 errors then
stop". Backoff can be added later if needed.
- No change to the 30s query timeout — separate concern; could be
raised in a follow-up if `nostr.wine`/`nostr.land` keep timing out
despite actually being reachable.
- No new UI button — the existing "Reset Backfill Progress" button
already covers the "re-run the whole cache" use case.
plans/caching_follows_status_plan.md
+275
View File
@@ -0,0 +1,275 @@
# Caching Follows Status UI with Username Resolution
## Goal
Show a list of all followed/cached pubkeys with their **usernames**, **event counts by kind**, **outbox relays**, and **backfill status** in the relay admin web UI. Also add username resolution to the existing stats top-pubkeys table.
## Architecture Overview
```mermaid
graph LR
A[Admin Web UI] -->|system_command: caching_follows_status| B[relay config.c]
B -->|SQL query| C[(PostgreSQL)]
C -->|caching_followed_pubkeys| D[follows + backfill status]
C -->|events kind=0| E[metadata: name, picture, about]
C -->|events kind=10002| F[outbox relay URLs]
C -->|events GROUP BY kind| G[event counts per kind]
B -->|JSON response| A
```
The relay already has DB access and an admin command pipeline. We add one new admin command that joins the caching tables with the events table to produce a rich per-follow status response.
## Data Sources
All data is already in PostgreSQL — no new tables needed:
| Data | Source Table | Query |
|---|---|---|
| Followed pubkeys + backfill status | `caching_followed_pubkeys` | `SELECT pubkey, is_root, backfill_complete, events_fetched, first_seen, last_seen` |
| Per-relay backfill progress | `caching_backfill_relay_progress` | `SELECT relay_url, until_cursor, complete, events_fetched WHERE author_pubkey = $1` |
| Username / profile metadata | `events` (kind=0) | `SELECT content FROM events WHERE pubkey = $1 AND kind = 0 ORDER BY created_at DESC LIMIT 1` — content is JSON with `name`, `picture`, `about`, `nip05` |
| Outbox relays | `events` (kind=10002) | `SELECT content, tags FROM events WHERE pubkey = $1 AND kind = 10002 ORDER BY created_at DESC LIMIT 1` — tags contain `["r", "wss://..."]` entries |
| Event counts by kind | `events` | `SELECT kind, count(*) FROM events WHERE pubkey = $1 GROUP BY kind ORDER BY kind` |
## Implementation Steps
### Step 1: Username/Profile Resolution Helper
**File:** `src/db_ops_postgres.c` (and `src/db_ops.h` for the declaration)
Add a reusable function that, given a pubkey, returns the most recent kind-0 metadata as a JSON object with `name`, `picture`, `about`, `nip05`, `display_name` fields parsed from the event content.
```c
// Returns a cJSON object: {"name":"...", "picture":"...", "about":"...", "nip05":"...", "display_name":"..."}
// Returns NULL if no kind-0 event exists for this pubkey.
cJSON* db_get_profile_metadata(const char* pubkey);
```
**SQL:**
```sql
SELECT content FROM events
WHERE pubkey = $1 AND kind = 0
ORDER BY created_at DESC LIMIT 1
```
The `content` column is a JSON string (NIP-01 metadata). Parse it with cJSON and extract the fields. This function will be used by both the caching follows command and the stats top-pubkeys enrichment.
**Also add to SQLite backend** (`src/db_ops_sqlite.c`) for parity, though the primary target is PostgreSQL.
### Step 2: Outbox Relay Resolution Helper
**File:** `src/db_ops_postgres.c` (and `src/db_ops.h`)
Add a function that returns the outbox relay URLs from the most recent kind-10002 event for a pubkey.
```c
// Returns a cJSON array of relay URL strings: ["wss://relay1", "wss://relay2", ...]
// Returns NULL if no kind-10002 event exists.
cJSON* db_get_outbox_relays(const char* pubkey);
```
**SQL:**
```sql
SELECT tags FROM events
WHERE pubkey = $1 AND kind = 10002
ORDER BY created_at DESC LIMIT 1
```
The `tags` column is JSONB. Extract all `["r", "url"]` tag entries. In PostgreSQL this can be done in SQL:
```sql
SELECT tag->>1 AS relay_url
FROM events e,
jsonb_array_elements(e.tags) AS tag
WHERE e.pubkey = $1 AND e.kind = 10002
AND jsonb_typeof(tag) = 'array'
AND jsonb_array_length(tag) >= 2
AND tag->>0 = 'r'
ORDER BY e.created_at DESC
LIMIT 1
```
Actually, simpler: fetch the tags JSONB for the latest kind-10002 and parse in C with cJSON (consistent with how the rest of the codebase works).
### Step 3: New Admin Command `caching_follows_status`
**File:** `src/config.c` (in `handle_system_command_unified()`, near the existing `caching_status` handler at line 4128)
Add a new command `caching_follows_status` that:
1. Queries `caching_followed_pubkeys` for all followed pubkeys
2. For each pubkey, calls `db_get_profile_metadata()` to get the username
3. For each pubkey, queries event counts by kind: `SELECT kind, count(*) FROM events WHERE pubkey = $1 GROUP BY kind ORDER BY kind`
4. For each pubkey, calls `db_get_outbox_relays()` to get outbox relay list
5. For each pubkey, queries `caching_backfill_relay_progress` for per-relay status
6. Builds a JSON response:
```json
{
"command": "caching_follows_status",
"status": "success",
"timestamp": 1785234000,
"follows": [
{
"pubkey": "4d7842051782...",
"npub": "npub1f4uyypgh...",
"name": "username_or_null",
"picture": "url_or_null",
"is_root": false,
"backfill_complete": false,
"events_fetched": 6965,
"first_seen": 1785233973,
"last_seen": 1785233973,
"event_counts": [
{"kind": 0, "count": 1},
{"kind": 1, "count": 658},
{"kind": 6, "count": 1}
],
"total_events_in_db": 660,
"outbox_relays": [
"wss://nostr-01.yakihonne.com",
"wss://nostr.land",
"wss://relay.snort.social"
],
"relay_progress": [
{"relay_url": "wss://nos.lol", "until_cursor": 1782686796, "complete": false, "events_fetched": 1000},
{"relay_url": "wss://relay.damus.io", "until_cursor": 1785234229, "complete": true, "events_fetched": 0}
]
}
],
"total_follows": 8,
"total_events_in_db": 9843
}
```
**Performance consideration:** With up to 5000 followed pubkeys (`caching_max_followed_pubkeys`), running per-pubkey queries in a loop could be slow. Two approaches:
- **Option A (simple, fine for <100 follows):** Loop in C, call the helper functions per pubkey. With 8-50 follows this is fast enough.
- **Option B (scalable, for large follow sets):** Use a single SQL query with CTEs and LEFT JOINs to get everything in one round-trip. This is more complex but handles 5000 follows.
**Recommendation:** Start with Option A. If follow counts grow large, optimize to Option B later. The per-pubkey queries are all indexed (`idx_events_pubkey_kind`, `idx_events_pubkey`).
### Step 4: Frontend — Caching Follows Table
**File:** `api/index.html`
Add a new section to the caching page (below the existing config form and service/inbox status), containing a table:
```html
<div id="cachingFollowsSection">
<h3>Followed Pubkeys</h3>
<table id="caching-follows-table">
<thead>
<tr>
<th>Name</th>
<th>npub</th>
<th>Root?</th>
<th>Events in DB</th>
<th>Backfill</th>
<th>Outbox Relays</th>
<th>Event Counts by Kind</th>
<th>Last Seen</th>
</tr>
</thead>
<tbody id="caching-follows-table-body">
<tr><td colspan="8">Loading...</td></tr>
</tbody>
</table>
</div>
```
**File:** `api/index.js`
1. Add a `fetchCachingFollowsStatus()` function that sends `['system_command', 'caching_follows_status']` via the admin WebSocket.
2. Add a `handleCachingFollowsResponse(responseData)` function that renders the table:
- Each row shows: name (or "unknown" with a muted style), npub (as a clickable njump.me link), root badge, total events in DB, backfill status (complete/pending with relay count), outbox relays (comma-separated or expandable), event counts by kind (e.g. "k0:1, k1:658, k6:1"), last seen timestamp.
- Sort by total_events_in_db descending (most active first).
3. Hook into the existing caching page auto-refresh: in the `if (pageName === 'caching')` block at line 5445, also call `fetchCachingFollowsStatus()` and add it to the `cachingRefreshInterval` polling.
4. Add response routing in `handleSystemCommandResponse()` near line 2082:
```js
if (responseData.command === 'caching_follows_status') {
handleCachingFollowsResponse(responseData);
}
```
### Step 5: Frontend — Username in Stats Top-Pubkeys Table
**File:** `api/index.js`
Modify `populateStatsPubkeysFromMonitoring()` (line 4882) and `populateStatsPubkeys()` (line 4844) to add a "Name" column.
**Challenge:** The top-pubkeys monitoring data (`src/api.c:470` `query_top_pubkeys()`) currently returns only `{pubkey, event_count}`. To add usernames, we have two options:
- **Option A (backend enrichment):** Modify `query_top_pubkeys()` in `src/api.c` to also call `db_get_profile_metadata()` for each of the top 10 pubkeys and include `name` in the response. This is the cleanest approach — 10 extra queries on a 30-second polling cycle is negligible.
- **Option B (frontend enrichment):** The frontend separately queries kind-0 metadata for each pubkey. This is messier and adds latency.
**Recommendation:** Option A. Modify `query_top_pubkeys()` to include `name` and `picture` fields.
**Changes:**
1. `src/api.c:470` — in `query_top_pubkeys()`, after getting each pubkey row, call `db_get_profile_metadata(pubkey)` and add `name` to the pubkey object.
2. `api/index.js:4882` — in `populateStatsPubkeysFromMonitoring()`, add a "Name" column before the npub column. Display `pubkey.name || 'unknown'`.
3. `api/index.js:4844` — same change in `populateStatsPubkeys()`.
4. `api/index.html` — add a `<th>Name</th>` column to the top-pubkeys table header.
### Step 6: npub Conversion
The frontend already has `window.NostrTools.nip19.npubEncode()` available (used at `api/index.js:4902`). The backend `caching_follows_status` response should include both the hex pubkey and the npub for convenience, but the frontend can also convert. Including npub in the backend response is preferred since the backend has access to the same nostr library.
**File:** `src/config.c` — use `nostr_bytes_to_hex` / `nostr_decode_npub` from nostr_core_lib, or simply return the hex pubkey and let the frontend convert (it already does this in the stats page). **Recommendation: let the frontend convert** — it already has the pattern at line 4902, and it avoids adding nostr_core_lib dependency to the config.c command handler.
## File Change Summary
| File | Change |
|---|---|
| `src/db_ops.h` | Declare `db_get_profile_metadata()` and `db_get_outbox_relays()` |
| `src/db_ops_postgres.c` | Implement both functions (PostgreSQL) |
| `src/db_ops_sqlite.c` | Implement both functions (SQLite, for parity) |
| `src/db_ops.c` | Wire up dispatch for both functions |
| `src/config.c` | Add `caching_follows_status` command handler in `handle_system_command_unified()` |
| `src/api.c` | Enrich `query_top_pubkeys()` with `name` field |
| `api/index.html` | Add follows table to caching page; add Name column to stats top-pubkeys table |
| `api/index.js` | Add `fetchCachingFollowsStatus()`, `handleCachingFollowsResponse()`, route the response, add Name column to pubkey tables, hook into caching page auto-refresh |
## Mermaid: Data Flow
```mermaid
sequenceDiagram
participant UI as Admin Web UI
participant Relay as relay config.c
participant DB as PostgreSQL
participant Cache as caching_relay process
UI->>Relay: system_command: caching_follows_status
Relay->>DB: SELECT * FROM caching_followed_pubkeys
DB-->>Relay: 8 rows
loop each pubkey
Relay->>DB: SELECT content FROM events WHERE kind=0 AND pubkey=$1
DB-->>Relay: metadata JSON
Relay->>DB: SELECT kind,count(*) FROM events WHERE pubkey=$1 GROUP BY kind
DB-->>Relay: event counts
Relay->>DB: SELECT tags FROM events WHERE kind=10002 AND pubkey=$1
DB-->>Relay: outbox relay tags
Relay->>DB: SELECT * FROM caching_backfill_relay_progress WHERE author_pubkey=$1
DB-->>Relay: per-relay status
end
Relay-->>UI: JSON response with follows array
UI->>UI: render table with usernames, event counts, relays
```
## Edge Cases
- **No kind-0 metadata for a pubkey:** Show "unknown" in the name column with muted styling. The pubkey is still followed and being cached; we just don't have their profile yet. The backfill should eventually fetch their kind-0.
- **No kind-10002 for a pubkey:** Show "none" in the outbox relays column. The caching service falls back to bootstrap relays for these authors.
- **Large follow sets (5000):** The per-pubkey loop could be slow. Add a `LIMIT` parameter to the command (default 100, configurable). The UI can paginate or show "showing first 100 of 5000".
- **Kind-0 content is not valid JSON:** Parse defensively — if cJSON parsing fails, treat as no metadata.
- **Replaceable events:** Kind 0 is replaceable, so `ORDER BY created_at DESC LIMIT 1` gets the latest. The unique index `uq_events_replaceable_pubkey_kind` ensures only one kind-0 per pubkey is stored, so this is already handled at the DB level.
## Future Enhancements (not in this plan)
- Add/remove follows manually via admin commands (currently follows come from root npub's kind-3)
- Delete individual events via admin command
- Real-time backfill tick log streaming
- Per-follow detail view (click a row to see all events for that author)
- Search/filter the follows table by name or npub
plans/php_admin_page_plan.md
+305
View File
@@ -0,0 +1,305 @@
# PHP Admin Page for Caching Service
## Problem
The existing Nostr-based admin API (`caching_follows_status` kind-23456
command) encrypts its JSON response with NIP-44, which has a hard 64KB
plaintext limit. With 679+ followed pubkeys, the response exceeds this
limit by ~2-4×, causing `Encryption result code: -15`
(`NOSTR_ERROR_NIP44_BUFFER_TOO_SMALL`) and the UI shows
"Failed to load".
Rather than paginating the NIP-44 admin command (complex, still limited),
we build a **separate traditional admin page** using PHP + nginx that
queries PostgreSQL directly. This bypasses the NIP-44 limit entirely,
scales to thousands of follows, and runs in parallel with the existing
Nostr admin API (no changes to the relay binary needed).
## Architecture
```mermaid
flowchart LR
Browser[Admin Browser] -->|HTTPS| Nginx
Nginx -->|/admin/ *.php| PHP_FPM[PHP-FPM]
Nginx -->|/ and /api| Relay[C-Relay-PG :8888]
PHP_FPM -->|PDO| PostgreSQL[(PostgreSQL crelay)]
Relay -->|libpq| PostgreSQL
```
- **nginx** fronts everything on 443 (existing SSL setup).
- `/` and `/api` → proxy to C-Relay-PG (existing, unchanged)
- `/admin/` → PHP-FPM (new)
- **PHP-FPM** runs as a pool (e.g. `www` or a dedicated `relay-admin`
pool). PHP files live in `/opt/c-relay-pg/admin/` (or similar).
- **PDO** connects to PostgreSQL using the same `crelay` credentials
the relay uses. Read-only queries for display; write queries only for
config updates (with auth guard).
- **No relay restart needed** — this is purely nginx + PHP-FPM
configuration + static files.
## Tech Stack
- **PHP 8.x** with PDO PostgreSQL extension (`php-pgsql`)
- **PHP-FPM** managed by systemd (or the existing nginx PHP-FPM pool)
- **nginx** `location /admin/` block with `fastcgi_pass` to PHP-FPM
- **Frontend**: vanilla HTML + JS (no build step), same visual style as
the existing `api/index.html` admin page. Optionally use HTMX or
Alpine.js for lightweight interactivity, but vanilla JS is sufficient.
## Authentication
The PHP admin page needs its own auth since it's not going through the
Nostr admin command path. Options (in order of simplicity):
1. **HTTP Basic Auth** (nginx-level) — simplest, sufficient for a
single-admin relay. Add `auth_basic` to the `/admin/` location block
with a `.htpasswd` file.
2. **Session-based login** — PHP login form, password hash in a config
file, session cookie. More flexible but more code.
3. **Nostr login (NIP-07 extension)** — the page could verify a signed
auth event from the admin's browser extension, matching against the
`admin_pubkey` in the config table. Most aligned with the existing
system but most complex.
**Recommendation:** Start with HTTP Basic Auth (option 1) for the MVP,
upgrade to Nostr login later if desired.
## Database Access
PHP connects via PDO with the same credentials the relay uses:
```
host=localhost port=5432 dbname=crelay user=crelay password=crelay
```
Store the connection string in a PHP config file
(`/opt/c-relay-pg/admin/config.php`) that is NOT web-accessible (outside
the document root or protected by nginx).
## Pages / Endpoints
### 1. Dashboard (`/admin/index.php`)
Overview cards:
- Service state, heartbeat age, config generation (from
`caching_service_state`)
- Followed author count, selected relay count, connected relay count
- Events fetched, inbox inserts, inbox pending count
- Backfill progress: X/Y authors complete
- Active backfill target (pubkey + relay) from `caching_backfill_active`
- Error relays summary (count of rows with `consecutive_errors >= 3`)
### 2. Follows (`/admin/follows.php`)
The main table that was failing via NIP-44. Paginated server-side:
- Page size: 50 or 100 follows per page
- Columns: Name (from kind-0), npub, Root?, Events in DB, Backfill
Complete, Relays (expandable), Last Seen
- JOIN with `events` table for kind-0 profile metadata (name,
display_name, picture, nip05)
- JOIN with `caching_backfill_relay_progress` for per-relay status
- Filter: all / incomplete only / root only / error relays only
- Search by name or pubkey prefix
- Sort by: events_fetched desc, name, last_seen, backfill_complete
SQL (paginated):
```sql
SELECT fp.pubkey, fp.is_root, fp.backfill_complete, fp.events_fetched,
fp.first_seen, fp.last_seen,
e.content::json->>'name' AS name,
e.content::json->>'display_name' AS display_name,
e.content::json->>'picture' AS picture,
e.content::json->>'nip05' AS nip05,
(SELECT COUNT(*) FROM events WHERE pubkey = fp.pubkey) AS total_events
FROM caching_followed_pubkeys fp
LEFT JOIN LATERAL (
SELECT content FROM events
WHERE pubkey = fp.pubkey AND kind = 0
ORDER BY created_at DESC LIMIT 1
) e ON true
ORDER BY fp.is_root DESC, fp.events_fetched DESC
LIMIT $page_size OFFSET $offset;
```
### 3. Relay Progress (`/admin/relays.php`)
Per (author, relay) backfill status from
`caching_backfill_relay_progress`:
- Columns: Author (name + npub), Relay URL, Until Cursor, Complete,
Events Fetched, Consecutive Errors, Last Status, Updated At
- Filter: incomplete only / error relays / complete / all
- Highlight rows with `consecutive_errors >= 3`
- Show `last_status` (now descriptive: `error: tls/ws handshake failed`,
`eose`, `timeout`, etc.)
### 4. Config (`/admin/config.php`)
Read and edit caching config values from the `config` table:
- Read: `caching_root_npubs`, `caching_bootstrap_relays`,
`caching_kinds`, `caching_admin_kinds`, all `caching_*` keys
- Edit: form to update values, bumps `caching_config_generation` on
save (so the caching service hot-reloads)
- This replaces the "Apply Configuration" button for caching settings
### 5. Inbox (`/admin/inbox.php`)
Monitor the `caching_event_inbox` queue:
- Pending count by source_class (live / backfill / discovery)
- Oldest pending age
- Recent events preview (last 20, with kind + pubkey + source)
- This helps diagnose if the poller is keeping up
## File Structure
```
/opt/c-relay-pg/admin/ # PHP document root (or symlink)
├── config.php # DB connection config (NOT web-served)
├── index.php # Dashboard
├── follows.php # Followed pubkeys table (paginated)
├── relays.php # Per-relay backfill progress
├── config.php # Caching config editor
├── inbox.php # Inbox queue monitor
├── api/
│ ├── follows.php # JSON endpoint for follows (AJAX)
│ ├── relays.php # JSON endpoint for relay progress
│ ├── status.php # JSON endpoint for dashboard stats
│ └── config.php # JSON endpoint for config get/set
├── assets/
│ ├── style.css # Shared CSS (match existing admin theme)
│ └── app.js # Shared JS (pagination, search, refresh)
└── .htpasswd # HTTP Basic Auth passwords
```
In the repo, these live under `admin/` (new top-level directory,
parallel to `api/`).
## nginx Configuration
Add to the existing `server` block in
[`examples/deployment/nginx-proxy/nginx.conf`](../examples/deployment/nginx-proxy/nginx.conf):
```nginx
# PHP admin page for caching service (direct PostgreSQL access)
location /admin/ {
root /opt/c-relay-pg;
index index.php;
# HTTP Basic Auth
auth_basic "Relay Admin";
auth_basic_user_file /opt/c-relay-pg/admin/.htpasswd;
# Pass .php files to PHP-FPM
location ~ \.php$ {
fastcgi_pass unix:/run/php/php8.2-fpm.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
# Deny access to config.php (contains DB credentials)
location = /admin/config.php {
deny all;
}
}
```
## Implementation Plan
### Step 1 — Server setup (one-time)
- Install PHP 8.x + PHP-FPM + `php-pgsql` on the relay server
- Create `/opt/c-relay-pg/admin/` directory
- Configure PHP-FPM pool (or use default `www` pool)
- Add the nginx `location /admin/` block
- Create `.htpasswd` with `htpasswd` / `openssl passwd`
- Reload nginx
### Step 2 — PHP config + DB connection helper
- `admin/lib/db.php` — PDO connection helper, read-only by default
- `admin/config.php` — connection string (protected from web access)
- Test: `php -r "echo 'ok';"` + simple `SELECT 1` query
### Step 3 — Dashboard page (`index.php`)
- Query `caching_service_state` for overview stats
- Query `caching_backfill_active` for active target
- Query `caching_backfill_relay_progress` for error summary
- Display as cards + summary table
- Auto-refresh every 15s via JS `setInterval` + `fetch()`
### Step 4 — Follows page (`follows.php`)
- Server-side pagination (page param in URL)
- JOIN with `events` for kind-0 profile metadata
- Sub-query for per-pubkey event count
- Expandable relay progress per follow (AJAX load on click)
- Search + filter controls
- This is the page that replaces the broken NIP-44 `caching_follows_status`
### Step 5 — Relay progress page (`relays.php`)
- Full `caching_backfill_relay_progress` table with pagination
- Filter by complete/incomplete/error
- Show `last_status` and `consecutive_errors` columns
- Color-code: green=eose, yellow=timeout, red=error, gray=complete
### Step 6 — Config editor page (`config.php` → `admin/config-edit.php`)
- Read all `caching_*` config keys
- Form to edit `caching_root_npubs`, `caching_bootstrap_relays`,
`caching_kinds`, etc.
- On save: `UPDATE config SET value = $1 WHERE key = $2` + bump
`caching_config_generation`
- Confirmation dialog before applying
### Step 7 — Inbox monitor page (`inbox.php`)
- Pending counts by source_class
- Oldest pending age
- Recent 20 events preview
- Poller stats (from relay log or a new stats table)
### Step 8 — Styling + polish
- Match the existing `api/index.css` dark theme
- Responsive layout (works on mobile)
- Auto-refresh for dashboard and follows pages
## Security Considerations
1. **HTTP Basic Auth** on `/admin/` — minimum barrier
2. **`config.php` denied** via nginx `location` block (contains DB password)
3. **PDO prepared statements** everywhere — no SQL injection
4. **Read-only DB user** for display pages (optional: create a
`crelay_readonly` role with SELECT-only on caching tables; use the
full `crelay` user only for the config editor)
5. **HTTPS only** — the existing nginx SSL config covers this
6. **No CORS needed** — same origin as the relay
## Out of Scope (for MVP)
- Nostr NIP-07 extension login (use HTTP Basic Auth first)
- Write operations beyond config editing (reset backfill, etc. — those
stay on the Nostr admin API)
- Real-time WebSocket updates (use polling instead — 15s interval is
sufficient for a monitoring dashboard)
- Multi-user / role-based access
## Files to Create
| File | Purpose |
|------|---------|
| `admin/config.php` | DB connection config (protected) |
| `admin/lib/db.php` | PDO connection helper |
| `admin/lib/helpers.php` | Shared helpers (pagination, formatting) |
| `admin/index.php` | Dashboard |
| `admin/follows.php` | Followed pubkeys table (paginated) |
| `admin/relays.php` | Per-relay backfill progress |
| `admin/config-edit.php` | Caching config editor |
| `admin/inbox.php` | Inbox queue monitor |
| `admin/api/status.php` | JSON: dashboard stats |
| `admin/api/follows.php` | JSON: paginated follows |
| `admin/api/relays.php` | JSON: relay progress |
| `admin/assets/style.css` | Shared CSS |
| `admin/assets/app.js` | Shared JS |
| `admin/.htpasswd` | HTTP Basic Auth passwords |
| `examples/deployment/nginx-proxy/nginx.conf` | Add `/admin/` location block |
## Deployment
1. Copy `admin/` to `/opt/c-relay-pg/admin/` on the server
2. Install `php8.2-fpm php8.2-pgsql` (or distro-equivalent)
3. Create `.htpasswd`: `htpasswd -c /opt/c-relay-pg/admin/.htpasswd admin`
4. Edit `admin/config.php` with the PostgreSQL connection string
5. Add the nginx `/admin/` location block, reload nginx
6. Visit `https://relay.yourdomain.com/admin/`
No relay restart required. The PHP admin page runs entirely in parallel
with the existing relay and Nostr admin API.
relay.pid
+1 -1
View File
@@ -1 +1 @@
594859
3125854
src/api.c
+47 -3
View File
@@ -493,10 +493,34 @@ cJSON* query_top_pubkeys(void) {
cJSON* pubkey_item = cJSON_GetObjectItemCaseSensitive(row, "pubkey");
cJSON* count_item = cJSON_GetObjectItemCaseSensitive(row, "count");
cJSON_AddStringToObject(pubkey_obj, "pubkey",
cJSON_IsString(pubkey_item) ? pubkey_item->valuestring : "");
const char* pk_str = cJSON_IsString(pubkey_item) ? pubkey_item->valuestring : "";
cJSON_AddStringToObject(pubkey_obj, "pubkey", pk_str);
cJSON_AddNumberToObject(pubkey_obj, "event_count",
cJSON_IsNumber(count_item) ? count_item->valuedouble : 0.0);
// Enrich with profile name from kind-0 metadata.
if (pk_str[0] != '\0') {
cJSON* metadata = db_get_profile_metadata(pk_str);
if (metadata) {
cJSON* name = cJSON_GetObjectItemCaseSensitive(metadata, "name");
cJSON* display_name = cJSON_GetObjectItemCaseSensitive(metadata, "display_name");
const char* display = NULL;
if (display_name && cJSON_IsString(display_name) && display_name->valuestring[0])
display = display_name->valuestring;
else if (name && cJSON_IsString(name) && name->valuestring[0])
display = name->valuestring;
cJSON_AddStringToObject(pubkey_obj, "name", display ? display : "");
cJSON* picture = cJSON_GetObjectItemCaseSensitive(metadata, "picture");
if (picture && cJSON_IsString(picture) && picture->valuestring[0])
cJSON_AddStringToObject(pubkey_obj, "picture", picture->valuestring);
cJSON_Delete(metadata);
} else {
cJSON_AddStringToObject(pubkey_obj, "name", "");
}
} else {
cJSON_AddStringToObject(pubkey_obj, "name", "");
}
cJSON_AddItemToArray(pubkeys_array, pubkey_obj);
}
cJSON_Delete(rows);
@@ -1613,9 +1637,29 @@ char* generate_stats_json(void) {
double count_val = cJSON_IsNumber(count_item) ? count_item->valuedouble : 0.0;
double pct = (total_events > 0) ? ((count_val * 100.0) / (double)total_events) : 0.0;
cJSON_AddStringToObject(pubkey_obj, "pubkey", cJSON_IsString(pubkey_item) ? pubkey_item->valuestring : "");
const char* pk_str = cJSON_IsString(pubkey_item) ? pubkey_item->valuestring : "";
cJSON_AddStringToObject(pubkey_obj, "pubkey", pk_str);
cJSON_AddNumberToObject(pubkey_obj, "event_count", count_val);
cJSON_AddNumberToObject(pubkey_obj, "percentage", pct);
// Enrich with profile name from kind-0 metadata.
if (pk_str[0] != '\0') {
cJSON* metadata = db_get_profile_metadata(pk_str);
if (metadata) {
cJSON* name = cJSON_GetObjectItemCaseSensitive(metadata, "name");
cJSON* display_name = cJSON_GetObjectItemCaseSensitive(metadata, "display_name");
const char* display = NULL;
if (display_name && cJSON_IsString(display_name) && display_name->valuestring[0])
display = display_name->valuestring;
else if (name && cJSON_IsString(name) && name->valuestring[0])
display = name->valuestring;
cJSON_AddStringToObject(pubkey_obj, "name", display ? display : "");
cJSON_Delete(metadata);
} else {
cJSON_AddStringToObject(pubkey_obj, "name", "");
}
} else {
cJSON_AddStringToObject(pubkey_obj, "name", "");
}
cJSON_AddItemToArray(top_pubkeys, pubkey_obj);
}
cJSON_Delete(pubkey_rows);
src/caching_inbox_poller.c
+1 -1
View File
@@ -36,7 +36,7 @@
// ---- Defaults ----------------------------------------------------------------
#define DEFAULT_BATCH_SIZE 25
#define DEFAULT_BATCH_SIZE 150
#define DEFAULT_ACTIVE_POLL_MS 200
#define DEFAULT_IDLE_POLL_MS 5000
src/config.c
+196 -2
View File
@@ -4197,6 +4197,17 @@ int handle_system_command_unified(cJSON* event, const char* command, char* error
int svc_live = 0, svc_backfill = 0, svc_oldest = 0;
db_caching_inbox_pending_counts(&svc_live, &svc_backfill);
(void)svc_live; (void)svc_backfill; (void)svc_oldest;
// Backfill author progress (per-author until-cursor drain model).
// The caching_service_state table carries backfill_authors_complete
// and backfill_authors_total columns (altered in via pg_schema.sql).
// Surface them on the service object so the frontend can render
// "Backfill: X/Y authors complete".
int bf_complete = 0, bf_total = 0;
if (db_caching_backfill_author_counts(&bf_complete, &bf_total) == DB_OK) {
cJSON_AddNumberToObject(service_obj, "backfill_authors_complete", bf_complete);
cJSON_AddNumberToObject(service_obj, "backfill_authors_total", bf_total);
}
}
#endif
@@ -4218,6 +4229,174 @@ int handle_system_command_unified(cJSON* event, const char* command, char* error
snprintf(error_message, error_size, "failed to send caching status response");
return -1;
}
else if (strcmp(command, "caching_follows_status") == 0) {
// Build per-follow status response with usernames, event counts,
// outbox relays, and backfill progress.
cJSON* response = cJSON_CreateObject();
cJSON_AddStringToObject(response, "command", "caching_follows_status");
cJSON_AddStringToObject(response, "status", "success");
cJSON_AddNumberToObject(response, "timestamp", (double)time(NULL));
// Read the active backfill target from the caching_backfill_active
// table (written by the caching_relay process) so the UI can
// highlight which author/relay is currently being queried.
{
char active_pk[65] = {0};
char active_relay[256] = {0};
db_stmt_t* at_stmt = NULL;
if (db_prepare("SELECT active_pubkey, active_relay "
"FROM caching_backfill_active WHERE id = 1", &at_stmt) == DB_OK) {
if (db_step_stmt(at_stmt) == DB_ROW) {
const char* apk = db_column_text_value(at_stmt, 0);
const char* arl = db_column_text_value(at_stmt, 1);
cJSON_AddStringToObject(response, "active_backfill_pubkey",
apk ? apk : "");
cJSON_AddStringToObject(response, "active_backfill_relay",
arl ? arl : "");
}
db_finalize_stmt(at_stmt);
}
// If the table doesn't exist or query fails, just omit the fields.
}
cJSON* follows_array = cJSON_CreateArray();
long total_events_in_db = 0;
// Query all followed pubkeys from caching_followed_pubkeys.
db_stmt_t* stmt = NULL;
if (db_prepare("SELECT pubkey, is_root, backfill_complete, events_fetched, "
"first_seen, last_seen FROM caching_followed_pubkeys "
"ORDER BY is_root DESC, pubkey", &stmt) == DB_OK) {
while (db_step_stmt(stmt) == DB_ROW) {
const char* pk = db_column_text_value(stmt, 0);
if (!pk || pk[0] == '\0') continue;
cJSON* follow = cJSON_CreateObject();
cJSON_AddStringToObject(follow, "pubkey", pk);
// is_root (column 1) - boolean
const char* is_root_str = db_column_text_value(stmt, 1);
int is_root = (is_root_str && (is_root_str[0] == 't' || is_root_str[0] == 'T' || is_root_str[0] == '1'));
cJSON_AddBoolToObject(follow, "is_root", is_root);
// backfill_complete (column 2) - boolean
const char* bf_complete_str = db_column_text_value(stmt, 2);
int bf_complete = (bf_complete_str && (bf_complete_str[0] == 't' || bf_complete_str[0] == 'T' || bf_complete_str[0] == '1'));
cJSON_AddBoolToObject(follow, "backfill_complete", bf_complete);
// events_fetched (column 3) - int
long long events_fetched = db_column_int64_value(stmt, 3);
cJSON_AddNumberToObject(follow, "events_fetched", (double)events_fetched);
// first_seen / last_seen (columns 4, 5)
long long first_seen = db_column_int64_value(stmt, 4);
long long last_seen = db_column_int64_value(stmt, 5);
cJSON_AddNumberToObject(follow, "first_seen", (double)first_seen);
cJSON_AddNumberToObject(follow, "last_seen", (double)last_seen);
// Profile metadata (kind-0) for username.
cJSON* metadata = db_get_profile_metadata(pk);
if (metadata) {
cJSON* name = cJSON_GetObjectItemCaseSensitive(metadata, "name");
cJSON* display_name = cJSON_GetObjectItemCaseSensitive(metadata, "display_name");
cJSON* picture = cJSON_GetObjectItemCaseSensitive(metadata, "picture");
cJSON* nip05 = cJSON_GetObjectItemCaseSensitive(metadata, "nip05");
// Prefer display_name, fall back to name.
const char* display = NULL;
if (display_name && cJSON_IsString(display_name) && display_name->valuestring[0])
display = display_name->valuestring;
else if (name && cJSON_IsString(name) && name->valuestring[0])
display = name->valuestring;
cJSON_AddStringToObject(follow, "name", display ? display : "");
if (picture && cJSON_IsString(picture) && picture->valuestring[0])
cJSON_AddStringToObject(follow, "picture", picture->valuestring);
if (nip05 && cJSON_IsString(nip05) && nip05->valuestring[0])
cJSON_AddStringToObject(follow, "nip05", nip05->valuestring);
cJSON_Delete(metadata);
} else {
cJSON_AddStringToObject(follow, "name", "");
}
// Event counts by kind for this pubkey.
cJSON* event_counts = cJSON_CreateArray();
long long total_for_pubkey = 0;
db_stmt_t* kind_stmt = NULL;
const char* kind_params[1] = { pk };
// Use db_prepare with parameterized query.
if (db_prepare("SELECT kind, COUNT(*) FROM events WHERE pubkey = ? "
"GROUP BY kind ORDER BY kind", &kind_stmt) == DB_OK) {
if (db_bind_text_param(kind_stmt, 1, pk) == DB_OK) {
while (db_step_stmt(kind_stmt) == DB_ROW) {
int kind = db_column_int_value(kind_stmt, 0);
long long count = db_column_int64_value(kind_stmt, 1);
cJSON* kc = cJSON_CreateObject();
cJSON_AddNumberToObject(kc, "kind", (double)kind);
cJSON_AddNumberToObject(kc, "count", (double)count);
cJSON_AddItemToArray(event_counts, kc);
total_for_pubkey += count;
}
}
db_finalize_stmt(kind_stmt);
}
cJSON_AddItemToObject(follow, "event_counts", event_counts);
cJSON_AddNumberToObject(follow, "total_events_in_db", (double)total_for_pubkey);
total_events_in_db += total_for_pubkey;
// Outbox relays from kind-10002.
cJSON* outbox = db_get_outbox_relays(pk);
if (outbox) {
cJSON_AddItemToObject(follow, "outbox_relays", outbox);
} else {
cJSON_AddItemToObject(follow, "outbox_relays", cJSON_CreateArray());
}
// Per-relay backfill progress.
cJSON* relay_progress = cJSON_CreateArray();
db_stmt_t* rp_stmt = NULL;
if (db_prepare("SELECT relay_url, until_cursor, complete, events_fetched, "
"COALESCE(last_status, '') AS last_status "
"FROM caching_backfill_relay_progress "
"WHERE author_pubkey = ? ORDER BY relay_url", &rp_stmt) == DB_OK) {
if (db_bind_text_param(rp_stmt, 1, pk) == DB_OK) {
while (db_step_stmt(rp_stmt) == DB_ROW) {
const char* relay_url = db_column_text_value(rp_stmt, 0);
long long until_cursor = db_column_int64_value(rp_stmt, 1);
const char* complete_str = db_column_text_value(rp_stmt, 2);
long long rp_events = db_column_int64_value(rp_stmt, 3);
const char* rp_status = db_column_text_value(rp_stmt, 4);
int rp_complete = (complete_str && (complete_str[0] == 't' || complete_str[0] == 'T' || complete_str[0] == '1'));
cJSON* rp = cJSON_CreateObject();
cJSON_AddStringToObject(rp, "relay_url", relay_url ? relay_url : "");
cJSON_AddNumberToObject(rp, "until_cursor", (double)until_cursor);
cJSON_AddBoolToObject(rp, "complete", rp_complete);
cJSON_AddNumberToObject(rp, "events_fetched", (double)rp_events);
cJSON_AddStringToObject(rp, "last_status", rp_status ? rp_status : "");
cJSON_AddItemToArray(relay_progress, rp);
}
}
db_finalize_stmt(rp_stmt);
}
cJSON_AddItemToObject(follow, "relay_progress", relay_progress);
cJSON_AddItemToArray(follows_array, follow);
}
db_finalize_stmt(stmt);
}
cJSON_AddItemToObject(response, "follows", follows_array);
cJSON_AddNumberToObject(response, "total_follows", (double)cJSON_GetArraySize(follows_array));
cJSON_AddNumberToObject(response, "total_events_in_db", (double)total_events_in_db);
if (send_admin_response_event(response, sender_pubkey, wsi) == 0) {
cJSON_Delete(response);
return 0;
}
cJSON_Delete(response);
snprintf(error_message, error_size, "failed to send caching follows status response");
return -1;
}
else if (strcmp(command, "caching_reset_progress") == 0) {
// Reset backfill progress
DEBUG_INFO("Admin requested caching backfill progress reset");
@@ -4227,8 +4406,23 @@ int handle_system_command_unified(cJSON* event, const char* command, char* error
cJSON_AddNumberToObject(response, "timestamp", (double)time(NULL));
#ifdef DB_BACKEND_POSTGRES
// Clear the caching_backfill_progress table
if (db_exec_sql("DELETE FROM caching_backfill_progress") == 0) {
// Clear the backfill progress tables and mark all authors incomplete.
// 1. caching_backfill_progress (legacy window table — may be empty)
// 2. caching_backfill_relay_progress (per-relay drain table)
// 3. caching_followed_pubkeys: reset backfill_complete=FALSE, until_cursor=0
int reset_ok = (db_exec_sql("DELETE FROM caching_backfill_progress") == 0);
if (reset_ok) {
reset_ok = (db_exec_sql("DELETE FROM caching_backfill_relay_progress") == 0);
}
if (reset_ok) {
/* Mark all followed authors as incomplete so the backfill picks them
* up again. Reset until_cursor to 0 so each relay starts fresh. */
reset_ok = (db_exec_sql(
"UPDATE caching_followed_pubkeys "
"SET backfill_complete = FALSE, until_cursor = 0, events_fetched = 0, "
" updated_at = EXTRACT(EPOCH FROM NOW())::BIGINT") == 0);
}
if (reset_ok) {
cJSON_AddStringToObject(response, "status", "success");
cJSON_AddStringToObject(response, "message", "Backfill progress reset. The caching service will restart backfill from the beginning.");
DEBUG_INFO("Caching backfill progress reset successfully");
src/db_ops.c
+25
View File
@@ -178,11 +178,20 @@ int db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count
int db_caching_inbox_oldest_age(int* out_oldest_age_seconds) {
return postgres_db_caching_inbox_oldest_age(out_oldest_age_seconds);
}
int db_caching_backfill_author_counts(int* out_complete, int* out_total) {
return postgres_db_caching_backfill_author_counts(out_complete, out_total);
}
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority) {
return postgres_db_caching_inbox_insert(event_id, event_json, source_relay, source_class, priority);
}
cJSON* db_get_profile_metadata(const char* pubkey) {
return postgres_db_get_profile_metadata(pubkey);
}
cJSON* db_get_outbox_relays(const char* pubkey) {
return postgres_db_get_outbox_relays(pubkey);
}
#else
@@ -365,6 +374,11 @@ int db_caching_inbox_oldest_age(int* out_oldest_age_seconds) {
if (out_oldest_age_seconds) *out_oldest_age_seconds = 0;
return DB_ERROR;
}
int db_caching_backfill_author_counts(int* out_complete, int* out_total) {
if (out_complete) *out_complete = 0;
if (out_total) *out_total = 0;
return DB_ERROR;
}
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority) {
@@ -372,4 +386,15 @@ int db_caching_inbox_insert(const char* event_id, const char* event_json,
return DB_ERROR;
}
cJSON* db_get_profile_metadata(const char* pubkey) {
/* SQLite backend: kind-0 metadata not supported in this context.
* The caching follows feature is PostgreSQL-only. */
(void)pubkey;
return NULL;
}
cJSON* db_get_outbox_relays(const char* pubkey) {
(void)pubkey;
return NULL;
}
#endif
src/db_ops.h
+15
View File
@@ -133,8 +133,23 @@ int db_wal_checkpoint_truncate(void);
cJSON* db_caching_inbox_dequeue_batch(int batch_size, int* out_count);
int db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count);
int db_caching_inbox_oldest_age(int* out_oldest_age_seconds);
// Backfill author progress from caching_service_state (per-author until-cursor
// drain model). Returns DB_OK and fills out_complete/out_total, or DB_ERROR.
int db_caching_backfill_author_counts(int* out_complete, int* out_total);
int db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority);
// Profile metadata and outbox relay helpers.
// db_get_profile_metadata: returns the most recent kind-0 metadata for a pubkey
// as a cJSON object with name/picture/about/nip05/display_name fields parsed
// from the event content JSON. Returns NULL if no kind-0 event exists.
// Caller must cJSON_Delete() the result.
cJSON* db_get_profile_metadata(const char* pubkey);
// db_get_outbox_relays: returns relay URLs from the most recent kind-10002
// event for a pubkey, as a cJSON array of strings. Returns NULL if no
// kind-10002 event exists. Caller must cJSON_Delete() the result.
cJSON* db_get_outbox_relays(const char* pubkey);
#endif // DB_OPS_H
src/db_ops_postgres.c
+172
View File
@@ -2470,3 +2470,175 @@ int postgres_db_caching_inbox_insert(const char* event_id, const char* event_jso
return DB_ERROR;
#endif
}
int postgres_db_caching_backfill_author_counts(int* out_complete, int* out_total) {
#if defined(DB_BACKEND_POSTGRES) && defined(HAVE_LIBPQ)
if (out_complete) *out_complete = 0;
if (out_total) *out_total = 0;
PGconn* conn = postgres_db_active_connection();
if (!conn || PQstatus(conn) != CONNECTION_OK) return DB_ERROR;
PGresult* res = PQexec(conn,
"SELECT backfill_authors_complete, backfill_authors_total "
"FROM caching_service_state WHERE id = 1");
if (!res || PQresultStatus(res) != PGRES_TUPLES_OK) {
if (res) {
postgres_set_error_text(PQresultErrorMessage(res));
PQclear(res);
} else {
postgres_set_error_from_conn(conn, "postgres_db_caching_backfill_author_counts failed");
}
return DB_ERROR;
}
if (PQntuples(res) > 0) {
if (out_complete) {
*out_complete = PQgetisnull(res, 0, 0) ? 0
: (int)strtol(PQgetvalue(res, 0, 0), NULL, 10);
}
if (out_total) {
*out_total = PQgetisnull(res, 0, 1) ? 0
: (int)strtol(PQgetvalue(res, 0, 1), NULL, 10);
}
}
PQclear(res);
return DB_OK;
#else
if (out_complete) *out_complete = 0;
if (out_total) *out_total = 0;
return DB_ERROR;
#endif
}
/* ------------------------------------------------------------------ */
/* Profile metadata and outbox relay helpers */
/* ------------------------------------------------------------------ */
/* Returns the most recent kind-0 metadata for a pubkey as a cJSON object
* with name/picture/about/nip05/display_name fields parsed from the event
* content JSON. Returns NULL if no kind-0 event exists or on error.
* Caller must cJSON_Delete() the result. */
cJSON* postgres_db_get_profile_metadata(const char* pubkey) {
#if defined(DB_BACKEND_POSTGRES) && defined(HAVE_LIBPQ)
if (!pubkey || pubkey[0] == '\0') return NULL;
PGconn* conn = postgres_db_active_connection();
if (!conn || PQstatus(conn) != CONNECTION_OK) return NULL;
const char* params[1] = { pubkey };
PGresult* res = PQexecParams(conn,
"SELECT content FROM events WHERE pubkey = $1 AND kind = 0 "
"ORDER BY created_at DESC LIMIT 1",
1, NULL, params, NULL, NULL, 0);
if (!res || PQresultStatus(res) != PGRES_TUPLES_OK) {
if (res) PQclear(res);
return NULL;
}
cJSON* result = NULL;
if (PQntuples(res) > 0 && !PQgetisnull(res, 0, 0)) {
const char* content = PQgetvalue(res, 0, 0);
if (content && content[0] != '\0') {
/* Parse the kind-0 content JSON and extract known fields. */
cJSON* parsed = cJSON_Parse(content);
if (parsed) {
result = cJSON_CreateObject();
if (!result) {
cJSON_Delete(parsed);
PQclear(res);
return NULL;
}
/* Extract standard NIP-01 metadata fields. */
const char* fields[] = {"name", "display_name", "picture",
"about", "nip05", "website", "lud16", "lud06"};
for (size_t i = 0; i < sizeof(fields) / sizeof(fields[0]); i++) {
cJSON* val = cJSON_GetObjectItemCaseSensitive(parsed, fields[i]);
if (val && cJSON_IsString(val) && val->valuestring[0] != '\0') {
cJSON_AddStringToObject(result, fields[i], val->valuestring);
}
}
cJSON_Delete(parsed);
}
}
}
PQclear(res);
return result;
#else
(void)pubkey;
return NULL;
#endif
}
/* Returns relay URLs from the most recent kind-10002 event for a pubkey,
* as a cJSON array of strings. Returns NULL if no kind-10002 event exists
* or on error. Caller must cJSON_Delete() the result. */
cJSON* postgres_db_get_outbox_relays(const char* pubkey) {
#if defined(DB_BACKEND_POSTGRES) && defined(HAVE_LIBPQ)
if (!pubkey || pubkey[0] == '\0') return NULL;
PGconn* conn = postgres_db_active_connection();
if (!conn || PQstatus(conn) != CONNECTION_OK) return NULL;
const char* params[1] = { pubkey };
PGresult* res = PQexecParams(conn,
"SELECT tags FROM events WHERE pubkey = $1 AND kind = 10002 "
"ORDER BY created_at DESC LIMIT 1",
1, NULL, params, NULL, NULL, 0);
if (!res || PQresultStatus(res) != PGRES_TUPLES_OK) {
if (res) PQclear(res);
return NULL;
}
cJSON* result = NULL;
if (PQntuples(res) > 0 && !PQgetisnull(res, 0, 0)) {
const char* tags_json = PQgetvalue(res, 0, 0);
if (tags_json && tags_json[0] != '\0') {
cJSON* tags = cJSON_Parse(tags_json);
if (tags && cJSON_IsArray(tags)) {
result = cJSON_CreateArray();
if (!result) {
cJSON_Delete(tags);
PQclear(res);
return NULL;
}
int n = cJSON_GetArraySize(tags);
for (int i = 0; i < n; i++) {
cJSON* tag = cJSON_GetArrayItem(tags, i);
if (!tag || !cJSON_IsArray(tag)) continue;
int tag_len = cJSON_GetArraySize(tag);
if (tag_len < 2) continue;
cJSON* tag_name = cJSON_GetArrayItem(tag, 0);
cJSON* tag_val = cJSON_GetArrayItem(tag, 1);
if (tag_name && cJSON_IsString(tag_name) &&
strcmp(tag_name->valuestring, "r") == 0 &&
tag_val && cJSON_IsString(tag_val) &&
tag_val->valuestring[0] != '\0') {
/* Check for a marker (3rd element) - skip "read" only
* relays, keep "write" and unmarked (both) relays. */
int is_write = 1;
if (tag_len >= 3) {
cJSON* marker = cJSON_GetArrayItem(tag, 2);
if (marker && cJSON_IsString(marker) &&
strcmp(marker->valuestring, "read") == 0) {
is_write = 0;
}
}
if (is_write) {
cJSON_AddItemToArray(result,
cJSON_CreateString(tag_val->valuestring));
}
}
}
cJSON_Delete(tags);
}
}
}
PQclear(res);
return result;
#else
(void)pubkey;
return NULL;
#endif
}
src/db_ops_postgres.h
+5
View File
@@ -102,8 +102,13 @@ int postgres_db_wal_checkpoint_truncate(void);
cJSON* postgres_db_caching_inbox_dequeue_batch(int batch_size, int* out_count);
int postgres_db_caching_inbox_pending_counts(int* out_live_count, int* out_backfill_count);
int postgres_db_caching_inbox_oldest_age(int* out_oldest_age_seconds);
int postgres_db_caching_backfill_author_counts(int* out_complete, int* out_total);
int postgres_db_caching_inbox_insert(const char* event_id, const char* event_json,
const char* source_relay, const char* source_class,
int priority);
// Profile metadata and outbox relay helpers.
cJSON* postgres_db_get_profile_metadata(const char* pubkey);
cJSON* postgres_db_get_outbox_relays(const char* pubkey);
#endif // DB_OPS_POSTGRES_H
src/default_config_event.h
+2 -2
View File
@@ -148,7 +148,7 @@ static const struct {
{"caching_live_resubscribe_seconds", "300"}, // Live subscription resubscribe interval
{"caching_backfill_enabled", "true"}, // Enable historical backfill
{"caching_backfill_windows", "86400,604800,2592000,7776000,31536000"}, // Backfill window schedule in seconds
{"caching_backfill_page_size", "200"}, // Initial backfill query page size
{"caching_backfill_page_size", "500"}, // Initial backfill query page size
{"caching_backfill_tick_interval_ms", "5000"}, // Minimum delay between backfill queries
{"caching_backfill_window_cooldown_seconds", "60"}, // Delay between completed windows
{"caching_follow_graph_refresh_seconds", "600"}, // Kind-3 follow graph refresh interval
@@ -158,7 +158,7 @@ static const struct {
{"caching_max_relays_per_pubkey", "5"}, // Maximum outbox relays per author
{"caching_query_timeout_ms", "15000"}, // Upstream query timeout
{"caching_inbox_enabled", "false"}, // Enable inbox poller in c-relay-pg
{"caching_inbox_batch_size", "25"}, // Inbox dequeue batch size
{"caching_inbox_batch_size", "150"}, // Inbox dequeue batch size
{"caching_inbox_active_poll_ms", "200"}, // Poll interval when inbox has events
{"caching_inbox_idle_poll_ms", "5000"}, // Poll interval when inbox is empty
{"caching_max_event_json_bytes", "65536"}, // Maximum event JSON size for inbox insert
src/embedded_web_content.c
+4 -4
View File
File diff suppressed because one or more lines are too long
src/main.c
+44 -5
View File
@@ -2921,8 +2921,27 @@ int main(int argc, char* argv[]) {
// Initialize kind-based index for fast subscription lookup
init_kind_index();
// Populate event_tags from existing events (for tag-based lookups)
populate_event_tags_from_existing();
// Populate event_tags from existing events (for tag-based lookups).
// This is a one-time backfill guarded by a config flag so it does NOT
// run on every startup — on large databases (tens of GB) the
// CROSS JOIN LATERAL jsonb_array_elements over every event can take a
// very long time and blocks the relay from binding its WebSocket port.
{
char* tags_backfilled = db_get_config_value_dup("event_tags_backfilled");
if (!tags_backfilled || strcmp(tags_backfilled, "true") != 0) {
DEBUG_INFO("One-time event_tags backfill: populating from existing events...");
populate_event_tags_from_existing();
/* Record completion in the config table so we skip it on future
* startups. db_set_config_value_full(key, value, type, ...). */
db_set_config_value_full("event_tags_backfilled", "true", "boolean",
"Internal flag: event_tags backfill completed",
"internal", 0);
DEBUG_INFO("event_tags backfill complete.");
} else {
DEBUG_LOG("event_tags backfill already done, skipping.");
}
if (tags_backfilled) free(tags_backfilled);
}
// Sync Web of Trust whitelist if enabled
int wot_level = get_config_int("wot_enabled", 0);
@@ -2970,12 +2989,32 @@ int main(int argc, char* argv[]) {
// Optionally reset backfill progress first if --reset-backfill was passed.
if (cli_options.start_caching) {
#ifdef DB_BACKEND_POSTGRES
/* Ensure caching_enabled and caching_inbox_enabled are set to true
* in the config table so the UI reflects the actual state and the
* inbox poller is active. We call db_update_config_value_only()
* directly (bypassing the config-layer wrapper) so we must also
* invalidate the in-memory config cache — otherwise the poller's
* get_config_bool("caching_inbox_enabled") will keep returning
* the stale pre-update value and never dequeue any events. */
db_update_config_value_only("caching_enabled", "true");
db_update_config_value_only("caching_inbox_enabled", "true");
invalidate_config_cache();
DEBUG_INFO("CLI: --start-caching: set caching_enabled=true, caching_inbox_enabled=true");
if (cli_options.reset_backfill) {
DEBUG_INFO("CLI: resetting caching backfill progress (--reset-backfill)");
if (db_exec_sql("DELETE FROM caching_backfill_progress") == 0) {
DEBUG_INFO("CLI: caching_backfill_progress cleared");
/* Clear both the legacy window table and the per-relay drain table,
* and mark all followed authors as incomplete. */
int rb_ok = (db_exec_sql("DELETE FROM caching_backfill_progress") == 0);
if (rb_ok) rb_ok = (db_exec_sql("DELETE FROM caching_backfill_relay_progress") == 0);
if (rb_ok) rb_ok = (db_exec_sql(
"UPDATE caching_followed_pubkeys "
"SET backfill_complete = FALSE, until_cursor = 0, events_fetched = 0, "
" updated_at = EXTRACT(EPOCH FROM NOW())::BIGINT") == 0);
if (rb_ok) {
DEBUG_INFO("CLI: caching backfill progress cleared (all tables reset)");
} else {
DEBUG_WARN("CLI: failed to clear caching_backfill_progress");
DEBUG_WARN("CLI: failed to clear caching backfill progress");
}
}
DEBUG_INFO("CLI: auto-starting caching service (--start-caching)");
src/main.h
+2 -2
View File
@@ -13,8 +13,8 @@
// Using CRELAY_ prefix to avoid conflicts with nostr_core_lib VERSION macros
#define CRELAY_VERSION_MAJOR 2
#define CRELAY_VERSION_MINOR 1
#define CRELAY_VERSION_PATCH 22
#define CRELAY_VERSION "v2.1.22"
#define CRELAY_VERSION_PATCH 23
#define CRELAY_VERSION "v2.1.23"
// Relay metadata (authoritative source for NIP-11 information)
#define RELAY_NAME "C-Relay-PG"
src/pg_schema.h
+42
View File
@@ -461,6 +461,48 @@ static const char* const EMBEDDED_PG_SCHEMA_SQL =
" ON caching_backfill_progress(window_index)\n"
" WHERE complete = FALSE;\n"
"\n"
"-- Durable followed-pubkey list with per-author backfill cursor state.\n"
"-- Replaces the window-coupled caching_backfill_progress table for the\n"
"-- per-author until-cursor drain backfill model.\n"
"CREATE TABLE IF NOT EXISTS caching_followed_pubkeys (\n"
" pubkey TEXT PRIMARY KEY,\n"
" is_root BOOLEAN NOT NULL DEFAULT FALSE,\n"
" until_cursor BIGINT NOT NULL DEFAULT 0,\n"
" backfill_complete BOOLEAN NOT NULL DEFAULT FALSE,\n"
" events_fetched INTEGER NOT NULL DEFAULT 0,\n"
" first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" last_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_caching_followed_incomplete\n"
" ON caching_followed_pubkeys(backfill_complete, last_seen)\n"
" WHERE backfill_complete = FALSE;\n"
"\n"
"-- Per-relay backfill cursor tracking for the relay-by-relay drain model.\n"
"-- Each (author, relay) pair has its own until_cursor and completion flag so\n"
"-- slow relays do not cause premature completion marking for the author.\n"
"CREATE TABLE IF NOT EXISTS caching_backfill_relay_progress (\n"
" author_pubkey TEXT NOT NULL,\n"
" relay_url TEXT NOT NULL,\n"
" until_cursor BIGINT NOT NULL DEFAULT 0,\n"
" complete BOOLEAN NOT NULL DEFAULT FALSE,\n"
" events_fetched INTEGER NOT NULL DEFAULT 0,\n"
" updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,\n"
" PRIMARY KEY (author_pubkey, relay_url)\n"
");\n"
"\n"
"CREATE INDEX IF NOT EXISTS idx_caching_relay_progress_incomplete\n"
" ON caching_backfill_relay_progress(author_pubkey)\n"
" WHERE complete = FALSE;\n"
"\n"
"-- Replace window-based progress columns with author-based progress.\n"
"ALTER TABLE caching_service_state\n"
" DROP COLUMN IF EXISTS current_window_index,\n"
" DROP COLUMN IF EXISTS backfill_cursor,\n"
" ADD COLUMN IF NOT EXISTS backfill_authors_complete INTEGER NOT NULL DEFAULT 0,\n"
" ADD COLUMN IF NOT EXISTS backfill_authors_total INTEGER NOT NULL DEFAULT 0;\n"
"\n"
"-- =====================================================================\n"
"-- LISTEN/NOTIFY support for api-worker monitoring (Phase 5)\n"
"-- Fires a notification on the 'event_stored' channel whenever a new event\n"
src/pg_schema.sql
+42
View File
@@ -360,4 +360,46 @@ CREATE INDEX IF NOT EXISTS idx_caching_backfill_progress_window
ON caching_backfill_progress(window_index)
WHERE complete = FALSE;
-- Durable followed-pubkey list with per-author backfill cursor state.
-- Replaces the window-coupled caching_backfill_progress table for the
-- per-author until-cursor drain backfill model.
CREATE TABLE IF NOT EXISTS caching_followed_pubkeys (
pubkey TEXT PRIMARY KEY,
is_root BOOLEAN NOT NULL DEFAULT FALSE,
until_cursor BIGINT NOT NULL DEFAULT 0,
backfill_complete BOOLEAN NOT NULL DEFAULT FALSE,
events_fetched INTEGER NOT NULL DEFAULT 0,
first_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
last_seen BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT
);
CREATE INDEX IF NOT EXISTS idx_caching_followed_incomplete
ON caching_followed_pubkeys(backfill_complete, last_seen)
WHERE backfill_complete = FALSE;
-- Per-relay backfill cursor tracking for the relay-by-relay drain model.
-- Each (author, relay) pair has its own until_cursor and completion flag so
-- slow relays do not cause premature completion marking for the author.
CREATE TABLE IF NOT EXISTS caching_backfill_relay_progress (
author_pubkey TEXT NOT NULL,
relay_url TEXT NOT NULL,
until_cursor BIGINT NOT NULL DEFAULT 0,
complete BOOLEAN NOT NULL DEFAULT FALSE,
events_fetched INTEGER NOT NULL DEFAULT 0,
updated_at BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
PRIMARY KEY (author_pubkey, relay_url)
);
CREATE INDEX IF NOT EXISTS idx_caching_relay_progress_incomplete
ON caching_backfill_relay_progress(author_pubkey)
WHERE complete = FALSE;
-- Replace window-based progress columns with author-based progress.
ALTER TABLE caching_service_state
DROP COLUMN IF EXISTS current_window_index,
DROP COLUMN IF EXISTS backfill_cursor,
ADD COLUMN IF NOT EXISTS backfill_authors_complete INTEGER NOT NULL DEFAULT 0,
ADD COLUMN IF NOT EXISTS backfill_authors_total INTEGER NOT NULL DEFAULT 0;
COMMIT;
src/websockets.h
+4 -1
View File
@@ -22,7 +22,10 @@
// Maximum number of messages allowed in a per-client write queue.
// When exceeded, new messages are dropped to prevent unbounded memory growth
// under high-traffic or slow-client conditions.
#define MAX_MESSAGE_QUEUE_SIZE 500
// Must be larger than max_limit (5000) so a full page of EVENT messages
// plus the trailing EOSE can always be queued — otherwise the EOSE gets
// dropped when a query returns exactly `limit` events.
#define MAX_MESSAGE_QUEUE_SIZE 5100
// Filter validation constants
#define MAX_FILTERS_PER_REQUEST 10
tests/test_ws_client
Executable
BIN
View File
Binary file not shown.
tests/test_ws_client.c
+181
View File
@@ -0,0 +1,181 @@
/*
* test_ws_client.c — Standalone test for nostr_core_lib's WebSocket client.
*
* Connects to a relay, sends a REQ with a filter, and counts how many
* EVENT messages and EOSE it receives. This isolates the WebSocket client
* from the caching backfill logic to diagnose why the C client only gets
* 1 event from laantungir.net/relay while node.js gets 500 + EOSE.
*
* Build: see test_ws_client_makefile or use the caching/ Makefile pattern.
* cd tests && gcc -o test_ws_client test_ws_client.c \
* -I../nostr_core_lib -I../nostr_core_lib/nostr_core \
* -I../nostr_core_lib/cjson -I../nostr_core_lib/nostr_websocket \
* ../nostr_core_lib/libnostr_core_x64.a \
* -lssl -lcrypto -lwebsockets -lsecp256k1 -lz -ldl -lpthread -lm
*
* Usage: ./test_ws_client <relay_url> <pubkey_hex> [limit] [kinds]
* limit defaults to 500, kinds defaults to "0,1,3,6,10000,10002,30023"
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "nostr_core.h"
#include "cjson/cJSON.h"
typedef struct {
int event_count;
int got_eose;
int got_timeout;
int got_error;
char last_status[256];
time_t start;
} test_ctx_t;
static void test_callback(const char* relay_url, const char* status,
const char* event_id, int events_received,
int total_relays, int completed_relays,
void* user_data) {
(void)relay_url; (void)event_id; (void)total_relays; (void)completed_relays;
test_ctx_t* ctx = (test_ctx_t*)user_data;
if (!ctx || !status) return;
double elapsed = difftime(time(NULL), ctx->start);
/* Track all status types */
if (strcmp(status, "event_found") == 0) {
ctx->event_count = events_received;
if (ctx->event_count % 50 == 0) {
printf(" [%5.0fs] %d events received\n", elapsed, ctx->event_count);
}
} else if (strcmp(status, "eose") == 0) {
ctx->got_eose = 1;
ctx->event_count = events_received;
printf(" [%5.0fs] EOSE received! events=%d\n", elapsed, ctx->event_count);
} else if (strcmp(status, "timeout") == 0) {
ctx->got_timeout = 1;
ctx->event_count = events_received;
printf(" [%5.0fs] TIMEOUT (relay timed out) events=%d\n", elapsed, ctx->event_count);
} else if (strcmp(status, "error") == 0) {
ctx->got_error = 1;
printf(" [%5.0fs] ERROR from relay\n", elapsed);
} else if (strcmp(status, "all_complete") == 0) {
printf(" [%5.0fs] all_complete (synthetic) events=%d\n", elapsed, events_received);
} else if (strcmp(status, "connecting") == 0) {
printf(" [%5.0fs] connecting...\n", elapsed);
} else if (strcmp(status, "subscribed") == 0) {
printf(" [%5.0fs] subscribed, REQ sent\n", elapsed);
} else {
printf(" [%5.0fs] status='%s' events=%d\n", elapsed, status, events_received);
}
snprintf(ctx->last_status, sizeof(ctx->last_status), "%s", status);
}
int main(int argc, char* argv[]) {
if (argc < 3) {
fprintf(stderr, "Usage: %s <relay_url> <pubkey_hex> [limit] [kinds_csv]\n", argv[0]);
fprintf(stderr, "Example: %s wss://laantungir.net/relay 1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139 500\n", argv[0]);
return 1;
}
const char* relay_url = argv[1];
const char* pubkey = argv[2];
int limit = (argc >= 4) ? atoi(argv[3]) : 500;
const char* kinds_csv = (argc >= 5) ? argv[4] : "0,1,3,6,10000,10002,30023";
printf("=== nostr_core_lib WebSocket Client Test ===\n");
printf("Relay: %s\n", relay_url);
printf("Pubkey: %s\n", pubkey);
printf("Limit: %d\n", limit);
printf("Kinds: %s\n", kinds_csv);
printf("Timeout: 30s\n\n");
/* Build the filter JSON */
cJSON* filter = cJSON_CreateObject();
cJSON* authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(pubkey));
cJSON_AddItemToObject(filter, "authors", authors);
/* Parse kinds CSV */
cJSON* kinds = cJSON_CreateArray();
char kinds_buf[256];
strncpy(kinds_buf, kinds_csv, sizeof(kinds_buf) - 1);
kinds_buf[sizeof(kinds_buf) - 1] = '\0';
char* tok = strtok(kinds_buf, ",");
while (tok) {
cJSON_AddItemToArray(kinds, cJSON_CreateNumber((double)atoi(tok)));
tok = strtok(NULL, ",");
}
cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)limit));
/* Use until=now to mimic backfill first query */
long now = (long)time(NULL);
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)now));
printf("Until: %ld (now)\n\n", now);
const char* one_relay = relay_url;
int result_count = 0;
test_ctx_t ctx = {0};
ctx.start = time(NULL);
printf("Calling synchronous_query_relays_with_progress...\n\n");
cJSON** events = synchronous_query_relays_with_progress(
&one_relay, 1, filter, RELAY_QUERY_ALL_RESULTS,
&result_count, 30, /* 30s timeout */
test_callback, &ctx,
0, NULL /* no NIP-42 */);
double total_elapsed = difftime(time(NULL), ctx.start);
printf("\n=== RESULTS ===\n");
printf("Total elapsed: %.0fs\n", total_elapsed);
printf("Events received: %d\n", ctx.event_count);
printf("Result count: %d\n", result_count);
printf("Got EOSE: %s\n", ctx.got_eose ? "YES" : "NO");
printf("Got timeout: %s\n", ctx.got_timeout ? "YES" : "NO");
printf("Got error: %s\n", ctx.got_error ? "YES" : "NO");
printf("Last status: %s\n", ctx.last_status);
if (events) {
printf("Returned event array: %d events\n", result_count);
for (int i = 0; i < result_count && i < 3; i++) {
if (events[i]) {
cJSON* id = cJSON_GetObjectItem(events[i], "id");
cJSON* kind = cJSON_GetObjectItem(events[i], "kind");
cJSON* created = cJSON_GetObjectItem(events[i], "created_at");
printf(" event[%d]: id=%s kind=%d created_at=%lld\n",
i,
(id && cJSON_IsString(id)) ? id->valuestring : "?",
(kind && cJSON_IsNumber(kind)) ? (int)kind->valuedouble : -1,
(created && cJSON_IsNumber(created)) ? (long long)created->valuedouble : 0);
cJSON_Delete(events[i]);
}
}
/* Free remaining */
for (int i = 3; i < result_count; i++) {
if (events[i]) cJSON_Delete(events[i]);
}
free(events);
}
cJSON_Delete(filter);
printf("\n");
if (ctx.got_eose) {
printf("✓ SUCCESS: EOSE received, WebSocket client works correctly\n");
return 0;
} else if (ctx.got_timeout) {
printf("✗ FAILURE: Timed out without EOSE — WebSocket client bug confirmed\n");
return 1;
} else {
printf("✗ FAILURE: No EOSE and no timeout — unexpected state\n");
return 2;
}
}
tests/test_ws_client_makefile
+40
View File
@@ -0,0 +1,40 @@
# test_ws_client Makefile — builds the standalone WebSocket client test
# using nostr_core_lib, matching the caching/ build pattern.
CC = gcc
CFLAGS = -Wall -Wextra -std=c99 -g -O0
NOSTR_CORE_DIR = ../nostr_core_lib
INCLUDES = -I. -I$(NOSTR_CORE_DIR) -I$(NOSTR_CORE_DIR)/nostr_core \
-I$(NOSTR_CORE_DIR)/cjson -I$(NOSTR_CORE_DIR)/nostr_websocket
LIBS = -lssl -lcrypto -lwebsockets -lsecp256k1 -lz -ldl -lpthread -lm -lsqlite3 -lcurl
ARCH = $(shell uname -m)
ifeq ($(ARCH),x86_64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
else ifeq ($(ARCH),aarch64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
else ifeq ($(ARCH),arm64)
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_arm64.a
else
NOSTR_CORE_LIB = $(NOSTR_CORE_DIR)/libnostr_core_x64.a
endif
TARGET = test_ws_client
all: $(TARGET)
$(TARGET): test_ws_client.c $(NOSTR_CORE_LIB)
@echo "Building test_ws_client..."
@rm -rf /tmp/twsc_lib && mkdir -p /tmp/twsc_lib && \
ar x $(NOSTR_CORE_LIB) --output=/tmp/twsc_lib && \
$(CC) $(CFLAGS) $(INCLUDES) test_ws_client.c /tmp/twsc_lib/*.o -o $(TARGET) $(LIBS) && \
rm -rf /tmp/twsc_lib && \
echo "Build complete: $(TARGET)"
clean:
rm -f $(TARGET)
.PHONY: all clean
tests/test_ws_raw
Executable
BIN
View File
Binary file not shown.
tests/test_ws_raw.c
+142
View File
@@ -0,0 +1,142 @@
/*
* test_ws_raw.c — Minimal raw WebSocket test using nostr_core_lib.
* Directly calls nostr_ws_connect + nostr_ws_receive to bypass
* synchronous_query_relays_with_progress and see exactly what the
* WebSocket client returns.
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include "nostr_websocket_tls.h"
#include "cjson/cJSON.h"
int main(int argc, char* argv[]) {
if (argc < 3) {
fprintf(stderr, "Usage: %s <relay_url> <pubkey_hex> [limit]\n", argv[0]);
return 1;
}
const char* relay_url = argv[1];
const char* pubkey = argv[2];
int limit = (argc >= 4) ? atoi(argv[3]) : 10;
printf("Connecting to %s ...\n", relay_url);
fflush(stdout);
nostr_ws_client_t* ws = nostr_ws_connect(relay_url);
if (!ws) {
printf("FAILED to connect\n");
return 1;
}
printf("Connected!\n");
fflush(stdout);
/* Build REQ message */
cJSON* filter = cJSON_CreateObject();
cJSON* authors = cJSON_CreateArray();
cJSON_AddItemToArray(authors, cJSON_CreateString(pubkey));
cJSON_AddItemToObject(filter, "authors", authors);
cJSON* kinds = cJSON_CreateArray();
int kind_list[] = {0, 1, 3, 6, 10000, 10002, 30023};
for (int i = 0; i < 7; i++) {
cJSON_AddItemToArray(kinds, cJSON_CreateNumber((double)kind_list[i]));
}
cJSON_AddItemToObject(filter, "kinds", kinds);
cJSON_AddItemToObject(filter, "limit", cJSON_CreateNumber((double)limit));
long now = (long)time(NULL);
cJSON_AddItemToObject(filter, "until", cJSON_CreateNumber((double)now));
cJSON* req = cJSON_CreateArray();
cJSON_AddItemToArray(req, cJSON_CreateString("REQ"));
cJSON_AddItemToArray(req, cJSON_CreateString("test_raw"));
cJSON_AddItemToArray(req, filter);
char* req_str = cJSON_PrintUnformatted(req);
printf("Sending REQ: %.200s...\n", req_str);
fflush(stdout);
int send_ret = nostr_ws_send_text(ws, req_str);
printf("send_text returned: %d\n", send_ret);
fflush(stdout);
free(req_str);
cJSON_Delete(req);
/* Receive loop */
char buffer[262144];
int event_count = 0;
int eose_received = 0;
time_t start = time(NULL);
printf("\n--- Receiving messages ---\n");
fflush(stdout);
while (1) {
double elapsed = difftime(time(NULL), start);
if (elapsed > 35) {
printf("[%.0fs] *** 35s total timeout reached ***\n", elapsed);
break;
}
printf("[%.0fs] calling nostr_ws_receive (timeout=2000ms)...\n", elapsed);
fflush(stdout);
int len = nostr_ws_receive(ws, buffer, sizeof(buffer) - 1, 2000);
elapsed = difftime(time(NULL), start);
if (len < 0) {
printf("[%.0fs] nostr_ws_receive returned %d (error/timeout)\n", elapsed, len);
fflush(stdout);
/* Keep trying — the relay might send data later */
continue;
}
if (len == 0) {
printf("[%.0fs] nostr_ws_receive returned 0 (connection closed)\n", elapsed);
break;
}
buffer[len] = '\0';
printf("[%.0fs] received %d bytes: %.200s\n", elapsed, len, buffer);
fflush(stdout);
/* Parse the message */
cJSON* msg = cJSON_Parse(buffer);
if (msg && cJSON_IsArray(msg)) {
cJSON* type = cJSON_GetArrayItem(msg, 0);
if (type && cJSON_IsString(type)) {
if (strcmp(type->valuestring, "EVENT") == 0) {
event_count++;
if (event_count % 50 == 0) {
printf(" >>> %d events so far\n", event_count);
}
} else if (strcmp(type->valuestring, "EOSE") == 0) {
eose_received = 1;
printf(" >>> EOSE received! total events=%d\n", event_count);
break;
} else if (strcmp(type->valuestring, "NOTICE") == 0) {
cJSON* notice = cJSON_GetArrayItem(msg, 1);
printf(" >>> NOTICE: %s\n", notice ? notice->valuestring : "?");
} else if (strcmp(type->valuestring, "CLOSED") == 0) {
printf(" >>> CLOSED\n");
break;
} else if (strcmp(type->valuestring, "AUTH") == 0) {
printf(" >>> AUTH challenge received\n");
}
}
}
if (msg) cJSON_Delete(msg);
}
printf("\n=== RESULTS ===\n");
printf("Events: %d\n", event_count);
printf("EOSE: %s\n", eose_received ? "YES" : "NO");
printf("Elapsed: %.0fs\n", difftime(time(NULL), start));
nostr_ws_close(ws);
return eose_received ? 0 : 1;
}