- New caching_custom_backfill_jobs and caching_custom_backfill_batches tables - Admin API (POST create/cancel, GET list) at admin/api/custom_backfill.php - Admin UI form with preset buttons and job status table on Backfill page - Caching daemon module (custom_backfill.c) processes jobs independently - pg_inbox functions for job/batch claim, progress update, and cancel - Three batch modes: author-batched, id-batched, time-window scan - Fixed done_batches overcount on job completion - Admin config now environment-variable driven (C_RELAY_DB_*) - admin/serve.sh supports separate instances for different databases - make_and_restart_relay.sh only kills relay on target port, not all relays
3.9 KiB
3.9 KiB
Profile Fetch Latency Test Results — Option A vs Profiles Table
Test Environment
- Relay: C-Relay-PG on
ws://localhost:7777(PostgreSQL backend) - Database: 1,070,741 kind-1 events, 8,734 kind-0 events, 8,734 profiles
- Client: Python 3.13,
websockets15.0.1, localhost (no network latency) - Test script:
tests/profile_fetch_latency_test.py
Option A — Two Standard REQs (measured)
| Metric | Time |
|---|---|
| Phase 1: kind-1 REQ | |
| REQ send → first EVENT | 20.85 ms |
| REQ send → EOSE | 23.81 ms |
| Inter-event gap (min/avg/max) | 0.01 / 0.01 / 0.54 ms |
| Events received | 200 |
| Unique authors | 115 |
| Phase 2: kind-0 REQ | |
| REQ send → first EVENT | 83.49 ms |
| REQ send → EOSE | 83.63 ms |
| Profiles received | 10 of 115 (8.7%) |
| End-to-end (kind-1 REQ → kind-0 EOSE) | 107.67 ms |
Key observations
- Phase 1 is fast (24ms EOSE for 200 events) — the local relay streams events at ~0.01ms inter-arrival.
- Phase 2 is slower (83ms to first kind-0 event) — the relay must scan the
eventstable, parse JSON content, verify signatures, etc. Even though it's a local relay, the full event pipeline has overhead. - Only 8.7% of authors had kind-0 events — most kind-1 authors never published a kind-0 to this relay. This is the real bottleneck, not latency.
Profiles Table Query Speed (measured)
EXPLAIN ANALYZE SELECT pubkey, name, display_name, about, picture, banner,
nip05, website, lud16, lud06
FROM profiles
WHERE pubkey = ANY($1::text[]);
| Metric | Time |
|---|---|
| Planning Time | 12.70 ms (one-time, cached) |
| Execution Time | 2.21 ms |
| Index type | Primary key (Bitmap Index Scan) |
| Rows returned | 7 (of 7 requested) |
The profiles table query is ~40x faster than the kind-0 REQ path (2.2ms vs 83ms).
Comparison: Option A vs Profiles Table
| Aspect | Option A (kind-0 REQ) | Profiles table query |
|---|---|---|
| Latency | 83ms to first event | 2.2ms |
| Payload size | Full kind-0 event (sig, tags, raw content) | Only parsed fields (name, picture, etc.) |
| Client work | Must parse kind-0 content JSON, handle replaceable dedup |
Ready-to-render fields |
| Missing profiles | Returns nothing for missing pubkeys | Returns nothing for missing pubkeys |
| Protocol | Standard Nostr WebSocket | Custom (HTTP or WebSocket) |
| Portability | Works with any relay | Relay-specific |
The Real Problem
The test revealed a more fundamental issue than latency: most kind-1 authors don't have kind-0 events on this relay. Only 8.7% of authors had profiles. This means:
- Option A would show blank names/pictures for 91% of posts.
- Profiles table has the same data (it's synced from kind-0 events), so it has the same gap.
- The caching daemon doesn't cache kind-0 by default — it caches whatever
caching_kindsis configured to.
To make either option viable, the relay needs more kind-0 data
The caching daemon should include kind-0 in its cached kinds, or a separate profile-caching pass should backfill kind-0 events for all known pubkeys.
Recommendation
Option B (batched HTTP profiles from the profiles table) is the clear winner for a no-storage local client:
- 2.2ms query time vs 83ms for kind-0 REQ
- Ready-to-render fields (no client-side kind-0 JSON parsing)
- Smaller payload (no sig, tags, raw content)
- Single HTTP call for all authors at once
But it requires:
- A new HTTP endpoint (e.g.,
POST /api/profilesaccepting{"pubkeys":[...]}) - The caching daemon to include kind-0 in its cache kinds (or a separate profile backfill pass)
Option C (relay auto-hydrates profiles before EOSE) would be even better for the client (one subscription, everything arrives before EOSE), but requires more relay-side work.