Laan Tungir
50653dc86a
- New caching_custom_backfill_jobs and caching_custom_backfill_batches tables - Admin API (POST create/cancel, GET list) at admin/api/custom_backfill.php - Admin UI form with preset buttons and job status table on Backfill page - Caching daemon module (custom_backfill.c) processes jobs independently - pg_inbox functions for job/batch claim, progress update, and cancel - Three batch modes: author-batched, id-batched, time-window scan - Fixed done_batches overcount on job completion - Admin config now environment-variable driven (C_RELAY_DB_*) - admin/serve.sh supports separate instances for different databases - make_and_restart_relay.sh only kills relay on target port, not all relays
97 lines
3.9 KiB
Markdown
97 lines
3.9 KiB
Markdown
# Profile Fetch Latency Test Results — Option A vs Profiles Table
|
|
|
|
## Test Environment
|
|
|
|
- **Relay**: C-Relay-PG on `ws://localhost:7777` (PostgreSQL backend)
|
|
- **Database**: 1,070,741 kind-1 events, 8,734 kind-0 events, 8,734 profiles
|
|
- **Client**: Python 3.13, `websockets` 15.0.1, localhost (no network latency)
|
|
- **Test script**: [`tests/profile_fetch_latency_test.py`](../tests/profile_fetch_latency_test.py)
|
|
|
|
---
|
|
|
|
## Option A — Two Standard REQs (measured)
|
|
|
|
| Metric | Time |
|
|
|---|---|
|
|
| **Phase 1: kind-1 REQ** | |
|
|
| REQ send → first EVENT | 20.85 ms |
|
|
| REQ send → EOSE | 23.81 ms |
|
|
| Inter-event gap (min/avg/max) | 0.01 / 0.01 / 0.54 ms |
|
|
| Events received | 200 |
|
|
| Unique authors | 115 |
|
|
| **Phase 2: kind-0 REQ** | |
|
|
| REQ send → first EVENT | 83.49 ms |
|
|
| REQ send → EOSE | 83.63 ms |
|
|
| Profiles received | 10 of 115 (8.7%) |
|
|
| **End-to-end** (kind-1 REQ → kind-0 EOSE) | **107.67 ms** |
|
|
|
|
### Key observations
|
|
|
|
1. **Phase 1 is fast** (24ms EOSE for 200 events) — the local relay streams events at ~0.01ms inter-arrival.
|
|
2. **Phase 2 is slower** (83ms to first kind-0 event) — the relay must scan the `events` table, parse JSON content, verify signatures, etc. Even though it's a local relay, the full event pipeline has overhead.
|
|
3. **Only 8.7% of authors had kind-0 events** — most kind-1 authors never published a kind-0 to this relay. This is the real bottleneck, not latency.
|
|
|
|
---
|
|
|
|
## Profiles Table Query Speed (measured)
|
|
|
|
```sql
|
|
EXPLAIN ANALYZE SELECT pubkey, name, display_name, about, picture, banner,
|
|
nip05, website, lud16, lud06
|
|
FROM profiles
|
|
WHERE pubkey = ANY($1::text[]);
|
|
```
|
|
|
|
| Metric | Time |
|
|
|---|---|
|
|
| Planning Time | 12.70 ms (one-time, cached) |
|
|
| Execution Time | **2.21 ms** |
|
|
| Index type | Primary key (Bitmap Index Scan) |
|
|
| Rows returned | 7 (of 7 requested) |
|
|
|
|
The profiles table query is **~40x faster** than the kind-0 REQ path (2.2ms vs 83ms).
|
|
|
|
---
|
|
|
|
## Comparison: Option A vs Profiles Table
|
|
|
|
| Aspect | Option A (kind-0 REQ) | Profiles table query |
|
|
|---|---|---|
|
|
| **Latency** | 83ms to first event | 2.2ms |
|
|
| **Payload size** | Full kind-0 event (sig, tags, raw content) | Only parsed fields (name, picture, etc.) |
|
|
| **Client work** | Must parse kind-0 `content` JSON, handle replaceable dedup | Ready-to-render fields |
|
|
| **Missing profiles** | Returns nothing for missing pubkeys | Returns nothing for missing pubkeys |
|
|
| **Protocol** | Standard Nostr WebSocket | Custom (HTTP or WebSocket) |
|
|
| **Portability** | Works with any relay | Relay-specific |
|
|
|
|
---
|
|
|
|
## The Real Problem
|
|
|
|
The test revealed a more fundamental issue than latency: **most kind-1 authors don't have kind-0 events on this relay**. Only 8.7% of authors had profiles. This means:
|
|
|
|
1. **Option A** would show blank names/pictures for 91% of posts.
|
|
2. **Profiles table** has the same data (it's synced from kind-0 events), so it has the same gap.
|
|
3. The caching daemon doesn't cache kind-0 by default — it caches whatever `caching_kinds` is configured to.
|
|
|
|
### To make either option viable, the relay needs more kind-0 data
|
|
|
|
The caching daemon should include kind-0 in its cached kinds, or a separate profile-caching pass should backfill kind-0 events for all known pubkeys.
|
|
|
|
---
|
|
|
|
## Recommendation
|
|
|
|
**Option B (batched HTTP profiles from the profiles table)** is the clear winner for a no-storage local client:
|
|
|
|
- **2.2ms** query time vs 83ms for kind-0 REQ
|
|
- Ready-to-render fields (no client-side kind-0 JSON parsing)
|
|
- Smaller payload (no sig, tags, raw content)
|
|
- Single HTTP call for all authors at once
|
|
|
|
But it requires:
|
|
1. A new HTTP endpoint (e.g., `POST /api/profiles` accepting `{"pubkeys":[...]}`)
|
|
2. The caching daemon to include kind-0 in its cache kinds (or a separate profile backfill pass)
|
|
|
|
**Option C (relay auto-hydrates profiles before EOSE)** would be even better for the client (one subscription, everything arrives before EOSE), but requires more relay-side work.
|