c-relay-pg/plans/caching_follows_status_plan.md

Caching Follows Status UI with Username Resolution

Goal

Show a list of all followed/cached pubkeys with their usernames, event counts by kind, outbox relays, and backfill status in the relay admin web UI. Also add username resolution to the existing stats top-pubkeys table.

Architecture Overview

graph LR
    A[Admin Web UI] -->|system_command: caching_follows_status| B[relay config.c]
    B -->|SQL query| C[(PostgreSQL)]
    C -->|caching_followed_pubkeys| D[follows + backfill status]
    C -->|events kind=0| E[metadata: name, picture, about]
    C -->|events kind=10002| F[outbox relay URLs]
    C -->|events GROUP BY kind| G[event counts per kind]
    B -->|JSON response| A

The relay already has DB access and an admin command pipeline. We add one new admin command that joins the caching tables with the events table to produce a rich per-follow status response.

Data Sources

All data is already in PostgreSQL — no new tables needed:

Data	Source Table	Query
Followed pubkeys + backfill status	caching_followed_pubkeys	SELECT pubkey, is_root, backfill_complete, events_fetched, first_seen, last_seen
Per-relay backfill progress	caching_backfill_relay_progress	SELECT relay_url, until_cursor, complete, events_fetched WHERE author_pubkey = $1
Username / profile metadata	events (kind=0)	SELECT content FROM events WHERE pubkey = $1 AND kind = 0 ORDER BY created_at DESC LIMIT 1 — content is JSON with name, picture, about, nip05
Outbox relays	events (kind=10002)	SELECT content, tags FROM events WHERE pubkey = $1 AND kind = 10002 ORDER BY created_at DESC LIMIT 1 — tags contain ["r", "wss://..."] entries
Event counts by kind	events	SELECT kind, count(*) FROM events WHERE pubkey = $1 GROUP BY kind ORDER BY kind

Implementation Steps

Step 1: Username/Profile Resolution Helper

File: src/db_ops_postgres.c (and src/db_ops.h for the declaration)

Add a reusable function that, given a pubkey, returns the most recent kind-0 metadata as a JSON object with name, picture, about, nip05, display_name fields parsed from the event content.

// Returns a cJSON object: {"name":"...", "picture":"...", "about":"...", "nip05":"...", "display_name":"..."}
// Returns NULL if no kind-0 event exists for this pubkey.
cJSON* db_get_profile_metadata(const char* pubkey);

SQL:

SELECT content FROM events
 WHERE pubkey = $1 AND kind = 0
 ORDER BY created_at DESC LIMIT 1

The content column is a JSON string (NIP-01 metadata). Parse it with cJSON and extract the fields. This function will be used by both the caching follows command and the stats top-pubkeys enrichment.

Also add to SQLite backend (src/db_ops_sqlite.c) for parity, though the primary target is PostgreSQL.

Step 2: Outbox Relay Resolution Helper

File: src/db_ops_postgres.c (and src/db_ops.h)

Add a function that returns the outbox relay URLs from the most recent kind-10002 event for a pubkey.

// Returns a cJSON array of relay URL strings: ["wss://relay1", "wss://relay2", ...]
// Returns NULL if no kind-10002 event exists.
cJSON* db_get_outbox_relays(const char* pubkey);

SQL:

SELECT tags FROM events
 WHERE pubkey = $1 AND kind = 10002
 ORDER BY created_at DESC LIMIT 1

The tags column is JSONB. Extract all ["r", "url"] tag entries. In PostgreSQL this can be done in SQL:

SELECT tag->>1 AS relay_url
  FROM events e,
       jsonb_array_elements(e.tags) AS tag
 WHERE e.pubkey = $1 AND e.kind = 10002
   AND jsonb_typeof(tag) = 'array'
   AND jsonb_array_length(tag) >= 2
   AND tag->>0 = 'r'
 ORDER BY e.created_at DESC
 LIMIT 1

Actually, simpler: fetch the tags JSONB for the latest kind-10002 and parse in C with cJSON (consistent with how the rest of the codebase works).

Step 3: New Admin Command caching_follows_status

File: src/config.c (in handle_system_command_unified(), near the existing caching_status handler at line 4128)

Add a new command caching_follows_status that:

1. Queries caching_followed_pubkeys for all followed pubkeys
2. For each pubkey, calls db_get_profile_metadata() to get the username
3. For each pubkey, queries event counts by kind: SELECT kind, count(*) FROM events WHERE pubkey = $1 GROUP BY kind ORDER BY kind
4. For each pubkey, calls db_get_outbox_relays() to get outbox relay list
5. For each pubkey, queries caching_backfill_relay_progress for per-relay status
6. Builds a JSON response:

{
  "command": "caching_follows_status",
  "status": "success",
  "timestamp": 1785234000,
  "follows": [
    {
      "pubkey": "4d7842051782...",
      "npub": "npub1f4uyypgh...",
      "name": "username_or_null",
      "picture": "url_or_null",
      "is_root": false,
      "backfill_complete": false,
      "events_fetched": 6965,
      "first_seen": 1785233973,
      "last_seen": 1785233973,
      "event_counts": [
        {"kind": 0, "count": 1},
        {"kind": 1, "count": 658},
        {"kind": 6, "count": 1}
      ],
      "total_events_in_db": 660,
      "outbox_relays": [
        "wss://nostr-01.yakihonne.com",
        "wss://nostr.land",
        "wss://relay.snort.social"
      ],
      "relay_progress": [
        {"relay_url": "wss://nos.lol", "until_cursor": 1782686796, "complete": false, "events_fetched": 1000},
        {"relay_url": "wss://relay.damus.io", "until_cursor": 1785234229, "complete": true, "events_fetched": 0}
      ]
    }
  ],
  "total_follows": 8,
  "total_events_in_db": 9843
}

Performance consideration: With up to 5000 followed pubkeys (caching_max_followed_pubkeys), running per-pubkey queries in a loop could be slow. Two approaches:

• Option A (simple, fine for <100 follows): Loop in C, call the helper functions per pubkey. With 8-50 follows this is fast enough.
• Option B (scalable, for large follow sets): Use a single SQL query with CTEs and LEFT JOINs to get everything in one round-trip. This is more complex but handles 5000 follows.

Recommendation: Start with Option A. If follow counts grow large, optimize to Option B later. The per-pubkey queries are all indexed (idx_events_pubkey_kind, idx_events_pubkey).

Step 4: Frontend — Caching Follows Table

File: api/index.html

Add a new section to the caching page (below the existing config form and service/inbox status), containing a table:

<div id="cachingFollowsSection">
  <h3>Followed Pubkeys</h3>
  <table id="caching-follows-table">
    <thead>
      <tr>
        <th>Name</th>
        <th>npub</th>
        <th>Root?</th>
        <th>Events in DB</th>
        <th>Backfill</th>
        <th>Outbox Relays</th>
        <th>Event Counts by Kind</th>
        <th>Last Seen</th>
      </tr>
    </thead>
    <tbody id="caching-follows-table-body">
      <tr><td colspan="8">Loading...</td></tr>
    </tbody>
  </table>
</div>

File: api/index.js

1. Add a fetchCachingFollowsStatus() function that sends ['system_command', 'caching_follows_status'] via the admin WebSocket.

2. Add a handleCachingFollowsResponse(responseData) function that renders the table:

   • Each row shows: name (or "unknown" with a muted style), npub (as a clickable njump.me link), root badge, total events in DB, backfill status (complete/pending with relay count), outbox relays (comma-separated or expandable), event counts by kind (e.g. "k0:1, k1:658, k6:1"), last seen timestamp.
   • Sort by total_events_in_db descending (most active first).

3. Hook into the existing caching page auto-refresh: in the if (pageName === 'caching') block at line 5445, also call fetchCachingFollowsStatus() and add it to the cachingRefreshInterval polling.

4. Add response routing in handleSystemCommandResponse() near line 2082:

   if (responseData.command === 'caching_follows_status') {
       handleCachingFollowsResponse(responseData);
   }
   

Step 5: Frontend — Username in Stats Top-Pubkeys Table

File: api/index.js

Modify populateStatsPubkeysFromMonitoring() (line 4882) and populateStatsPubkeys() (line 4844) to add a "Name" column.

Challenge: The top-pubkeys monitoring data (src/api.c:470 query_top_pubkeys()) currently returns only {pubkey, event_count}. To add usernames, we have two options:

• Option A (backend enrichment): Modify query_top_pubkeys() in src/api.c to also call db_get_profile_metadata() for each of the top 10 pubkeys and include name in the response. This is the cleanest approach — 10 extra queries on a 30-second polling cycle is negligible.

• Option B (frontend enrichment): The frontend separately queries kind-0 metadata for each pubkey. This is messier and adds latency.

Recommendation: Option A. Modify query_top_pubkeys() to include name and picture fields.

Changes:

1. src/api.c:470 — in query_top_pubkeys(), after getting each pubkey row, call db_get_profile_metadata(pubkey) and add name to the pubkey object.
2. api/index.js:4882 — in populateStatsPubkeysFromMonitoring(), add a "Name" column before the npub column. Display pubkey.name || 'unknown'.
3. api/index.js:4844 — same change in populateStatsPubkeys().
4. api/index.html — add a <th>Name</th> column to the top-pubkeys table header.

Step 6: npub Conversion

The frontend already has window.NostrTools.nip19.npubEncode() available (used at api/index.js:4902). The backend caching_follows_status response should include both the hex pubkey and the npub for convenience, but the frontend can also convert. Including npub in the backend response is preferred since the backend has access to the same nostr library.

File: src/config.c — use nostr_bytes_to_hex / nostr_decode_npub from nostr_core_lib, or simply return the hex pubkey and let the frontend convert (it already does this in the stats page). Recommendation: let the frontend convert — it already has the pattern at line 4902, and it avoids adding nostr_core_lib dependency to the config.c command handler.

File Change Summary

File	Change
src/db_ops.h	Declare db_get_profile_metadata() and db_get_outbox_relays()
src/db_ops_postgres.c	Implement both functions (PostgreSQL)
src/db_ops_sqlite.c	Implement both functions (SQLite, for parity)
src/db_ops.c	Wire up dispatch for both functions
src/config.c	Add caching_follows_status command handler in handle_system_command_unified()
src/api.c	Enrich query_top_pubkeys() with name field
api/index.html	Add follows table to caching page; add Name column to stats top-pubkeys table
api/index.js	Add fetchCachingFollowsStatus(), handleCachingFollowsResponse(), route the response, add Name column to pubkey tables, hook into caching page auto-refresh

Mermaid: Data Flow

sequenceDiagram
    participant UI as Admin Web UI
    participant Relay as relay config.c
    participant DB as PostgreSQL
    participant Cache as caching_relay process

    UI->>Relay: system_command: caching_follows_status
    Relay->>DB: SELECT * FROM caching_followed_pubkeys
    DB-->>Relay: 8 rows
    loop each pubkey
        Relay->>DB: SELECT content FROM events WHERE kind=0 AND pubkey=$1
        DB-->>Relay: metadata JSON
        Relay->>DB: SELECT kind,count(*) FROM events WHERE pubkey=$1 GROUP BY kind
        DB-->>Relay: event counts
        Relay->>DB: SELECT tags FROM events WHERE kind=10002 AND pubkey=$1
        DB-->>Relay: outbox relay tags
        Relay->>DB: SELECT * FROM caching_backfill_relay_progress WHERE author_pubkey=$1
        DB-->>Relay: per-relay status
    end
    Relay-->>UI: JSON response with follows array
    UI->>UI: render table with usernames, event counts, relays

Edge Cases

• No kind-0 metadata for a pubkey: Show "unknown" in the name column with muted styling. The pubkey is still followed and being cached; we just don't have their profile yet. The backfill should eventually fetch their kind-0.
• No kind-10002 for a pubkey: Show "none" in the outbox relays column. The caching service falls back to bootstrap relays for these authors.
• Large follow sets (5000): The per-pubkey loop could be slow. Add a LIMIT parameter to the command (default 100, configurable). The UI can paginate or show "showing first 100 of 5000".
• Kind-0 content is not valid JSON: Parse defensively — if cJSON parsing fails, treat as no metadata.
• Replaceable events: Kind 0 is replaceable, so ORDER BY created_at DESC LIMIT 1 gets the latest. The unique index uq_events_replaceable_pubkey_kind ensures only one kind-0 per pubkey is stored, so this is already handled at the DB level.

Future Enhancements (not in this plan)

• Add/remove follows manually via admin commands (currently follows come from root npub's kind-3)
• Delete individual events via admin command
• Real-time backfill tick log streaming
• Per-follow detail view (click a row to see all events for that author)
• Search/filter the follows table by name or npub