c-relay-pg/plans/caching_backfill_separation_plan.md

Caching/Backfill Separation: Remaining Work

Current State

The admin page now has separate Backfill and Caching nav items. The Backfill page works as before. The Caching page has a subscription design UI but no relay status display and the live subscriber isn't actually receiving events properly.

Three Issues to Fix

1. Caching Page: Add Relay Status Display

The backfill page shows upstream relay connection status (connected/disconnected/error per relay). The caching page should show the same relay status, since the live subscriber uses the same upstream pool.

Changes needed:

• admin/api/live_subscription.php — Add upstream relay status query to the GET response
• admin/index.php — Add relay status HTML section to the caching page
• admin/assets/app.js — Render relay status in loadCaching()

2. Backfill Auto-Shutdown When Complete

The backfill currently runs forever in a "steady state" loop, checking for incomplete authors every tick interval even when all are done. It should stop itself when complete.

Current behavior (caching/src/backfill.c):

if (pg_inbox_pick_next_author_with_incomplete_relays(...) != 0) {
    bf->in_progress = 0;  // sets to steady state
    return -2;             // caller keeps looping
}

The main loop (caching/src/main.c) calls cr_backfill_tick() every iteration regardless:

int brc = cr_backfill_tick(&bf, &cfg, upstream, &followed, &sink, &relay_map);
(void)brc;  // return value is ignored!

Changes needed:

• caching/src/main.c — Check bf.in_progress before calling cr_backfill_tick(). When backfill is complete (bf.in_progress == 0), skip the call entirely.
• caching/src/backfill.c — When bf.in_progress transitions to 0, log a clear "Backfill complete" message.
• admin/api/caching.php — The backfill page already shows backfill_authors_complete / backfill_authors_total. When complete, show a clear "Backfill complete — stopped" message instead of "listening for live events".

3. Live Subscriber Fast Dedup Ring

The existing cr_seen_ring_t is 4096 entries and shared between backfill and live. The user wants a separate, smaller, faster ring specifically for the live subscriber — 100 entries — to act as a fast debounce before events hit the inbox.

The problem: when the live subscriber receives events from multiple relays, the same event can arrive from different relays within seconds. The inbox ON CONFLICT DO NOTHING handles this at the DB level, but it's wasteful to serialize and insert events that will be rejected. A small in-memory ring catches duplicates quickly.

Changes needed:

• caching/src/state.h — Add a new ring size constant CR_LIVE_RING_SIZE 100
• caching/src/live_subscriber.h — Add a cr_seen_ring_t live_seen field to cr_live_t
• caching/src/live_subscriber.c — In live_on_event(), check the live ring before publishing. Only publish if the event ID is new to the live ring.
• caching/src/main.c — Initialize the live ring in cr_live_open()

Implementation Order

Step 1: Caching Page Relay Status

1. Update admin/api/live_subscription.php GET to query caching_upstream_relays table
2. Add relay status HTML to admin/index.php caching section
3. Add relay status rendering to admin/assets/app.js loadCaching()

Step 2: Backfill Auto-Shutdown

4. Modify caching/src/main.c to skip cr_backfill_tick() when bf.in_progress == 0
5. Add clear "Backfill complete" log message in caching/src/backfill.c
6. Update backfill page status display to show "Backfill complete — stopped" when done

Step 3: Live Subscriber Fast Dedup

7. Add CR_LIVE_RING_SIZE 100 to caching/src/state.h
8. Add cr_seen_ring_t live_seen to caching/src/live_subscriber.h
9. Initialize live ring in caching/src/main.c before opening live sub
10. Check live ring in caching/src/live_subscriber.c live_on_event() before publishing

Files Changed

File	Change
admin/api/live_subscription.php	Add upstream relay status to GET response
admin/index.php	Add relay status HTML to caching section
admin/assets/app.js	Render relay status in loadCaching()
caching/src/main.c	Skip backfill tick when complete, init live ring
caching/src/backfill.c	Log "Backfill complete" message
caching/src/state.h	Add CR_LIVE_RING_SIZE constant
caching/src/live_subscriber.h	Add live_seen ring field
caching/src/live_subscriber.c	Check live ring before publishing