c-relay-pg/plans/custom_backfill_feature_plan.md

Custom Backfill Feature Plan

Overview

A fully general ad-hoc backfill system that lets the admin define a backfill job using any combination of NIP-01 filter fields — kinds, authors, event IDs, time range, tag filters, and limit — plus relay selection. The caching daemon executes the job, batching as needed, and inserts results through the existing inbox pipeline.

The kind-0 profile backfill ("cache all kind-0 for these authors") and the time-window scan ("cache all kind-0 from date X to date Y") are both just presets of this one general feature.

Motivation

The current backfill only covers the followed set (688 pubkeys). There are 23,257 pubkeys in the local DB missing kind-0 profiles, and the user wants a general way to backfill arbitrary filters. Use cases:

• "Cache all kind-0 from date X to date Y" — time-window scan, no authors
• "Backfill missing profiles for kind-1 authors" — author-batched, kind-0
• "Backfill kind-10002 relay lists for all pubkeys in DB" — author-batched
• "Fetch all events from pubkey X" — single author, all kinds
• "Fetch events matching #t:nostr from last 30 days" — tag filter + time
• "Re-fetch a specific event by ID" — IDs filter
• "Backfill kind-3 contact lists for the followed set" — author-batched

NIP-01 Filter Coverage

The feature supports every standard NIP-01 filter field:

Field	UI Input	Batching Strategy
kinds	CSV text (e.g., 0 or 0,1,3)	Included in every batch REQ
authors	Source selector + manual list	Author-batched: split into groups of N per REQ
ids	Textarea, one hex ID per line	ID-batched: split into groups of N per REQ
since	datetime-local picker	Included in every batch REQ
until	datetime-local picker	Included in every batch REQ
limit	number (optional)	Included in every batch REQ
#e, #p, #t, #r, etc.	JSON textarea (advanced)	Included in every batch REQ

Batching logic

The daemon picks the batching strategy based on which array fields are populated:

1. authors populated → split authors into batches of batch_size (default 100). Each batch REQ includes the full kinds/since/until/tags/limit.
2. ids populated (and no authors) → split IDs into batches of batch_size. Each batch REQ includes kinds/since/until/tags/limit.
3. Neither authors nor ids → one REQ per relay (time-window scan mode). No batching; the filter is sent as-is to each selected relay. This is the "cache all kind-0 from date X to date Y" case.

flowchart TD
    A[Submit job] --> B{authors or ids populated?}
    B -->|yes| C[Split into batches of N]
    C --> D[One REQ per batch per relay]
    B -->|no| E[One REQ per relay - time window scan]
    D --> F[Collect events until EOSE/timeout]
    E --> F
    F --> G[Insert into caching_event_inbox]
    G --> H{More batches?}
    H -->|yes| D
    H -->|no| I[Mark job complete]
    G --> J[Relay inbox poller stores events]
    J --> K[Triggers update profiles table etc]

Database Schema

caching_custom_backfill_jobs

One row per submitted job. Stores the raw filter JSON plus metadata for the UI. The daemon reconstructs the NIP-01 filter from this.

CREATE TABLE IF NOT EXISTS caching_custom_backfill_jobs (
    job_id          BIGSERIAL PRIMARY KEY,
    label           TEXT NOT NULL DEFAULT '',

    -- The complete NIP-01 filter as JSON. This is the source of truth;
    -- the daemon builds REQ filters directly from this.
    -- Example: {"kinds":[0],"since":1700000000,"until":1700086400}
    -- Example: {"kinds":[0],"authors":["abc...","def..."]}
    -- Example: {"kinds":[1],"#t":["nostr"],"since":1700000000}
    filter_json     JSONB NOT NULL,

    -- Author/ID source metadata (for UI display and batch planning).
    -- 'manual' | 'all_in_db' | 'kind1_authors' | 'followed' | 'none'
    authors_source  TEXT NOT NULL DEFAULT 'none',
    -- When source produces a derived list (all_in_db, kind1_authors, etc.),
    -- the resolved author list is stored here so the daemon doesn't need
    -- to re-derive it. CSV of hex pubkeys.
    authors_resolved TEXT NOT NULL DEFAULT '',
    -- Same for IDs (if the job is ID-based). CSV of hex IDs.
    ids_resolved    TEXT NOT NULL DEFAULT '',
    -- Optional filter: only include authors missing a kind-0 event.
    authors_missing_kind0 INTEGER NOT NULL DEFAULT 0,

    -- Batching
    -- 'authors' | 'ids' | 'none' (none = one REQ per relay)
    batch_mode      TEXT NOT NULL DEFAULT 'none',
    batch_size      INTEGER NOT NULL DEFAULT 100,

    -- Relay selection
    -- 'bootstrap' | 'outbox' | 'custom'
    relay_mode      TEXT NOT NULL DEFAULT 'bootstrap',
    relay_list      TEXT NOT NULL DEFAULT '',  -- CSV when mode=custom

    -- State
    status          TEXT NOT NULL DEFAULT 'pending',
        -- 'pending' | 'running' | 'complete' | 'failed' | 'cancelled'
    total_batches   INTEGER NOT NULL DEFAULT 0,
    done_batches    INTEGER NOT NULL DEFAULT 0,
    events_fetched  INTEGER NOT NULL DEFAULT 0,
    error_message   TEXT,
    created_at      BIGINT NOT NULL DEFAULT EXTRACT(EPOCH FROM NOW())::BIGINT,
    started_at      BIGINT,
    completed_at    BIGINT,
    created_by      TEXT NOT NULL DEFAULT 'admin'
);

CREATE INDEX IF NOT EXISTS idx_custom_backfill_jobs_status
    ON caching_custom_backfill_jobs(status) WHERE status IN ('pending', 'running');

caching_custom_backfill_batches

One row per batch. For time-window mode (batch_mode='none'), there is one batch per relay.

CREATE TABLE IF NOT EXISTS caching_custom_backfill_batches (
    batch_id    BIGSERIAL PRIMARY KEY,
    job_id      BIGINT NOT NULL REFERENCES caching_custom_backfill_jobs(job_id) ON DELETE CASCADE,
    batch_index INTEGER NOT NULL,
    -- For author-batched: CSV of hex pubkeys in this batch
    -- For id-batched: CSV of hex IDs in this batch
    -- For time-window: empty (filter_json has everything)
    batch_items TEXT NOT NULL DEFAULT '',
    -- Relay this batch is assigned to
    relay_url   TEXT NOT NULL,
    -- State
    status      TEXT NOT NULL DEFAULT 'pending',
        -- 'pending' | 'running' | 'complete' | 'failed'
    events_fetched INTEGER NOT NULL DEFAULT 0,
    error_message TEXT,
    started_at  BIGINT,
    completed_at BIGINT,
    UNIQUE(job_id, batch_index)
);

CREATE INDEX IF NOT EXISTS idx_custom_backfill_batches_pending
    ON caching_custom_backfill_batches(job_id, status) WHERE status = 'pending';

Admin UI

Add a new card to the existing Backfill section in admin/index.php, below the "Re-run All Backfill" button.

Form fields

Field	Type	Description
Label	text	User-friendly name (e.g., "Profile backfill Jan 2024")
Kinds	text (CSV)	Comma-separated kinds (e.g., 0 or 0,1,3). Help text lists common kinds.
Authors source	select	none / manual list / all pubkeys in DB / kind-1 authors only / followed set
Authors list	textarea	Hex pubkeys, one per line (only when source=manual)
Only missing kind-0	checkbox	Filter author list to those without a kind-0 event
Event IDs	textarea	Hex event IDs, one per line (optional, alternative to authors)
Since	datetime-local	Lower bound (optional)
Until	datetime-local	Upper bound (optional)
Limit	number	Max events per REQ (optional, 0 = no limit)
Tag filters	text (JSON)	e.g., {"#t":["nostr"],"#p":["abc..."]} (optional, advanced)
Batch size	number	Authors/IDs per REQ (default 100, ignored for time-window mode)
Relay mode	select	bootstrap relays / outbox relays / custom list
Custom relays	textarea	Relay URLs, one per line (only when mode=custom)

Preset buttons

Quick-fill the form for common scenarios:

• "Backfill missing profiles (kind-1 authors)" — kinds=0, authors source=kind-1 authors only, only-missing-kind-0=checked, relay mode=bootstrap
• "Backfill missing profiles (all pubkeys)" — kinds=0, authors source=all pubkeys in DB, only-missing-kind-0=checked, relay mode=bootstrap
• "Cache all kind-0 from date range" — kinds=0, authors source=none, since/until = user picks, relay mode=bootstrap
• "Backfill relay lists" — kinds=10002, authors source=all pubkeys in DB, relay mode=bootstrap
• "Backfill contact lists" — kinds=3, authors source=followed set, relay mode=bootstrap

Job status table

Below the form, a table showing recent jobs:

Label	Filter	Batches	Status	Events	Started	Actions
Missing profiles	kinds:0 authors:9178	92	running 45/92	4,012	2m ago	Cancel
Kind-0 Jan 2024	kinds:0 since:.. until:..	4	complete 4/4	1,203	1h ago	—
Relay lists	kinds:10002 authors:31994	320	complete 320/320	28,103	3h ago	—

Admin API

admin/api/custom_backfill.php

POST — Create a new job:

{
  "action": "create",
  "label": "Missing profiles",
  "kinds": "0",
  "authors_source": "kind1_authors",
  "authors_list": "",
  "authors_missing_kind0": true,
  "ids_list": "",
  "since_ts": null,
  "until_ts": null,
  "limit": 0,
  "tag_filters": "",
  "batch_size": 100,
  "relay_mode": "bootstrap",
  "relay_list": ""
}

The API:

1. Resolves authors_source to a concrete author list:
   • all_in_db → SELECT DISTINCT pubkey FROM events
   • kind1_authors → SELECT DISTINCT pubkey FROM events WHERE kind = 1
   • followed → SELECT pubkey FROM caching_followed_pubkeys
   • manual → parse authors_list
   • none → empty (time-window mode)
2. If authors_missing_kind0, filter to: NOT EXISTS (SELECT 1 FROM events WHERE kind = 0 AND pubkey = ...)
3. Builds the filter_json from kinds/since/until/limit/tag_filters.
4. Determines batch_mode:
   • If authors resolved → authors
   • Else if IDs provided → ids
   • Else → none (time-window)
5. Creates batch rows:
   • authors/ids mode → split into groups of batch_size, one batch per group per relay
   • none mode → one batch per selected relay
6. Inserts job + batches, returns {"ok": true, "job_id": 42}.

POST — Cancel a job:

{"action": "cancel", "job_id": 42}

Sets status to cancelled; the daemon checks status between batches.

GET — List jobs (for the status table):

GET /api/custom_backfill.php?limit=20

Returns recent jobs with progress + batch counts.

Caching Daemon

New module: caching/src/custom_backfill.c / .h

typedef struct {
    int has_pending;
    long active_job_id;
    int active_batch_index;
    time_t last_tick;
} cr_custom_backfill_t;

void cr_custom_backfill_init(cr_custom_backfill_t *cb);
int cr_custom_backfill_tick(cr_custom_backfill_t *cb,
                            nostr_relay_pool_t *upstream,
                            cr_sink_t *sink);

Main loop integration

In caching/src/main.c main loop, add a tick after the existing backfill tick:

/* Custom backfill tick — runs independently of the followed-set backfill. */
cr_custom_backfill_tick(&custom_bf, upstream, &sink);

Tick logic

1. If no active job, poll caching_custom_backfill_jobs for a pending job.
2. If found, atomically claim it (set status=running, started_at=now).
3. Pick the next pending batch for this job.
4. Build the NIP-01 REQ filter:
   • Start from the job's filter_json (kinds, since, until, limit, tags).
   • If batch_mode='authors': add "authors": [batch_items]
   • If batch_mode='ids': add "ids": [batch_items]
   • If batch_mode='none': filter is already complete (time-window scan)
5. Resolve relays based on relay_mode:
   • bootstrap → use the upstream pool's relays
   • outbox → look up each author's kind-10002 outbox relays (only for author-batched mode; falls back to bootstrap if no outbox found)
   • custom → use the job's relay_list
6. Send REQ to the selected relay(s), collect events until EOSE or timeout.
7. Insert each event via pg_inbox_insert_event() with source_class="backfill".
8. Update batch row: events_fetched, status=complete.
9. Update job row: increment done_batches, add to events_fetched.
10. If all batches done, set job status=complete, completed_at=now.
11. Throttle: one batch per caching_backfill_tick_interval_ms (reuse existing config).

New pg_inbox functions

/* Poll for a pending custom backfill job. Atomically claims it by setting
 * status='running'. Returns 0 on success, -1 if none available or error. */
int pg_inbox_claim_custom_backfill_job(long *out_job_id);

/* Get job parameters for a claimed job. Returns 0 on success, -1 on error.
 * Caller must free returned strings. filter_json is the raw JSONB string. */
int pg_inbox_get_custom_backfill_job(long job_id,
                                     char **out_filter_json,
                                     char **out_batch_mode,
                                     int *out_batch_size,
                                     char **out_relay_mode,
                                     char **out_relay_list);

/* Pick the next pending batch for a job. Atomically sets status='running'.
 * Returns 0 on success, -1 if no more batches. */
int pg_inbox_claim_custom_backfill_batch(long job_id,
                                         int *out_batch_index,
                                         char **out_batch_items,
                                         char **out_relay_url);

/* Update batch progress after a REQ completes. */
int pg_inbox_update_custom_backfill_batch(long job_id, int batch_index,
                                          int events_fetched,
                                          const char *status,
                                          const char *error);

/* Update job aggregate progress and optionally mark complete. */
int pg_inbox_update_custom_backfill_job(long job_id,
                                        int events_fetched,
                                        const char *status,
                                        const char *error);

/* Cancel a job (set status='cancelled'). Batches will be skipped. */
int pg_inbox_cancel_custom_backfill_job(long job_id);

Relay Inbox Poller

No changes needed. The existing caching_inbox_poller already consumes caching_event_inbox rows and inserts them into the events table. The sync_profile_from_event() trigger auto-populates the profiles table for kind-0 events. Custom backfill events flow through the same pipeline.

Implementation Steps

1. Schema migration — Add caching_custom_backfill_jobs and caching_custom_backfill_batches tables to src/pg_schema.sql.

2. Admin API — Create admin/api/custom_backfill.php with POST (create/cancel) and GET (list) handlers. The create handler resolves author sources, builds filter_json, determines batch mode, and creates batch rows.

3. Admin UI — Add the custom backfill form + preset buttons + job status table to the Backfill section in admin/index.php and handlers in admin/assets/app.js.

4. pg_inbox functions — Add the claim/progress/update functions to caching/src/pg_inbox.c and caching/src/pg_inbox.h.

5. Custom backfill module — Create caching/src/custom_backfill.c and .h with the tick logic. Handles all three batch modes (authors/ids/none).

6. Main loop integration — Add the custom backfill tick to caching/src/main.c.

7. Build & test — Rebuild with ./build_static.sh, start relay, submit jobs via the UI:

   • Test "Backfill missing profiles" (author-batched)
   • Test "Cache all kind-0 from date range" (time-window)
   • Verify profiles table populates and job status updates

Safety Considerations

• Throttle: Custom backfill respects the existing caching_backfill_tick_interval_ms (5s default) to avoid hammering upstream relays.
• One job at a time: Only one custom backfill job runs at a time; others stay pending until the active one completes.
• Cancellation: The daemon checks job status between batches; a cancelled job stops within one batch.
• Dedup: The caching_event_inbox unique constraint on event_id prevents duplicate inserts. The events table primary key does the same at the relay level.
• Author list size: For all_in_db (31,994 pubkeys), batching at 100 authors/REQ = 320 batches × 5s = ~27 minutes. Acceptable for a one-time job.
• Time-window mode: One REQ per relay. For 4 bootstrap relays = 4 batches × 5s = 20 seconds. Very fast.
• Relay limits: Some relays cap authors array length. The batch size (default 100) stays under common limits. UI warns if batch size > 500.
• Large time windows: A time-window scan with no limit could return huge result sets. The UI should warn if no limit is set and kinds include high-volume kinds (like 1). The daemon enforces a caching_max_event_json_bytes check per event (existing config).

Future Extensions

• Scheduled jobs: Run a job on a recurring schedule (e.g., "backfill missing profiles every hour"). Add a schedule_cron column.
• Job templates: Save a job configuration as a reusable template.
• Dry run: Show the resolved author list, batch count, and estimated time before submitting.
• Progress streaming: WebSocket or SSE for real-time batch progress instead of polling.
• Per-relay parallelism: Run batches against different relays concurrently instead of sequentially.