# caching_relay

A C99 daemon that caches Nostr events from people you follow into a local relay,
so your Nostr client can point at a single fast local relay instead of fanning
out to dozens of upstream relays.

## What it does

1. Reads a `.jsonc` config file listing your **root npub(s)**, upstream relays,
   a local relay URL, and the event kinds to cache.
2. For each root npub, fetches its kind-3 contact list to discover **followed
   pubkeys** (your follows + their follows).
3. **Live-subscribes** to new events of the configured kinds from the union of
   followed pubkeys.
4. **Backfills** historical events using a progressive window-expansion strategy
   (24h → 7d → 30d → 90d → 365d), round-robin per pubkey, throttled to be polite
   to upstream relays.
5. **Re-publishes** every fetched event to the local relay via a plain WebSocket
   `EVENT` client connection (relay-agnostic - works with c-relay, c-relay-pg,
   or any Nostr relay).
6. Persists its backfill progress **in the config file itself** so a restart
   resumes where it left off.

## Build

### Static binary (recommended, c-relay style)

Produces a truly portable statically-linked MUSL binary with zero runtime
dependencies:

```bash
./build_static.sh
# or with debug symbols:
./build_static.sh --debug
# or cross-compile for arm64:
./build_static.sh --arch arm64
```

Output: `caching_relay` (in the project root).

Requires Docker. The build runs in an Alpine container that compiles
libsecp256k1, libwebsockets, and nostr_core_lib from source, then statically
links everything.

### Local build (if you have the shared libs installed)

```bash
make
```

Output: `caching_relay` (in the project root).

Requires: `libwebsockets`, `openssl`, `libsecp256k1`, `libcurl`, `zlib` shared
libraries, and a pre-built `../nostr_core_lib/libnostr_core_x64.a`.

## Usage

```bash
./caching_relay -d 3
```

The daemon looks for `caching_relay_config.jsonc` in the current directory by default.
You can override with `-c`:

```bash
./caching_relay -c /path/to/my_config.jsonc -d 3
```

Options:
- `-c, --config <file>` - Path to `.jsonc` config file (default: `./caching_relay_config.jsonc`)
- `-d, --debug <level>` - Log level 1-4 (1=error, 2=warn, 3=info, 4=debug). Default 3.
- `-h, --help` - Show help

Signals:
- `SIGINT` / `SIGTERM` - Graceful shutdown (saves state, closes connections)
- `SIGHUP` - Reload config (preserves backfill state)

## Config file

The config file [`caching_relay_config.jsonc`](caching_relay_config.jsonc) lives in the
project root. It is JSONC (JSON with `//` and `/* */` comments). The daemon
rewrites it as plain JSON when saving state (comments are not preserved on
rewrite).

Key fields:

| Field | Description |
|-------|-------------|
| `root_npubs` | npubs whose kind-3 follows list we crawl |
| `upstream_relays` | bootstrap relays for initial discovery (kind-3, kind-10002). Outbox relays are discovered dynamically via NIP-65. |
| `local_relay` | relay to publish cached events into (never queried) |
| `kinds` | event kinds to cache for followed people (e.g. `[1, 3, 6, 10000, 30023]`) |
| `admin_kinds` | kinds to follow specifically for root (admin) npubs. Use `["*"]` for all kinds. If omitted, admin uses same `kinds` as everyone else. |
| `backfill.window_schedule_seconds` | progressive window sizes in seconds |
| `backfill.events_per_tick` | max events per pubkey per backfill tick |
| `backfill.tick_interval_seconds` | delay between pubkey backfills (throttle) |
| `live.enabled` | enable live subscription |
| `follow_graph_refresh_seconds` | how often to re-resolve the follow graph |
| `state.*` | managed by the daemon - do not hand-edit |

## Architecture

See [`plans/plan.md`](plans/plan.md) for the full architecture document with
Mermaid diagrams.

The daemon uses a **NIP-65 outbox model**: instead of querying a fixed set of
bootstrap relays for everything, it discovers which relays each followed pubkey
actually posts to (via kind 10002 relay lists) and connects to those relays
dynamically. The `upstream_relays` in the config serve only as **bootstrap
relays** -- the initial set used to discover kind-3 and kind-10002 events before
the outbox relay map is built.

### Relay pool selection: minimum covering set

After discovering each followed pubkey's outbox relays (from their kind 10002),
the daemon computes the **minimum set of relays that covers all followed
pubkeys**. This is the classic set cover problem, solved with a greedy
approximation:

1. Build a map: `{relay_url -> set of pubkeys that list it in their 10002}`
2. Greedily pick the relay that covers the most uncovered pubkeys
3. Repeat until all pubkeys are covered (or no more relays to pick)
4. Always include the bootstrap relays in the final set (they may have events
   from pubkeys that don't publish a kind 10002)

This minimizes the number of WebSocket connections while ensuring every
followed pubkey is reachable. The daemon logs the selected relay set and which
pubkeys each relay covers.

Pubkeys that have no kind 10002 (or whose 10002 lists no relays) are covered by
the bootstrap relays as a fallback.

```
                    ┌─────────────────────────────────────────┐
                    │              caching_relay               │
                    │                                          │
   upstream relays  │  upstream_pool          sink_pool        │  local relay
   (damus, nos.lol) │  (query + subscribe)    (publish only)   │  (c-relay)
        │           │          │                   │           │      │
        │           │   follow_graph         relay_sink        │      │
        └──────────►│   live_subscriber ──────►│              ├─────►│
                    │   backfill ──────────────►│              │      │
                    │          │                   │           │      │
                    │       state + seen ring      │           │      │
                    └──────────────────────────────────────────┘      │
```

Two `nostr_relay_pool_t` instances:
- **upstream_pool** - holds bootstrap relays initially, then dynamically adds
  outbox relays discovered from kind 10002. Used for `query_sync` (kind-3 fetch,
  kind-10002 fetch, backfill) and the long-lived live subscription.
- **sink_pool** - holds only the local relay; used exclusively for
  `publish_async`. Never queried.

## Startup Flow

### First-time startup (empty local relay)

The daemon has never run before. The local relay has no cached events. The
daemon must bootstrap from the config's `upstream_relays` to discover the
outbox relays for each followed pubkey.

```
 START
   |
   v
 Load config (caching_relay_config.jsonc)
   |  - root_npubs, upstream_relays (bootstrap), kinds, admin_kinds
   |  - state.backfilled_until == 0  =>  first-time startup
   |
   v
 Create upstream_pool with bootstrap relays only
 Create sink_pool with local_relay
   |
   v
 Phase 1: Resolve follow graph (from bootstrap relays)
   |  - For each root npub: query_sync kind=3 from bootstrap relays
   |  - Parse "p" tags => followed pubkey set
   |  - Log: "follow: resolved N followed pubkeys"
   |
   v
 Phase 2: Discover outbox relays (NIP-65, kind 10002) from bootstrap relays
   |  - For each followed pubkey: query_sync kind=10002 from bootstrap relays
   |  - Parse "r" tags => per-pubkey relay list
   |  - Publish all kind-10002 events to local relay (cache them)
   |  - Dynamically add discovered relays to upstream_pool
   |  - Log: "relay_discovery: found N relays from M pubkeys"
   |  - Log: "upstream_pool: now connected to N relays" + list them
   |
   v
 Phase 3: Open live subscriptions on upstream_pool
   |  - follows_sub: non-admin pubkeys + regular kinds
   |  - admin_sub: admin npubs + admin_kinds (or all kinds)
   |
   v
 Phase 4: Begin progressive backfill
   |  - Window 0: 24h  -> query each pubkey from their outbox relays
   |  - Window 1: 7d   -> ...
   |  - Window 2: 30d  -> ...
   |  - Publish all fetched events to local relay
   |  - Save state to config file as each window completes
   |
   v
 Steady-state: live sub + periodic follow-graph refresh + periodic
   kind-10002 refresh + backfill re-cycle
   |
   v
 SHUTDOWN (SIGINT/SIGTERM) -> save state -> clean exit
```

### Subsequent startup (local relay already has cached events)

The daemon has run before. The local relay already has kind-3 and kind-10002
events cached from the previous run. The daemon can read these from the local
relay directly (fast, no network round-trip to bootstrap relays) and only falls
back to bootstrap relays for pubkeys it cannot find locally.

```
 START
   |
   v
 Load config (caching_relay_config.jsonc)
   |  - state.backfilled_until > 0  =>  subsequent startup
   |  - state.current_window_index, backfill_cursor preserved
   |
   v
 Create upstream_pool with bootstrap relays
 Create sink_pool with local_relay
   |
   v
 Phase 1: Resolve follow graph (from LOCAL relay first, bootstrap fallback)
   |  - Query local relay for kind=3 per root npub
   |  - If found locally: use it (fast, no upstream query)
   |  - If not found: fall back to bootstrap relays
   |  - Parse "p" tags => followed pubkey set
   |  - Log: "follow: resolved N followed pubkeys (L local, B bootstrap)"
   |
   v
 Phase 2: Discover outbox relays (from LOCAL relay first, bootstrap fallback)
   |  - Query local relay for kind=10002 per followed pubkey
   |  - If found locally: use it (fast)
   |  - If not found: fall back to bootstrap relays, cache result to local
   |  - Parse "r" tags => per-pubkey relay list
   |  - Dynamically add discovered relays to upstream_pool
   |  - Log: "relay_discovery: found N relays (L local, B bootstrap)"
   |  - Log: "upstream_pool: now connected to N relays" + list them
   |
   v
 Phase 3: Open live subscriptions on upstream_pool
   |  - (same as first-time)
   |
   v
 Phase 4: Resume backfill from saved state
   |  - Resume at state.current_window_index, state.backfill_cursor
   |  - No re-pull of already-backfilled windows
   |  - Continue progressive window expansion from where it left off
   |
   v
 Steady-state (same as first-time)
   |
   v
 SHUTDOWN -> save state -> clean exit
```

### Relay logging

The daemon logs all relay activity so you can see exactly which relays are
being used:

- `upstream: added wss://relay.damus.io` -- each relay added to the pool
- `relay_discovery: found 47 relays from 195 pubkeys` -- outbox discovery summary
- `upstream_pool: 50 relays connected:` -- full relay list at startup
- `relay_discovery: pubkey X -> wss://relay.example.com (local)` -- per-pubkey
  relay source (local cache vs bootstrap)
- `backfill: pubkey[3/195] abc123 -> wss://relay.example.com (12 events)` --
  which relay served each backfill query

## Dependencies

- [nostr_core_lib](../nostr_core_lib) - built with NIPs 1, 4, 6, 19, 42, 44
- libwebsockets, openssl, libsecp256k1, libcurl, zlib (all statically linked
  in the Docker build)

## Testing

Start a local c-relay, then run the daemon:

```bash
# Start local relay (from c-relay project)
cd ../c-relay/build && ./c_relay_static_x86_64 -p 8888 &

# Run the caching daemon (uses ./caching_relay_config.jsonc by default)
./caching_relay -d 3

# Verify events are cached (from another terminal)
nak req -k 1 -l 10 ws://127.0.0.1:8888
```
