laantungir/c-relay-pg
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
master
c-relay-pg/plans/cache_all_feasibility_test_plan.md
T

17 KiB
Raw Permalink Blame History

Cache-All Feasibility Test Program

Goal

Build a standalone C test program that connects to all outbox relays of followed pubkeys and attempts to subscribe to all events (no author filter) to determine if relays will tolerate this level of data flow. The program logs all relay interactions (disconnections, rate limiting, CLOSED messages, NOTICE messages) so we can assess feasibility before modifying the caching daemon.

Background

The current caching daemon subscribes only to events from followed pubkeys using authors=[followed_set] filters. The proposed "cache everything" approach would subscribe to all events on those relays and prune old non-followed events later. The key unknown is whether relays will allow this — they may disconnect or rate-limit clients that request too much data.

Preliminary Probe Results (60s on 3 major relays)

Before designing the test program, we ran 60-second probes using nak req --stream with no filter (all events) against 3 major relays to measure real-world data volume and kind distribution.

Relay Connectivity

Relay Events/60s Avg Bytes Max Event Unique Pubkeys Notes
nos.lol 2,634 1,713 44,372 (kind 30089) 1,228 Connected OK
relay.primal.net 1,547 1,920 207,377 (kind 3) 613 Connected OK
relay.damus.io 0 — — — HTTP 503 (unavailable)

Kind Distribution (nos.lol + primal combined, ~4,181 events)

Kind Description Count % Avg Bytes Notes
21059 Gift wrap (NIP-59) 1,063 25.4% 2,574 Encrypted DMs — large, noisy
5 Deletion requests 592 14.2% 447 Very common
30078 App-specific data 173 4.1% 3,670 Variable size, can be large
20001 Ephemeral (key exchange?) 253 6.0% 849 Short-lived
22580 WebRTC signaling 193 4.6% 795 Noise
1059 Gift wrap seal (NIP-59) 154 3.7% 3,023 Encrypted wrapper
22734 WebRTC signaling 168 4.0% 427 Noise
1 Text notes 58 1.4% 812 Primary content
7 Reactions 59 1.4% 534 Likes
0 Profile metadata 33 0.8% 985 Replaceable
3 Follow lists 10 0.2% 43,077 Replaceable, can be huge
10002 Relay lists 55 1.3% 418 Small
9735 Zap receipts 22 0.5% 2,110 Payment confirmations
6 Reposts 11 0.3% 1,876
30023 Long-form articles 3 0.1% 6,119
30089 Chunked data 16 0.4% 43,907 Very large (avg 44KB!)
30001 Lists (encrypted) 4 0.1% 30,538 Large encrypted content
30815 Ephemeral large 4 0.1% 37,534 Very large (avg 37KB!)
25555 App data 31 0.7% 5,115
13194 NWC info 15 0.4% 514 Wallet connect
445 Encrypted group msg 22 0.5% 1,452
Other 90+ other kinds ~600 ~14% varies Long tail of app-specific kinds

Key Findings

  1. Gift wrap (kind 21059) dominates: 25% of all events are NIP-59 gift wraps (encrypted DMs). These are large (~2.5KB avg) and mostly noise for a caching relay. They have expiration tags and are ephemeral by nature.

  2. Deletion requests (kind 5) are extremely common: 14% of all events. This is surprising — relays are very busy processing deletes.

  3. WebRTC signaling (kinds 22580, 22734, etc.) is ~10% of traffic: These are ephemeral signaling events for video/voice calls. Pure noise for caching.

  4. "Social" content (kinds 1, 7, 6, 0, 3) is only ~4% of total events: The vast majority of relay traffic is NOT the content users actually see.

  5. Some kinds are extremely large: Kind 30089 averages 44KB, kind 30001 averages 30KB, kind 30815 averages 37KB. These would consume significant storage.

  6. Data volume is manageable: ~2,600-4,200 events/min across 2 relays. At this rate, a 24h test would collect ~3.7-6M events. Storage would be ~6-10 GB of raw JSON.

  7. Damus.io is unavailable: Returns HTTP 503. This may be temporary or may indicate they block unfiltered subscriptions.

Implications for the Test Program

  • Subscribe to ALL kinds for the test — we need to measure which kinds relays actually send and whether they tolerate the volume.
  • Do NOT save full event JSON during the test — at ~2MB/min, a 24h run would produce ~3GB of raw data per relay. Instead, log kind/pubkey/size summaries.
  • Track kind 21059 (gift wrap) separately — it's the dominant kind and may need special handling in the real implementation.
  • Monitor for CLOSED messages — if relays start closing subscriptions due to volume, we'll see it in the raw relay log.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                  cache_all_feasibility_test                  │
│                                                             │
│  1. Load root npubs from config.jsonc                       │
│  2. Resolve follow graph (kind-3) → followed set            │
│  3. Discover outbox relays (kind-10002) for each follow     │
│  4. Compute minimum covering set of relays                  │
│  5. Connect to all covering relays                          │
│  6. Subscribe to ALL events (no authors filter) on each     │
│  7. Log every event received (counts + stats, not full JSON)│
│  8. Log every relay message: CLOSED, NOTICE, EOSE, error    │
│  9. Run for configurable duration (default 24h)             │
│ 10. Save summary report + per-relay stats to files          │
└─────────────────────────────────────────────────────────────┘

File Structure

All new files go in caching/ directory, alongside the existing daemon:

caching/
├── Makefile                          # Modified: add test target
├── src/
│   ├── main.c                        # Existing daemon (unchanged)
│   ├── cache_all_test.c              # NEW: test program entry point
│   ├── cache_all_test.h              # NEW: test program header
│   ├── debug.c / debug.h             # Existing (reused)
│   ├── config.c / config.h           # Existing (reused for config loading)
│   ├── state.c / state.h             # Existing (reused for pubkey set)
│   ├── follow_graph.c / follow_graph.h # Existing (reused)
│   ├── relay_discovery.c / relay_discovery.h # Existing (reused)
│   └── ...                           # Other existing files unchanged

Detailed Design

1. Config Loading (reuse existing)

Reuse caching/src/config.c and caching/src/config.h to load root npubs, upstream relays, kinds, and follow graph settings from the same .jsonc config file the daemon uses.

2. Follow Graph Resolution (reuse existing)

Reuse caching/src/follow_graph.c to:

  • Decode root npubs to hex
  • Query each root's most recent kind-3 contact list from bootstrap relays
  • Build the cr_pubkey_set_t of followed pubkeys

3. Relay Discovery (reuse existing)

Reuse caching/src/relay_discovery.c to:

  • Query kind-10002 for each followed pubkey
  • Parse "r" tags to build outbox relay map
  • Compute minimum covering set of relays

4. Subscription Strategy

For each relay in the covering set, open a subscription with:

// Filter: ALL events (no authors, no kinds, no limit)
cJSON *filter = cJSON_CreateObject();
cJSON_AddItemToObject(filter, "since", cJSON_CreateNumber((double)time(NULL)));

This requests every new event from now onward on that relay.

Key difference from existing daemon: No authors filter. This means we receive events from everyone on that relay, not just followed pubkeys.

5. Event Handling & Logging

The on_event callback should:

  1. Count events per relay (maintain a per-relay counter)
  2. Count events per kind (maintain a kind distribution map)
  3. Count events per pubkey (track which pubkeys are most active)
  4. Log at INFO level every N events (e.g., every 1000) with summary stats
  5. Do NOT save full event JSON to disk during the run (would be too much data)
  6. Periodically checkpoint (every 5 min) a summary to a file

6. Relay Interaction Logging — Raw Relay Responses

Critical requirement: Log the actual raw response text from the relay, not an interpreted summary. The relay's own words are what matter.

The on_event, on_eose, and subscription status callbacks from nostr_core_lib provide status strings and message content. These must be logged verbatim.

What to log and how:

Trigger Log Level What to Log (verbatim relay text)
EOSE INFO [RELAY] <url> EOSE
CLOSED WARN [RELAY] <url> CLOSED: <relay's exact reason text>
NOTICE WARN [RELAY] <url> NOTICE: <relay's exact notice text>
Error ERROR [RELAY] <url> ERROR: <relay's exact error message>
Disconnect WARN [RELAY] <url> DISCONNECTED: <transport-level error if any>
Reconnect INFO [RELAY] <url> RECONNECTED
OK (publish) INFO [RELAY] <url> OK: <event_id> <relay's exact message>

Do NOT categorize or summarize. If a relay says:

"rate-limited: please wait 60 seconds before sending new requests"

log that exact string. Do NOT log just "rate-limited".

Raw Relay Log File

In addition to the normal debug log output, write a raw relay log file:

cache_all_test_raw_relay_<timestamp>.log

This file contains only relay responses, one per line, in this format:

[TIMESTAMP] [RELAY] <relay_url> <RAW_RESPONSE>

Example:

[2026-08-02 13:01:00] [RELAY] wss://relay.damus.io CLOSED: "rate-limited: please wait 60 seconds before sending new requests"
[2026-08-02 13:01:05] [RELAY] wss://relay.damus.io RECONNECTED
[2026-08-02 13:01:10] [RELAY] wss://relay.damus.io NOTICE: "too many subscriptions, closing oldest"
[2026-08-02 13:02:00] [RELAY] wss://relay.primal.net EOSE
[2026-08-02 13:02:01] [RELAY] wss://relay.primal.net ERROR: "connection closed unexpectedly"

This file is append-only and can be tail -f'd during the test run.

Event Logging

For events received, log at TRACE level (not to the raw relay log):

[TRACE] [EVENT] relay=<url> kind=<N> pubkey=<first8chars>... id=<first8chars>...

This gives enough to correlate without flooding the log with full JSON. Every 1000 events, log a summary at INFO level:

[INFO] [STATS] 5000 events received total | relayA: 3200 relayB: 1800 | kinds: 1=4500 7=500

7. Summary Report

At the end of the run (or on SIGINT/SIGTERM), write a report file:

cache_all_test_report_<timestamp>.txt

Contents:

=== Cache-All Feasibility Test Report ===
Duration: 24h 3m 12s
Config: ./caching_relay_config.jsonc

=== Relay Summary ===
Relay                    Events  EOSE  CLOSED  NOTICE  Errors  Status
wss://relay.example.com  124532  12    0       2       0       OK
wss://relay2.example.com 0       0     3       5       2       BLOCKED

=== Kind Distribution ===
Kind    Count     %
0       1,234     0.5%
1       234,567   94.2%
3       567       0.2%
7       12,345    5.0%
9734    234       0.1%

=== Top 10 Pubkeys by Event Count ===
pubkey_hex_here...        12,345 events
pubkey_hex_here...        8,901 events
...

=== Raw Relay Response Log ===
[2026-08-02 13:01:00] wss://relay.damus.io CLOSED: "rate-limited: please wait 60 seconds before sending new requests"
[2026-08-02 13:01:05] wss://relay.damus.io RECONNECTED
[2026-08-02 13:01:10] wss://relay.damus.io NOTICE: "too many subscriptions, closing oldest"
[2026-08-02 13:02:00] wss://relay.primal.net EOSE
[2026-08-02 13:02:01] wss://relay.primal.net ERROR: "connection closed unexpectedly"
...

The Raw Relay Response Log section is a copy of the raw relay log file (cache_all_test_raw_relay_<timestamp>.log). It contains the verbatim text from each relay response, not interpreted or summarized.

8. Per-Relay Stats File

Additionally, write a JSON file with per-relay detailed stats:

cache_all_test_stats_<timestamp>.json

This can be used for programmatic analysis. It includes the raw relay response text for each interaction, not just counts.

Implementation Steps

Step 1: Create caching/src/cache_all_test.h

Header file declaring the test program's public interface:

#ifndef CACHE_ALL_TEST_H
#define CACHE_ALL_TEST_H

/* Run the cache-all feasibility test.
 * config_path: path to .jsonc config file
 * duration_seconds: how long to run (0 = run until SIGINT)
 * log_level: debug level 0-5
 * Returns 0 on success, -1 on error.
 */
int run_cache_all_test(const char *config_path, 
                       long duration_seconds, 
                       int log_level);

#endif

Step 2: Create caching/src/cache_all_test.c

Main implementation file with these sections:

  1. Includes and forward declarations
  2. Per-relay stats tracking structure
  3. Global stats accumulator
  4. Callback implementations (on_event, on_eose, on_status)
  5. Report generation (write summary + JSON stats)
  6. Main entry point (run_cache_all_test)

Step 3: Modify caching/Makefile

Add a new target cache_all_test that compiles the test program:

TEST_SRC = src/cache_all_test.c src/debug.c src/jsonc_strip.c src/config.c \
           src/state.c src/follow_graph.c src/relay_discovery.c

cache_all_test: $(TEST_SRC) $(NOSTR_CORE_LIB)
    # ... compile to ../build/cache_all_test

Step 4: Build and Run

cd caching && make cache_all_test
./build/cache_all_test -c caching_relay_config.jsonc -d 3 -t 86400

Output Files

File Contents
cache_all_test_raw_relay_<timestamp>.log Primary output. One line per relay response, verbatim text. Can be tail -f'd live.
cache_all_test_report_<timestamp>.txt Summary report with counts, kind distribution, top pubkeys, and raw relay log section.
cache_all_test_stats_<timestamp>.json Machine-readable JSON with per-relay stats including raw response texts.

What We're Measuring

  1. Relay tolerance: Do relays CLOSE our subscription or disconnect us?
  2. Rate limiting: How often do we get rate-limited? What are the cooldown periods?
  3. Data volume: How many events per hour per relay? What's the kind distribution?
  4. Connection stability: How often do relays drop us? Do they allow reconnection?
  5. Pubkey diversity: How many unique pubkeys are posting? What's the ratio of followed vs non-followed events?

Success Criteria

The test is considered a success (feasible) if:

  • At least 80% of relays maintain the subscription for the full duration
  • Rate limiting events are infrequent (< 5 per relay per day)
  • No relay permanently bans or blacklists the connection
  • Data volume is manageable (under ~1M events/day total)

The test is considered a failure (not feasible) if:

  • Most relays CLOSE the subscription within minutes
  • Rate limiting is constant (every few minutes)
  • Multiple relays permanently disconnect

Non-Goals

  • This test does NOT save events to PostgreSQL
  • This test does NOT modify the existing caching daemon
  • This test does NOT implement pruning logic
  • This test does NOT need to be efficient for production use

Mermaid Diagram

flowchart TD
    A[Start] --> B[Load config from .jsonc]
    B --> C[Resolve follow graph kind-3]
    C --> D[Discover outbox relays kind-10002]
    D --> E[Compute min covering set]
    E --> F[Connect to all covering relays]
    F --> G[Subscribe to ALL events no authors filter]
    G --> H{Test duration reached or SIGINT?}
    H -->|No| I[Pump relay pool]
    I --> J[Count events per relay/kind/pubkey]
    J --> K[Log relay interactions CLOSED/NOTICE/errors]
    K --> L[Periodic checkpoint every 5 min]
    L --> H
    H -->|Yes| M[Write summary report]
    M --> N[Write per-relay JSON stats]
    N --> O[Cleanup and exit]

Files to Create

File Purpose
caching/src/cache_all_test.h Header with public API
caching/src/cache_all_test.c Main implementation (~400-500 lines)

Files to Modify

File Change
caching/Makefile Add cache_all_test target

Files NOT Modified

The existing caching daemon files are not touched:

  • caching/src/main.c — unchanged
  • caching/src/backfill.c — unchanged
  • caching/src/live_subscriber.c — unchanged
  • caching/src/pg_inbox.c — unchanged
  • caching/src/forward_catchup.c — unchanged
  • caching/src/pg_config.c — unchanged
Reference in New Issue View Git Blame Copy Permalink