amethyst/relayBench/plans/2026-07-04-sync-throughput-1m.md

1M-event sync throughput: strfry↔strfry vs geode↔geode vs strfry→geode

Date: 2026-07-04

Four-pair NEGENTROPY comparison (2026-07-05, apples-to-apples)

All four source→sink pairings, all NIP-77 negentropy, empty sink pulls the same ~1M damus.io corpus, run sequentially (no contention). geode sink configured like strfry: verify ON, FTS OFF (-DsyncVerify=true -DsyncFts=false); strfry sync verifies too and has no FTS. Sinks: geode = its negentropy client; strfry = strfry sync --dir down. Sources: a geode relay seeded with geode import <corpus.ndjson> --db … then served normally, and a strfry relay (strfry import + serve) — each holding the corpus.

#	source → sink	sink engine	synced	time	throughput
A	geode → geode	geode neg client	994,162 / 994,832	142.6 s	6,971 ev/s
B	strfry → geode	geode neg client	994,828 / 994,917	148.7 s	6,690 ev/s
C	strfry → strfry	strfry sync	893,000	333.0 s	2,682 ev/s
D	geode → strfry	strfry sync	994,151	467.3 s	2,127 ev/s

Reads:

• The SINK sets the rate, not the source. geode sink ≈ 6.7–7.0k ev/s from either source; strfry sink ≈ 2.1–2.7k ev/s from either source. Sync throughput is an ingest property.
• geode ingests ≈ 2.6–3.3× faster than strfry via negentropy, verifying.
• Full geode↔strfry NIP-77 interop, both directions: B (geode client ↔ strfry server) and D (strfry client ↔ geode server) both complete. geode's negentropy server and client are wire-compatible with strfry's.
• Count artifacts (throughput is unaffected): C synced 893k, not ~994k, because strfry's negentropy snapshot excludes NIP-40-expired events, so a strfry source only offers ~893k. geode has no expiration cutoff ("no limits"), so a geode source offers its full ~994k (D transfers all of it). Same NIP-62/09/40 fairness family as 2026-07-04-sync-fairness-nip62-09-40.md.

REQ-mode counterpart (2026-07-05)

Same corpus + same geode sink config (verify ON, FTS OFF), but a plain paged REQ pull instead of negentropy (-DsyncMode=req, 20k-event until windows).

#	source → sink	mechanism	synced	time	throughput
A	geode → geode	geode paged REQ	994,144 / 994,832	200.3 s	4,963 ev/s ✅
B	strfry → geode	geode paged REQ	994,813 / 994,898	171.5 s	5,801 ev/s ✅
C	strfry → strfry	—	—	—	n/a
D	geode → strfry	—	—	—	n/a

• strfry has no REQ bulk sync. Its REQ path is strfry stream, which sends ["REQ","sub",{"limit":0}] — live-only, no historical replay; strfry sync is negentropy. So the strfry-sink REQ pairs (C, D) have no strfry-native mechanism — that gap is the finding: geode can bulk-pull over REQ, strfry can't.
• strfry→geode over REQ now completes (B) — earlier it stalled at ~310k because FTS-on ingest fell behind strfry's page serving and strfry killed the slow client at its 32 MB pending cap. With FTS off the sink keeps pace, so strfry never trips the cap. (Completion over REQ is contingent on the sink out-running the source; negentropy is immune because it's pull-paced.)
• REQ is ~15–30% slower than negentropy for the geode sink (4,963–5,801 vs 6,690–6,971): the paged until windows re-fetch boundary duplicates and range- scan the source, vs negentropy's direct id fetch. And geode-as-a-REQ-source is slower than strfry-as-a-source (A 4,963 < B 5,801) — SQLite range scans vs strfry's LMDB.

---

Question

When syncing a full 1,000,000-event corpus between two local relays, what is the end-to-end throughput (events/second) for each pairing:

1. strfry → strfry (strfry's native strfry sync, NIP-77 negentropy)
2. geode → geode (geode's production MirrorWorker, REQ backfill)
3. strfry → geode (geode ingesting a strfry-served corpus)

Corpus: the latest ~1M events from relay.damus.io (relayBench/.corpus-cache/corpus-download-relay.damus.io-n1000000.ndjson). After strfry import + de-dup the source holds 997,980 events (a few thousand of the raw 1M are duplicates / superseded replaceables).

How each number is measured

Every run starts with an empty sink and pulls the whole corpus from a full source over a real loopback WebSocket, timed from first byte to the sink reaching the target (or plateauing). Sink = geode uses its real relay config: deferred-FTS indexing + the live negentropy index on (i.e. the same work a production geode does on every insert).

• strfry→strfry: strfry sync ws://src --dir down against a running source strfry. Native negentropy reconcile + strfry ingest. Script: relayBench/sync-throughput-strfry.sh (and the reuse variant that syncs a fresh sink from an already-running source).
• geode→geode: MirrorSyncThroughputTest default mode. An in-process KtorRelay preloaded with N events is mirrored by the production MirrorWorker (REQ backfill, trusted/skip-verify) into an empty geode.
• strfry→geode: MirrorSyncThroughputTest -DsyncSourceUrl=ws://…. See the "MirrorWorker can't do a bulk backfill from strfry" finding below — this mode drives NostrServer.ingest directly from a backpressured socket drain instead of MirrorWorker.

Results (1M corpus, single 16 GB host)

Pair	Mechanism	Synced	Time	Throughput	Completes?
strfry→strfry	strfry sync (negentropy)	997,977 / 997,980	~391 s	~2,550 ev/s	✅
geode→geode	MirrorWorker REQ (synthetic)	1,000,000 / 1M	76.0 s	13,161 ev/s	✅
strfry→geode	negentropy (geode client)	994,936 / 997,980	171.0 s	5,818 ev/s	✅
strfry→geode	paged REQ drain (real corpus)	~310k / 997,980	—	~7,000 ev/s *	❌ (strfry kills conn)

* the paged-REQ drain sustains ~7,000 ev/s (peaks ~11k warm) but does not finish — see "strfry kills a slow REQ client" below. Switching geode to a NIP-77 negentropy client (the same mechanism strfry sync uses) does finish: reconcile against strfry's full set took 11.4 s / 64 rounds (empty local → 995,024 need-ids), then a client-paced fetch (2,000-id REQ batches) + ingest completed at ~7k ev/s steady-state, 5,818 ev/s end-to-end. (The 994,936 vs 997,980 gap is the NEG-scope/expiry artifact — strfry's negentropy set excludes a few thousand expired/superseded events it still exports — same family as the strfry→strfry 3-event shortfall, not data loss.)

The headline, apples-to-apples comparison (both negentropy, same corpus, empty→full): strfry→geode 5,818 ev/s vs strfry→strfry ~2,550 ev/s — geode ingests real content ≈2.3× faster than strfry, and finishes the pull.

Two things make the raw table misleading — read the caveats:

1. geode→geode's 13,161 is inflated by synthetic content. That source preloads events with 4-byte content ("e$i"), empty tags, and a fake sig. Real damus.io events are ~2.4 KB on the wire with real content + tags, so FTS tokenization, tag indexing, and store growth all cost more. The real-content geode ingest rate is the ~7,000 ev/s measured on strfry→geode, not 13k. So the honest real-content ingest comparison is geode ~7,000 ev/s vs strfry ~2,550 ev/s (≈2.7× — consistent with the earlier ingest benchmarks; strfry's figure also carries negentropy reconcile).
2. strfry→strfry's 3-event shortfall is the known NIP-62/09/40 harness artifact (2026-07-04-sync-fairness-nip62-09-40.md), not data loss.

Neither geode number is a like-for-like protocol comparison with strfry→strfry (REQ streaming vs negentropy reconcile); they're each relay's real "pull the whole corpus" path.

Finding: MirrorWorker can't bulk-backfill from a fast foreign relay

The first strfry→geode attempts via the production MirrorWorker stalled at ~500, then ~16k of 1M and never converged. Root cause, in order of discovery:

1. strfry's maxFilterLimit (default 500). MirrorWorker's down path is a plain REQ with no limit. strfry caps an unbounded REQ at maxFilterLimit stored events, sends EOSE, then streams only live events. geode-as-source has no such cap, which is why geode→geode fully converges. Fix for the benchmark: boot the source strfry with --set relay.maxFilterLimit=5000000 --set relay.maxPendingOutboundBytes=0. A raw REQ then returns all 997,986 events in 12.8 s (~78k ev/s) — strfry is not the bottleneck.

2. MirrorWorker's unbounded intake channel. Its SubscriptionListener callback can't suspend, so events funnel through a Channel.UNLIMITED via trySend into a consumer that calls the (bounded, backpressured) IngestQueue. This is a deliberate trade — blocking the shared OkHttp reader would park other upstreams' backlogs (see MirrorWorker kdoc). For live tailing it's correct. For a 1M bulk backfill from a 78k-ev/s source into a ~7k-ev/s sink, the unbounded channel balloons past the heap and the JVM dies GC-thrashing (observed: consumer stops at ~16k/1M). IngestQueue itself is bounded (capacity 1024) and submit suspends, so a direct feeder gets proper backpressure — the fragility is only the unbounded hop in front of it.

3. …and even a correctly-backpressured, client-paced REQ can't finish. The benchmark's external drain replaces MirrorWorker with a dedicated socket whose reader we are free to block (bounded hand-off channel), walking the corpus newest→oldest in bounded limit=20000 windows. That pulls cleanly at ~7,000 ev/s… until ~310k events, where it stalls every time (index on/off, 2 GB heap or 10 GB — same wall). Root cause, from the strfry source log: every drain connection disconnects at exactly Pending: 32.01M (1006). geode's real-content ingest gradually slows as its in-memory store grows; once it drops below strfry's send rate, strfry's outbound buffer for the in-flight page crosses its maxPendingOutboundBytes = 32 MB cap and strfry kills the connection. The pager then waits forever for an EOSE that never arrives. Raising strfry's cap just moves the wall (it OOM-buffers gigabytes instead — the Pending: 1.05G / Broken pipe seen earlier).

Conclusion: cross-relay bulk sync needs negentropy, not REQ

strfry's REQ serving is structurally hostile to any client slower than its scan: it buffers outbound and hard-kills (or OOM-buffers) the laggard. No amount of client-side backpressure or paging fixes this, because the failure is on strfry's send side. NIP-77 negentropy is the only mechanism that completes: it is pull/reconcile-based and client-paced (strfry's own sync fetches the diff 50 events at a time), so a slow client is never killed. This is exactly why strfry sync — the strfry→strfry path — is negentropy, and it is the direct answer to "why aren't we using negentropy for geode too": we should, and over plain REQ geode structurally cannot finish a 1M pull from strfry.

Implemented: MirrorWorker now mirrors strfry's two-phase model

MirrorWorker gained a NIP-77 "sync" catch-up phase (geode's equivalent of strfry sync --dir down) that runs once per down/both upstream before the live REQ tail:

• Catch-up reconciles the local set against the upstream over the historical [now - backfill_seconds, now] window and downloads only the diff, via the ready-made INostrClient.negentropySyncOrFetch — client-paced, so strfry can't overrun/kill us, and it completes the bulk pull. A small localEntries param was added to the public negentropySync/ negentropySyncOrFetch so the reconcile diffs against what we already hold (incremental, like strfry sync) instead of re-downloading.
• Either mode, transparently: negentropySyncOrFetch auto-falls back to paged REQ for an upstream that doesn't speak NIP-77 — no config toggle.
• Live tail unchanged: the REQ subscription now starts at now when catch-up is on (history is the sync's job); the two windows overlap at now and the store's unique-id constraint dedups the seam.

Both directions, matching strfry sync --dir both (which strfry's source confirms is bidirectional negentropy — doUp = both||up, doDown = both||down):

• down catch-up pulls what the upstream has and we lack (negentropySyncOrFetch, with paged fallback), then the live REQ sub tails.
• up catch-up reconciles and pushes what we have and the upstream lacks (streaming negentropyReconcile, publishing each onHaveIds batch straight to client.publish), then the live up-session tails. Negentropy-only (no paged fallback needed — the live up-session covers a non-NIP-77 upstream). The streaming reconcile keeps peak memory at one id batch rather than materializing the whole have/need diff (an important distinction at 1M — the full diff is ~100 MB of id strings). The push runs as a reconcile→push convergence loop: client.publish's outbox is best-effort under a bulk burst (each publish also churns a reconnect; measured ~1–2% dropped per pass), so each round re-reconciles — the reconcile is the delivery check — and re-pushes only the stragglers until the have diff is empty. Observed on the 3000-event up test: 3000 → 69 → 2 → 0 across 3 rounds, lossless.

Same vocabulary as strfry throughout — one [[mirror]] entry, one dir (down/up/both) driving both phases; negentropy-vs-REQ is an internal transport detail. The geode binary opts in (Main passes negentropyBackfill = true + the store); the MirrorWorker default stays off so existing live-REQ tests are unchanged. See MirrorNegentropyCatchUpTest: the down test isolates catch-up from the live tail by preloading historical events a live-only sub can't deliver (3000 + 1 live); the up test pushes 3000 local events to an empty sink.

Structural note: strfry keeps these as separate commands (sync = negentropy both-ways; router/stream = REQ live both-ways). geode folds both into one MirrorWorker under a single dir — more integrated, same semantics.

Remaining follow-ups: liveNegentropySnapshot-based local enumeration for very large mirrors (the up path's snapshotIdsForNegentropy currently scans); a single-pass reconcile for dir=both (today it runs one reconcile per direction); and optionally bounding the live-tail intake per-upstream.

Separately worth a look: geode's real-content ingest decays from ~11k→~7k ev/s as the in-memory store grows to a few hundred k — expected B-tree/FTS cost, but worth confirming it's not superlinear at 1M on a disk-backed store.

Why the geode number is a paged REQ, not negentropy — and why it's the same

strfry→strfry already is negentropy (strfry sync). The natural question: shouldn't strfry→geode use negentropy too, for parity?

NIP-77 sync is two phases: (1) reconcile — exchange range fingerprints to compute the set difference, no events moved; (2) transfer — fetch the missing events. Into an empty sink phase 1 trivially says "you're missing everything," so the whole cost collapses onto phase 2: fetch + ingest all ~998k events in bounded batches (strfry's own sync pulls 50 at a time — DOWN: 50 events in its log). Therefore:

• strfry→strfry's ~2,600 ev/s is ingest/transfer-bound, not reconcile-bound.
• Into an empty geode, negentropy and the paged-REQ drain measure the same thing — geode's paced ingest rate. The paged drain is negentropy's transfer phase minus a near-free reconcile round.
• Negentropy's reconcile only wins when the sink is already mostly full (small diff). Geode's reconcile performance there is measured separately — at 1M, geode's negentropy (kmp-negentropy v1.2.0, PrefixSumStorageVector) is ~1.5–4× faster than strfry's C implementation (see the negentropy-speedup tasks / 2026-07-04-sync-fairness-nip62-09-40.md).

So the paged-REQ figure is the correct empty-sink strfry→geode number; a full negentropy-client run lands at the same place plus a cheap reconcile.