Acts on the measured gaps from TagAuthorIndexBenchmark and
FsDriverSelectionBenchmark:
- FsQueryPlanner: replace the fixed tags -> kinds -> authors driver
order with a cost-based pick. Every legal driver (each tagsAll value,
each tags key's value union, the kind set, the author set) opens a
lazy directory iterator; all are drained in lockstep and the first to
exhaust (the smallest listing) drives, so a giant idx/kind tree is
never read past ~the smallest candidate's size. Fixes the 149 ms vs
3.4 ms (~44x at 30k events) authors+kinds+limit regression.
- EventIndexesModule.ensureOptionalIndexes + SQLiteEventStore: flag-
gated indexes are runtime config, not schema. An idempotent
CREATE INDEX IF NOT EXISTS pass now runs on every open, so flipping
an IndexingStrategy flag on an existing DB builds the index without
a user_version bump.
- Desktop LocalRelayStore: enable indexEventsByPubkeyAlone. Shared
ViewModels (Nip65RelayList, PrivateOutboxRelayList, VanishRequests)
replay authors-only filters that full-scanned without
(pubkey, created_at); existing DBs pick the index up on next open.
- relayBench Scenarios: add "conversation" (tag ∩ author ∩ kind, the
DM-room shape, 65 client assembler call sites) and "reactions-watch"
(kind 7 + #e IN 150 hottest notes) so the uncovered archetypes get
head-to-head numbers vs strfry.
- quartz build: forward tagBenchScale/fsBenchScale to the test JVM.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RzkoN3SJHCZWRAiadXXG4w
Clears real Kotlin compiler warnings surfaced across quartz, cli,
relayBench, amethyst, and desktopApp:
- quartz Sha256/EventHasher/ScratchLocal: ThreadLocal.get() is nullable
in Kotlin; assert non-null (withInitial never yields null).
- quartz GitHttpClient: PriorityQueue.poll() under isNotEmpty() is
non-null; assert it.
- relayBench CorpusDownloader: drop redundant !! on smart-cast Long;
Jackson fields() -> properties().
- cli GrapeRankCommand: drop redundant ?. where latest is smart-cast.
- PodcastRemoteContent: OkHttp body is non-null; drop dead elvis.
- Dead/redundant expressions: remove no-op when-branch values and a
redundant trailing Unit (HomeScreen, LocalCache, EmbeddedTabLayer,
ParticipantHostActionsSheet, NestActionBar, ControlWhenPlayerIsActive,
ShareNoteAsImageScreen exhaustive-when else).
- CalendarEventDetailScreen / SetPasswordDialog / ProfileClinkOfferResolver:
drop always-true conditions (reorder to keep smart-casts).
- WalletColumnScreen: OkHttp body non-null; drop unreachable null-guards.
- PcmTapRegistry: the @OptIn used androidx.annotation.OptIn, which does
not opt into Kotlin's ExperimentalCoroutinesApi; use kotlin.OptIn.
- GitRepositoryScreen: suppress the standard ViewModel-factory cast.
- PushNotificationReceiverService: suppress override-of-deprecated.
- Desktop GlobalScope call sites: @OptIn(DelicateCoroutinesApi::class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
Two benchmark-integrity bugs that could invalidate sync numbers while a run
reports success:
- CorpusServerMain keyed its serve-existing decision only on port + row
count, so re-running a port with a different corpus/maxCount, or after a
load crashed mid-way, silently served a stale/partial DB. Gate reuse on a
completion sentinel keyed on corpus identity (path + byte length) and
maxCount, written only after a full load; on any mismatch the prior DB is
dropped and reloaded.
- CorpusDownloader treated CLOSED identically to EOSE, so a relay ending a
sub early (rate-limit/policy) after sending a partial page advanced the
cursor past the unsent tail — silent corpus loss. Treat CLOSED as a soft
failure (null → reconnect and retry the same cursor; the id dedup set
absorbs the re-fetch), distinct from EOSE which means the page is
complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Report, per sync pair: fetch coverage (did the peer REQ return every needed id,
without duplicates?), publish acks (accepted/rejected/unacked per target), and —
for any event a relay ends up missing — whether it was in the delta batch
delivered to that relay. In-batch-but-absent isolates ingest loss
(ack-without-persist) from a fetch/read gap, turning a vague "missing N" into a
pinned layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
When the identical-set reconcile shows a side missing events the reference set
has, log those events grouped by kind (plus a few samples with their tag keys)
instead of leaving a bare "did not converge". The harness already holds every
event by id, so it can resolve exactly what a relay dropped — turning a vague
verdict into an actionable ingest-semantics diagnosis (NIP-09 deletion / NIP-40
expiration / validation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The corpus downloader previously sampled ~100k events max (40-page cap
per kind bucket) and held everything in memory with no failure recovery.
Rework it for full-depth timeline pulls:
- page the latest events newest-first with an inclusive until cursor
(id-dedup absorbs the same-second overlap) instead of kind buckets
- no page cap: keep paging until the --limit target is met
- stream every unique event to an on-disk NDJSON spill instead of RAM
- checkpoint the pagination cursor per relay; interrupted downloads
resume where they left off
- reconnect with exponential backoff on socket drops/timeouts
- filter deterministically droppable events (kind-5 deletions are ~40%
of a live firehose, ephemerals, oversize) at page time so they never
count toward the download goal
Verified with a 1M-event pull from relay.damus.io (~2.1 GB raw,
4100 pages, ~22 min through a proxy) feeding a full geode vs strfry
run; corpus prepared to exactly 1,000,000 events, fingerprint
141e746599d901f5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
A NIP-77 server session rebuilt its reconciliation structure from
scratch on every NEG-OPEN: full id+created_at scan, per-entry hex
decode into a fresh StorageVector, O(n log n) seal. That cost grows
with the corpus and is paid even when nothing changed — the exact
shape of a periodic mirror's heartbeat, where N peers reconcile the
same broad filter over and over. relayBench measured 342 ms per
identical-set reconcile at 50k events vs strfry's 26 ms off its
always-current LMDB tree.
Reconciliation only *reads* the sealed storage, so one instance can
back any number of concurrent sessions:
- NegentropyServerSession now accepts a pre-sealed IStorage (the
List<IdAndTime> constructor remains and delegates).
- SessionBackend.sealedNegentropyStorage() builds + seals (null when
the set exceeds maxSyncEvents); LiveEventStore overrides it with a
single-slot cache keyed by (filter set, write generation) plus a 30s
TTL. The generation bumps on every accepted ingest; the TTL bounds
staleness from delete paths the counter can't see (expiration
sweeps, admin purges) — negentropy snapshots are point-in-time sets,
so seconds of staleness only means a peer briefly re-offers ids.
- NegSessionRegistry.open consumes the shared sealed storage;
over-cap NEG-ERR behavior unchanged (strfry parity).
relayBench gains a 'heartbeat' measurement — the identical-set
reconcile repeated immediately with no writes in between. At 50k
events: geode 342 ms -> 27.8 ms vs strfry 21.1 ms (near parity; was
13x). Cold reconciles (first open after a write) are unchanged.
Also fixes the GeodeVsStrfryNegentropySyncTest fixture to write
'nofiles = 0' so the opt-in interop test can boot strfry inside
containers with a low RLIMIT_NOFILE hard cap; the interop suite passes
against strfry v1-b80cda3 with the cache in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Reviewed strfry's LMDB indices (golpe.yaml) against geode's SQLite set:
the two are nearly isomorphic — time, id, kind+time, author+kind+time,
tag+time, plus conditional deletion/expiration/replaceable entries
(geode's are partial indexes, so ordinary events don't pay for them).
Nothing to drop. One real hole: strfry maintains a plain
pubkey(+created_at) index and geode had none, so an authors-only filter
(no kinds) — archive pulls, account-migration tools, 'everything by
these pubkeys' — degraded to a full walk of the time index. EXPLAIN
confirmed: SCAN query_by_created_at_id.
- quartz: IndexingStrategy.indexEventsByPubkeyAlone (default false —
clients query their supported kinds and can skip it) gates a new
query_by_pubkey_created index; DATABASE_VERSION 2→3 with an
idempotent migration that backfills it for opted-in strategies.
- geode: relayIndexingStrategy turns it on.
- relayBench: new 'author-archive' scenario — every kind by 3 *quiet*
pubkeys. Quiet is the point: prolific authors are dense in the time
index and a scan finds them quickly, which is why the suite never
caught this; sparse authors force the full walk.
Measured (50k corpus): author-archive EOSE p50 42.8 ms -> 3.6 ms (12x,
and the old path grows linearly with table size); ingest 5,337 -> 5,156
events/s (~3%, the one extra B-tree per event). strfry reference on the
same scenario: 0.52 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
strfry implements no NIP-50 at all, so geode's default setup pays FTS
tokenization on every searchable event for a feature the other side of a
benchmark isn't providing. The new switch removes it cleanly:
- geode: --no-search CLI flag and [options].full_text_search TOML key
(default true). When off, the store skips the FTS index entirely,
NIP-11 stops advertising 50 (an explicit [info] nips list stays
operator-authoritative), and search filters match nothing via the
existing QueryBuilder guard. RelayIndexingStrategy becomes
relayIndexingStrategy(fullTextSearch) with the stock val kept.
- relayBench: --geode-no-search runs the geode entry with the flag (relay
named geode-nosearch in reports); README documents the apples-to-apples
rationale and a recipe for benchmarking both geode flavors side by side.
Alternating A/B on the 10k corpus: 4,767/4,792 ev/s without search vs
4,111/3,977 with — NIP-50 costs ~18% of ingest throughput at current
write-path speed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Main's giant-REQ fix (18bd3600) replaced query()'s copy-on-add immutable
dedupe set with a spin-locked HashSet, but the rebase left queryRaw —
now the default REQ path — on the old pattern, which would have
reintroduced the O(n²) crawl for large replays. Both paths now share
the mutable-set-under-spinlock shape.
relayBench's sync driver moves to NegentropySession.fromEvents(),
following the session's new List<IdAndTime> constructor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
New :relayBench module that boots relay implementations as real external
processes under equivalent setups (persistent storage, sig verification on,
no auth) and compares them on the same corpus:
- Ingest: receipt->queryable-by-REQ latency (publish on one connection,
hammer-poll REQ{ids} on another), OK-ack latency, and pipelined corpus
replay throughput over N connections.
- Queries: client-realistic filters derived from the corpus (home feed,
thread, notifications, profiles, hashtag, by-ids, ...), time-to-EOSE
percentiles plus aggregate events/sec under concurrency; result-set
counts are cross-checked between relays and mismatches flagged.
- NIP-77 negentropy sync between every relay pair: 80%/80% slices with 60%
overlap, reconcile timing/rounds/wire-bytes per server side, delta
transfer, steady-state identical-set reconcile, convergence verified.
- Storage footprint after full ingest.
Corpora (all cached as NDJSON + manifest with a sha256 id fingerprint so
results are comparable across runs and machines):
- synthetic (default): deterministic to the byte — seeded keys, fixed
timestamps, seed-derived BIP-340 aux nonces — with a realistic social
shape (zipf authors, threads, reactions, reposts, zap request/receipt
pairs, hashtags);
- the checked-in real dump (quartz test fixture, ~31k unique 2024 events);
- external dumps (NDJSON or JSON array, gzip sniffed by magic bytes),
e.g. the 2.1M contact-list archive, with --max-event-bytes/--max-tags
raising both the corpus filter and the strfry config together;
- live download from public relays.
Every source runs through the same preparation: dedup, drop unsigned/
ephemeral/kind-5, enforce relay ingest caps, parallel Schnorr verify,
chronological sort.
relayBench/run.sh is the one-command entry point: builds geode + harness,
resolves strfry (STRFRY_BIN, PATH, or source build into .cache), runs the
suite and renders an ANSI report with per-metric bars and winners, plus
report.md and results.json under relayBench/results/<timestamp>/.
The harness client disables Nagle (TCP_NODELAY): with the JDK default, a
REQ following the previous round's CLOSE stalls a full delayed-ACK
interval and every latency floors at ~44 ms against both geode and strfry
(verified: ~0.3 ms with it off).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Claude
davotoula