Two relayBench runs (50k corpus, both geode variants side by side, then
order-reversed): identical-set reconcile 56/57 ms with the index vs
110/130 ms without in both orders (~2.2x, the post-write NEG-OPEN the
old cache always missed); ingest and first-ever reconcile deltas flip
with relay order, i.e. run-order noise — no regression. Also records
the run-order-bias protocol note for future A/Bs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestones 2-3 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
SQLiteEventStore now maintains the LiveNegentropyIndex when the strategy
opts in (geode's does by default; [negentropy].live_index = false turns
it off; app-side stores are untouched):
- Write paths collect a LiveIndexDelta and apply it after COMMIT while
still holding the writer mutex — rolled-back savepoint rows never
reach the index and updates land in exact commit order (also vs the
rebuild, which runs under the same mutex).
- Replaceable/addressable overwrites report the row their BEFORE-INSERT
trigger displaces via one indexed pre-SELECT that mirrors the trigger
predicate (including the NIP-01 lowest-id tie-break and the
d_tag-NULL case).
- Paths that can't itemize (kind-5, vanish, delete-by-filter/id,
expiration sweeps, clearDB) invalidate; the next NEG-OPEN rebuilds
from one scan on the writer connection.
- Until that first NEG-OPEN populates the index, ingest pays zero
bookkeeping — the populated check happens under the writer mutex so
it can't race the rebuild.
LiveEventStore serves a single unconstrained filter (the relay-relay
sync default; relayBench sends exactly this) from the index; everything
else keeps the scan+seal path and its single-slot cache.
Micro-benchmark at 50k events (LiveNegentropyBenchmark, in-container):
scan+seal cold path 80-100 ms; index post-write open 9-16 ms (~5-10x).
The relayBench A/B is the acceptance gate and comes next.
Correctness: LiveNegentropyIndexStoreTest asserts index content ==
snapshotIdsForNegentropy scan after every mutation pattern (overwrites,
losers, kind-5 rebuilds, filter deletes, mixed-outcome batches,
transactions); the full geode suite (NIP-77 + interop sync tests) runs
with the index on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestone 1 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
Sorted array with binary-search insert (near-tail in the common case),
itemized remove for displaced rows, wholesale invalidate for delete
paths that can't itemize, and sealed snapshots memoized per mutation
generation — reconcile only reads, so one snapshot backs any number of
concurrent sessions and stays immutable under later writes. Over-cap
answers null so the caller keeps the strfry-parity NEG-ERR.
Not wired into any store yet; next milestones plumb displaced-row
deltas from the SQLite modules and serve index-total filters from it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Design for backlog item 3 of the relay performance campaign: an
always-current (created_at, id) index maintained from the store's write
path so cold NEG-OPENs stop paying the full scan + O(n log n) seal
(~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers
the snapshot/COW model, the removal-correctness split (RETURNING deltas
for replaceable overwrites, wholesale invalidation for rare delete
paths), IndexingStrategy gating so app-side stores are untouched, and
the micro + relayBench A/B measurement plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Adds a public local-ingest entry to NostrServer for events that don't
arrive over a client connection (mirror/sync workers, import jobs). It
routes through the same group-commit IngestQueue and live fanout as a
client publish, and each Submission can opt out of the parallel
Schnorr-verify hook — the relay-to-relay trust model, for events
streamed from an upstream that already verified them (verify profiles
at ~8% of busy ingest CPU).
Library defaults are unchanged: skipVerify defaults to false everywhere
and nothing in the client-publish path can set it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
The worker's writer-mutex fast path can slip past its own scope
cancellation and run one batch against a store that close() already
freed. The resulting closed-connection exception escaped scope.launch
under a SupervisorJob with no handler — harmless in production, but the
kotlinx-coroutines-test global handler attributes it to whatever runTest
starts next (seen on CI as EventSourceServerTest.countUsesSource failing
with UncaughtExceptionsBeforeTest). Catch, log, and stop the pass:
nothing depends on it — the pre-search drain covers search correctness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Two issues surfaced by rebasing onto main:
- SQLiteConnectionPool.close() freed the writer's native handle without
taking the writer mutex, so a block still running on another thread
(the deferred-FTS catch-up worker's current batch outlives its scope
cancellation) could call sqlite3_prepare on a freed sqlite3* — native
heap corruption, SIGSEGV in sqlite3DbMallocRawNN (reproduced twice in
:geode:test). close() now reclaims every reader from the channel and
acquires the writer mutex before closing handles, then releases so
stragglers get the managed closed-connection exception. Idempotent
via a closed flag.
- main's NIP-50 fix (7deda28d) strips search-extension tokens before
every store query, but the zero-decode queryRaw replay path added on
this branch predates it and passed raw filters through — an
extensions-only search would hit SQLite FTS as column syntax and
error. queryRaw now strips like query/count/snapshot do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
FTS indexing ran inside every insert's transaction — a measurable slice
of write cost (relayBench: ~18% of ingest throughput) paid at publish
time for a feature only search queries read. It now runs as a watermark
catch-up:
- IndexingStrategy.deferFullTextSearchIndexing (default false; geode's
relay strategy enables it with search). Deferred inserts skip
tokenization entirely.
- FullTextSearchModule keeps a fts_catchup_state watermark (everything
<= last_row_id is indexed) and gains catchUpBatch(): scan past the
watermark, tokenize, advance — one write transaction per batch, so
publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at
MAX(row_id) for existing (synchronously indexed) databases.
- NostrServer runs the catch-up worker, poked by IngestQueue's new
onBatchCommitted hook, and *yields to publish traffic*: it only
drains while the queue has no backlog (IngestQueue.hasBacklog()), so
bursts ingest at no-FTS speed and tokenization fills the gaps.
- LiveEventStore drains the backlog synchronously before serving any
filter with a search term (query, queryRaw, count) — NIP-50 results
stay exactly as fresh as the synchronous path; the deferral is
invisible to correctness. Geode's existing search tests pass
unchanged through this path.
The first implementation reused reindexBatch and collapsed ingest 8x —
its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?'
matches on a plain FTS5 column, i.e. a full FTS-table scan per row
(O(n²) overall), and the worker competed with the replay for the writer
mutex. catchUpBatch therefore inserts without the delete (rows past the
watermark are never indexed; switching a DB between deferred and
synchronous strategies requires reindexAll, same rule as a
searchable-kinds change), and the worker backs off whenever publishes
are pending.
Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136
events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the
--no-search ceiling while keeping NIP-50 advertised and fresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
A NIP-77 server session rebuilt its reconciliation structure from
scratch on every NEG-OPEN: full id+created_at scan, per-entry hex
decode into a fresh StorageVector, O(n log n) seal. That cost grows
with the corpus and is paid even when nothing changed — the exact
shape of a periodic mirror's heartbeat, where N peers reconcile the
same broad filter over and over. relayBench measured 342 ms per
identical-set reconcile at 50k events vs strfry's 26 ms off its
always-current LMDB tree.
Reconciliation only *reads* the sealed storage, so one instance can
back any number of concurrent sessions:
- NegentropyServerSession now accepts a pre-sealed IStorage (the
List<IdAndTime> constructor remains and delegates).
- SessionBackend.sealedNegentropyStorage() builds + seals (null when
the set exceeds maxSyncEvents); LiveEventStore overrides it with a
single-slot cache keyed by (filter set, write generation) plus a 30s
TTL. The generation bumps on every accepted ingest; the TTL bounds
staleness from delete paths the counter can't see (expiration
sweeps, admin purges) — negentropy snapshots are point-in-time sets,
so seconds of staleness only means a peer briefly re-offers ids.
- NegSessionRegistry.open consumes the shared sealed storage;
over-cap NEG-ERR behavior unchanged (strfry parity).
relayBench gains a 'heartbeat' measurement — the identical-set
reconcile repeated immediately with no writes in between. At 50k
events: geode 342 ms -> 27.8 ms vs strfry 21.1 ms (near parity; was
13x). Cold reconciles (first open after a write) are unchanged.
Also fixes the GeodeVsStrfryNegentropySyncTest fixture to write
'nofiles = 0' so the opt-in interop test can boot strfry inside
containers with a low RLIMIT_NOFILE hard cap; the interop suite passes
against strfry v1-b80cda3 with the cache in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Reviewed strfry's LMDB indices (golpe.yaml) against geode's SQLite set:
the two are nearly isomorphic — time, id, kind+time, author+kind+time,
tag+time, plus conditional deletion/expiration/replaceable entries
(geode's are partial indexes, so ordinary events don't pay for them).
Nothing to drop. One real hole: strfry maintains a plain
pubkey(+created_at) index and geode had none, so an authors-only filter
(no kinds) — archive pulls, account-migration tools, 'everything by
these pubkeys' — degraded to a full walk of the time index. EXPLAIN
confirmed: SCAN query_by_created_at_id.
- quartz: IndexingStrategy.indexEventsByPubkeyAlone (default false —
clients query their supported kinds and can skip it) gates a new
query_by_pubkey_created index; DATABASE_VERSION 2→3 with an
idempotent migration that backfills it for opted-in strategies.
- geode: relayIndexingStrategy turns it on.
- relayBench: new 'author-archive' scenario — every kind by 3 *quiet*
pubkeys. Quiet is the point: prolific authors are dense in the time
index and a scan finds them quickly, which is why the suite never
caught this; sparse authors force the full walk.
Measured (50k corpus): author-archive EOSE p50 42.8 ms -> 3.6 ms (12x,
and the old path grows linearly with table size); ingest 5,337 -> 5,156
events/s (~3%, the one extra B-tree per event). strfry reference on the
same scenario: 0.52 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
The drain loop ran verify(batch N) and insert(batch N) strictly
back-to-back: the SQLite writer idled during every Schnorr verify and
the CPU cores idled during every commit. Split it into two stage
coroutines joined by a capacity-1 channel — a verifier that collects
and checks batch N+1 while the writer commits batch N. Same shape as
strfry's ingester/writer thread split.
Ordering (single verifier, single writer, FIFO handoff), OK-after-commit
semantics, per-row error isolation and submit() backpressure are all
unchanged; total in-flight grows by at most one batch.
Alternating A/B on the 10k corpus, 4-core host with the benchmark client
competing for the same cores: sequential 4,111/3,977/4,443 vs pipelined
4,050/5,020/5,092 events/s (~+13% mean). The overlap should widen on
dedicated relay hosts where verify has its own cores. Feature-parity
run after this change: geode --no-search 5,911 vs strfry 9,374 events/s
(1.59x, down from 2.2x at the start of the perf work).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Main's giant-REQ fix (18bd3600) replaced query()'s copy-on-add immutable
dedupe set with a spin-locked HashSet, but the rebase left queryRaw —
now the default REQ path — on the old pattern, which would have
reintroduced the O(n²) crawl for large replays. Both paths now share
the mutable-set-under-spinlock shape.
relayBench's sync driver moves to NegentropySession.fromEvents(),
following the session's new List<IdAndTime> constructor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Three changes, each validated head-to-head against strfry with relayBench
(same 10k-event corpus a1cd3517a8296911, stock configs, sig verify on):
- geode: RelayIndexingStrategy turns on indexEventsByCreatedAtAlone for
the relay's stores (quartz's DefaultIndexingStrategy stays off for
client-side stores). A relay can't predict client filters, and the
cheapest REQ of all — {"limit":N} — was a full-table scan + top-N
sort: 40 ms and O(table) growth before; 11 ms and index-streamed
(first event 30 ms -> 1.9 ms) after.
- quartz: zero-decode REQ replay. Stored events now stream as RawEvent
(tags kept in serialized form) and are spliced directly into wire
frames — no tags parse, no EventFactory dispatch, no re-serialize per
row. Gated on the new IRelayPolicy.filtersOutgoingEvents capability:
policies that can veto per-event delivery (none today) keep the
materialized path; everyone else skips it. Live post-EOSE delivery is
unchanged (live matching needs Event objects).
- quartz: StatementCachingConnection wraps the pool's writer and reader
connections, replaying prepared statements instead of re-preparing per
event/REQ (eager reset on return keeps cursors from holding table
locks; a 256-statement cap bounds client-controlled filter-shape
variety). Ingest went 3,000 -> 4,700 events/s (+55%) — prepare
overhead was the single largest non-crypto write cost.
Net effect on the benchmark: ingest gap vs strfry narrowed from 2.2x to
1.8x, the firehose latency gap from 4x to 1.2x, and geode now wins 4 of
9 query-latency scenarios (notifications, hashtag, by-ids,
recent-window) plus most concurrent-throughput scenarios, while keeping
the smaller on-disk footprint. Result sets stayed byte-identical across
relays and NIP-77 sync still converges.
Measured but deliberately NOT taken: per-row SAVEPOINT elision (+4%,
within run noise — not worth weakening batch error isolation), FTS-off
(+25% ingest but drops NIP-50), --no-verify (+45% but unfair/unsafe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Raw key:value tokens like include:spam reached SQLite FTS MATCH, where
the colon is column-filter syntax — any REQ carrying an extension token
died with CLOSED "no such column: include" instead of matching.
Adds SearchQuery.stripExtensions() plus Filter/List<Filter>
.strippingSearchExtensions() so EventStore users can drop the tokens
before querying, and applies them in LiveEventStore (query, count,
negentropy snapshots). Per NIP-50, unsupported extensions are ignored:
an extensions-only search becomes unconstrained, not match-nothing.
EventSource-backed search relays still receive the raw string since a
real search backend wants the extensions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tujoyfc2kNLZNVgiLZAR7F
Replaces the hand-rolled raw-WebSocket NIP-77 negotiate loop (single
un-windowed session — a strfry max_sync_events overflow was a hard
error) with quartz's negentropyReconcile: created_at window splitting
on overflow, keep-alive connection pinning, and streaming id batches.
Downloads and uploads now pipeline with the remaining reconcile
rounds: need-id batches feed 4 concurrent by-id drains, have-ids feed
an uploader (peak 7 subscriptions, under the common relay cap of 20).
Every downloaded event still funnels through the verify-and-store
path. Output field 'rounds' (protocol round-trips) is now 'windows'
(created_at splits).
Verified end-to-end against embedded geode relays: down-only 25/25,
up-only 5/5, and bidirectional re-runs converge to a zero diff.
Also records both adoptions in the perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Same-JVM interleaved comparison against the resurrected spin-lock
implementation: single-threaded parity (the uncontended lock was ~free),
but 8 threads sharing one decoder run 3.0-4.5x faster lock-free — the
spin lock serialized the concurrent hit path just like the old global
PoolRequests lock did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The speedup assertion depends on AVAILABLE parallelism, which a
full-suite run (the pre-push hook) can eat — it flaked at 1.14x under
load. Now retries up to 3 measurement passes and enforces a hard 1.05x
floor that a genuine serialization regression (the per-event-async
version measured 0.94x) can never pass, warning instead of failing in
the noise band between floor and the 1.5x target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Replaces the decoder's spin lock with lock-free concurrent maps so the
hot per-frame duplicate check never serializes across the pool's relay
consumer coroutines. New minimal ConcurrentHashCache expect/actual
(get/put/size/clear) following LargeCache's per-platform choices:
ConcurrentHashMap on JVM/Android, CacheMap on Apple, copy-on-write on
the CI-only Linux target. Counters become atomics; the generational
rotation keeps its deliberately-tolerated benign races, now documented
per failure mode (each is at worst a redundant re-parse, never a wrong
message).
CachingEventDecoderConcurrencyTest hammers one shared decoder from 8
threads with 80k duplicate-heavy frames and capacity 256 (rotations
fire constantly): zero wrong messages, exact parsed+reused accounting.
The 7 scan-safety tests and DedupDecodeBenchmark's enforced speedup
pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Code review confirmed each relay connection owns its own consumer
coroutine on Dispatchers.IO with no downstream funnel or shared lock
before justVerify (LargeCache is a ConcurrentSkipListMap; the
PoolRequests lock is per-subscription now), so multi-relay bursts
already verify in parallel across cores. Corrects the earlier follow-up
suggesting a CacheClientConnector integration: the accessory's scope is
single-connection bulk streams, plus a possible future dispatcher-
hygiene fix if on-device profiling shows CPU-bound verifies
oversubscribing the 64-thread IO pool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Deliberate design decision reversing the 4096-frame receive bound:
1. The remote infrastructure isn't ours — TCP backpressure parks the
backlog in the RELAY's outbound buffers. A client should release the
relay from its duties as fast as it can send and own the buffering
itself.
2. The app holds 2000+ simultaneous relay connections; a bounded buffer
under a slow consumer blocks OkHttp reader threads, and at that
connection count blocked readers are a thread-starvation hazard far
worse than the heap growth they prevent.
The UNLIMITED channels now carry an explicit do-not-bound comment with
this rationale, and the slow-consumer risk is addressed from the other
side: keep the consumer faster than any relay's send rate
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).
BoundedReceiveBufferTest removed with the bound it tested; plan doc
records the decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Two server bugs masking each other made a single giant REQ crawl and
then wedge:
LiveEventStore's historical-replay dedup used an immutable Set under an
AtomicReference with copy-on-add — set + id copies the whole set per
streamed event, so large replays were accidentally O(n²) (100k-event
REQ: ~700 events/s, degrading as the response grew). Replaced with a
spin-lock-guarded mutable HashSet (same threads, single contains/add
per critical section).
Fixing that unmasked WebSocketSessionPump's slow-client policy: a fast
replay instantly overflowed the 8192-frame cap — which conflated 'slow
client' with 'replay outruns the socket writer', normal for bulk — and
the 'drop' only closed the internal queue, leaving the socket half-dead
(no EOSE, no close frame, tail silently missing: the likely cause of
the benchmark's 99,998/100,000). Producers are now paced against a full
backlog (bounded blocking wait, consistent with the documented ingest
fanout behavior) and only a client still behind after 30s is dropped,
by actually cancelling the socket.
GiantReqStreamTest guards the regression: 20k-event REQ pre-fix 8.4s
(~2.4k events/s), post-fix 0.7s (~27k events/s), all events + EOSE
delivered. 96 geode tests and the quartz relay/server suites pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The per-connection-wall test prints the negotiated
Sec-WebSocket-Extensions header; against strfry OkHttp negotiates
permessage-deflate (client_no_context_takeover) out of the box, closing
the 'is compression actually on?' question from the optimization list —
it is, no change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
One reconcile feeds by-id download batches to N clients (one socket
each) x reqsPerClient workers; reconcile windows also round-robin
across the connections, since a single connection produced need-ids at
only ~9k/s on a 2.6M corpus and starved the downloads. Events funnel
through a bounded channel to a single consumer (exact maxEvents,
single-threaded onEvent); all stages backpressure; localEntries diffing
and have-counting match negentropyReconcile. reconcileWindows became
multi-client internally; single-client paths pass listOf(this).
Production shootout (same-run pairs, 100k cap): +18% to +64% over the
tuned single client, capped by the relay's server-side reconcile id
production rather than download parallelism (which the by-id matrix
shows scales 2.7x with connections). 4 in-process multi-client tests;
18 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Client-side mirror of IngestQueue.parallelVerify: submit() is a cheap
bounded-channel send from the relay consumer coroutine; a drain loop
batches greedily (up to 256) and fans each batch across
Dispatchers.Default in core-sized chunks, dispatching callbacks in
submission order. preVerified short-circuits already-trusted ids; the
bounded channel backpressures the socket instead of growing heap.
Batch/chunk sizing is measurement-driven: per-event async cost ~40us of
scheduling each (swallowing the gain), and the per-batch join barrier
at 64 still cost half (64 -> 1.2x, 256 -> 2.2x, 1024 -> 3.2x on 4
cores). ParallelVerifyBenchmark (fresh signed events per pass — Event
caches derived state after first verify, so passes must not share
instances) measures 1.8x vs sequential with an enforced >=1.5x
assertion. 4 correctness tests cover valid/tampered routing, ordering,
preVerified and callback-crash resilience.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The reader-thread-to-consumer channel in BasicOkHttpWebSocket and the
app's OkHttpWebSocket was UNLIMITED: a consumer slower than the socket
accumulated frame Strings without bound (gigabytes over a multi-million
event download). Now capped at 4096 frames — when full, OkHttp's reader
thread blocks and TCP flow control pushes back on the relay instead of
the heap. The trade-off (a blocked reader delays PING/PONG handling) is
documented on the constant; the bound is deep enough that only a
pathologically slow consumer hits it.
BoundedReceiveBufferTest forces sustained backpressure (8-frame buffer,
sleeping consumer, real socket to a local geode relay) and asserts all
events plus EOSE arrive in order with no drops or deadlock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
BasicRelayClient's decode step becomes a pluggable MessageDecoder
(default unchanged). The opt-in CachingEventDecoder scans EVENT frames
for their id (~0.3us, JSON-escape-safe so embedded event JSON in repost
content cannot confuse it; any irregularity falls back to full parse)
and on a cache hit synthesizes the EventMessage from the already-parsed
Event with the frame's own subId — every subscription still gets its
delivery and per-relay bookkeeping is unchanged; only the redundant
parse is skipped. Production traffic measured 14-57% duplicate frames.
DedupDecodeBenchmark (60k frames, 67% dups): 10.0us/frame full parse vs
1.5us/frame cached — 6.5x, with an in-benchmark assertion so the gain
is enforced. 7 scan-safety unit tests in CachingEventDecoderTest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The single global spin lock serialized every EVENT frame from every
relay and measured negative scaling (4 concurrent relay consumers
pushed 3.6M deliveries/s aggregate vs 11.1M for one thread alone). The
lock now lives in RequestSubscriptionState — one per subscription —
since all compound mutations are per-subId and different subs share no
wire state. decideCommandLocked takes the state instance to avoid
re-entering the non-reentrant lock; all-subs iterations lock one sub at
a time; withLock is inline to keep the hot path allocation-free.
DispatchStageBenchmark (PoolRequests-only, 1 -> 4 feeders): scaling
flips from 0.33x to 3.4-7.3x across runs. PoolRequests concurrency,
NostrClient and negentropy suites pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Splits the reconcile out of negentropySync so callers decide how to
load: negentropyReconcile streams needIds (relay has, local lacks —
download) and haveIds (local has, relay lacks — publish) in batchSize
chunks with back-pressure, taking local state as List<IdAndTime> and
slicing it per created_at window on overflow splits; the accumulating
negentropyReconcileIds convenience returns both lists. negentropySync
now delegates to the same window engine.
NegentropySession's primary constructor takes List<IdAndTime> (JVM
erasure forbids a List<Event> overload); the event-list form moved to
NegentropySession.fromEvents, mirroring NegentropyServerSession, with
all call sites migrated.
Adds NostrClientNegentropyReconcileTest (empty local set, partial
overlap both directions, identical sets, batch streaming, since/until
window slicing) — 49 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Restructures the sync around the measured bottlenecks: one download
worker pool now spans the whole sync (windows no longer join before the
next reconcile starts), overflow-split windows are reconciled by a
caller-set number of concurrent NEG sessions (reconcileConcurrency,
default 1) from a shared work queue, and the reconcile-to-download
buffer depth is exposed (idBufferBatches). No NIP-11 auto-detection:
peak subscription usage (maxConcurrentReqs + reconcileConcurrency + 1)
is documented and budgeting it against the relay's max_subscriptions is
the caller's call.
All 44 negentropy tests pass unchanged. Production shootout on a 2.6M
corpus, single connection: 3.6k events/s with old-equivalent params,
4.5k/s tuned (12 reqs + 4 reconcilers) — ~82% of the measured ~5.5k/s
per-connection by-id ceiling, vs 1.6k/s for the old implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The relay was backfilled from 33k to 2.6M kind-30382 events between
runs, and the picture changed: on the big corpus one connection
saturates at ~8 in-flight REQs (~5.5k events/s) and the relay tops out
at ~15k events/s around 40 total in-flight — past that, added
concurrency only inflates per-REQ latency. Axis 1 now stops at the
relay's NIP-11 max_subscriptions (20; exceeding it wedged the
connection in the first extended run), cells carry a hard 120s budget
with one retry per batch, and per-REQ latency is computed over
completed batches so partial cells report honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Downloads the full 33k kind-30382 corpus from nip85.nosfabrica.com per
cell over a (connections x concurrent REQs) matrix with pre-connected
sockets. One connection scales near-linearly to 16 in-flight REQs
(1,970 -> 30,735 events/s) with no wall at 4, and 1x16 matches 4x4 —
total in-flight REQs is the real variable. Every slow path measured so
far (serial page cursors ~2-3.7k/s, negentropySync 1.6k/s) is a
pipelining/serialization problem: negentropySync starves its download
workers on reconcile cadence and recurses overflow windows
sequentially. Findings and implied negentropySync improvements in the
plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
A raw no-parse socket and the full quartz stack page the same
production query on one connection at the same ~3.4-3.8k events/s,
proving the single-connection ceiling reported by a user is the
relay's per-connection response cadence rather than the client's
serial parse (which is ~1% busy at that rate). Findings and the
assessment of the proposed parallel-parse + ordered-dispatch pipeline
are in the plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Measures how to download millions of events from a single relay as fast
as possible, download+parse only: local geode ceilings (giant REQ vs
until-cursor paging vs created_at-sharded connections, quartz stack vs
raw frames), offline per-frame strategies (full parse, parallel parse,
id-scan for raw archiving), and a production case syncing kind 30382
from nip85.nosfabrica.com via NIP-77 negentropy against plain paging.
Headline results in the plan doc: paging beats giant REQs 26x (which
also dropped frames), created_at sharding stacks ~2x on top, parse is
2-5% of the budget, and negentropy is 2.4x slower than paging for a
cold download (its win is incremental re-sync).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Offline microbenchmark of NostrClient's dispatch stage — PoolRequests
state machine, listener fan-out, id dedup and handoff to a verify
stage — under 1 and 4 concurrent relay feeders. Key results: ~100ns per
message uncontended, but negative scaling under concurrency (the
PoolRequests busy-wait spin lock makes 4 feeders slower in aggregate
than 1), per-event channel handoff costs ~180ns vs a free 64-batch,
and early dedup before the locked path gives 2.7-4x aggregate
throughput at production duplicate factors. Findings appended to the
receiver-perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Gated JVM test (-PprodRelayBench=1) that connects to live relays with
realistic filters and measures per-relay queue delay, processing time,
consumer busy fraction, EOSE latency and duplicate rates, comparing the
current inline verification against a parallel verify stage, plus
offline single-thread parse/verify ceilings on captured frames.
Findings from a first run are written up in
quartz/plans/2026-07-02-nostrclient-receiver-perf.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Part C of the dispatchers/thread-caps audit. Both LnurlEndpointCache and
DesktopCachedRichTextParser were bounded caches backed by a LinkedHashMap
behind a single monitor (@Synchronized / Collections.synchronizedMap with
accessOrder). An access-order map structurally mutates on get, so every
read took the lock — serializing all readers on paths that are hot
(kind-9735 zap-receipt validation; feed rich-text rendering).
Add ConcurrentLruCache<K, V> in quartz utils: storage is a
ConcurrentHashMap so get is lock-free; writes + eviction run under a small
write lock that is off the read path. Eviction is least-recently-put order
(get does not refresh recency) — exactly what LnurlEndpointCache already
did, and fine for the deterministic rich-text parse cache.
Point both caches at the shared helper. Covered by a new
ConcurrentLruCacheTest (round-trip, eviction order, re-put recency
refresh, get-does-not-refresh, clear, and a concurrent size-bound smoke
test); the existing LnurlEndpointCacheTest still passes unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANuUziXKRafSTBxbh4SMoq
Part of the dispatchers/thread-caps audit. Two low-risk fixes that remove
thread-blocking work from paths hit on every event / every packet:
- LargeCache (iOS actual): drop the runBlocking wrapper around
createIfAbsent. The block contained only synchronous CacheMap ops (the
same get/put getOrCreate already calls without runBlocking), so it was
pure dispatcher-blocking overhead on the per-event ingest path. Now
mirrors the JVM actual's plain-function shape.
- QUIC JCA AEADs (AES-GCM + ChaCha20-Poly1305): split the single
`synchronized(this)` monitor into disjoint encryptLock / decryptLock.
seal-family touches only encryptCipher + recentEncryptNonces; open-family
touches only decryptCipher, so a connection's send loop and read loop no
longer serialize against each other through crypto on every packet. The
documented defence-in-depth against cross-coroutine Cipher corruption is
preserved per direction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANuUziXKRafSTBxbh4SMoq
printStackTrace() dumps straight to stderr, bypassing both Log.minLevel
and the consumer's Log.sink — the very thing the LogSink work exists to
control. Migrate the five production call sites:
- Lud06: drop two printStackTrace() calls that sat directly above an
existing Log.w(..., t) carrying the same throwable (pure duplication).
- ElectrumXClient: the swallowed-lookup catch said "Log but don't crash"
yet used printStackTrace(); route it through Log.w with context.
- OpenTimestamps: log the swallowed merge failure via Log.w; drop the
print-then-rethrow (the rethrown exception already carries the trace).
Socket-protocol writer.println(...) and README/KDoc println examples are
left as-is — they are wire I/O and documentation, not logging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EK3TrDkP1EXj1d62oKJMdc
Quartz already funnels every diagnostic through the `Log` facade, but the
sink was hardcoded per platform (android.util.Log / System.err / NSLog /
println), so a consuming app couldn't route Quartz logs into its own stack
(Timber, SLF4J, Crashlytics, a file, a test buffer, or /dev/null).
Add a `LogSink` fun interface and a replaceable `Log.sink`, defaulting to
`PlatformLogSink` which reproduces the historical per-platform behavior.
All ~225 call sites and the `Log.*` signatures are unchanged; the lazy
`() -> String` overloads still short-circuit on `minLevel` before the
lambda runs, preserving the allocation-free fast path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EK3TrDkP1EXj1d62oKJMdc
Fold the full-text-search on/off switch into `IndexingStrategy` as
`indexFullTextSearch` (default `true`) instead of a separate top-level
`enableFullTextSearch` constructor param on `EventStore`/`SQLiteEventStore`.
`IndexingStrategy` is already the single place that decides which indexes
the store builds — every field is a per-index toggle with a size/speed
tradeoff, and `QueryBuilder` already receives it — so FTS, being just
another index, belongs there rather than split across two config surfaces.
Behaviour is unchanged: the module's no-op path and the QueryBuilder
"search matches nothing" guards now read the flag via the strategy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjzUpY8H31c7ux669zytWg
Add an `enableFullTextSearch` flag (default `true`) to `EventStore` and
`SQLiteEventStore` so deployments that never serve NIP-50 search from
SQLite — e.g. a relay that offloads search to an external engine like
Vespa — can skip the full-text-search write cost.
When disabled:
- `FullTextSearchModule` becomes an inert no-op: the `event_fts` virtual
table and its `fts_foreign_key` delete trigger are never created,
inserts skip `indexableContent()` + tokenization, and both reindex
entry points return immediately.
- `QueryBuilder` short-circuits any query/count/delete filter carrying a
non-empty `search` term to a "matches nothing" result (an empty-string
search still imposes no constraint), so no SQL ever references the
absent `event_fts` table. In a multi-filter union the search branch
contributes nothing while the other filters resolve normally.
Everything else (replaceable/addressable handling, deletions,
expirations, right-to-vanish, negentropy) is unchanged, and the default
keeps FTS on for existing callers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjzUpY8H31c7ux669zytWg
Kotlin/Native (the iOS test target) rejects commas in backtick function
names, so test-quartz-ios failed to compile even though jvmTest — which
allows them — passed. Rename the two offending tests. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JGa1EM5KWyDo1o5Yr6sS18
Claude
Vitor PamplonaandGitHub