- only parse commonName when the alt tag has the Birdstar prefix
- summary() on both Birdstar events now uses the canonical NIP-31
tags.alt() helper instead of a raw firstTagValue("alt") lookup
- speciesReference() only returns http(s) URLs since UIs render it as
a clickable link (rejects e.g. javascript: schemes), with a test
- Detection card: parse tags once into a single remember slot, drop
the near-dead '?: summary' title fallback, stop rebuilding the
italic TextStyle every recomposition
- Hoist the duplicated bird-emoji literal into a shared BIRD_PREFIX
- Trim the redundant factory test to the assertIs idiom
- fetch Birdex life lists in home and profile relay REQs
- richer Birdstar cards — common name title, Wikidata link, bird emoji
- surface Birdstar bird detections in home and profile feeds
On connect, syncFilters re-sends every desired REQ through
PoolRequests.syncState. It previously sent the frame and only recorded
the subscription as SENT afterward, in the post-send onSent callback. A
relay that answers faster than that callback runs — the in-process
transport used by the desktop launch-optimization tests, or any relay on
a fast path — can deliver the EOSE while the per-sub state still reads
"nothing in flight" (onConnecting cleared it, onSent hasn't recorded it).
The EOSE handler then sees empty filters, concludes it never sent a REQ,
and fires a duplicate, replaying the whole page a second time.
Pre-mark the sub as SENT under its lock before the frame leaves, mirroring
the decideCommandLocked pre-mark already used by sendToRelayIfChanged, so
a response can never race ahead of the record. The send stays
unconditional: this is a fresh-connection sync (onConnecting always
cleared the per-relay state first), so there is no in-flight REQ on the
new socket to dedupe against.
Fixes the flaky SubscribeBeforeConnectTest, which asserted a pre-connect
subscription delivers exactly its events once and intermittently saw them
doubled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4kmtZoNUSXxwG23wD2JwP
TcpNoDelaySocketFactory's connecting overloads used
`socket().apply { connect(...) }`, which leaks the file descriptor if
bind/connect throws (the JDK's connecting Socket constructors close on
failure; ours didn't). Wrapped in a helper that closes on throw. OkHttp
only calls the no-arg overload, so this guards any other direct caller.
DesktopHttpClient's pre-init `simpleClient` (direct relay sockets opened
before setInstance) now gets the same TcpNoDelaySocketFactory as
directClient. failClosedClient is left as-is: it's a SOCKS client and
OkHttp bypasses the socket factory for SOCKS proxies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
ingest() bypasses the per-connection policy chain, which is where
VerifyPolicy lives. The IngestQueue verify hook only exists when
parallelVerify is true, so with parallelVerify = false and
skipVerify = false, ingest() previously verified nothing — an untrusted
mirror upstream on a relay running the legacy in-policy verify path could
inject forgeries. ingest() now verifies inline in that configuration
(same rejection reason as the queue), so the documented "default keeps
verify-everything semantics" holds regardless of parallelVerify. KDoc
also spells out that ingest() skips the entire policy chain (blacklists,
size limits), which callers must screen for themselves.
Test covers the parallelVerify = false server: forged rejected, trusted
skip and valid still land.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Audit follow-ups on the live NIP-77 index, all with the store/scan
equivalence test extended to cover them:
- Same-batch replaceable displacement left a dead id in the index.
applyAfterCommit applied all removes before all adds, so when a later
row in one transaction displaced an earlier row of the same batch (two
versions of one replaceable — the mirror-backfill hot path), the
displaced row's remove no-op'd against an index that hadn't taken its
add yet, then the add re-inserted it: the index advertised an id the
trigger had already deleted. recordAccepted now cancels the pending add
instead of queueing a remove (added is a LinkedHashSet for O(1)
cancel).
- A kind-5 that deleted nothing (a delete broadcast for events this relay
never stored — the common case) still invalidated the whole index,
forcing a full-scan rebuild under the writer mutex on the next
NEG-OPEN. DeletionRequestModule.insert now returns the rows it deleted;
recordAccepted only invalidates when that count is > 0, else records the
kind-5 as a plain row.
- The first NEG-OPEN over a corpus larger than the serve cap scanned the
whole table uncapped, built a full index that could never produce a
snapshot, and then maintained it forever for zero benefit.
liveNegentropySnapshot now caps the rebuild scan at maxEntries + 1 and
leaves the index unpopulated when the corpus is over-cap (the scan path
answers NEG-ERR, as before).
- delete/deleteExpired/clearDB invalidate() moved inside the writer mutex
so no NEG-OPEN can seal a snapshot of just-deleted rows, and no
concurrent rebuild can be discarded by a late invalidate.
Also corrects the ~40 B/event heap figure to ~140 B (IdAndTime keeps the
id as a 64-char hex string, not 32 bytes) in the strategy/plan docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog items 4-5 plumbing, config-gated and off by default so quartz
library defaults stay untouched for the app-side stores:
- quartz: SQLiteEventStore/EventStore accept extraPragmas (applied on
every pooled connection AFTER the built-in configuration, so they
can override it) and expose optimize() — an analysis_limit-bounded
PRAGMA optimize for incremental planner-stats refresh.
- geode: [database] readers / mmap_size / temp_store_memory map onto
the store; optimize_interval_seconds drives a maintenance coroutine
(cancelled first in the shutdown hook, before the store closes).
The A/B verdict on whether the example config should RECOMMEND any of
these on container-class hardware follows in the next commit — the
knobs themselves are operator tools worth having either way, since
mmap/temp-store value is hardware-dependent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Found while attributing the small-REQ wire floor (backlog item 6,
latency half): geode's new WireReqFloorBenchmark measured a flat
43.7 ms per REQ round trip that survived every server-side change —
store configs, dispatchers, the pump — and then vanished when the
round's preceding CLOSE was dropped. Root cause is client-side: OkHttp
does not set TCP_NODELAY, relays never answer a CLOSE (NIP-01), so its
bytes sit unACKed for the peer's ~40 ms delayed-ACK window and Nagle
holds the next REQ behind them. CLOSE-then-REQ is a Nostr client's
hottest pattern — every feed/filter switch.
relayBench's harness client already shipped a no-delay socket factory
(which is why benchmark numbers never showed the stall) but the
production clients did not. New TcpNoDelaySocketFactory (quartz
jvmAndroid, next to BasicOkHttpWebSocket) is now used by the Android
relay pool factory, the Desktop relay client, amy's relay connections,
and geode's mirror worker. Direct connections only — SOCKS/Tor paths
are untouched.
With the factory, the benchmark puts geode's ~21-row REQ at ~1.25 ms
on the wire (matching relayBench): ~0.6 ms Ktor CIO+OkHttp loopback
floor, ~0.5 ms per-REQ server work (already investigated). Per-frame
burst cost measured negligible and the pump adds ~nothing, so the
send-path latency angle of backlog item 6 is closed as not-a-problem;
its ingest-CPU share remains a separate throughput question. Findings
recorded in quartz/plans/2026-07-04-small-req-floor.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Reverts the queryRawInline fast path (fb29d655, 1b786f31) per the
keep-only-winners rule. Three relayBench runs at 50k (baseline, cap-256
where the path never engaged, cap-512 where author-archive/by-ids/
500-limit feeds genuinely took it) showed no movement outside the
container drift band — strfry's own numbers drifted ±30% between runs
and inline-eligible scenarios moved the same as ineligible ones.
The in-process win was real but small (~17%, 0.60 -> 0.50 ms per
~21-row REQ); the wire-level p50 is 1.2-1.7 ms, so the missing ~1 ms
per REQ sits in the transport (Ktor frame send path + client round
trip) — backlog item 6 territory, not dispatch. Findings, numbers, and
the do-not-retry note live in quartz/plans/2026-07-04-small-req-floor.md;
SmallReqFloorBenchmark stays as the measurement tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
The first cut's rule (every filter needs limit, sum <= 256) missed the
shapes relays actually receive: relayBench's author-archive carries
limit=500 and by-ids carries only an ids list, so the fast path never
engaged in the acceptance run. Bounds now come from limit OR the ids
count (ids are unique keys, the result cannot exceed the list), summed
against a 512 cap — a 500-row inline replay measures single-digit ms,
nothing a CLOSE could meaningfully preempt. Provably unbounded filters
(no limit, no ids — e.g. a bare tag filter) keep the launched path.
Pending: relayBench verdict decides whether the fast path stays at all,
per the keep-only-winners rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog item 2 (small-REQ dispatch floor), first cut. relayBench at 50k
shows geode ~2.5x slower than strfry when a REQ returns ~20 events
(author-archive 1.18 vs 0.48 ms EOSE p50) while WINNING the 500-event
scenarios — with tiny results the fixed per-REQ cost dominates. A new
stage benchmark (SmallReqFloorBenchmark) decomposes the floor: at ~21
rows, raw SQL is 0.18 ms and the remaining ~0.42 ms is live-subscription
machinery plus per-REQ coroutine dispatch.
A REQ whose filters all carry a limit summing to <= 256 now runs its
stored replay inline on the receive coroutine and only retains a
live-tail handle (SessionBackend.queryRawInline; LiveEventStore's
queryRaw is re-expressed on the same core) — no per-REQ launch, no Job,
no dispatcher handoffs, no awaitCancellation scaffolding. Unbounded
REQs keep the launched path so CLOSE can always interrupt a giant
replay. In-process time-to-EOSE for ~21-row REQs drops 0.60 -> 0.50 ms;
the bigger effect expected under concurrency (no per-REQ Job+dispatch
churn) is relayBench's to judge.
InlineReqFastPathTest pins wire-behavior parity: stored-then-EOSE
ordering, live tail delivery and CLOSE detachment, same-subId
replacement, launched fallback for unbounded and over-cap REQs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestones 2-3 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
SQLiteEventStore now maintains the LiveNegentropyIndex when the strategy
opts in (geode's does by default; [negentropy].live_index = false turns
it off; app-side stores are untouched):
- Write paths collect a LiveIndexDelta and apply it after COMMIT while
still holding the writer mutex — rolled-back savepoint rows never
reach the index and updates land in exact commit order (also vs the
rebuild, which runs under the same mutex).
- Replaceable/addressable overwrites report the row their BEFORE-INSERT
trigger displaces via one indexed pre-SELECT that mirrors the trigger
predicate (including the NIP-01 lowest-id tie-break and the
d_tag-NULL case).
- Paths that can't itemize (kind-5, vanish, delete-by-filter/id,
expiration sweeps, clearDB) invalidate; the next NEG-OPEN rebuilds
from one scan on the writer connection.
- Until that first NEG-OPEN populates the index, ingest pays zero
bookkeeping — the populated check happens under the writer mutex so
it can't race the rebuild.
LiveEventStore serves a single unconstrained filter (the relay-relay
sync default; relayBench sends exactly this) from the index; everything
else keeps the scan+seal path and its single-slot cache.
Micro-benchmark at 50k events (LiveNegentropyBenchmark, in-container):
scan+seal cold path 80-100 ms; index post-write open 9-16 ms (~5-10x).
The relayBench A/B is the acceptance gate and comes next.
Correctness: LiveNegentropyIndexStoreTest asserts index content ==
snapshotIdsForNegentropy scan after every mutation pattern (overwrites,
losers, kind-5 rebuilds, filter deletes, mixed-outcome batches,
transactions); the full geode suite (NIP-77 + interop sync tests) runs
with the index on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestone 1 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
Sorted array with binary-search insert (near-tail in the common case),
itemized remove for displaced rows, wholesale invalidate for delete
paths that can't itemize, and sealed snapshots memoized per mutation
generation — reconcile only reads, so one snapshot backs any number of
concurrent sessions and stays immutable under later writes. Over-cap
answers null so the caller keeps the strfry-parity NEG-ERR.
Not wired into any store yet; next milestones plumb displaced-row
deltas from the SQLite modules and serve index-total filters from it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Adds a public local-ingest entry to NostrServer for events that don't
arrive over a client connection (mirror/sync workers, import jobs). It
routes through the same group-commit IngestQueue and live fanout as a
client publish, and each Submission can opt out of the parallel
Schnorr-verify hook — the relay-to-relay trust model, for events
streamed from an upstream that already verified them (verify profiles
at ~8% of busy ingest CPU).
Library defaults are unchanged: skipVerify defaults to false everywhere
and nothing in the client-publish path can set it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
A single-sample wall-clock speedup assertion (>1.5x) flakes when the
shared CI machine is loaded. Apply the same pattern already used by
ParallelVerifyBenchmark: retry up to 3 measurement attempts taking the
best, keep a hard 1.05x floor that a real regression (~1.0x when dedup
saves no work) can never pass, and warn instead of fail in the noise
band. The deterministic parsed/reused-count correctness gate still
fails hard on every attempt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzMix4ZivxoYrcn88TMqtF
The worker's writer-mutex fast path can slip past its own scope
cancellation and run one batch against a store that close() already
freed. The resulting closed-connection exception escaped scope.launch
under a SupervisorJob with no handler — harmless in production, but the
kotlinx-coroutines-test global handler attributes it to whatever runTest
starts next (seen on CI as EventSourceServerTest.countUsesSource failing
with UncaughtExceptionsBeforeTest). Catch, log, and stop the pass:
nothing depends on it — the pre-search drain covers search correctness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Two issues surfaced by rebasing onto main:
- SQLiteConnectionPool.close() freed the writer's native handle without
taking the writer mutex, so a block still running on another thread
(the deferred-FTS catch-up worker's current batch outlives its scope
cancellation) could call sqlite3_prepare on a freed sqlite3* — native
heap corruption, SIGSEGV in sqlite3DbMallocRawNN (reproduced twice in
:geode:test). close() now reclaims every reader from the channel and
acquires the writer mutex before closing handles, then releases so
stragglers get the managed closed-connection exception. Idempotent
via a closed flag.
- main's NIP-50 fix (7deda28d) strips search-extension tokens before
every store query, but the zero-decode queryRaw replay path added on
this branch predates it and passed raw filters through — an
extensions-only search would hit SQLite FTS as column syntax and
error. queryRaw now strips like query/count/snapshot do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
FTS indexing ran inside every insert's transaction — a measurable slice
of write cost (relayBench: ~18% of ingest throughput) paid at publish
time for a feature only search queries read. It now runs as a watermark
catch-up:
- IndexingStrategy.deferFullTextSearchIndexing (default false; geode's
relay strategy enables it with search). Deferred inserts skip
tokenization entirely.
- FullTextSearchModule keeps a fts_catchup_state watermark (everything
<= last_row_id is indexed) and gains catchUpBatch(): scan past the
watermark, tokenize, advance — one write transaction per batch, so
publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at
MAX(row_id) for existing (synchronously indexed) databases.
- NostrServer runs the catch-up worker, poked by IngestQueue's new
onBatchCommitted hook, and *yields to publish traffic*: it only
drains while the queue has no backlog (IngestQueue.hasBacklog()), so
bursts ingest at no-FTS speed and tokenization fills the gaps.
- LiveEventStore drains the backlog synchronously before serving any
filter with a search term (query, queryRaw, count) — NIP-50 results
stay exactly as fresh as the synchronous path; the deferral is
invisible to correctness. Geode's existing search tests pass
unchanged through this path.
The first implementation reused reindexBatch and collapsed ingest 8x —
its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?'
matches on a plain FTS5 column, i.e. a full FTS-table scan per row
(O(n²) overall), and the worker competed with the replay for the writer
mutex. catchUpBatch therefore inserts without the delete (rows past the
watermark are never indexed; switching a DB between deferred and
synchronous strategies requires reindexAll, same rule as a
searchable-kinds change), and the worker backs off whenever publishes
are pending.
Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136
events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the
--no-search ceiling while keeping NIP-50 advertised and fresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
A NIP-77 server session rebuilt its reconciliation structure from
scratch on every NEG-OPEN: full id+created_at scan, per-entry hex
decode into a fresh StorageVector, O(n log n) seal. That cost grows
with the corpus and is paid even when nothing changed — the exact
shape of a periodic mirror's heartbeat, where N peers reconcile the
same broad filter over and over. relayBench measured 342 ms per
identical-set reconcile at 50k events vs strfry's 26 ms off its
always-current LMDB tree.
Reconciliation only *reads* the sealed storage, so one instance can
back any number of concurrent sessions:
- NegentropyServerSession now accepts a pre-sealed IStorage (the
List<IdAndTime> constructor remains and delegates).
- SessionBackend.sealedNegentropyStorage() builds + seals (null when
the set exceeds maxSyncEvents); LiveEventStore overrides it with a
single-slot cache keyed by (filter set, write generation) plus a 30s
TTL. The generation bumps on every accepted ingest; the TTL bounds
staleness from delete paths the counter can't see (expiration
sweeps, admin purges) — negentropy snapshots are point-in-time sets,
so seconds of staleness only means a peer briefly re-offers ids.
- NegSessionRegistry.open consumes the shared sealed storage;
over-cap NEG-ERR behavior unchanged (strfry parity).
relayBench gains a 'heartbeat' measurement — the identical-set
reconcile repeated immediately with no writes in between. At 50k
events: geode 342 ms -> 27.8 ms vs strfry 21.1 ms (near parity; was
13x). Cold reconciles (first open after a write) are unchanged.
Also fixes the GeodeVsStrfryNegentropySyncTest fixture to write
'nofiles = 0' so the opt-in interop test can boot strfry inside
containers with a low RLIMIT_NOFILE hard cap; the interop suite passes
against strfry v1-b80cda3 with the cache in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Reviewed strfry's LMDB indices (golpe.yaml) against geode's SQLite set:
the two are nearly isomorphic — time, id, kind+time, author+kind+time,
tag+time, plus conditional deletion/expiration/replaceable entries
(geode's are partial indexes, so ordinary events don't pay for them).
Nothing to drop. One real hole: strfry maintains a plain
pubkey(+created_at) index and geode had none, so an authors-only filter
(no kinds) — archive pulls, account-migration tools, 'everything by
these pubkeys' — degraded to a full walk of the time index. EXPLAIN
confirmed: SCAN query_by_created_at_id.
- quartz: IndexingStrategy.indexEventsByPubkeyAlone (default false —
clients query their supported kinds and can skip it) gates a new
query_by_pubkey_created index; DATABASE_VERSION 2→3 with an
idempotent migration that backfills it for opted-in strategies.
- geode: relayIndexingStrategy turns it on.
- relayBench: new 'author-archive' scenario — every kind by 3 *quiet*
pubkeys. Quiet is the point: prolific authors are dense in the time
index and a scan finds them quickly, which is why the suite never
caught this; sparse authors force the full walk.
Measured (50k corpus): author-archive EOSE p50 42.8 ms -> 3.6 ms (12x,
and the old path grows linearly with table size); ingest 5,337 -> 5,156
events/s (~3%, the one extra B-tree per event). strfry reference on the
same scenario: 0.52 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
The drain loop ran verify(batch N) and insert(batch N) strictly
back-to-back: the SQLite writer idled during every Schnorr verify and
the CPU cores idled during every commit. Split it into two stage
coroutines joined by a capacity-1 channel — a verifier that collects
and checks batch N+1 while the writer commits batch N. Same shape as
strfry's ingester/writer thread split.
Ordering (single verifier, single writer, FIFO handoff), OK-after-commit
semantics, per-row error isolation and submit() backpressure are all
unchanged; total in-flight grows by at most one batch.
Alternating A/B on the 10k corpus, 4-core host with the benchmark client
competing for the same cores: sequential 4,111/3,977/4,443 vs pipelined
4,050/5,020/5,092 events/s (~+13% mean). The overlap should widen on
dedicated relay hosts where verify has its own cores. Feature-parity
run after this change: geode --no-search 5,911 vs strfry 9,374 events/s
(1.59x, down from 2.2x at the start of the perf work).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Main's giant-REQ fix (18bd3600) replaced query()'s copy-on-add immutable
dedupe set with a spin-locked HashSet, but the rebase left queryRaw —
now the default REQ path — on the old pattern, which would have
reintroduced the O(n²) crawl for large replays. Both paths now share
the mutable-set-under-spinlock shape.
relayBench's sync driver moves to NegentropySession.fromEvents(),
following the session's new List<IdAndTime> constructor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Three changes, each validated head-to-head against strfry with relayBench
(same 10k-event corpus a1cd3517a8296911, stock configs, sig verify on):
- geode: RelayIndexingStrategy turns on indexEventsByCreatedAtAlone for
the relay's stores (quartz's DefaultIndexingStrategy stays off for
client-side stores). A relay can't predict client filters, and the
cheapest REQ of all — {"limit":N} — was a full-table scan + top-N
sort: 40 ms and O(table) growth before; 11 ms and index-streamed
(first event 30 ms -> 1.9 ms) after.
- quartz: zero-decode REQ replay. Stored events now stream as RawEvent
(tags kept in serialized form) and are spliced directly into wire
frames — no tags parse, no EventFactory dispatch, no re-serialize per
row. Gated on the new IRelayPolicy.filtersOutgoingEvents capability:
policies that can veto per-event delivery (none today) keep the
materialized path; everyone else skips it. Live post-EOSE delivery is
unchanged (live matching needs Event objects).
- quartz: StatementCachingConnection wraps the pool's writer and reader
connections, replaying prepared statements instead of re-preparing per
event/REQ (eager reset on return keeps cursors from holding table
locks; a 256-statement cap bounds client-controlled filter-shape
variety). Ingest went 3,000 -> 4,700 events/s (+55%) — prepare
overhead was the single largest non-crypto write cost.
Net effect on the benchmark: ingest gap vs strfry narrowed from 2.2x to
1.8x, the firehose latency gap from 4x to 1.2x, and geode now wins 4 of
9 query-latency scenarios (notifications, hashtag, by-ids,
recent-window) plus most concurrent-throughput scenarios, while keeping
the smaller on-disk footprint. Result sets stayed byte-identical across
relays and NIP-77 sync still converges.
Measured but deliberately NOT taken: per-row SAVEPOINT elision (+4%,
within run noise — not worth weakening batch error isolation), FTS-off
(+25% ingest but drops NIP-50), --no-verify (+45% but unfair/unsafe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Raw key:value tokens like include:spam reached SQLite FTS MATCH, where
the colon is column-filter syntax — any REQ carrying an extension token
died with CLOSED "no such column: include" instead of matching.
Adds SearchQuery.stripExtensions() plus Filter/List<Filter>
.strippingSearchExtensions() so EventStore users can drop the tokens
before querying, and applies them in LiveEventStore (query, count,
negentropy snapshots). Per NIP-50, unsupported extensions are ignored:
an extensions-only search becomes unconstrained, not match-nothing.
EventSource-backed search relays still receive the raw string since a
real search backend wants the extensions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tujoyfc2kNLZNVgiLZAR7F
The speedup assertion depends on AVAILABLE parallelism, which a
full-suite run (the pre-push hook) can eat — it flaked at 1.14x under
load. Now retries up to 3 measurement passes and enforces a hard 1.05x
floor that a genuine serialization regression (the per-event-async
version measured 0.94x) can never pass, warning instead of failing in
the noise band between floor and the 1.5x target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Replaces the decoder's spin lock with lock-free concurrent maps so the
hot per-frame duplicate check never serializes across the pool's relay
consumer coroutines. New minimal ConcurrentHashCache expect/actual
(get/put/size/clear) following LargeCache's per-platform choices:
ConcurrentHashMap on JVM/Android, CacheMap on Apple, copy-on-write on
the CI-only Linux target. Counters become atomics; the generational
rotation keeps its deliberately-tolerated benign races, now documented
per failure mode (each is at worst a redundant re-parse, never a wrong
message).
CachingEventDecoderConcurrencyTest hammers one shared decoder from 8
threads with 80k duplicate-heavy frames and capacity 256 (rotations
fire constantly): zero wrong messages, exact parsed+reused accounting.
The 7 scan-safety tests and DedupDecodeBenchmark's enforced speedup
pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Deliberate design decision reversing the 4096-frame receive bound:
1. The remote infrastructure isn't ours — TCP backpressure parks the
backlog in the RELAY's outbound buffers. A client should release the
relay from its duties as fast as it can send and own the buffering
itself.
2. The app holds 2000+ simultaneous relay connections; a bounded buffer
under a slow consumer blocks OkHttp reader threads, and at that
connection count blocked readers are a thread-starvation hazard far
worse than the heap growth they prevent.
The UNLIMITED channels now carry an explicit do-not-bound comment with
this rationale, and the slow-consumer risk is addressed from the other
side: keep the consumer faster than any relay's send rate
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).
BoundedReceiveBufferTest removed with the bound it tested; plan doc
records the decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Two server bugs masking each other made a single giant REQ crawl and
then wedge:
LiveEventStore's historical-replay dedup used an immutable Set under an
AtomicReference with copy-on-add — set + id copies the whole set per
streamed event, so large replays were accidentally O(n²) (100k-event
REQ: ~700 events/s, degrading as the response grew). Replaced with a
spin-lock-guarded mutable HashSet (same threads, single contains/add
per critical section).
Fixing that unmasked WebSocketSessionPump's slow-client policy: a fast
replay instantly overflowed the 8192-frame cap — which conflated 'slow
client' with 'replay outruns the socket writer', normal for bulk — and
the 'drop' only closed the internal queue, leaving the socket half-dead
(no EOSE, no close frame, tail silently missing: the likely cause of
the benchmark's 99,998/100,000). Producers are now paced against a full
backlog (bounded blocking wait, consistent with the documented ingest
fanout behavior) and only a client still behind after 30s is dropped,
by actually cancelling the socket.
GiantReqStreamTest guards the regression: 20k-event REQ pre-fix 8.4s
(~2.4k events/s), post-fix 0.7s (~27k events/s), all events + EOSE
delivered. 96 geode tests and the quartz relay/server suites pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The per-connection-wall test prints the negotiated
Sec-WebSocket-Extensions header; against strfry OkHttp negotiates
permessage-deflate (client_no_context_takeover) out of the box, closing
the 'is compression actually on?' question from the optimization list —
it is, no change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
One reconcile feeds by-id download batches to N clients (one socket
each) x reqsPerClient workers; reconcile windows also round-robin
across the connections, since a single connection produced need-ids at
only ~9k/s on a 2.6M corpus and starved the downloads. Events funnel
through a bounded channel to a single consumer (exact maxEvents,
single-threaded onEvent); all stages backpressure; localEntries diffing
and have-counting match negentropyReconcile. reconcileWindows became
multi-client internally; single-client paths pass listOf(this).
Production shootout (same-run pairs, 100k cap): +18% to +64% over the
tuned single client, capped by the relay's server-side reconcile id
production rather than download parallelism (which the by-id matrix
shows scales 2.7x with connections). 4 in-process multi-client tests;
18 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Client-side mirror of IngestQueue.parallelVerify: submit() is a cheap
bounded-channel send from the relay consumer coroutine; a drain loop
batches greedily (up to 256) and fans each batch across
Dispatchers.Default in core-sized chunks, dispatching callbacks in
submission order. preVerified short-circuits already-trusted ids; the
bounded channel backpressures the socket instead of growing heap.
Batch/chunk sizing is measurement-driven: per-event async cost ~40us of
scheduling each (swallowing the gain), and the per-batch join barrier
at 64 still cost half (64 -> 1.2x, 256 -> 2.2x, 1024 -> 3.2x on 4
cores). ParallelVerifyBenchmark (fresh signed events per pass — Event
caches derived state after first verify, so passes must not share
instances) measures 1.8x vs sequential with an enforced >=1.5x
assertion. 4 correctness tests cover valid/tampered routing, ordering,
preVerified and callback-crash resilience.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The reader-thread-to-consumer channel in BasicOkHttpWebSocket and the
app's OkHttpWebSocket was UNLIMITED: a consumer slower than the socket
accumulated frame Strings without bound (gigabytes over a multi-million
event download). Now capped at 4096 frames — when full, OkHttp's reader
thread blocks and TCP flow control pushes back on the relay instead of
the heap. The trade-off (a blocked reader delays PING/PONG handling) is
documented on the constant; the bound is deep enough that only a
pathologically slow consumer hits it.
BoundedReceiveBufferTest forces sustained backpressure (8-frame buffer,
sleeping consumer, real socket to a local geode relay) and asserts all
events plus EOSE arrive in order with no drops or deadlock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
BasicRelayClient's decode step becomes a pluggable MessageDecoder
(default unchanged). The opt-in CachingEventDecoder scans EVENT frames
for their id (~0.3us, JSON-escape-safe so embedded event JSON in repost
content cannot confuse it; any irregularity falls back to full parse)
and on a cache hit synthesizes the EventMessage from the already-parsed
Event with the frame's own subId — every subscription still gets its
delivery and per-relay bookkeeping is unchanged; only the redundant
parse is skipped. Production traffic measured 14-57% duplicate frames.
DedupDecodeBenchmark (60k frames, 67% dups): 10.0us/frame full parse vs
1.5us/frame cached — 6.5x, with an in-benchmark assertion so the gain
is enforced. 7 scan-safety unit tests in CachingEventDecoderTest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The single global spin lock serialized every EVENT frame from every
relay and measured negative scaling (4 concurrent relay consumers
pushed 3.6M deliveries/s aggregate vs 11.1M for one thread alone). The
lock now lives in RequestSubscriptionState — one per subscription —
since all compound mutations are per-subId and different subs share no
wire state. decideCommandLocked takes the state instance to avoid
re-entering the non-reentrant lock; all-subs iterations lock one sub at
a time; withLock is inline to keep the hot path allocation-free.
DispatchStageBenchmark (PoolRequests-only, 1 -> 4 feeders): scaling
flips from 0.33x to 3.4-7.3x across runs. PoolRequests concurrency,
NostrClient and negentropy suites pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Splits the reconcile out of negentropySync so callers decide how to
load: negentropyReconcile streams needIds (relay has, local lacks —
download) and haveIds (local has, relay lacks — publish) in batchSize
chunks with back-pressure, taking local state as List<IdAndTime> and
slicing it per created_at window on overflow splits; the accumulating
negentropyReconcileIds convenience returns both lists. negentropySync
now delegates to the same window engine.
NegentropySession's primary constructor takes List<IdAndTime> (JVM
erasure forbids a List<Event> overload); the event-list form moved to
NegentropySession.fromEvents, mirroring NegentropyServerSession, with
all call sites migrated.
Adds NostrClientNegentropyReconcileTest (empty local set, partial
overlap both directions, identical sets, batch streaming, since/until
window slicing) — 49 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Restructures the sync around the measured bottlenecks: one download
worker pool now spans the whole sync (windows no longer join before the
next reconcile starts), overflow-split windows are reconciled by a
caller-set number of concurrent NEG sessions (reconcileConcurrency,
default 1) from a shared work queue, and the reconcile-to-download
buffer depth is exposed (idBufferBatches). No NIP-11 auto-detection:
peak subscription usage (maxConcurrentReqs + reconcileConcurrency + 1)
is documented and budgeting it against the relay's max_subscriptions is
the caller's call.
All 44 negentropy tests pass unchanged. Production shootout on a 2.6M
corpus, single connection: 3.6k events/s with old-equivalent params,
4.5k/s tuned (12 reqs + 4 reconcilers) — ~82% of the measured ~5.5k/s
per-connection by-id ceiling, vs 1.6k/s for the old implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The relay was backfilled from 33k to 2.6M kind-30382 events between
runs, and the picture changed: on the big corpus one connection
saturates at ~8 in-flight REQs (~5.5k events/s) and the relay tops out
at ~15k events/s around 40 total in-flight — past that, added
concurrency only inflates per-REQ latency. Axis 1 now stops at the
relay's NIP-11 max_subscriptions (20; exceeding it wedged the
connection in the first extended run), cells carry a hard 120s budget
with one retry per batch, and per-REQ latency is computed over
completed batches so partial cells report honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Downloads the full 33k kind-30382 corpus from nip85.nosfabrica.com per
cell over a (connections x concurrent REQs) matrix with pre-connected
sockets. One connection scales near-linearly to 16 in-flight REQs
(1,970 -> 30,735 events/s) with no wall at 4, and 1x16 matches 4x4 —
total in-flight REQs is the real variable. Every slow path measured so
far (serial page cursors ~2-3.7k/s, negentropySync 1.6k/s) is a
pipelining/serialization problem: negentropySync starves its download
workers on reconcile cadence and recurses overflow windows
sequentially. Findings and implied negentropySync improvements in the
plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
A raw no-parse socket and the full quartz stack page the same
production query on one connection at the same ~3.4-3.8k events/s,
proving the single-connection ceiling reported by a user is the
relay's per-connection response cadence rather than the client's
serial parse (which is ~1% busy at that rate). Findings and the
assessment of the proposed parallel-parse + ordered-dispatch pipeline
are in the plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Measures how to download millions of events from a single relay as fast
as possible, download+parse only: local geode ceilings (giant REQ vs
until-cursor paging vs created_at-sharded connections, quartz stack vs
raw frames), offline per-frame strategies (full parse, parallel parse,
id-scan for raw archiving), and a production case syncing kind 30382
from nip85.nosfabrica.com via NIP-77 negentropy against plain paging.
Headline results in the plan doc: paging beats giant REQs 26x (which
also dropped frames), created_at sharding stacks ~2x on top, parse is
2-5% of the budget, and negentropy is 2.4x slower than paging for a
cold download (its win is incremental re-sync).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Offline microbenchmark of NostrClient's dispatch stage — PoolRequests
state machine, listener fan-out, id dedup and handoff to a verify
stage — under 1 and 4 concurrent relay feeders. Key results: ~100ns per
message uncontended, but negative scaling under concurrency (the
PoolRequests busy-wait spin lock makes 4 feeders slower in aggregate
than 1), per-event channel handoff costs ~180ns vs a free 64-batch,
and early dedup before the locked path gives 2.7-4x aggregate
throughput at production duplicate factors. Findings appended to the
receiver-perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Gated JVM test (-PprodRelayBench=1) that connects to live relays with
realistic filters and measures per-relay queue delay, processing time,
consumer busy fraction, EOSE latency and duplicate rates, comparing the
current inline verification against a parallel verify stage, plus
offline single-thread parse/verify ceilings on captured frames.
Findings from a first run are written up in
quartz/plans/2026-07-02-nostrclient-receiver-perf.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Part C of the dispatchers/thread-caps audit. Both LnurlEndpointCache and
DesktopCachedRichTextParser were bounded caches backed by a LinkedHashMap
behind a single monitor (@Synchronized / Collections.synchronizedMap with
accessOrder). An access-order map structurally mutates on get, so every
read took the lock — serializing all readers on paths that are hot
(kind-9735 zap-receipt validation; feed rich-text rendering).
Add ConcurrentLruCache<K, V> in quartz utils: storage is a
ConcurrentHashMap so get is lock-free; writes + eviction run under a small
write lock that is off the read path. Eviction is least-recently-put order
(get does not refresh recency) — exactly what LnurlEndpointCache already
did, and fine for the deterministic rich-text parse cache.
Point both caches at the shared helper. Covered by a new
ConcurrentLruCacheTest (round-trip, eviction order, re-put recency
refresh, get-does-not-refresh, clear, and a concurrent size-bound smoke
test); the existing LnurlEndpointCacheTest still passes unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANuUziXKRafSTBxbh4SMoq
Part of the dispatchers/thread-caps audit. Two low-risk fixes that remove
thread-blocking work from paths hit on every event / every packet:
- LargeCache (iOS actual): drop the runBlocking wrapper around
createIfAbsent. The block contained only synchronous CacheMap ops (the
same get/put getOrCreate already calls without runBlocking), so it was
pure dispatcher-blocking overhead on the per-event ingest path. Now
mirrors the JVM actual's plain-function shape.
- QUIC JCA AEADs (AES-GCM + ChaCha20-Poly1305): split the single
`synchronized(this)` monitor into disjoint encryptLock / decryptLock.
seal-family touches only encryptCipher + recentEncryptNonces; open-family
touches only decryptCipher, so a connection's send loop and read loop no
longer serialize against each other through crypto on every packet. The
documented defence-in-depth against cross-coroutine Cipher corruption is
preserved per direction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANuUziXKRafSTBxbh4SMoq
printStackTrace() dumps straight to stderr, bypassing both Log.minLevel
and the consumer's Log.sink — the very thing the LogSink work exists to
control. Migrate the five production call sites:
- Lud06: drop two printStackTrace() calls that sat directly above an
existing Log.w(..., t) carrying the same throwable (pure duplication).
- ElectrumXClient: the swallowed-lookup catch said "Log but don't crash"
yet used printStackTrace(); route it through Log.w with context.
- OpenTimestamps: log the swallowed merge failure via Log.w; drop the
print-then-rethrow (the rethrown exception already carries the trace).
Socket-protocol writer.println(...) and README/KDoc println examples are
left as-is — they are wire I/O and documentation, not logging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EK3TrDkP1EXj1d62oKJMdc
Quartz already funnels every diagnostic through the `Log` facade, but the
sink was hardcoded per platform (android.util.Log / System.err / NSLog /
println), so a consuming app couldn't route Quartz logs into its own stack
(Timber, SLF4J, Crashlytics, a file, a test buffer, or /dev/null).
Add a `LogSink` fun interface and a replaceable `Log.sink`, defaulting to
`PlatformLogSink` which reproduces the historical per-platform behavior.
All ~225 call sites and the `Log.*` signatures are unchanged; the lazy
`() -> String` overloads still short-circuit on `minLevel` before the
lambda runs, preserving the allocation-free fast path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EK3TrDkP1EXj1d62oKJMdc
davotoula
Claude
Vitor PamplonaandGitHub