Extend the showOnchainWallet preference to the zap flows, which also surface an
on-chain rail:
- the on-chain rail on each zap-amount chip in ZapAmountChoicePopup (folded into
the shared railCapability so every caller — reaction row, ReusableZapButton —
is covered)
- the "Send on-chain instead" hand-off button in the custom zap dialog
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
Some users don't want the on-chain wallet surfaced. Add a `showOnchainWallet`
UI preference (default true, so behavior is unchanged) that hides it across the
app when turned off:
- the "Bitcoin" card on the Wallet screen (OnchainSection)
- the on-chain chip on profiles (DisplayPaymentRailChips)
- the on-chain rail in the Send Payment screen
The flag lives in the existing UiSettings/UiSettingsFlow display-preferences
system (persisted to the shared-settings DataStore, alongside the other
show/hide profile toggles) and is exposed as a switch on the Profile UI
settings screen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
A relay that refuses negentropy with a NEG-ERR whose reason merely starts
with "blocked" (e.g. "blocked: Negentropy sync is disabled" from a relay
that has NIP-77 turned off, or an auth/ban refusal) was misclassified as a
strfry `max_sync_events` overflow by `isOverflow`. Overflow triggers
created_at window-splitting, so every split re-opened, was refused again,
and the splitter fanned out breadth-first across the whole created_at range
(~2^31 windows). The call therefore never threw NegentropySyncException (so
`negentropySyncOrFetch` never took its paging fallback) and never tripped
the idle watchdog (the relay answered every NEG-OPEN promptly), so it hung
indefinitely. A second relay whose refusal string did not start with
"blocked" fell through to `Failed` -> paging and completed, which is why the
two behaved differently despite advertising the same NIPs.
Narrow `isOverflow` to genuine "result set too large" signals only; a bare
`blocked:` refusal now maps to a hard failure and fails over to paging.
Adds a regression test driving an in-process relay that refuses every
NEG-OPEN with "blocked: Negentropy sync is disabled" while still serving
plain REQ: negentropySyncOrFetch now pages and delivers every event instead
of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015UF3eh76rRiwAuPg32rwiz
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):
geode import [--db …] [--no-verify] [FILE…] # files, or stdin when none
geode export [--db …] # NDJSON to stdout
Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.
This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.
Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.
Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two benchmark-integrity bugs that could invalidate sync numbers while a run
reports success:
- CorpusServerMain keyed its serve-existing decision only on port + row
count, so re-running a port with a different corpus/maxCount, or after a
load crashed mid-way, silently served a stale/partial DB. Gate reuse on a
completion sentinel keyed on corpus identity (path + byte length) and
maxCount, written only after a full load; on any mismatch the prior DB is
dropped and reloaded.
- CorpusDownloader treated CLOSED identically to EOSE, so a relay ending a
sub early (rate-limit/policy) after sending a partial page advanced the
cursor past the unsent tail — silent corpus loss. Treat CLOSED as a soft
failure (null → reconnect and retry the same cursor; the id dedup set
absorbs the re-fetch), distinct from EOSE which means the page is
complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
runCatchUpUp used negentropyReconcileIds, which builds the FULL need+have
id lists in memory even though the up direction only needs haveIds — on a
large window (e.g. 1M local events against an empty upstream) that is a
~100 MB+ heap spike per convergence round, plus a needIds list built and
immediately discarded.
Switch to the streaming negentropyReconcile: publish each haveIds batch as
it arrives (bounded to one batch of ids) and drop the need direction via a
no-op onNeedIds. The publish still suspends the reconcile round, so the
back-pressure and the reconcile-as-delivery-check convergence loop are
unchanged — only the peak memory drops from O(window) to O(batch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:
- id tie-break: thread the IndexingStrategy through run()/prepareStreams
and append ", id ASC" to the per-stream ORDER BY when
useAndIndexIdOnOrderBy is set — matching every sibling query in
QueryBuilder. The id-indexed order comes straight off the index (no
extra sort, lazy cursor preserved), so the merge now matches the
single-SQL path byte-for-byte on same-second same-author events. Without
the id index the tie stays in rowid order (a valid NIP-01 newest-N);
documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
kinds, so a filter with a repeated pubkey can no longer open two
identical cursors and emit each matching event twice (the single-SQL
IN(…) path already dedups).
Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode paged-REQ sink (verify on, FTS off):
geode→geode 4,963 ev/s (completes) strfry→geode 5,801 ev/s (completes)
Findings: strfry has no REQ bulk sync (stream is live-only limit:0), so the
strfry-sink REQ pairs are n/a. strfry→geode over REQ now completes (earlier
FTS-on stall was strfry killing the slow client at its 32MB pending cap; FTS-off
keeps pace). REQ is ~15-30% slower than negentropy for the geode sink, and
geode-as-a-REQ-source is slower than strfry (SQLite range scans vs LMDB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
All four source→sink pairings via NIP-77, geode sink configured like strfry
(verify on, FTS off), run sequentially:
geode→geode 6,971 ev/s strfry→geode 6,690 ev/s
strfry→strfry 2,682 ev/s geode→strfry 2,127 ev/s
Key findings: the sink sets the rate (geode ~6.7-7.0k, strfry ~2.1-2.7k from
either source — geode ingests ~2.6-3.3x faster); full geode↔strfry negentropy
interop both directions; strfry-source count is lower because its negentropy
snapshot drops NIP-40-expired events (geode, no cutoff, offers its full set).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Reuse the file-backed source DB when it already holds events instead of
deleting + reloading the corpus every boot, so re-running a single sync pair
skips the multi-minute load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
- MirrorSyncThroughputTest: add -DsyncVerify (default true — `strfry sync` always
verifies received events, so the negentropy sink verifies too for an
apples-to-apples comparison) and -DsyncFts (default true; pass false to match
strfry, which has no NIP-50). Both forwarded through the geode test task.
- CorpusServerMain: a benchmark-only tool that boots a real geode relay
(geode's default indexing) preloaded with an NDJSON corpus over a file-backed
store and serves forever, so `strfry sync` / another geode / the negentropy
sink can reconcile against a geode source holding the same 1M corpus a strfry
source does.
Used to run the 4-pair 1M negentropy sync comparison (geode↔geode, strfry→geode,
strfry↔strfry, geode→strfry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
strfry's `sync --dir` is bidirectional (its source: doUp = both||up,
doDown = both||down), but geode's catch-up was down-only. Add the up half so
`dir=up`/`dir=both` reconcile-and-push matches `strfry sync --dir both`.
runCatchUpUp reconciles the local set against the upstream (negentropyReconcileIds)
and publishes the events we hold that the upstream lacks (the reconcile's `have`
ids). Symmetric to the down catch-up: same one `dir`, live up-session starts at
`now` when the up catch-up covers history.
Reliability: client.publish's outbox is best-effort under a bulk burst (each
publish also churns a reconnect — measured ~1-2% dropped per pass), so the push
runs as a reconcile→push convergence loop. Each round re-reconciles — the
reconcile IS the delivery check against the upstream — and re-pushes only the
stragglers until the have-diff is empty. Test observed 3000 → 69 → 2 → 0 across
3 rounds, lossless.
Test: MirrorNegentropyCatchUpTest.negentropyCatchUpPushesUp pushes 3000 local
events to an empty (no-verify) sink and asserts all 3000 land. The four existing
mirror tests still pass (up catch-up needs a store + negentropyBackfill, both off
by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).
MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:
- Catch-up reconciles the local set against the upstream over the
[now - backfill_seconds, now] window and downloads only the diff via the
existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
overrun us) and it completes the pull. Reconcile-against-local means a warm
restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
the sync's job). The windows overlap at `now`; the store's unique-id
constraint dedups the seam.
Changes:
- quartz: add a backward-compatible `localEntries` param to the public
negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
trusted-scope re-check as the live path; failure is non-fatal). New `store` +
`negentropyBackfill` ctor params; default off so existing live-REQ tests are
unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
preloading historical events a live-only sub cannot deliver, then proves the
post-boot event still arrives (3000 catch-up + 1 live = 3001).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a NIP-77 negentropy client path to MirrorSyncThroughputTest (default for
external sources; `-DsyncMode=req` keeps the paged-REQ drain). geode reconciles
its empty set against strfry, then client-paced fetch-by-ids + ingest.
Result on the 1M damus.io corpus (strfry source):
- reconcile (empty local → 995,024 need-ids): 11.4 s, 64 rounds.
- fetch + ingest: ~7,000 ev/s steady-state.
- overall: 994,936 / 997,980 in 171.0 s => 5,818 ev/s — COMPLETES.
This is the apples-to-apples counterpart to strfry→strfry's `strfry sync`
(both negentropy, same corpus, empty→full): strfry→geode 5,818 ev/s vs
strfry→strfry ~2,550 ev/s — geode ingests real content ~2.3x faster and
finishes the pull. The paged-REQ path, by contrast, stalls at ~310k every time
(strfry kills a slow REQ client at its 32 MB maxPendingOutboundBytes cap),
confirming that cross-relay bulk sync from strfry requires negentropy, not REQ.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a strfry→geode drain mode to MirrorSyncThroughputTest and documents the
three-way 1M-corpus sync throughput comparison.
Findings (relayBench/plans/2026-07-04-sync-throughput-1m.md):
- strfry→strfry (strfry sync / negentropy): ~2,550 ev/s, completes.
- geode→geode (MirrorWorker REQ): 13,161 ev/s, completes — but inflated by
synthetic 4-byte content; real-content ingest is ~7,000 ev/s.
- strfry→geode (paged REQ drain, real corpus): sustains ~7,000 ev/s but does
NOT finish — strfry hard-kills a slower REQ client at its 32 MB
maxPendingOutboundBytes cap (~310k events in; confirmed by the source log's
"Pending: 32.01M" disconnects). Independent of heap (2G/10G) and the live
negentropy index (on/off).
Conclusion: strfry's REQ serving is structurally hostile to any client slower
than its scan (buffers outbound, then kills or OOMs). Only NIP-77 negentropy —
pull/reconcile-based and client-paced — completes a cross-relay bulk pull, which
is why strfry's own sync uses it. A production geode backfill from a foreign
relay should use negentropy, not the live-tail REQ path.
Also documents MirrorWorker's unbounded intake channel (a deliberate live-tail
trade) OOMing under a 1M bulk backfill, and forwards the sync* system
properties through the geode test task.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.
When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds -DsyncLiveIndex and a fast (no-live-index) source preload. Rate curves show
geode's mirror sustains ~20k ev/s with the live index off, but collapses to
O(n^2) with it on: the LiveNegentropyIndex is a sorted ArrayList whose per-event
insert is O(n), and a backfill delivers historical (non-near-tail) events, so
every insert memmoves ~n/2 entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Default EventStore(null) tokenizes FTS synchronously on every insert, which
dominates ingest and misrepresents mirror sync throughput. Use geode's actual
RelayIndexingStrategy (deferred FTS, live negentropy index) for both source and
sink, and log instantaneous events/s every 3s so the rate is visible during the
run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two pieces measuring how fast each sink pulls a large corpus from a source using
its native sync client:
- relayBench/sync-throughput-strfry.sh: `strfry import` N events into a source
strfry, boot it, `strfry sync --dir down` an empty sink, report events/s.
- MirrorSyncThroughputTest: geode downstream pulls via the real MirrorWorker
(WebSocket). Default in-process geode source (geode→geode); with
-DsyncSourceUrl it mirrors an external relay (e.g. strfry) for strfry→geode.
Sized by -DsyncN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The ~19/40k event shortfall in the geode↔geode sync is not a geode event-loss
bug: geode is proven lossless across the store, the concurrent IngestQueue
pipeline, RelaySession (OK-true only post-commit), and the real MirrorWorker
WebSocket path (50k/50k). The shortfall is in the benchmark's hand-rolled
fetchByIds+publish delta transfer. Points the fix at the harness (or at driving
convergence through geode's real mirror) rather than geode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Streams 50k events from an upstream KtorRelay to a downstream via the production
MirrorWorker (real OkHttp WebSocket + trusted skipVerify ingest) and asserts the
downstream receives every one — 50000/50000, 0 missing.
This closes the last untested layer: the in-process guards (BatchInsertLossTest,
ConcurrentIngestLossTest) call IngestQueue.submit directly, bypassing the wire.
With this, geode is proven lossless end-to-end — store, concurrent pipeline,
background pool contention, RelaySession, and the real WS mirror path. The
~0.05% shortfall seen in the geode↔geode relayBench sync is therefore an
artifact of the harness's hand-rolled fetchByIds+publish delta transfer, not a
geode event-loss bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the full IngestQueue pipeline — parallel verify, greedy-drain group
commit, a concurrent deferred-FTS catch-up worker taking the pool writer, and
windowed concurrent submits via the trusted (skipVerify) mirror path — over the
clean 200k corpus, asserting every Accepted regular event is queryable after.
Passes (199,612 in/accepted/stored, 0 lost), together with BatchInsertLossTest
proving geode's ingest is lossless at every in-process layer. The geode↔geode
sync event-loss therefore lives above the store+queue — in the real Ktor
WebSocket path or the benchmark harness's hand-rolled delta transfer, which the
in-process paths bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the exact store path — batchInsertEvents with geode's indexing strategy
in 64-event batches — over the clean 200k corpus and asserts every Accepted
kind-1 (regular, never replaced/deleted here) is queryable afterward. Passes
(199,612 in, 199,612 stored), which is the point: it proves the sequential
store path is lossless and narrows the geode↔geode sync event-loss to the
concurrent IngestQueue pipeline (async verify + greedy-drain batching +
concurrent WS submits), not the store itself.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Report, per sync pair: fetch coverage (did the peer REQ return every needed id,
without duplicates?), publish acks (accepted/rejected/unacked per target), and —
for any event a relay ends up missing — whether it was in the delta batch
delivered to that relay. In-batch-but-absent isolates ingest loss
(ack-without-persist) from a fetch/read gap, turning a vague "missing N" into a
pinned layer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The harness's reconcile reference only collapses replaceable kinds; it ignores
NIP-62 Request to Vanish, NIP-09 deletions, and NIP-40 expiration. So a
compliant relay (geode, which honors NIP-62) is falsely flagged "did not
converge" and the phantom events inflate its reconcile round count, masking
negentropy speedups at the wire. Documents the symptom (traced to one
ALL_RELAYS vanish pubkey in the damus corpus), the census (7 vanish pubkeys,
381 expiry events, 0 deletions in the first 200k), and the fix design for
whoever implements it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
When the identical-set reconcile shows a side missing events the reference set
has, log those events grouped by kind (plus a few samples with their tag keys)
instead of leaving a bare "did not converge". The harness already holds every
event by id, so it can resolve exactly what a relay dropped — turning a vague
verdict into an actionable ingest-semantics diagnosis (NIP-09 deletion / NIP-40
expiration / validation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Op instances key Timestamp.ops (MutableMap<Op, Timestamp>), so contract
violations corrupt hash-map behavior:
- OpKECCAK256 defined equals without hashCode, so equal instances hashed
by identity — two equal keys could land in different buckets, producing
duplicate branches or failed lookups in keccak256 timestamp trees. Add
hashCode = TAG, mirroring OpSHA1/OpSHA256/OpRIPEMD160.
- OpBinary defined hashCode without equals — and its TAG referenced
Op.TAG (0x00), a no-op XOR. Define the equals/hashCode pair once on
OpBinary using tag() and drop the duplicated overrides from
OpAppend/OpPrepend (behavior unchanged: same tag + same arg content).
- VerifyResult.equals cast without a type test (ClassCastException on
foreign types instead of false) and hashCode force-cast the nullable
timestamp (NPE for null-timestamp results). Convert to a data class;
the custom toString and compareTo stay.
v1.2.0 (on Maven Central) speeds up the library's own reconcile and fingerprint
walk on top of the v1.1.1 PrefixSumStorageVector wiring. At the 1M relayBench
slice shape (NegentropyReconcileBenchmark, converges exactly, need/have=200k):
client reconcile 264 → 178 ms, seal 424 → 320 ms, and the library's O(range)
fingerprint walk 447 → 252 ms (~1.8×). Our prefix-sum path still answers each
range fingerprint in 0.7 ms (356× the now-faster walk). All NIP-77 tests pass.
Update the reconcile-profiling plan: the fix shipped via the upstream
IStorage.fingerprint seam (v1.1.1) rather than a quartz-side fast server; record
the v1.0.2 → v1.1.1 → v1.2.0 progression.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
`SyncBenchmark.effectiveEvents` materializes the whole corpus as Event objects
plus a dedup LinkedHashMap to derive the 80% slices, so a million-event run
blew past the 2g default with an OutOfMemoryError before the negentropy sync
could start. -Xmx is a ceiling, not a reservation, so smaller runs don't pay
for the higher limit; JAVA_OPTS still overrides it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
DesktopLocalCache stores Users in a WeakReference-backed LargeSoftCache. The
followee User in kind3IsHydratedBeforeKind0SoMetadataLoadsForFollowedAuthors is
created only during hydrate's kind:0 phase and has no Note referencing it, so it
is only weakly reachable once hydrate returns. A GC landing between hydrate()
and the assertions evicted it, flaking the test (reproduced deterministically by
forcing System.gc()).
Pin a strong reference to the followee's User for the duration of the test so
the cache cannot evict it, mirroring how followed users stay reachable via live
account/UI state in the running app. The ordering invariant the test asserts is
unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NHQ3g7wD9WbDvj7NspiAWW
kmp-negentropy v1.1.1 adds the `IStorage.fingerprint(begin, end)` seam we needed
and ships `PrefixSumStorageVector` — a drop-in IStorage that builds an additive
prefix-sum table on seal() and answers any range fingerprint in O(1) instead of
re-walking the range. Range fingerprints are the CPU-bound core of a NEG-MSG on
a large snapshot; profiling pinned them as the steady-state reconcile cost geode
lost multiples on (not serialization).
Seal a `PrefixSumStorageVector` in both `NegentropyServerSession.sealVector`
(server / relay-relay responder, also backs the `LiveNegentropyIndex` snapshot
cache) and `NegentropySession` (initiator). Byte-identical to the plain vector —
only the fingerprint path is accelerated. `NegentropyPrefixFingerprintTest` now
also asserts the library's `PrefixSumStorageVector.fingerprint` matches the plain
walk over 2000 random ranges + boundaries at 50k; the reconcile-shaped mix
measures 601× (447 ms → 0.7 ms).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The `rawQueryPathMatchesDecodedQuery` case called `store.rawQuery(filter)`, but
`EventStore` only exposes the streaming `rawQuery(filters, onEach)` — the
list-returning overload lives on the inner `SQLiteEventStore`. Point the check
at `store.store.rawQuery(filter)` so the zero-decode path is actually exercised.
Record the shipped k-way merge result in the plan doc: a fresh 1M relayBench run
has geode `follow-feed` at 18.8 ms vs strfry 17.7 ms (down from 97.7 ms, now at
parity) and 46,258 ev/s vs strfry 15,365 @8conn, both returning the same 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The home-feed REQ (`authors=[…] (+ kinds=[…]) [+ since/until] limit=N`,
newest-first) is one of the most common relay queries. SQLite serves it by
seeking every `(kind, pubkey)` combo and feeding *all* matching rows through a
LIMIT-bounded sorter, so it reads O(the followed set's whole matching history)
— on a cold on-disk 1M corpus that was the `follow-feed` regression (relayBench:
97.7 ms vs strfry 17.6 ms).
Add `MergeQueryExecutor`, an app-level k-way merge that opens one lazy
newest-first cursor per stream off the existing composite indexes
(`query_by_kind_pubkey_created`, or `query_by_pubkey_created` for authors-only),
merges their heads `(created_at DESC, id ASC)` and stops at the limit — reading
only O(limit + streams) rows regardless of how much history the authors have.
It reuses indexes that already exist, so write throughput and on-disk size are
untouched. Eligibility is narrow (2..2048 streams, simple filter, explicit
limit, no ids/d-tags); everything else falls through to the single-SQL plan.
Wired into both `query` and the zero-decode `rawQuery` paths (the relay REQ hot
path) and the single-element filter-list variants, so `LiveEventStore` REQs go
through it.
`MergeQueryCorrectnessTest` proves the merge returns exactly the single-SQL
top-N — vs an independent Kotlin reference and vs the SQL path — across
distinct/tied created_at, since/until windows, authors-only, fewer-than-limit,
streaming onEach, and the raw path. `FollowFeedReadBenchmark` gains a `merge`
variant: at 1.05M events it's flat ~10-12 ms across both prolific-recent and
sparse-old, where `scan` is catastrophic on sparse follows (1995 ms) and
`current` is disk-bound on prolific ones.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bumps the negentropy-kmp dependency from v1.0.2 to v1.1.1. The release is
backward compatible for consumers (StorageVector still provided; the codebase
only consumes IStorage and never implements it). Quartz compiles and all
NIP-77 negentropy tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M3659Rs6ayVm4XmxvbtpWE
follow-feed (kinds=[1,6] × 150 authors, ORDER BY created_at DESC LIMIT 500)
was geode's 5.5× loss (97.7ms vs strfry 17.6ms). Investigated whether any
change is worth it, across read + write + size.
Read (FollowFeedReadBenchmark, in-memory, scale 5 ≈ 1.05M events):
prolific-recent sparse-old
current 5.7 ms 1.9 ms
scan (strfry) 1.0 ms 1601.9 ms
union 316.9 ms 20.0 ms
- scan (created_at index + early LIMIT) wins for active follows but is
catastrophic for sparse/inactive follows AND grows with corpus size
(234ms→1601ms from scale 1→5) — following rarely-posting accounts is
common, so it'd be a severe regression.
- union (300 per-branch subqueries) is dominated by branch overhead.
- current is the only robust option — flat across scale, bounded by the
followed set, never catastrophic. The 97.7ms is a worst case (the 150
MOST prolific authors, disk-bound reading all their matching rows).
No safe SQL-level swap exists; each alternative trades geode's worst case
for a worse one on a common workload. The only universal improvement is
strfry's app-level k-way merge (O(LIMIT+streams)) — a real new executor,
not a SQL tweak.
Write & size: neutral for every candidate — all reuse existing indexes
(query_by_kind_pubkey_created / query_by_created_at_id), none adds a
CREATE INDEX, so ingest throughput and storage are untouched regardless of
choice. A new index was considered and rejected (taxes every write, helps
one shape, reverts under ANALYZE).
Decision: keep the current composite plan. Full write-up in
quartz/plans/2026-07-04-follow-feed-read-tradeoff.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Answers 'could a new index beat the INDEXED BY pin without the pin?' for
the profiles shape (kind=0 AND pubkey IN(...) ORDER BY created_at DESC).
Measured:
- stock unhinted: scans query_by_kind_created (1.40ms) — the bug.
- pinned composite: seek + tiny sort (0.23ms) — the shipped fix.
- new (pubkey,kind,created_at) index, unhinted: STILL scans — no help.
- new (kind,pubkey,created_at ASC) index, unhinted: picks the seek (0.23ms)
BUT that's a no-stats cost-model artifact — ANALYZE reverts it to the
scan (1.47ms). Fragile, and a full duplicate of the DESC composite.
Root reason: ORDER BY created_at over a multi-value pubkey IN(...) needs a
sort no matter the index (no B-tree gives global created_at order across
pubkeys), and SQLite prefers the one sort-free plan — the full-kind scan.
Only the explicit pin reliably overrides that. A new index would add
write+storage cost on every event for the whole relay, help only this one
shape, and break under ANALYZE — so the free, deterministic, scoped pin is
strictly better. Diagnostic evidence for the design choice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The server's per-round reconcile spends a large slice turning the ~1MB hex
reconcile frame into wire JSON: the generic (Jackson) serializer wraps the
hex string in a value node and scans every char for JSON escapes a
[0-9a-f] payload can never contain, then re-copies.
NegMsgMessage.toJson() now builds ["NEG-MSG","<sub>","<hex>"] directly —
no node tree, no escape scan of the hex. Fast path fires only for
escape-free printable-ASCII subIds (what the JSON encoder emits verbatim);
exotic subIds fall back to the generic serializer, so output is
byte-identical. RelaySession.send routes through message.toJson() (default
unchanged for every other message type).
Measured (toJson + UTF-8, per frame): 64KiB 2.5×, 250KiB 2.6×, 500KiB
(strfry cap) 2.8× — ~2.5ms saved per NEG-MSG, ~35ms over a 14-round
reconcile. Correctness: a subId battery asserts byte-identity with the
generic path, and GeodeVsStrfryNegentropySyncTest (real strfry) reconciles
against the fast-built frames.
Also records the ingest-latency candidate as measured-not-worth-it: the
IngestQueue pipeline overhead is only ~0.17ms p50, <10% of the ~2.4ms
receipt→queryable gap — that gap lives in the REQ-visibility path, not the
writer. Full write-up in
quartz/plans/2026-07-04-sync-serialization-and-ingest-latency.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Vitor PamplonaandGitHub
Claude
davotoula