Audit of the authorsMissingOutbox anti-join surfaced one correctness bug and
one performance win:
- Bug (semantic): kind-1059 giftwraps store a random one-time key in
event_headers.pubkey (the real recipient is only a hash), so the query
returned an unbounded set of ephemeral keys that can never own a 10002 —
junk for the outbox model this feeds. Both the SQLite path and the generic
default now exclude kind 1059 from the "authors" set.
- Performance: replaced the DISTINCT + correlated NOT EXISTS scan with an
index-only EXCEPT (all authors minus 10002 owners). Both sides ride the
unconditional query_by_kind_pubkey_created covering index — so it does NOT
depend on the optional pubkey-alone index — and measured ~3x faster
(44ms vs 137ms at 152k events / 20k authors); the gap widens with author
count, since the old form paid one seek per distinct author. A loose-index
skip-scan was rejected: it needs the pubkey-alone index and degrades to a
full scan per author without it.
Also: corrected the KDocs (the old text implied an efficient index-only
distinct that wasn't guaranteed), added a giftwrap-exclusion test, and added
FsAuthorsMissingOutboxTest — the only coverage of the IEventStore DEFAULT
implementation, which EventStore always overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Adds AuthorsMissingOutboxBenchmark (gated behind -PprodRelayBench=1, like the
other prod benches). It syncs a real sample from relay.damus.io (kind 1 notes +
kind 10002 relay lists), replicates it to 1,000,000 stored rows while preserving
the real author set and outbox-owner set, then times the two shipping
implementations of authorsMissingOutbox() on the same store:
- generic: the IEventStore interface default (decodes every event via
query(Filter()))
- sqlite: EventStore's SELECT DISTINCT pubkey ... NOT EXISTS
Both are asserted to return the same set, matching the seeded ground truth.
Measured on a 4-core container, 1,000,000 events (best of 3):
generic 138,880 ms
sqlite 2,623 ms → ~53x faster
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:
- id tie-break: thread the IndexingStrategy through run()/prepareStreams
and append ", id ASC" to the per-stream ORDER BY when
useAndIndexIdOnOrderBy is set — matching every sibling query in
QueryBuilder. The id-indexed order comes straight off the index (no
extra sort, lazy cursor preserved), so the merge now matches the
single-SQL path byte-for-byte on same-second same-author events. Without
the id index the tie stays in rowid order (a valid NIP-01 newest-N);
documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
kinds, so a filter with a repeated pubkey can no longer open two
identical cursors and emit each matching event twice (the single-SQL
IN(…) path already dedups).
Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the full IngestQueue pipeline — parallel verify, greedy-drain group
commit, a concurrent deferred-FTS catch-up worker taking the pool writer, and
windowed concurrent submits via the trusted (skipVerify) mirror path — over the
clean 200k corpus, asserting every Accepted regular event is queryable after.
Passes (199,612 in/accepted/stored, 0 lost), together with BatchInsertLossTest
proving geode's ingest is lossless at every in-process layer. The geode↔geode
sync event-loss therefore lives above the store+queue — in the real Ktor
WebSocket path or the benchmark harness's hand-rolled delta transfer, which the
in-process paths bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the exact store path — batchInsertEvents with geode's indexing strategy
in 64-event batches — over the clean 200k corpus and asserts every Accepted
kind-1 (regular, never replaced/deleted here) is queryable afterward. Passes
(199,612 in, 199,612 stored), which is the point: it proves the sequential
store path is lossless and narrows the geode↔geode sync event-loss to the
concurrent IngestQueue pipeline (async verify + greedy-drain batching +
concurrent WS submits), not the store itself.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
kmp-negentropy v1.1.1 adds the `IStorage.fingerprint(begin, end)` seam we needed
and ships `PrefixSumStorageVector` — a drop-in IStorage that builds an additive
prefix-sum table on seal() and answers any range fingerprint in O(1) instead of
re-walking the range. Range fingerprints are the CPU-bound core of a NEG-MSG on
a large snapshot; profiling pinned them as the steady-state reconcile cost geode
lost multiples on (not serialization).
Seal a `PrefixSumStorageVector` in both `NegentropyServerSession.sealVector`
(server / relay-relay responder, also backs the `LiveNegentropyIndex` snapshot
cache) and `NegentropySession` (initiator). Byte-identical to the plain vector —
only the fingerprint path is accelerated. `NegentropyPrefixFingerprintTest` now
also asserts the library's `PrefixSumStorageVector.fingerprint` matches the plain
walk over 2000 random ranges + boundaries at 50k; the reconcile-shaped mix
measures 601× (447 ms → 0.7 ms).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The home-feed REQ (`authors=[…] (+ kinds=[…]) [+ since/until] limit=N`,
newest-first) is one of the most common relay queries. SQLite serves it by
seeking every `(kind, pubkey)` combo and feeding *all* matching rows through a
LIMIT-bounded sorter, so it reads O(the followed set's whole matching history)
— on a cold on-disk 1M corpus that was the `follow-feed` regression (relayBench:
97.7 ms vs strfry 17.6 ms).
Add `MergeQueryExecutor`, an app-level k-way merge that opens one lazy
newest-first cursor per stream off the existing composite indexes
(`query_by_kind_pubkey_created`, or `query_by_pubkey_created` for authors-only),
merges their heads `(created_at DESC, id ASC)` and stops at the limit — reading
only O(limit + streams) rows regardless of how much history the authors have.
It reuses indexes that already exist, so write throughput and on-disk size are
untouched. Eligibility is narrow (2..2048 streams, simple filter, explicit
limit, no ids/d-tags); everything else falls through to the single-SQL plan.
Wired into both `query` and the zero-decode `rawQuery` paths (the relay REQ hot
path) and the single-element filter-list variants, so `LiveEventStore` REQs go
through it.
`MergeQueryCorrectnessTest` proves the merge returns exactly the single-SQL
top-N — vs an independent Kotlin reference and vs the SQL path — across
distinct/tied created_at, since/until windows, authors-only, fewer-than-limit,
streaming onEach, and the raw path. `FollowFeedReadBenchmark` gains a `merge`
variant: at 1.05M events it's flat ~10-12 ms across both prolific-recent and
sparse-old, where `scan` is catastrophic on sparse follows (1995 ms) and
`current` is disk-bound on prolific ones.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
follow-feed (kinds=[1,6] × 150 authors, ORDER BY created_at DESC LIMIT 500)
was geode's 5.5× loss (97.7ms vs strfry 17.6ms). Investigated whether any
change is worth it, across read + write + size.
Read (FollowFeedReadBenchmark, in-memory, scale 5 ≈ 1.05M events):
prolific-recent sparse-old
current 5.7 ms 1.9 ms
scan (strfry) 1.0 ms 1601.9 ms
union 316.9 ms 20.0 ms
- scan (created_at index + early LIMIT) wins for active follows but is
catastrophic for sparse/inactive follows AND grows with corpus size
(234ms→1601ms from scale 1→5) — following rarely-posting accounts is
common, so it'd be a severe regression.
- union (300 per-branch subqueries) is dominated by branch overhead.
- current is the only robust option — flat across scale, bounded by the
followed set, never catastrophic. The 97.7ms is a worst case (the 150
MOST prolific authors, disk-bound reading all their matching rows).
No safe SQL-level swap exists; each alternative trades geode's worst case
for a worse one on a common workload. The only universal improvement is
strfry's app-level k-way merge (O(LIMIT+streams)) — a real new executor,
not a SQL tweak.
Write & size: neutral for every candidate — all reuse existing indexes
(query_by_kind_pubkey_created / query_by_created_at_id), none adds a
CREATE INDEX, so ingest throughput and storage are untouched regardless of
choice. A new index was considered and rejected (taxes every write, helps
one shape, reverts under ANALYZE).
Decision: keep the current composite plan. Full write-up in
quartz/plans/2026-07-04-follow-feed-read-tradeoff.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Answers 'could a new index beat the INDEXED BY pin without the pin?' for
the profiles shape (kind=0 AND pubkey IN(...) ORDER BY created_at DESC).
Measured:
- stock unhinted: scans query_by_kind_created (1.40ms) — the bug.
- pinned composite: seek + tiny sort (0.23ms) — the shipped fix.
- new (pubkey,kind,created_at) index, unhinted: STILL scans — no help.
- new (kind,pubkey,created_at ASC) index, unhinted: picks the seek (0.23ms)
BUT that's a no-stats cost-model artifact — ANALYZE reverts it to the
scan (1.47ms). Fragile, and a full duplicate of the DESC composite.
Root reason: ORDER BY created_at over a multi-value pubkey IN(...) needs a
sort no matter the index (no B-tree gives global created_at order across
pubkeys), and SQLite prefers the one sort-free plan — the full-kind scan.
Only the explicit pin reliably overrides that. A new index would add
write+storage cost on every event for the whole relay, help only this one
shape, and break under ANALYZE — so the free, deterministic, scoped pin is
strictly better. Diagnostic evidence for the design choice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The server's per-round reconcile spends a large slice turning the ~1MB hex
reconcile frame into wire JSON: the generic (Jackson) serializer wraps the
hex string in a value node and scans every char for JSON escapes a
[0-9a-f] payload can never contain, then re-copies.
NegMsgMessage.toJson() now builds ["NEG-MSG","<sub>","<hex>"] directly —
no node tree, no escape scan of the hex. Fast path fires only for
escape-free printable-ASCII subIds (what the JSON encoder emits verbatim);
exotic subIds fall back to the generic serializer, so output is
byte-identical. RelaySession.send routes through message.toJson() (default
unchanged for every other message type).
Measured (toJson + UTF-8, per frame): 64KiB 2.5×, 250KiB 2.6×, 500KiB
(strfry cap) 2.8× — ~2.5ms saved per NEG-MSG, ~35ms over a 14-round
reconcile. Correctness: a subId battery asserts byte-identity with the
generic path, and GeodeVsStrfryNegentropySyncTest (real strfry) reconciles
against the fast-built frames.
Also records the ingest-latency candidate as measured-not-worth-it: the
IngestQueue pipeline overhead is only ~0.17ms p50, <10% of the ~2.4ms
receipt→queryable gap — that gap lives in the REQ-visibility path, not the
writer. Full write-up in
quartz/plans/2026-07-04-sync-serialization-and-ingest-latency.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Measurement harnesses for the two remaining relayBench gaps, isolating each
cost so a fix can be judged on the delta:
- IngestLatencyBenchmark: times single-event submit→onComplete through the
group-commit IngestQueue vs a direct batchInsert. The delta is the
pipeline's coroutine-handoff overhead — the receipt→queryable latency the
1M run measured geode losing (4.68ms vs strfry 2.32ms).
- NegMsgSerializationBenchmark: times the NEG-MSG wire path (Hex.encode →
MessageKSerializer JsonElement tree → UTF-8) against a direct StringBuilder
build, asserting byte-identical output. Isolates the ~40% serialization
slice of the server reconcile the JFR flagged.
Both compile and are self-contained; results + fixes to follow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
After the profiles fix, sweep every relayBench query shape with EXPLAIN
QUERY PLAN against a realistic multi-kind, tagged store to check for other
scan-that-should-seek mis-costs. Result: clean — every scenario seeks an
index (or does an index scan + LIMIT, like firehose's created_at walk).
The remaining TEMP B-TREE sorts are all over bounded result sets
(author/id count or LIMIT), not the profiles-scale pathology. The tag
queries (thread/notifications/hashtag) do an inherent subquery + sort;
hashtag is the slowest scenario but strfry is equally slow there, so it's
absolute cost, not a competitive gap.
Kept as a regression guard: asserts no scenario full-table-scans a base
table (bare SCAN without USING INDEX).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The profiles scenario — Filter(kinds=[0], authors=[50]), no limit — was
geode's ~100x loss to strfry on the 1M corpus (99.5ms vs 0.94ms).
Root cause: the REQ path always appends ORDER BY created_at DESC. For a
multi-author pubkey IN(...) filter, query_by_kind_created (kind,
created_at) satisfies that order for free by scanning an ENTIRE kind, so
SQLite prefers it over the selective query_by_kind_pubkey_created (which
would need a sort). The scan is O(all kind-0 profiles) — cheap at 2k, the
99.5ms at 1M. ANALYZE does not fix it (verified: even a reopened store
reading fresh sqlite_stat1 keeps the scan, since the ORDER BY genuinely
lets the scan skip a sort). A single author is costed right and already
seeks; only the IN-list of >1 is mis-costed.
Fix: pin INDEXED BY query_by_kind_pubkey_created for exactly that shape —
multi-author + kinds, no ids, no d-tags, no limit — keeping the ORDER BY.
SQLite seeks the authors and sorts the small result: identical rows,
identical newest-first order (zero behavior change), ~8x at 2k profiles,
growing to ~100x at 1M. Limited feeds (home/global) keep the created_at
scan + early LIMIT; single-author and d-tag queries are untouched. The
index is created unconditionally so the hint never dangles.
(Considered dropping the ORDER BY for no-limit author queries — faster and
hint-free, but it changes on-the-wire result ordering, which broke
FsParityTest's ordered-parity assertions, so it's client-visible. Rejected
in favor of the order-preserving pin.)
Adds ProfilesQueryPlanBenchmark (regression guard: asserts the live REQ
plan seeks the composite index, not the kind scan) and
plans/2026-07-04-profiles-query-plan.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
profiles (Filter(kinds=[0], authors=[50]), no limit) was the one query
geode lost to strfry on the 1M corpus — 99.5ms vs 0.94ms (~100x).
Root cause: the REQ path appends ORDER BY created_at DESC even with no
limit. query_by_kind_created (kind, created_at) satisfies that order for
free while scanning EVERY kind-0 profile, so SQLite prefers it over the
ideal query_by_kind_pubkey_created (which would need a sort). The scan is
O(all profiles) — cheap at 2k, the 99.5ms at 1M.
ANALYZE does not help: verified that even a reopened store reading fresh
sqlite_stat1 keeps the scan, because the ORDER BY genuinely lets the scan
avoid a sort.
Two fixes measured (both return identical rows), scoped to no-limit
kinds+authors filters:
- Fix A: drop ORDER BY when limit==null -> planner picks the composite
index itself (~7x at 2k profiles, ~100x at 1M). Changes result order
across authors (a NIP-01 SHOULD; clients re-sort).
- Fix B: force INDEXED BY query_by_kind_pubkey_created + keep ORDER BY
(~same speed, newest-first preserved, at the cost of a scoped hint).
Adds ProfilesQueryPlanBenchmark (prints plans+timings, asserts row-count
equivalence) and plans/2026-07-04-profiles-query-plan.md. No production
change yet — the fix is a core QueryBuilder behavior/ordering decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The 1M relayBench run had geode losing the negentropy phase to strfry
(initial reconcile 6066ms/27r vs 1270ms/14r; identical-set 1947ms vs
557ms). Three layered benchmarks pin where the time actually goes:
- NegentropyReconcileBenchmark (quartz): the kmp-negentropy server loop
in isolation is ~200ms for the full 14-round exchange — the reconcile
ALGORITHM is not the bottleneck. (An early version showed 22s/139r;
that was a benchmark bug — index slices over randomly-sorted ids
scatter the diff. Real relayBench slices are contiguous time ranges;
monotonic created_at fixes it and matches strfry's round count.)
- NegentropyServerReconcileBenchmark (geode): the real in-process geode
server over loopback is 3214ms — 15x the library loop. JFR of the
server call-trees: ~40% hex/UTF-8/JSON serialization of the payloads,
~26% actual reconcile, rest allocation. The gap is the JVM
constant-factor tax on hex-in-JSON, which strfry pays in C++, not a
single hotspot.
- NegentropyPrefixFingerprintTest (quartz): the one algorithmic lever.
Negentropy's fingerprint is an additive sum mod 2^256, so a prefix-sum
table answers any range in O(1). Proven bit-for-bit identical to the
library over 2000 random ranges, and 460x faster per call — the fix
for the ~26% reconcile slice (dominant in the identical-set case).
Not yet wired: the library instantiates FingerprintCalculator
internally, so shipping prefix-sum needs a kmp-negentropy change (or a
quartz-side fast server). Full write-up + artifacts in
quartz/plans/2026-07-04-negentropy-reconcile-profiling.md.
Benchmarks are CI-safe (small defaults / opt-in gates); JFR via
-PnegProfile, scale via -DnegBenchN, geode server bench via
-DnegServerBench=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Backlog item 2 (small-REQ dispatch floor), first cut. relayBench at 50k
shows geode ~2.5x slower than strfry when a REQ returns ~20 events
(author-archive 1.18 vs 0.48 ms EOSE p50) while WINNING the 500-event
scenarios — with tiny results the fixed per-REQ cost dominates. A new
stage benchmark (SmallReqFloorBenchmark) decomposes the floor: at ~21
rows, raw SQL is 0.18 ms and the remaining ~0.42 ms is live-subscription
machinery plus per-REQ coroutine dispatch.
A REQ whose filters all carry a limit summing to <= 256 now runs its
stored replay inline on the receive coroutine and only retains a
live-tail handle (SessionBackend.queryRawInline; LiveEventStore's
queryRaw is re-expressed on the same core) — no per-REQ launch, no Job,
no dispatcher handoffs, no awaitCancellation scaffolding. Unbounded
REQs keep the launched path so CLOSE can always interrupt a giant
replay. In-process time-to-EOSE for ~21-row REQs drops 0.60 -> 0.50 ms;
the bigger effect expected under concurrency (no per-REQ Job+dispatch
churn) is relayBench's to judge.
InlineReqFastPathTest pins wire-behavior parity: stored-then-EOSE
ordering, live tail delivery and CLOSE detachment, same-subId
replacement, launched fallback for unbounded and over-cap REQs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestones 2-3 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
SQLiteEventStore now maintains the LiveNegentropyIndex when the strategy
opts in (geode's does by default; [negentropy].live_index = false turns
it off; app-side stores are untouched):
- Write paths collect a LiveIndexDelta and apply it after COMMIT while
still holding the writer mutex — rolled-back savepoint rows never
reach the index and updates land in exact commit order (also vs the
rebuild, which runs under the same mutex).
- Replaceable/addressable overwrites report the row their BEFORE-INSERT
trigger displaces via one indexed pre-SELECT that mirrors the trigger
predicate (including the NIP-01 lowest-id tie-break and the
d_tag-NULL case).
- Paths that can't itemize (kind-5, vanish, delete-by-filter/id,
expiration sweeps, clearDB) invalidate; the next NEG-OPEN rebuilds
from one scan on the writer connection.
- Until that first NEG-OPEN populates the index, ingest pays zero
bookkeeping — the populated check happens under the writer mutex so
it can't race the rebuild.
LiveEventStore serves a single unconstrained filter (the relay-relay
sync default; relayBench sends exactly this) from the index; everything
else keeps the scan+seal path and its single-slot cache.
Micro-benchmark at 50k events (LiveNegentropyBenchmark, in-container):
scan+seal cold path 80-100 ms; index post-write open 9-16 ms (~5-10x).
The relayBench A/B is the acceptance gate and comes next.
Correctness: LiveNegentropyIndexStoreTest asserts index content ==
snapshotIdsForNegentropy scan after every mutation pattern (overwrites,
losers, kind-5 rebuilds, filter deletes, mixed-outcome batches,
transactions); the full geode suite (NIP-77 + interop sync tests) runs
with the index on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
A single-sample wall-clock speedup assertion (>1.5x) flakes when the
shared CI machine is loaded. Apply the same pattern already used by
ParallelVerifyBenchmark: retry up to 3 measurement attempts taking the
best, keep a hard 1.05x floor that a real regression (~1.0x when dedup
saves no work) can never pass, and warn instead of fail in the noise
band. The deterministic parsed/reused-count correctness gate still
fails hard on every attempt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzMix4ZivxoYrcn88TMqtF
FTS indexing ran inside every insert's transaction — a measurable slice
of write cost (relayBench: ~18% of ingest throughput) paid at publish
time for a feature only search queries read. It now runs as a watermark
catch-up:
- IndexingStrategy.deferFullTextSearchIndexing (default false; geode's
relay strategy enables it with search). Deferred inserts skip
tokenization entirely.
- FullTextSearchModule keeps a fts_catchup_state watermark (everything
<= last_row_id is indexed) and gains catchUpBatch(): scan past the
watermark, tokenize, advance — one write transaction per batch, so
publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at
MAX(row_id) for existing (synchronously indexed) databases.
- NostrServer runs the catch-up worker, poked by IngestQueue's new
onBatchCommitted hook, and *yields to publish traffic*: it only
drains while the queue has no backlog (IngestQueue.hasBacklog()), so
bursts ingest at no-FTS speed and tokenization fills the gaps.
- LiveEventStore drains the backlog synchronously before serving any
filter with a search term (query, queryRaw, count) — NIP-50 results
stay exactly as fresh as the synchronous path; the deferral is
invisible to correctness. Geode's existing search tests pass
unchanged through this path.
The first implementation reused reindexBatch and collapsed ingest 8x —
its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?'
matches on a plain FTS5 column, i.e. a full FTS-table scan per row
(O(n²) overall), and the worker competed with the replay for the writer
mutex. catchUpBatch therefore inserts without the delete (rows past the
watermark are never indexed; switching a DB between deferred and
synchronous strategies requires reindexAll, same rule as a
searchable-kinds change), and the worker backs off whenever publishes
are pending.
Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136
events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the
--no-search ceiling while keeping NIP-50 advertised and fresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
The speedup assertion depends on AVAILABLE parallelism, which a
full-suite run (the pre-push hook) can eat — it flaked at 1.14x under
load. Now retries up to 3 measurement passes and enforces a hard 1.05x
floor that a genuine serialization regression (the per-event-async
version measured 0.94x) can never pass, warning instead of failing in
the noise band between floor and the 1.5x target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Replaces the decoder's spin lock with lock-free concurrent maps so the
hot per-frame duplicate check never serializes across the pool's relay
consumer coroutines. New minimal ConcurrentHashCache expect/actual
(get/put/size/clear) following LargeCache's per-platform choices:
ConcurrentHashMap on JVM/Android, CacheMap on Apple, copy-on-write on
the CI-only Linux target. Counters become atomics; the generational
rotation keeps its deliberately-tolerated benign races, now documented
per failure mode (each is at worst a redundant re-parse, never a wrong
message).
CachingEventDecoderConcurrencyTest hammers one shared decoder from 8
threads with 80k duplicate-heavy frames and capacity 256 (rotations
fire constantly): zero wrong messages, exact parsed+reused accounting.
The 7 scan-safety tests and DedupDecodeBenchmark's enforced speedup
pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Deliberate design decision reversing the 4096-frame receive bound:
1. The remote infrastructure isn't ours — TCP backpressure parks the
backlog in the RELAY's outbound buffers. A client should release the
relay from its duties as fast as it can send and own the buffering
itself.
2. The app holds 2000+ simultaneous relay connections; a bounded buffer
under a slow consumer blocks OkHttp reader threads, and at that
connection count blocked readers are a thread-starvation hazard far
worse than the heap growth they prevent.
The UNLIMITED channels now carry an explicit do-not-bound comment with
this rationale, and the slow-consumer risk is addressed from the other
side: keep the consumer faster than any relay's send rate
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).
BoundedReceiveBufferTest removed with the bound it tested; plan doc
records the decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Two server bugs masking each other made a single giant REQ crawl and
then wedge:
LiveEventStore's historical-replay dedup used an immutable Set under an
AtomicReference with copy-on-add — set + id copies the whole set per
streamed event, so large replays were accidentally O(n²) (100k-event
REQ: ~700 events/s, degrading as the response grew). Replaced with a
spin-lock-guarded mutable HashSet (same threads, single contains/add
per critical section).
Fixing that unmasked WebSocketSessionPump's slow-client policy: a fast
replay instantly overflowed the 8192-frame cap — which conflated 'slow
client' with 'replay outruns the socket writer', normal for bulk — and
the 'drop' only closed the internal queue, leaving the socket half-dead
(no EOSE, no close frame, tail silently missing: the likely cause of
the benchmark's 99,998/100,000). Producers are now paced against a full
backlog (bounded blocking wait, consistent with the documented ingest
fanout behavior) and only a client still behind after 30s is dropped,
by actually cancelling the socket.
GiantReqStreamTest guards the regression: 20k-event REQ pre-fix 8.4s
(~2.4k events/s), post-fix 0.7s (~27k events/s), all events + EOSE
delivered. 96 geode tests and the quartz relay/server suites pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The per-connection-wall test prints the negotiated
Sec-WebSocket-Extensions header; against strfry OkHttp negotiates
permessage-deflate (client_no_context_takeover) out of the box, closing
the 'is compression actually on?' question from the optimization list —
it is, no change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
One reconcile feeds by-id download batches to N clients (one socket
each) x reqsPerClient workers; reconcile windows also round-robin
across the connections, since a single connection produced need-ids at
only ~9k/s on a 2.6M corpus and starved the downloads. Events funnel
through a bounded channel to a single consumer (exact maxEvents,
single-threaded onEvent); all stages backpressure; localEntries diffing
and have-counting match negentropyReconcile. reconcileWindows became
multi-client internally; single-client paths pass listOf(this).
Production shootout (same-run pairs, 100k cap): +18% to +64% over the
tuned single client, capped by the relay's server-side reconcile id
production rather than download parallelism (which the by-id matrix
shows scales 2.7x with connections). 4 in-process multi-client tests;
18 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Client-side mirror of IngestQueue.parallelVerify: submit() is a cheap
bounded-channel send from the relay consumer coroutine; a drain loop
batches greedily (up to 256) and fans each batch across
Dispatchers.Default in core-sized chunks, dispatching callbacks in
submission order. preVerified short-circuits already-trusted ids; the
bounded channel backpressures the socket instead of growing heap.
Batch/chunk sizing is measurement-driven: per-event async cost ~40us of
scheduling each (swallowing the gain), and the per-batch join barrier
at 64 still cost half (64 -> 1.2x, 256 -> 2.2x, 1024 -> 3.2x on 4
cores). ParallelVerifyBenchmark (fresh signed events per pass — Event
caches derived state after first verify, so passes must not share
instances) measures 1.8x vs sequential with an enforced >=1.5x
assertion. 4 correctness tests cover valid/tampered routing, ordering,
preVerified and callback-crash resilience.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The reader-thread-to-consumer channel in BasicOkHttpWebSocket and the
app's OkHttpWebSocket was UNLIMITED: a consumer slower than the socket
accumulated frame Strings without bound (gigabytes over a multi-million
event download). Now capped at 4096 frames — when full, OkHttp's reader
thread blocks and TCP flow control pushes back on the relay instead of
the heap. The trade-off (a blocked reader delays PING/PONG handling) is
documented on the constant; the bound is deep enough that only a
pathologically slow consumer hits it.
BoundedReceiveBufferTest forces sustained backpressure (8-frame buffer,
sleeping consumer, real socket to a local geode relay) and asserts all
events plus EOSE arrive in order with no drops or deadlock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
BasicRelayClient's decode step becomes a pluggable MessageDecoder
(default unchanged). The opt-in CachingEventDecoder scans EVENT frames
for their id (~0.3us, JSON-escape-safe so embedded event JSON in repost
content cannot confuse it; any irregularity falls back to full parse)
and on a cache hit synthesizes the EventMessage from the already-parsed
Event with the frame's own subId — every subscription still gets its
delivery and per-relay bookkeeping is unchanged; only the redundant
parse is skipped. Production traffic measured 14-57% duplicate frames.
DedupDecodeBenchmark (60k frames, 67% dups): 10.0us/frame full parse vs
1.5us/frame cached — 6.5x, with an in-benchmark assertion so the gain
is enforced. 7 scan-safety unit tests in CachingEventDecoderTest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Restructures the sync around the measured bottlenecks: one download
worker pool now spans the whole sync (windows no longer join before the
next reconcile starts), overflow-split windows are reconciled by a
caller-set number of concurrent NEG sessions (reconcileConcurrency,
default 1) from a shared work queue, and the reconcile-to-download
buffer depth is exposed (idBufferBatches). No NIP-11 auto-detection:
peak subscription usage (maxConcurrentReqs + reconcileConcurrency + 1)
is documented and budgeting it against the relay's max_subscriptions is
the caller's call.
All 44 negentropy tests pass unchanged. Production shootout on a 2.6M
corpus, single connection: 3.6k events/s with old-equivalent params,
4.5k/s tuned (12 reqs + 4 reconcilers) — ~82% of the measured ~5.5k/s
per-connection by-id ceiling, vs 1.6k/s for the old implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The relay was backfilled from 33k to 2.6M kind-30382 events between
runs, and the picture changed: on the big corpus one connection
saturates at ~8 in-flight REQs (~5.5k events/s) and the relay tops out
at ~15k events/s around 40 total in-flight — past that, added
concurrency only inflates per-REQ latency. Axis 1 now stops at the
relay's NIP-11 max_subscriptions (20; exceeding it wedged the
connection in the first extended run), cells carry a hard 120s budget
with one retry per batch, and per-REQ latency is computed over
completed batches so partial cells report honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Downloads the full 33k kind-30382 corpus from nip85.nosfabrica.com per
cell over a (connections x concurrent REQs) matrix with pre-connected
sockets. One connection scales near-linearly to 16 in-flight REQs
(1,970 -> 30,735 events/s) with no wall at 4, and 1x16 matches 4x4 —
total in-flight REQs is the real variable. Every slow path measured so
far (serial page cursors ~2-3.7k/s, negentropySync 1.6k/s) is a
pipelining/serialization problem: negentropySync starves its download
workers on reconcile cadence and recurses overflow windows
sequentially. Findings and implied negentropySync improvements in the
plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
A raw no-parse socket and the full quartz stack page the same
production query on one connection at the same ~3.4-3.8k events/s,
proving the single-connection ceiling reported by a user is the
relay's per-connection response cadence rather than the client's
serial parse (which is ~1% busy at that rate). Findings and the
assessment of the proposed parallel-parse + ordered-dispatch pipeline
are in the plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Measures how to download millions of events from a single relay as fast
as possible, download+parse only: local geode ceilings (giant REQ vs
until-cursor paging vs created_at-sharded connections, quartz stack vs
raw frames), offline per-frame strategies (full parse, parallel parse,
id-scan for raw archiving), and a production case syncing kind 30382
from nip85.nosfabrica.com via NIP-77 negentropy against plain paging.
Headline results in the plan doc: paging beats giant REQs 26x (which
also dropped frames), created_at sharding stacks ~2x on top, parse is
2-5% of the budget, and negentropy is 2.4x slower than paging for a
cold download (its win is incremental re-sync).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Offline microbenchmark of NostrClient's dispatch stage — PoolRequests
state machine, listener fan-out, id dedup and handoff to a verify
stage — under 1 and 4 concurrent relay feeders. Key results: ~100ns per
message uncontended, but negative scaling under concurrency (the
PoolRequests busy-wait spin lock makes 4 feeders slower in aggregate
than 1), per-event channel handoff costs ~180ns vs a free 64-batch,
and early dedup before the locked path gives 2.7-4x aggregate
throughput at production duplicate factors. Findings appended to the
receiver-perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Gated JVM test (-PprodRelayBench=1) that connects to live relays with
realistic filters and measures per-relay queue delay, processing time,
consumer busy fraction, EOSE latency and duplicate rates, comparing the
current inline verification against a parallel verify stage, plus
offline single-thread parse/verify ceilings on captured frames.
Findings from a first run are written up in
quartz/plans/2026-07-02-nostrclient-receiver-perf.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
A full FTS rebuild can run for a long time on a big store, so add a
resumable, batched overload alongside the one-shot:
reindexFullTextSearch(resumeFrom: String?, batchSize): FtsReindexProgress
Each call processes ~batchSize events in its own write transaction and
returns an opaque cursor + done flag. The caller loops until done and may
stop at any point — the cursor is durable across crash/app-restart, and
the writer lock is released between batches, so "pause" is just "don't
make the next call". The path is additive/refresh and keeps search usable
throughout (no up-front wipe); the one-shot variant remains for a
guaranteed-clean rebuild.
- SQLite: FullTextSearchModule.reindexBatch walks event_headers ordered
by the monotonic row_id (a free, stable cursor), restricted to
searchable kinds, delete-then-insert per event so batches are
idempotent and never duplicate rows.
- Filesystem: FsEventStore walks one idx/kind/<k>/ dir per step (linear,
no re-sort); cursor is the next searchable kind. Idempotent linkFts, so
nothing is wiped. Pauses between kinds.
- Wrappers delegate; new FtsReindexProgress value type carries cursor +
progress + done.
- cli: `amy store reindex-fts` now loops the batched path to completion
and reports processed/batch counts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZqPFds2TPPUKkMmBngwys
The set of event kinds that implement SearchableEvent — and the text
each contributes via indexableContent() — is baked into the quartz
build, so it changes across app versions. Events stored under older code
keep their old (or missing) NIP-50 full-text-search rows, so search
silently misses them after an upgrade.
Add IEventStore.reindexFullTextSearch() so the app can wipe and rebuild
the FTS index from already-stored events when it has spare cycles.
Speed: only kinds that currently map to a SearchableEvent are scanned.
Kind alone selects the event class in EventFactory, so a single probe per
distinct kind is authoritative, letting us push a `kind IN (...)` filter
(SQLite) / skip whole idx/kind dirs (filesystem) so the non-searchable
bulk — reactions, zaps, follow lists — is never deserialised.
- SQLite: FullTextSearchModule.reindexAll drops+recreates the virtual
table (O(1) wipe) then streams only searchable-kind rows in one write
transaction, reusing a single INSERT statement.
- Filesystem: rebuilds only idx/fts/, driving the walk from
idx/kind/<searchable kind>/ via the new FsIndexer.linkFts.
- Wrappers (EventStore, ObservableEventStore, InterningEventStore)
delegate; the observable layer emits nothing since no event changes.
- cli: `amy store reindex-fts`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BZqPFds2TPPUKkMmBngwys
The "title: ", "summary: ", "name: ", "Subject: ", "Option: " etc. prefixes
were tokenized into the single FTS content column as literal words, so every
event of a type matched bare terms like "title" or "summary" and the index
carried useless tokens — the same field-name pollution we avoid for JSON
kinds. The FTS table has one content column and search is plain-text MATCH,
so the labels enabled no fielded search; they were pure noise.
Index bare field values instead (listOfNotNull(...).joinToString("\n")),
which also drops the "null" token the older one-liners produced for absent
fields. Applied across all SearchableEvent implementations for consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RFdWREvyvixRXnmNXNzmmN
Kinds 0 (profile), 40/41 (channel create/metadata) and 31990 (app handler)
store their data as JSON in content. Implement SearchableEvent on them by
parsing the JSON (via the existing UserMetadata/ChannelData/AppMetadata
accessors) and indexing only the meaningful fields — names, bio/about, and
the addresses people search by: nip05 email, lightning addresses
(lud06/lud16), and website/picture/banner URLs. This avoids indexing the
JSON keys and structural punctuation that raw-content indexing would add.
Updates the FsSearchTest "non-searchable" case to use an unknown kind, since
MetadataEvent is now searchable, and adds SearchTest coverage for profile
and channel JSON fields (name, about, email, lightning, URL).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RFdWREvyvixRXnmNXNzmmN
meltToLightning selects proofs covering amount+fee_reserve and, on
overshoot, swaps them down to exactly that before melting. But the melt
mints its inputs on the active keyset and the mint then charges its own
NUT-02 input fee on them — which the swap-down target didn't include. On a
fee-charging mint (mint.coinos.io active keyset = 100 ppk) the melt was
left a sat short and threw "Inputs total X < required Y", so fixing the
per-keyset swap fee alone just moved the failure from the swap to the melt.
Reserve activeKeysetInputFeeFor(required) on top of amount+fee_reserve for
both proof selection and the swap-down target so the subsequent melt has
room for its input fee.
Also fix the reported fee in MeltCompleted: the pre-paid swap "keep" was
split off before the melt and never spent, so subtract it instead of
counting the whole selected total minus change (which overstated fees by
the keep amount in the swap path).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Ffjz3doZ5CtFtAWSpvmqR
When melting/swapping Cashu proofs, the wallet computed the NUT-02 input
fee from the mint's currently-active keyset (`keyset.inputFeePpk`) for
every input. But NUT-02 charges the fee per the keyset each input proof
was minted under, which can be an inactive, rotated-out keyset with a
different fee.
On mint.coinos.io the old keyset (004f7adf2a04356c) charges 0 ppk and the
active keyset (007311aa2fa58cc8) charges 100 ppk. A wallet holding proofs
on the old keyset reserved a fee the mint never takes, leaving the swap
outputs one sat short — surfacing as "Mint Error (HTTP 400), inputs 84 -
fees 0 vs output (83) are not balanced" when sending over Lightning.
Fee is now `ceil(sum(input_fee_ppk_i) / 1000)` over each input's own
keyset, read from /v1/keysets (which lists inactive keysets too, unlike
/v1/keys). Applied in swap, swapToLocked, and meltProofs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Ffjz3doZ5CtFtAWSpvmqR
When the user picks the Namecoin Core RPC backend and points at a
self-hosted node behind a self-signed cert (StartOS / Start9, umbrel,
LAN reverse proxy, …) the previous flow only worked if the cert's CA
was already in the device trust store. There was no in-app way to
inspect or pin the certificate, so users had to install the StartOS
root CA at the OS level — or settle for an unencrypted onion path.
This change brings the Namecoin Core RPC path up to parity with the
existing ElectrumX path:
- NamecoinCoreRpcClient.probe() now opens a short-lived, no-auth TLS
socket alongside the JSON-RPC call to capture the server's leaf
certificate (PEM + SHA-256 fingerprint). Capture is best-effort
and only runs for https:// URLs. Credentials are never sent over
the inspection socket.
- RpcProbeResult exposes serverCertPem, certFingerprint, and
tlsHandshakeFailed so the Settings UI can react. New fields are
nullable / default false so existing callers compile unchanged.
- NamecoinCoreRpcClient maintains its own dynamic-cert keystore and
a lazy pinned SSLSocketFactory (same shape as ElectrumXClient's,
minus the hardcoded list — Core RPC has no public defaults). When
cfg.usePinnedTrustStore is true and the URL is https, callRpc()
routes through the pinned factory with a permissive hostname
verifier (LAN/onion certs commonly carry IP-only SANs).
- NamecoinSettingsSection's Namecoin Core RPC card now shows a
'Trust Server Certificate?' AlertDialog after Test RPC when the
probe captured a cert and the user hasn't pinned yet, reusing the
existing namecoin_pin_cert_* strings. Accept persists the PEM
AND flips usePinnedTrustStore=true on the config. The result
card also displays the captured fingerprint and a '(pinned)'
marker so the user can see the current trust state at a glance.
- The pinned PEM list is stored in the existing
KEY_PINNED_CERTS DataStore entry, so a single TOFU confirmation
covers both backends. AppModules' namecoinCoreRpcClient init now
bootstraps the pinned list on app start, matching ElectrumX.
- Tests cover the new probe fields' defaults and the addPinnedCert
/ setDynamicCerts surface.
Local verification: builds clean (assembleFdroidDebug), :quartz:jvmTest
NamecoinCoreRpcClientTest all green, :amethyst:testFdroidDebugUnitTest
namecoin suites all green.
Newer mints charge a per-input fee on swap and melt. Without reserving
it from the output total, the mint rejects every swap/melt with
"amount mismatch" the moment it has any fee configured. We were
reading input_fee_ppk into the KeysetSummaryDto but never threading
it through the actual ops math — fee-charging mints simply didn't
work for us.
Per NUT-02 the fee is `ceil(numInputs * input_fee_ppk / 1000)`. The
ceiling is load-bearing: floor undercharges by one sat in the common
case (numInputs * ppk not exactly divisible by 1000), which is also
exactly what mints reject. New `computeInputFee` helper does the
ceiling-division in pure Long math — `(n * ppk + 999) / 1000` — with
defensive zeroing for null / zero / negative ppk.
Applied in three paths:
- swap(): output total = inputs - fee. The change bucket shrinks by
fee; the send bucket (when split) stays whole.
- swapToLocked() (nutzap send): change shrinks by fee, recipient
still gets exactly targetSplit sats locked.
- meltProofs(): required inputs grow by fee (separate from
quote.feeReserve, which bounds LN routing fees, not the mint's
processing fee). Change-output upper bound shrinks accordingly.
Also exposes input_fee_ppk on the full KeysetDto (was only on the
summary) so the fee-aware paths can read it from the same /v1/keys
call we already make.
Tests: 10 cases on the ceiling-division helper covering null/zero
ppk, exact-divide boundaries (999 / 1000 / 1001 inputs at 1 ppk),
typical and large fees, and defensive negative-ppk handling.
https://claude.ai/code/session_01MdWddiar819f8XYt5N8BjP
When a Namecoin record's value isn't valid JSON (a real failure mode
when an operator hand-builds the value and miscounts braces), the
NIP-05 path used to silently swallow the parser exception and surface
a misleading "no nostr field" message. That sends the publisher
chasing a phantom missing field when the actual problem is the value
itself.
Concrete case that triggered this: a `name_update` published a
474-byte d/testls value with one closing brace short of balanced. The
string parses up to the missing brace, after which kotlinx.serialization
throws "Unfinished JSON term at EOF at line 1, column 474". That error
was previously dropped, leaving the operator to debug "no nostr field"
without ever seeing the underlying JSON parse failure.
Changes:
- New NamecoinResolveOutcome.MalformedRecord(name, error). Distinct
from NoNostrField. The `error` field is the parser's own diagnostic
(e.g. "Unfinished JSON term at EOF at line 1, column 474") so the
publisher can locate the broken byte without spelunking.
- NamecoinNameResolver.performLookupDetailed: parse via a new
parseValueOrError helper and surface MalformedRecord instead of
collapsing into NoNostrField. Also rejects non-object top-level
values (arrays, primitives, null) with a useful diagnostic
("top-level value is JsonArray, expected JSON object").
- DesktopSearchScreen handles the new outcome by surfacing the parser
error verbatim in the Namecoin status banner, so the column number
reaches the publisher's screen.
Tests (commonTest / NamecoinImportTest):
- "NIP-05 lookup surfaces MalformedRecord with parser detail when
value is broken JSON": a deliberately one-brace-short value yields
MalformedRecord with a non-empty diagnostic.
- "NIP-05 lookup surfaces MalformedRecord when top-level value is a
JSON array": ensures non-object top-level values are rejected with
a useful "expected JSON object" message rather than silently
parsing as something unusable.
Tests don't pin the exact parser wording (kotlinx.serialization can
change it across versions); they only pin that the message is
attributed to JSON parsing rather than to a missing field.
Adds a new `NamecoinImportResolver` that follows the `import` field of a
Namecoin Domain Name Object value, recursively merging the imported
values into the importing object before `NamecoinNameResolver` reads
the `nostr` field.
Why
---
The 520-byte per-name limit on Namecoin is tight when a single record
needs `ip`, NIP-05 names, TLS, and per-subdomain map entries. A common
workaround (used by `testls.bit` and others) is to keep the apex record
small by delegating shared blocks into a sibling name via
`"import":"dd/<name>"` per ifa-0001 §"import". Without import support,
NIP-05 search for `testls.bit` (and the like) fails: the resolver sees
no `nostr` field at `d/testls` and gives up before consulting the
imported `dd/testls` that actually carries the names block.
What
----
Per ifa-0001 §"import":
* Importer items take precedence over imported items (including `null`,
which suppresses the imported counterpart).
* Recursion is supported up to 4 levels (the spec minimum) by default.
Cycles are broken by a visited-set; over-budget chains are silently
truncated, with the importer's own items still taking effect.
* Multiple imports in one array merge later-wins (the array's own
order), with the importing object on top of all of them.
* Subdomain selectors (the optional 2nd element of each inner array)
are resolved through a `map` walk inside the imported value, with
the standard exact-label > `*` wildcard > `""` default rules.
* Three short-hand value forms are accepted alongside the canonical
array-of-arrays: `"import":"d/foo"`, `"import":["d/foo"]`, and
`"import":["d/foo","selector"]`. They map onto the canonical
representation losslessly.
* A failed import lookup (name not found, malformed JSON, network
error) is treated as `{}` rather than failing the whole record, so
transient ElectrumX hiccups don't kill resolution. The importing
object's own items still apply.
Wires the expander into `NamecoinNameResolver.performLookup` and
`performLookupDetailed`. Records without an `import` key skip the
resolver entirely (zero extra I/O), so non-import names pay no cost.
Tests
-----
20 new tests in `NamecoinImportTest`:
* Unit (resolver in isolation): no-import passthrough, all four
shorthand value forms, importer precedence, null suppression,
depth-4 recursion happy path, over-budget truncation, lookup
failure, malformed JSON, malformed `import` value, cycle
protection, multi-label selector descending in DNS order, wildcard
fallback.
* Integration with `NamecoinNameResolver`: bare and named NIP-05 across
an import (mirroring the real-world `testls.bit -> dd/testls`
deployment); regression guard that no-import records issue exactly
one ElectrumX query; importer-wins on `nostr.names`; lenient
failure when the imported boilerplate is unreachable;
`resolveDetailed` returns `Success` and `NoNostrField` correctly
across imports.
All 15 existing `NamecoinNameResolverTest` cases continue to pass.
Scope note
----------
This PR is intentionally scoped to the NIP-05 / search path. The
relay-resolution path (#2595) also benefits from import support; that
wiring is left to land alongside #2595 so the two PRs can move
independently.
Groups EventInterner + InternedEventStore + the platform actuals + the
test under a focused interning sub-package. Leaves nip01Core/cache/ as
the namespace for cache primitives generally; future cache types
(LargeWeakCache, etc.) get sibling sub-packages.
Files moved:
cache/EventInterner.kt → cache/interning/EventInterner.kt
cache/InternedEventStore.kt → cache/interning/InternedEventStore.kt
cache/EventInterner.jvmAndroid → cache/interning/EventInterner.jvmAndroid
cache/EventInterner.apple → cache/interning/EventInterner.apple
cache/EventInterner.linux → cache/interning/EventInterner.linux
cache/EventInternerTest.kt → cache/interning/EventInternerTest.kt
No consumers to update — the previous decorator refactor already
removed the interner from all the store ctors. 7/7 interner +
240/240 store + projection tests still pass.
https://claude.ai/code/session_01Jny85MTu1ynKgFBgysfWu5
Claude
Vitor PamplonaandGitHub
m
mstrofnone