Third refactor slice (plan: quartz/plans/2026-07-11-concord-event-classes.md).
The kind-33301 public invite bundle is now a proper addressable Event class:
- New cord05Invites/bundle/ConcordInviteBundleEvent (BaseAddressableEvent), with a
build{} template that emits the ['d',''] + ['vsk','6'] tags via the standard DTag
ext and the shared VskTag (ControlEntityKind.INVITE_LIVE). Registered kind 33301
-> ConcordInviteBundleEvent in EventFactory so fetched bundles parse as the class.
- ConcordInviteBundle.build now signs ConcordInviteBundleEvent.build(...) with the
per-link key; the raw TAG_D/TAG_VSK/VSK_LIVE constants are gone. Tag order (d, vsk)
is preserved so the bundle event id / naddr are unchanged — the cord05Invites
tests (bundle round-trip + join flow) pass.
Remaining invite kinds: 3313 direct-invite (rumor has empty tags; p/k index is on
the giftwrap) and 13303 invite-list (unimplemented stub) — tracked in the plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
First slice of the nip88Polls-structure refactor (plan in quartz/plans/
2026-07-11-concord-event-classes.md). A Concord chat rumor IS a standard Nostr
event plus a channel/epoch binding, so stop re-deriving it by kind number:
- New cord03Channels/tags/ChannelTag + EpochTag (typed, parse/assemble), plus
TagArrayBuilderExt (channel/epoch/channelBinding) and TagArrayExt
(concordChannel/concordEpoch/isConcordBoundTo) — the poll-package shape.
- ChannelChat.message/reply now build via ChatEvent.build{ channelBinding(...) }
and assemble the template into a rumor; reaction reuses ReactionEvent.KIND. The
minimal e/p/k reaction tags and q/p reply tags are kept byte-identical for Armada
interop (guarded by the Concord round-trip tests, which pass).
- Drop the ConcordKinds.MESSAGE/COMMENT/REACTION/DELETE aliases (they shadowed
ChatEvent/CommentEvent/ReactionEvent/DeletionEvent KINDs); callers use the real
event KINDs. VoicePresence uses the new tag classes instead of bindingTags.
Remaining Concord-specific kinds (control 3308, invites, guestbook, rekey, voice,
seals/wrap) get their own per-kind packages in later slices per the plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
CachingDns duplicated 20% of what the app's SurgeDns already did better
(stale-while-revalidate, single-flight, jittered 24-48h positive TTL,
persistence, poison filtering). Consolidate on SurgeDns and fix what the
verification pass found along the way:
- move SurgeDns + SurgeDnsStore (+ their 41 tests) to quartz jvmAndroid so
the CLI can share them; delete CachingDns
- negative TTL 10s -> 10min (class default): a dead domain burns 10-30s of
getaddrinfo per re-dial and both the relay pool and a crawl re-dial dead
hosts continuously; the relay pool's own backoff already reaches 5min
- stop the call-failure listeners from erasing negative entries they were
just written from: callFailed caused by UnknownHostException must not
invalidate (it deleted every negative entry milliseconds after creation,
silently defeating the negative TTL entirely). The invalidation remains
for its real purpose - stale positives with rotated IPs
- new SurgeDns.staleAll(): soft-expire everything WITHOUT discarding -
positives serve stale and revalidate in the background on next use,
negatives re-try on first touch. Wired to network-identity changes in
AppModules (same trigger as Tor's onNetworkChange), replacing nothing:
the full-clear invalidate() was never actually called on network change
- amy: SurgeDns wired into the CLI OkHttp client, snapshot persisted at
~/.amy/shared/dns-cache.bin across runs (best-effort load/save)
Verified: all SurgeDns/SurgeDnsStore tests pass in the new location incl.
two new staleAll tests; :amethyst compiles (main + unit tests); cli suite
green. Caveat recorded in the plan doc: behind an HTTP CONNECT proxy OkHttp
never consults the client resolver (hostname goes to the proxy), so this
layer pays off on direct-connect deployments only - which also means the
branch's A/B numbers never depended on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Per-relay sub cap 32 made the hop-3 crawl 52% slower with slightly fewer
lists (27 relays demoted us vs 5 at cap 16) - the Phase-B plateau is
relay-side service rate, so 16 stays the default and AMY_RELAY_SUB_CAP
remains an experiment hatch. Final validation: cold hop-5 completed in
85.4 min with 203,903 contact lists / 391,549 users / 626,599 events -
the workload that never finished before this branch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Cold hop-3 A/B (fresh store per leg, same observer): 48 workers beat 24
twice at identical contact-list counts (579s->559s and 474s->396s). The
old result that made 64 look 2x slower predated the multithreaded
dispatcher fix - more coroutines on one starved event-loop thread. Plan
doc gains the full A/B table (kept: background-park degating, sweep
dedup, tail overlap, dc48; rejected: 5s fast window).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Most of a from-scratch crawl's wall clock was connection setup, re-paid
serially: the relay pool tears down a relay's socket ~300ms after its last
drain unsubscribes, OkHttp allowed only 256 concurrent WS handshakes, and
every dial re-ran an uncached blocking getaddrinfo (10-30s per dead domain,
once per per-user path URL of the same host). Open the connections once, in
parallel, and keep them:
- GrapeRankCrawler: the warm-pool trick (never-matching REQ that only holds
the socket) now covers the whole candidate universe instead of the top 20 -
seeded at crawl start from the reachability cache's live set (one parallel
connection storm, Config.knownLiveRelays) and refreshed each round with
newly learned outbox relays, capped by Config.preconnectCap (FD-budget
aware, --preconnect-cap / --no-preconnect).
- CachingDns (quartz jvmAndroid): 10-min positive + negative DNS cache with
in-flight per-host dedup; dead domains fail in microseconds instead of
re-burning resolver timeouts, path URLs of one host resolve once.
- cli Context: dispatcher and pre-connect caps derived from the process's
open-files limit (UnixOperatingSystemMXBean), warning when ulimit is low.
- amy graperank probe: relay census - mass-connects every relay the store
knows (kind:10002 universe deduped per authority + cached verdicts) in
waves, records live/dead with real measured rtt-open into the NIP-66
reachability cache (RelayProber + RelayReachabilityStore.recordProbed),
so the next crawl skips dead relays and waits once for the slow-but-alive.
Single-server limits (FDs, ephemeral ports, DNS, threads, conntrack) and the
design are documented in quartz/plans/2026-07-10-graperank-connect-storm.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
v1.2.0 (on Maven Central) speeds up the library's own reconcile and fingerprint
walk on top of the v1.1.1 PrefixSumStorageVector wiring. At the 1M relayBench
slice shape (NegentropyReconcileBenchmark, converges exactly, need/have=200k):
client reconcile 264 → 178 ms, seal 424 → 320 ms, and the library's O(range)
fingerprint walk 447 → 252 ms (~1.8×). Our prefix-sum path still answers each
range fingerprint in 0.7 ms (356× the now-faster walk). All NIP-77 tests pass.
Update the reconcile-profiling plan: the fix shipped via the upstream
IStorage.fingerprint seam (v1.1.1) rather than a quartz-side fast server; record
the v1.0.2 → v1.1.1 → v1.2.0 progression.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The `rawQueryPathMatchesDecodedQuery` case called `store.rawQuery(filter)`, but
`EventStore` only exposes the streaming `rawQuery(filters, onEach)` — the
list-returning overload lives on the inner `SQLiteEventStore`. Point the check
at `store.store.rawQuery(filter)` so the zero-decode path is actually exercised.
Record the shipped k-way merge result in the plan doc: a fresh 1M relayBench run
has geode `follow-feed` at 18.8 ms vs strfry 17.7 ms (down from 97.7 ms, now at
parity) and 46,258 ev/s vs strfry 15,365 @8conn, both returning the same 500.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
follow-feed (kinds=[1,6] × 150 authors, ORDER BY created_at DESC LIMIT 500)
was geode's 5.5× loss (97.7ms vs strfry 17.6ms). Investigated whether any
change is worth it, across read + write + size.
Read (FollowFeedReadBenchmark, in-memory, scale 5 ≈ 1.05M events):
prolific-recent sparse-old
current 5.7 ms 1.9 ms
scan (strfry) 1.0 ms 1601.9 ms
union 316.9 ms 20.0 ms
- scan (created_at index + early LIMIT) wins for active follows but is
catastrophic for sparse/inactive follows AND grows with corpus size
(234ms→1601ms from scale 1→5) — following rarely-posting accounts is
common, so it'd be a severe regression.
- union (300 per-branch subqueries) is dominated by branch overhead.
- current is the only robust option — flat across scale, bounded by the
followed set, never catastrophic. The 97.7ms is a worst case (the 150
MOST prolific authors, disk-bound reading all their matching rows).
No safe SQL-level swap exists; each alternative trades geode's worst case
for a worse one on a common workload. The only universal improvement is
strfry's app-level k-way merge (O(LIMIT+streams)) — a real new executor,
not a SQL tweak.
Write & size: neutral for every candidate — all reuse existing indexes
(query_by_kind_pubkey_created / query_by_created_at_id), none adds a
CREATE INDEX, so ingest throughput and storage are untouched regardless of
choice. A new index was considered and rejected (taxes every write, helps
one shape, reverts under ANALYZE).
Decision: keep the current composite plan. Full write-up in
quartz/plans/2026-07-04-follow-feed-read-tradeoff.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The server's per-round reconcile spends a large slice turning the ~1MB hex
reconcile frame into wire JSON: the generic (Jackson) serializer wraps the
hex string in a value node and scans every char for JSON escapes a
[0-9a-f] payload can never contain, then re-copies.
NegMsgMessage.toJson() now builds ["NEG-MSG","<sub>","<hex>"] directly —
no node tree, no escape scan of the hex. Fast path fires only for
escape-free printable-ASCII subIds (what the JSON encoder emits verbatim);
exotic subIds fall back to the generic serializer, so output is
byte-identical. RelaySession.send routes through message.toJson() (default
unchanged for every other message type).
Measured (toJson + UTF-8, per frame): 64KiB 2.5×, 250KiB 2.6×, 500KiB
(strfry cap) 2.8× — ~2.5ms saved per NEG-MSG, ~35ms over a 14-round
reconcile. Correctness: a subId battery asserts byte-identity with the
generic path, and GeodeVsStrfryNegentropySyncTest (real strfry) reconciles
against the fast-built frames.
Also records the ingest-latency candidate as measured-not-worth-it: the
IngestQueue pipeline overhead is only ~0.17ms p50, <10% of the ~2.4ms
receipt→queryable gap — that gap lives in the REQ-visibility path, not the
writer. Full write-up in
quartz/plans/2026-07-04-sync-serialization-and-ingest-latency.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The profiles scenario — Filter(kinds=[0], authors=[50]), no limit — was
geode's ~100x loss to strfry on the 1M corpus (99.5ms vs 0.94ms).
Root cause: the REQ path always appends ORDER BY created_at DESC. For a
multi-author pubkey IN(...) filter, query_by_kind_created (kind,
created_at) satisfies that order for free by scanning an ENTIRE kind, so
SQLite prefers it over the selective query_by_kind_pubkey_created (which
would need a sort). The scan is O(all kind-0 profiles) — cheap at 2k, the
99.5ms at 1M. ANALYZE does not fix it (verified: even a reopened store
reading fresh sqlite_stat1 keeps the scan, since the ORDER BY genuinely
lets the scan skip a sort). A single author is costed right and already
seeks; only the IN-list of >1 is mis-costed.
Fix: pin INDEXED BY query_by_kind_pubkey_created for exactly that shape —
multi-author + kinds, no ids, no d-tags, no limit — keeping the ORDER BY.
SQLite seeks the authors and sorts the small result: identical rows,
identical newest-first order (zero behavior change), ~8x at 2k profiles,
growing to ~100x at 1M. Limited feeds (home/global) keep the created_at
scan + early LIMIT; single-author and d-tag queries are untouched. The
index is created unconditionally so the hint never dangles.
(Considered dropping the ORDER BY for no-limit author queries — faster and
hint-free, but it changes on-the-wire result ordering, which broke
FsParityTest's ordered-parity assertions, so it's client-visible. Rejected
in favor of the order-preserving pin.)
Adds ProfilesQueryPlanBenchmark (regression guard: asserts the live REQ
plan seeks the composite index, not the kind scan) and
plans/2026-07-04-profiles-query-plan.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
profiles (Filter(kinds=[0], authors=[50]), no limit) was the one query
geode lost to strfry on the 1M corpus — 99.5ms vs 0.94ms (~100x).
Root cause: the REQ path appends ORDER BY created_at DESC even with no
limit. query_by_kind_created (kind, created_at) satisfies that order for
free while scanning EVERY kind-0 profile, so SQLite prefers it over the
ideal query_by_kind_pubkey_created (which would need a sort). The scan is
O(all profiles) — cheap at 2k, the 99.5ms at 1M.
ANALYZE does not help: verified that even a reopened store reading fresh
sqlite_stat1 keeps the scan, because the ORDER BY genuinely lets the scan
avoid a sort.
Two fixes measured (both return identical rows), scoped to no-limit
kinds+authors filters:
- Fix A: drop ORDER BY when limit==null -> planner picks the composite
index itself (~7x at 2k profiles, ~100x at 1M). Changes result order
across authors (a NIP-01 SHOULD; clients re-sort).
- Fix B: force INDEXED BY query_by_kind_pubkey_created + keep ORDER BY
(~same speed, newest-first preserved, at the cost of a scoped hint).
Adds ProfilesQueryPlanBenchmark (prints plans+timings, asserts row-count
equivalence) and plans/2026-07-04-profiles-query-plan.md. No production
change yet — the fix is a core QueryBuilder behavior/ordering decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The 1M relayBench run had geode losing the negentropy phase to strfry
(initial reconcile 6066ms/27r vs 1270ms/14r; identical-set 1947ms vs
557ms). Three layered benchmarks pin where the time actually goes:
- NegentropyReconcileBenchmark (quartz): the kmp-negentropy server loop
in isolation is ~200ms for the full 14-round exchange — the reconcile
ALGORITHM is not the bottleneck. (An early version showed 22s/139r;
that was a benchmark bug — index slices over randomly-sorted ids
scatter the diff. Real relayBench slices are contiguous time ranges;
monotonic created_at fixes it and matches strfry's round count.)
- NegentropyServerReconcileBenchmark (geode): the real in-process geode
server over loopback is 3214ms — 15x the library loop. JFR of the
server call-trees: ~40% hex/UTF-8/JSON serialization of the payloads,
~26% actual reconcile, rest allocation. The gap is the JVM
constant-factor tax on hex-in-JSON, which strfry pays in C++, not a
single hotspot.
- NegentropyPrefixFingerprintTest (quartz): the one algorithmic lever.
Negentropy's fingerprint is an additive sum mod 2^256, so a prefix-sum
table answers any range in O(1). Proven bit-for-bit identical to the
library over 2000 random ranges, and 460x faster per call — the fix
for the ~26% reconcile slice (dominant in the identical-set case).
Not yet wired: the library instantiates FingerprintCalculator
internally, so shipping prefix-sum needs a kmp-negentropy change (or a
quartz-side fast server). Full write-up + artifacts in
quartz/plans/2026-07-04-negentropy-reconcile-profiling.md.
Benchmarks are CI-safe (small defaults / opt-in gates); JFR via
-PnegProfile, scale via -DnegBenchN, geode server bench via
-DnegServerBench=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Audit follow-ups on the live NIP-77 index, all with the store/scan
equivalence test extended to cover them:
- Same-batch replaceable displacement left a dead id in the index.
applyAfterCommit applied all removes before all adds, so when a later
row in one transaction displaced an earlier row of the same batch (two
versions of one replaceable — the mirror-backfill hot path), the
displaced row's remove no-op'd against an index that hadn't taken its
add yet, then the add re-inserted it: the index advertised an id the
trigger had already deleted. recordAccepted now cancels the pending add
instead of queueing a remove (added is a LinkedHashSet for O(1)
cancel).
- A kind-5 that deleted nothing (a delete broadcast for events this relay
never stored — the common case) still invalidated the whole index,
forcing a full-scan rebuild under the writer mutex on the next
NEG-OPEN. DeletionRequestModule.insert now returns the rows it deleted;
recordAccepted only invalidates when that count is > 0, else records the
kind-5 as a plain row.
- The first NEG-OPEN over a corpus larger than the serve cap scanned the
whole table uncapped, built a full index that could never produce a
snapshot, and then maintained it forever for zero benefit.
liveNegentropySnapshot now caps the rebuild scan at maxEntries + 1 and
leaves the index unpopulated when the corpus is over-cap (the scan path
answers NEG-ERR, as before).
- delete/deleteExpired/clearDB invalidate() moved inside the writer mutex
so no NEG-OPEN can seal a snapshot of just-deleted rows, and no
concurrent rebuild can be discarded by a late invalidate.
Also corrects the ~40 B/event heap figure to ~140 B (IdAndTime keeps the
id as a 64-char hex string, not 32 bytes) in the strategy/plan docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Found while attributing the small-REQ wire floor (backlog item 6,
latency half): geode's new WireReqFloorBenchmark measured a flat
43.7 ms per REQ round trip that survived every server-side change —
store configs, dispatchers, the pump — and then vanished when the
round's preceding CLOSE was dropped. Root cause is client-side: OkHttp
does not set TCP_NODELAY, relays never answer a CLOSE (NIP-01), so its
bytes sit unACKed for the peer's ~40 ms delayed-ACK window and Nagle
holds the next REQ behind them. CLOSE-then-REQ is a Nostr client's
hottest pattern — every feed/filter switch.
relayBench's harness client already shipped a no-delay socket factory
(which is why benchmark numbers never showed the stall) but the
production clients did not. New TcpNoDelaySocketFactory (quartz
jvmAndroid, next to BasicOkHttpWebSocket) is now used by the Android
relay pool factory, the Desktop relay client, amy's relay connections,
and geode's mirror worker. Direct connections only — SOCKS/Tor paths
are untouched.
With the factory, the benchmark puts geode's ~21-row REQ at ~1.25 ms
on the wire (matching relayBench): ~0.6 ms Ktor CIO+OkHttp loopback
floor, ~0.5 ms per-REQ server work (already investigated). Per-frame
burst cost measured negligible and the pump adds ~nothing, so the
send-path latency angle of backlog item 6 is closed as not-a-problem;
its ingest-CPU share remains a separate throughput question. Findings
recorded in quartz/plans/2026-07-04-small-req-floor.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Reverts the queryRawInline fast path (fb29d655, 1b786f31) per the
keep-only-winners rule. Three relayBench runs at 50k (baseline, cap-256
where the path never engaged, cap-512 where author-archive/by-ids/
500-limit feeds genuinely took it) showed no movement outside the
container drift band — strfry's own numbers drifted ±30% between runs
and inline-eligible scenarios moved the same as ineligible ones.
The in-process win was real but small (~17%, 0.60 -> 0.50 ms per
~21-row REQ); the wire-level p50 is 1.2-1.7 ms, so the missing ~1 ms
per REQ sits in the transport (Ktor frame send path + client round
trip) — backlog item 6 territory, not dispatch. Findings, numbers, and
the do-not-retry note live in quartz/plans/2026-07-04-small-req-floor.md;
SmallReqFloorBenchmark stays as the measurement tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Same container, 50k corpus, geode --no-search, strfry built from
source: identical-set reconcile 41.4 ms (geode) vs 30.1 ms (strfry) —
down to ~1.4x from the campaign-opening full-scan-per-open; cold
reconcile 177 vs 112 ms (geode's first open pays the lazy rebuild);
ingest at parity in this container; storage 72.8 vs 106 MiB. Notes the
zero-copy IStorage follow-up that would close the remaining ~11 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Two relayBench runs (50k corpus, both geode variants side by side, then
order-reversed): identical-set reconcile 56/57 ms with the index vs
110/130 ms without in both orders (~2.2x, the post-write NEG-OPEN the
old cache always missed); ingest and first-ever reconcile deltas flip
with relay order, i.e. run-order noise — no regression. Also records
the run-order-bias protocol note for future A/Bs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Design for backlog item 3 of the relay performance campaign: an
always-current (created_at, id) index maintained from the store's write
path so cold NEG-OPENs stop paying the full scan + O(n log n) seal
(~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers
the snapshot/COW model, the removal-correctness split (RETURNING deltas
for replaceable overwrites, wholesale invalidation for rare delete
paths), IndexingStrategy gating so app-side stores are untouched, and
the micro + relayBench A/B measurement plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Replaces the hand-rolled raw-WebSocket NIP-77 negotiate loop (single
un-windowed session — a strfry max_sync_events overflow was a hard
error) with quartz's negentropyReconcile: created_at window splitting
on overflow, keep-alive connection pinning, and streaming id batches.
Downloads and uploads now pipeline with the remaining reconcile
rounds: need-id batches feed 4 concurrent by-id drains, have-ids feed
an uploader (peak 7 subscriptions, under the common relay cap of 20).
Every downloaded event still funnels through the verify-and-store
path. Output field 'rounds' (protocol round-trips) is now 'windows'
(created_at splits).
Verified end-to-end against embedded geode relays: down-only 25/25,
up-only 5/5, and bidirectional re-runs converge to a zero diff.
Also records both adoptions in the perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Same-JVM interleaved comparison against the resurrected spin-lock
implementation: single-threaded parity (the uncontended lock was ~free),
but 8 threads sharing one decoder run 3.0-4.5x faster lock-free — the
spin lock serialized the concurrent hit path just like the old global
PoolRequests lock did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Replaces the decoder's spin lock with lock-free concurrent maps so the
hot per-frame duplicate check never serializes across the pool's relay
consumer coroutines. New minimal ConcurrentHashCache expect/actual
(get/put/size/clear) following LargeCache's per-platform choices:
ConcurrentHashMap on JVM/Android, CacheMap on Apple, copy-on-write on
the CI-only Linux target. Counters become atomics; the generational
rotation keeps its deliberately-tolerated benign races, now documented
per failure mode (each is at worst a redundant re-parse, never a wrong
message).
CachingEventDecoderConcurrencyTest hammers one shared decoder from 8
threads with 80k duplicate-heavy frames and capacity 256 (rotations
fire constantly): zero wrong messages, exact parsed+reused accounting.
The 7 scan-safety tests and DedupDecodeBenchmark's enforced speedup
pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Code review confirmed each relay connection owns its own consumer
coroutine on Dispatchers.IO with no downstream funnel or shared lock
before justVerify (LargeCache is a ConcurrentSkipListMap; the
PoolRequests lock is per-subscription now), so multi-relay bursts
already verify in parallel across cores. Corrects the earlier follow-up
suggesting a CacheClientConnector integration: the accessory's scope is
single-connection bulk streams, plus a possible future dispatcher-
hygiene fix if on-device profiling shows CPU-bound verifies
oversubscribing the 64-thread IO pool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Deliberate design decision reversing the 4096-frame receive bound:
1. The remote infrastructure isn't ours — TCP backpressure parks the
backlog in the RELAY's outbound buffers. A client should release the
relay from its duties as fast as it can send and own the buffering
itself.
2. The app holds 2000+ simultaneous relay connections; a bounded buffer
under a slow consumer blocks OkHttp reader threads, and at that
connection count blocked readers are a thread-starvation hazard far
worse than the heap growth they prevent.
The UNLIMITED channels now carry an explicit do-not-bound comment with
this rationale, and the slow-consumer risk is addressed from the other
side: keep the consumer faster than any relay's send rate
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).
BoundedReceiveBufferTest removed with the bound it tested; plan doc
records the decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Two server bugs masking each other made a single giant REQ crawl and
then wedge:
LiveEventStore's historical-replay dedup used an immutable Set under an
AtomicReference with copy-on-add — set + id copies the whole set per
streamed event, so large replays were accidentally O(n²) (100k-event
REQ: ~700 events/s, degrading as the response grew). Replaced with a
spin-lock-guarded mutable HashSet (same threads, single contains/add
per critical section).
Fixing that unmasked WebSocketSessionPump's slow-client policy: a fast
replay instantly overflowed the 8192-frame cap — which conflated 'slow
client' with 'replay outruns the socket writer', normal for bulk — and
the 'drop' only closed the internal queue, leaving the socket half-dead
(no EOSE, no close frame, tail silently missing: the likely cause of
the benchmark's 99,998/100,000). Producers are now paced against a full
backlog (bounded blocking wait, consistent with the documented ingest
fanout behavior) and only a client still behind after 30s is dropped,
by actually cancelling the socket.
GiantReqStreamTest guards the regression: 20k-event REQ pre-fix 8.4s
(~2.4k events/s), post-fix 0.7s (~27k events/s), all events + EOSE
delivered. 96 geode tests and the quartz relay/server suites pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The per-connection-wall test prints the negotiated
Sec-WebSocket-Extensions header; against strfry OkHttp negotiates
permessage-deflate (client_no_context_takeover) out of the box, closing
the 'is compression actually on?' question from the optimization list —
it is, no change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
One reconcile feeds by-id download batches to N clients (one socket
each) x reqsPerClient workers; reconcile windows also round-robin
across the connections, since a single connection produced need-ids at
only ~9k/s on a 2.6M corpus and starved the downloads. Events funnel
through a bounded channel to a single consumer (exact maxEvents,
single-threaded onEvent); all stages backpressure; localEntries diffing
and have-counting match negentropyReconcile. reconcileWindows became
multi-client internally; single-client paths pass listOf(this).
Production shootout (same-run pairs, 100k cap): +18% to +64% over the
tuned single client, capped by the relay's server-side reconcile id
production rather than download parallelism (which the by-id matrix
shows scales 2.7x with connections). 4 in-process multi-client tests;
18 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Client-side mirror of IngestQueue.parallelVerify: submit() is a cheap
bounded-channel send from the relay consumer coroutine; a drain loop
batches greedily (up to 256) and fans each batch across
Dispatchers.Default in core-sized chunks, dispatching callbacks in
submission order. preVerified short-circuits already-trusted ids; the
bounded channel backpressures the socket instead of growing heap.
Batch/chunk sizing is measurement-driven: per-event async cost ~40us of
scheduling each (swallowing the gain), and the per-batch join barrier
at 64 still cost half (64 -> 1.2x, 256 -> 2.2x, 1024 -> 3.2x on 4
cores). ParallelVerifyBenchmark (fresh signed events per pass — Event
caches derived state after first verify, so passes must not share
instances) measures 1.8x vs sequential with an enforced >=1.5x
assertion. 4 correctness tests cover valid/tampered routing, ordering,
preVerified and callback-crash resilience.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The reader-thread-to-consumer channel in BasicOkHttpWebSocket and the
app's OkHttpWebSocket was UNLIMITED: a consumer slower than the socket
accumulated frame Strings without bound (gigabytes over a multi-million
event download). Now capped at 4096 frames — when full, OkHttp's reader
thread blocks and TCP flow control pushes back on the relay instead of
the heap. The trade-off (a blocked reader delays PING/PONG handling) is
documented on the constant; the bound is deep enough that only a
pathologically slow consumer hits it.
BoundedReceiveBufferTest forces sustained backpressure (8-frame buffer,
sleeping consumer, real socket to a local geode relay) and asserts all
events plus EOSE arrive in order with no drops or deadlock.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
BasicRelayClient's decode step becomes a pluggable MessageDecoder
(default unchanged). The opt-in CachingEventDecoder scans EVENT frames
for their id (~0.3us, JSON-escape-safe so embedded event JSON in repost
content cannot confuse it; any irregularity falls back to full parse)
and on a cache hit synthesizes the EventMessage from the already-parsed
Event with the frame's own subId — every subscription still gets its
delivery and per-relay bookkeeping is unchanged; only the redundant
parse is skipped. Production traffic measured 14-57% duplicate frames.
DedupDecodeBenchmark (60k frames, 67% dups): 10.0us/frame full parse vs
1.5us/frame cached — 6.5x, with an in-benchmark assertion so the gain
is enforced. 7 scan-safety unit tests in CachingEventDecoderTest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The single global spin lock serialized every EVENT frame from every
relay and measured negative scaling (4 concurrent relay consumers
pushed 3.6M deliveries/s aggregate vs 11.1M for one thread alone). The
lock now lives in RequestSubscriptionState — one per subscription —
since all compound mutations are per-subId and different subs share no
wire state. decideCommandLocked takes the state instance to avoid
re-entering the non-reentrant lock; all-subs iterations lock one sub at
a time; withLock is inline to keep the hot path allocation-free.
DispatchStageBenchmark (PoolRequests-only, 1 -> 4 feeders): scaling
flips from 0.33x to 3.4-7.3x across runs. PoolRequests concurrency,
NostrClient and negentropy suites pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Splits the reconcile out of negentropySync so callers decide how to
load: negentropyReconcile streams needIds (relay has, local lacks —
download) and haveIds (local has, relay lacks — publish) in batchSize
chunks with back-pressure, taking local state as List<IdAndTime> and
slicing it per created_at window on overflow splits; the accumulating
negentropyReconcileIds convenience returns both lists. negentropySync
now delegates to the same window engine.
NegentropySession's primary constructor takes List<IdAndTime> (JVM
erasure forbids a List<Event> overload); the event-list form moved to
NegentropySession.fromEvents, mirroring NegentropyServerSession, with
all call sites migrated.
Adds NostrClientNegentropyReconcileTest (empty local set, partial
overlap both directions, identical sets, batch streaming, since/until
window slicing) — 49 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Restructures the sync around the measured bottlenecks: one download
worker pool now spans the whole sync (windows no longer join before the
next reconcile starts), overflow-split windows are reconciled by a
caller-set number of concurrent NEG sessions (reconcileConcurrency,
default 1) from a shared work queue, and the reconcile-to-download
buffer depth is exposed (idBufferBatches). No NIP-11 auto-detection:
peak subscription usage (maxConcurrentReqs + reconcileConcurrency + 1)
is documented and budgeting it against the relay's max_subscriptions is
the caller's call.
All 44 negentropy tests pass unchanged. Production shootout on a 2.6M
corpus, single connection: 3.6k events/s with old-equivalent params,
4.5k/s tuned (12 reqs + 4 reconcilers) — ~82% of the measured ~5.5k/s
per-connection by-id ceiling, vs 1.6k/s for the old implementation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The relay was backfilled from 33k to 2.6M kind-30382 events between
runs, and the picture changed: on the big corpus one connection
saturates at ~8 in-flight REQs (~5.5k events/s) and the relay tops out
at ~15k events/s around 40 total in-flight — past that, added
concurrency only inflates per-REQ latency. Axis 1 now stops at the
relay's NIP-11 max_subscriptions (20; exceeding it wedged the
connection in the first extended run), cells carry a hard 120s budget
with one retry per batch, and per-REQ latency is computed over
completed batches so partial cells report honestly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Downloads the full 33k kind-30382 corpus from nip85.nosfabrica.com per
cell over a (connections x concurrent REQs) matrix with pre-connected
sockets. One connection scales near-linearly to 16 in-flight REQs
(1,970 -> 30,735 events/s) with no wall at 4, and 1x16 matches 4x4 —
total in-flight REQs is the real variable. Every slow path measured so
far (serial page cursors ~2-3.7k/s, negentropySync 1.6k/s) is a
pipelining/serialization problem: negentropySync starves its download
workers on reconcile cadence and recurses overflow windows
sequentially. Findings and implied negentropySync improvements in the
plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
A raw no-parse socket and the full quartz stack page the same
production query on one connection at the same ~3.4-3.8k events/s,
proving the single-connection ceiling reported by a user is the
relay's per-connection response cadence rather than the client's
serial parse (which is ~1% busy at that rate). Findings and the
assessment of the proposed parallel-parse + ordered-dispatch pipeline
are in the plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Measures how to download millions of events from a single relay as fast
as possible, download+parse only: local geode ceilings (giant REQ vs
until-cursor paging vs created_at-sharded connections, quartz stack vs
raw frames), offline per-frame strategies (full parse, parallel parse,
id-scan for raw archiving), and a production case syncing kind 30382
from nip85.nosfabrica.com via NIP-77 negentropy against plain paging.
Headline results in the plan doc: paging beats giant REQs 26x (which
also dropped frames), created_at sharding stacks ~2x on top, parse is
2-5% of the budget, and negentropy is 2.4x slower than paging for a
cold download (its win is incremental re-sync).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Offline microbenchmark of NostrClient's dispatch stage — PoolRequests
state machine, listener fan-out, id dedup and handoff to a verify
stage — under 1 and 4 concurrent relay feeders. Key results: ~100ns per
message uncontended, but negative scaling under concurrency (the
PoolRequests busy-wait spin lock makes 4 feeders slower in aggregate
than 1), per-event channel handoff costs ~180ns vs a free 64-batch,
and early dedup before the locked path gives 2.7-4x aggregate
throughput at production duplicate factors. Findings appended to the
receiver-perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Gated JVM test (-PprodRelayBench=1) that connects to live relays with
realistic filters and measures per-relay queue delay, processing time,
consumer busy fraction, EOSE latency and duplicate rates, comparing the
current inline verification against a parallel verify stage, plus
offline single-thread parse/verify ceilings on captured frames.
Findings from a first run are written up in
quartz/plans/2026-07-02-nostrclient-receiver-perf.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Audited all 143 plan files across the 10 plans/ folders. Each plan now
carries a Status header (shipped | in-progress | queued | abandoned)
backed by codebase evidence, and every folder has a README.md index
grouping plans by status.
Shipped plans were moved into a per-folder plans/archive/ (via git mv,
history preserved) so each plans/ folder surfaces only live work:
shipped (archived): 122 in-progress: 8 queued: 7 abandoned: 4
docs/plans/ is the frozen legacy folder; its plans were stamped and
indexed in place (48 of 52 archived) but it remains closed to new plans.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016hpUivtmq4pgzqRbY6MYrA
Render the repository README in the first tab and add a Code tab that
browses the repo's file tree and renders source files (syntax-highlighted),
reading directly from the NIP-34 clone URL over the git smart-HTTP v2
protocol (works with GRASP/ngit bare servers as well as GitHub/GitLab).
quartz (jvmAndroid): a from-scratch git smart-HTTP v2 client — pkt-line
codec, packfile parser with OFS/REF delta resolution and SHA-1 oids,
tree/commit parsers, and a high-level browser that fetches a shallow
filter=blob:none snapshot (one request for the whole tree) and lazily
pulls file blobs on demand. Offline tests run against real captured
GitHub wire bytes plus a git-generated OFS-delta pack.
amethyst: README tab (rich markdown), Code tab (folders-first browser
with breadcrumb navigation + a file viewer that renders markdown or
syntax-highlighted source), a browser ViewModel, new UI strings, and a
Folder material symbol (font subset regenerated). Syntax highlighting
uses dev.snipme:highlights (Apache-2.0, permissive).
Tabs are now: README, Code, Overview, Issues, Patches & PRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DpNmN8CvP6HnEsdTGAjVUr
Render real exercise names in POWR workout cards instead of slug-derived
labels by fetching the referenced kind-33401 exercise templates.
- ExerciseTemplateEvent (kind 33401, addressable) with title/format/
format_units/equipment/difficulty accessors; registered in EventFactory.
- WorkoutRecordEvent now implements AddressHintProvider: linkedAddressIds
(deduped 33401 + 33402 coordinates) seed the gatherer so the card
re-renders when a template arrives, and addressHints feed the relay-hint
index with the relay.powr.build hints so the fetch reaches where POWR
published the templates.
- WorkoutDisplay resolves each exercise via LoadAddressableNote +
observeNoteEvent<ExerciseTemplateEvent>, showing the template title once
fetched and the slug until then.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJwXz6CWez8r7trgHXT545
POWR and RUNSTR both publish kind 1301 but with incompatible tag schemas.
A POWR event previously rendered with the raw "33401:...:back-squat-bb"
coordinate as its activity label, no duration, and none of the set data.
Parse the POWR / NIP-101e dialect in quartz and render it in Amethyst:
- type tag for the activity (strength/circuit/emom/amrap), preferred over
the RUNSTR exercise verb; coordinate-form exercise tags no longer leak as
a verb.
- start/end session timestamps -> derived duration; completed flag.
- structured per-set exercise tags (kg weights, reps, rpe, set_type),
grouped per exercise template with volume/top-weight aggregates.
- WorkoutDisplay now shows Exercises/Sets/Volume stats and a per-exercise
breakdown (e.g. "Back Squat Bb -> 3 x 8 x 84 kg") in the viewer's unit.
Rendering interop only; Amethyst still publishes the RUNSTR-canonical form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QJwXz6CWez8r7trgHXT545
Adds a platform-agnostic resolver for NIP-5A static-website / napplet (NIP-5D)
manifests in quartz commonMain, under nip5aStaticWebsites/resolver/:
- StaticSitePathLookup: request-path normalization (query/fragment stripping,
leading-slash insensitivity, root/dir -> index.html), slash-insensitive
path lookup over a manifest's path tags, and a web-asset Content-Type guess.
- StaticSiteResolver: hash verification, Blossom candidate-URL assembly, and a
suspend resolve() that downloads each listed server in order and accepts the
first blob whose recomputed sha256 matches the manifest pin. HTTP is injected
via a BlobFetcher typealias so quartz keeps no HTTP dependency.
The trust model is the point: the signed manifest is the authority, the Blossom
server is untrusted. A server that substitutes/corrupts a blob fails
verification and is skipped -- it can withhold content but never forge it.
Tests cover normalization, lookup, MIME guessing, and the security cases
(tampered server skipped -> falls through to honest server; all-tampered ->
Unresolvable; undeclared path -> PathNotInManifest without fetching).
Also adds quartz/plans/2026-06-19-napplet-nip5a-resolver.md documenting the
design and the open event-shape alignment questions (35128 vs 35129 manifest
kind, capability declaration vs NIP-89, aggregate build hash, server ordering)
to raise with the napplet author before the nsite/napplet event shape forks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CdAJMbnHJfiMY7UcS99T6C
Claude