Milestone 1 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
Sorted array with binary-search insert (near-tail in the common case),
itemized remove for displaced rows, wholesale invalidate for delete
paths that can't itemize, and sealed snapshots memoized per mutation
generation — reconcile only reads, so one snapshot backs any number of
concurrent sessions and stays immutable under later writes. Over-cap
answers null so the caller keeps the strfry-parity NEG-ERR.
Not wired into any store yet; next milestones plumb displaced-row
deltas from the SQLite modules and serve index-total filters from it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Design for backlog item 3 of the relay performance campaign: an
always-current (created_at, id) index maintained from the store's write
path so cold NEG-OPENs stop paying the full scan + O(n log n) seal
(~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers
the snapshot/COW model, the removal-correctness split (RETURNING deltas
for replaceable overwrites, wholesale invalidation for rare delete
paths), IndexingStrategy gating so app-side stores are untouched, and
the micro + relayBench A/B measurement plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Measured over the real OkHttp transport (upstream killed mid-mirror and
restarted on the same port): NostrClient re-dials once on disconnect and
then falls back to its 60s keep-alive, which showed up as a 61s mirror
blackout when the immediate re-dial raced the port rebind.
- Retry pump: the worker nudges the pool every 5s. Cheap and safe —
reconnectIfNeedsTo skips connected relays and each relay's
exponential backoff (1s doubling, 5min cap) still gates real dial
attempts, so dead upstreams aren't hammered. Restart recovery drops
from 61s to ~6s in the new reconnect test.
- WebSocket pings (120s, same as the Android relay pool): without
them a half-open connection (network drop, no FIN) never fires
onDisconnected and the mirror would stall silently forever.
- MirrorWorkerReconnectTest: end-to-end over a real Ktor port — ride
through the drop, re-subscribe, drop the duplicate replay, pull the
event that only exists on the new instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Each [[mirror]] entry takes an optional NIP-01 filter as a JSON object
string, like strfry-router's per-stream filter. It is applied twice,
matching strfry's design (cmd_router.cpp):
- it shapes the REQ sent upstream (the mirror still owns since via
backfill_seconds and strips limit — the subscription is unbounded);
- every delivered event is re-checked against it before ingest, so an
upstream answering outside its REQ — including a trusted one whose
events skip signature verification — can only inject events inside
the operator-declared scope. Out-of-scope deliveries surface on a
'filtered' counter.
Malformed filter JSON fails the boot, not the first delivery. Several
disjoint scopes for one upstream = repeat [[mirror]] with the same url.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog item 1 of the relay performance campaign: skip signature
verification for events ingested from explicitly configured trusted
upstream relays, strfry-router style.
geode now dials each [[mirror]] url from the config, subscribes to
everything newer than now - backfill_seconds, and feeds the stream
through NostrServer.ingest (same group-commit writer + live fanout as
client publishes). The NostrClient underneath owns reconnects, backoff
and REQ re-sync; duplicate replays after a reconnect are rejected by
the store's unique-id constraint and only surface as counters.
trusted = true is the per-upstream trust switch: events from that
connection skip Schnorr verify. The trusted identity is the URL this
relay dialed (TLS-authenticated for wss://), never anything an inbound
peer claims. Default is false — mirror-but-verify — and a relay with no
[[mirror]] entries behaves exactly as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Adds a public local-ingest entry to NostrServer for events that don't
arrive over a client connection (mirror/sync workers, import jobs). It
routes through the same group-commit IngestQueue and live fanout as a
client publish, and each Submission can opt out of the parallel
Schnorr-verify hook — the relay-to-relay trust model, for events
streamed from an upstream that already verified them (verify profiles
at ~8% of busy ingest CPU).
Library defaults are unchanged: skipVerify defaults to false everywhere
and nothing in the client-publish path can set it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
The worker's writer-mutex fast path can slip past its own scope
cancellation and run one batch against a store that close() already
freed. The resulting closed-connection exception escaped scope.launch
under a SupervisorJob with no handler — harmless in production, but the
kotlinx-coroutines-test global handler attributes it to whatever runTest
starts next (seen on CI as EventSourceServerTest.countUsesSource failing
with UncaughtExceptionsBeforeTest). Catch, log, and stop the pass:
nothing depends on it — the pre-search drain covers search correctness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Two issues surfaced by rebasing onto main:
- SQLiteConnectionPool.close() freed the writer's native handle without
taking the writer mutex, so a block still running on another thread
(the deferred-FTS catch-up worker's current batch outlives its scope
cancellation) could call sqlite3_prepare on a freed sqlite3* — native
heap corruption, SIGSEGV in sqlite3DbMallocRawNN (reproduced twice in
:geode:test). close() now reclaims every reader from the channel and
acquires the writer mutex before closing handles, then releases so
stragglers get the managed closed-connection exception. Idempotent
via a closed flag.
- main's NIP-50 fix (7deda28d) strips search-extension tokens before
every store query, but the zero-decode queryRaw replay path added on
this branch predates it and passed raw filters through — an
extensions-only search would hit SQLite FTS as column syntax and
error. queryRaw now strips like query/count/snapshot do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
FTS indexing ran inside every insert's transaction — a measurable slice
of write cost (relayBench: ~18% of ingest throughput) paid at publish
time for a feature only search queries read. It now runs as a watermark
catch-up:
- IndexingStrategy.deferFullTextSearchIndexing (default false; geode's
relay strategy enables it with search). Deferred inserts skip
tokenization entirely.
- FullTextSearchModule keeps a fts_catchup_state watermark (everything
<= last_row_id is indexed) and gains catchUpBatch(): scan past the
watermark, tokenize, advance — one write transaction per batch, so
publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at
MAX(row_id) for existing (synchronously indexed) databases.
- NostrServer runs the catch-up worker, poked by IngestQueue's new
onBatchCommitted hook, and *yields to publish traffic*: it only
drains while the queue has no backlog (IngestQueue.hasBacklog()), so
bursts ingest at no-FTS speed and tokenization fills the gaps.
- LiveEventStore drains the backlog synchronously before serving any
filter with a search term (query, queryRaw, count) — NIP-50 results
stay exactly as fresh as the synchronous path; the deferral is
invisible to correctness. Geode's existing search tests pass
unchanged through this path.
The first implementation reused reindexBatch and collapsed ingest 8x —
its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?'
matches on a plain FTS5 column, i.e. a full FTS-table scan per row
(O(n²) overall), and the worker competed with the replay for the writer
mutex. catchUpBatch therefore inserts without the delete (rows past the
watermark are never indexed; switching a DB between deferred and
synchronous strategies requires reindexAll, same rule as a
searchable-kinds change), and the worker backs off whenever publishes
are pending.
Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136
events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the
--no-search ceiling while keeping NIP-50 advertised and fresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
A NIP-77 server session rebuilt its reconciliation structure from
scratch on every NEG-OPEN: full id+created_at scan, per-entry hex
decode into a fresh StorageVector, O(n log n) seal. That cost grows
with the corpus and is paid even when nothing changed — the exact
shape of a periodic mirror's heartbeat, where N peers reconcile the
same broad filter over and over. relayBench measured 342 ms per
identical-set reconcile at 50k events vs strfry's 26 ms off its
always-current LMDB tree.
Reconciliation only *reads* the sealed storage, so one instance can
back any number of concurrent sessions:
- NegentropyServerSession now accepts a pre-sealed IStorage (the
List<IdAndTime> constructor remains and delegates).
- SessionBackend.sealedNegentropyStorage() builds + seals (null when
the set exceeds maxSyncEvents); LiveEventStore overrides it with a
single-slot cache keyed by (filter set, write generation) plus a 30s
TTL. The generation bumps on every accepted ingest; the TTL bounds
staleness from delete paths the counter can't see (expiration
sweeps, admin purges) — negentropy snapshots are point-in-time sets,
so seconds of staleness only means a peer briefly re-offers ids.
- NegSessionRegistry.open consumes the shared sealed storage;
over-cap NEG-ERR behavior unchanged (strfry parity).
relayBench gains a 'heartbeat' measurement — the identical-set
reconcile repeated immediately with no writes in between. At 50k
events: geode 342 ms -> 27.8 ms vs strfry 21.1 ms (near parity; was
13x). Cold reconciles (first open after a write) are unchanged.
Also fixes the GeodeVsStrfryNegentropySyncTest fixture to write
'nofiles = 0' so the opt-in interop test can boot strfry inside
containers with a low RLIMIT_NOFILE hard cap; the interop suite passes
against strfry v1-b80cda3 with the cache in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Reviewed strfry's LMDB indices (golpe.yaml) against geode's SQLite set:
the two are nearly isomorphic — time, id, kind+time, author+kind+time,
tag+time, plus conditional deletion/expiration/replaceable entries
(geode's are partial indexes, so ordinary events don't pay for them).
Nothing to drop. One real hole: strfry maintains a plain
pubkey(+created_at) index and geode had none, so an authors-only filter
(no kinds) — archive pulls, account-migration tools, 'everything by
these pubkeys' — degraded to a full walk of the time index. EXPLAIN
confirmed: SCAN query_by_created_at_id.
- quartz: IndexingStrategy.indexEventsByPubkeyAlone (default false —
clients query their supported kinds and can skip it) gates a new
query_by_pubkey_created index; DATABASE_VERSION 2→3 with an
idempotent migration that backfills it for opted-in strategies.
- geode: relayIndexingStrategy turns it on.
- relayBench: new 'author-archive' scenario — every kind by 3 *quiet*
pubkeys. Quiet is the point: prolific authors are dense in the time
index and a scan finds them quickly, which is why the suite never
caught this; sparse authors force the full walk.
Measured (50k corpus): author-archive EOSE p50 42.8 ms -> 3.6 ms (12x,
and the old path grows linearly with table size); ingest 5,337 -> 5,156
events/s (~3%, the one extra B-tree per event). strfry reference on the
same scenario: 0.52 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
The drain loop ran verify(batch N) and insert(batch N) strictly
back-to-back: the SQLite writer idled during every Schnorr verify and
the CPU cores idled during every commit. Split it into two stage
coroutines joined by a capacity-1 channel — a verifier that collects
and checks batch N+1 while the writer commits batch N. Same shape as
strfry's ingester/writer thread split.
Ordering (single verifier, single writer, FIFO handoff), OK-after-commit
semantics, per-row error isolation and submit() backpressure are all
unchanged; total in-flight grows by at most one batch.
Alternating A/B on the 10k corpus, 4-core host with the benchmark client
competing for the same cores: sequential 4,111/3,977/4,443 vs pipelined
4,050/5,020/5,092 events/s (~+13% mean). The overlap should widen on
dedicated relay hosts where verify has its own cores. Feature-parity
run after this change: geode --no-search 5,911 vs strfry 9,374 events/s
(1.59x, down from 2.2x at the start of the perf work).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
strfry implements no NIP-50 at all, so geode's default setup pays FTS
tokenization on every searchable event for a feature the other side of a
benchmark isn't providing. The new switch removes it cleanly:
- geode: --no-search CLI flag and [options].full_text_search TOML key
(default true). When off, the store skips the FTS index entirely,
NIP-11 stops advertising 50 (an explicit [info] nips list stays
operator-authoritative), and search filters match nothing via the
existing QueryBuilder guard. RelayIndexingStrategy becomes
relayIndexingStrategy(fullTextSearch) with the stock val kept.
- relayBench: --geode-no-search runs the geode entry with the flag (relay
named geode-nosearch in reports); README documents the apples-to-apples
rationale and a recipe for benchmarking both geode flavors side by side.
Alternating A/B on the 10k corpus: 4,767/4,792 ev/s without search vs
4,111/3,977 with — NIP-50 costs ~18% of ingest throughput at current
write-path speed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Main's giant-REQ fix (18bd3600) replaced query()'s copy-on-add immutable
dedupe set with a spin-locked HashSet, but the rebase left queryRaw —
now the default REQ path — on the old pattern, which would have
reintroduced the O(n²) crawl for large replays. Both paths now share
the mutable-set-under-spinlock shape.
relayBench's sync driver moves to NegentropySession.fromEvents(),
following the session's new List<IdAndTime> constructor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Three changes, each validated head-to-head against strfry with relayBench
(same 10k-event corpus a1cd3517a8296911, stock configs, sig verify on):
- geode: RelayIndexingStrategy turns on indexEventsByCreatedAtAlone for
the relay's stores (quartz's DefaultIndexingStrategy stays off for
client-side stores). A relay can't predict client filters, and the
cheapest REQ of all — {"limit":N} — was a full-table scan + top-N
sort: 40 ms and O(table) growth before; 11 ms and index-streamed
(first event 30 ms -> 1.9 ms) after.
- quartz: zero-decode REQ replay. Stored events now stream as RawEvent
(tags kept in serialized form) and are spliced directly into wire
frames — no tags parse, no EventFactory dispatch, no re-serialize per
row. Gated on the new IRelayPolicy.filtersOutgoingEvents capability:
policies that can veto per-event delivery (none today) keep the
materialized path; everyone else skips it. Live post-EOSE delivery is
unchanged (live matching needs Event objects).
- quartz: StatementCachingConnection wraps the pool's writer and reader
connections, replaying prepared statements instead of re-preparing per
event/REQ (eager reset on return keeps cursors from holding table
locks; a 256-statement cap bounds client-controlled filter-shape
variety). Ingest went 3,000 -> 4,700 events/s (+55%) — prepare
overhead was the single largest non-crypto write cost.
Net effect on the benchmark: ingest gap vs strfry narrowed from 2.2x to
1.8x, the firehose latency gap from 4x to 1.2x, and geode now wins 4 of
9 query-latency scenarios (notifications, hashtag, by-ids,
recent-window) plus most concurrent-throughput scenarios, while keeping
the smaller on-disk footprint. Result sets stayed byte-identical across
relays and NIP-77 sync still converges.
Measured but deliberately NOT taken: per-row SAVEPOINT elision (+4%,
within run noise — not worth weakening batch error isolation), FTS-off
(+25% ingest but drops NIP-50), --no-verify (+45% but unfair/unsafe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
New :relayBench module that boots relay implementations as real external
processes under equivalent setups (persistent storage, sig verification on,
no auth) and compares them on the same corpus:
- Ingest: receipt->queryable-by-REQ latency (publish on one connection,
hammer-poll REQ{ids} on another), OK-ack latency, and pipelined corpus
replay throughput over N connections.
- Queries: client-realistic filters derived from the corpus (home feed,
thread, notifications, profiles, hashtag, by-ids, ...), time-to-EOSE
percentiles plus aggregate events/sec under concurrency; result-set
counts are cross-checked between relays and mismatches flagged.
- NIP-77 negentropy sync between every relay pair: 80%/80% slices with 60%
overlap, reconcile timing/rounds/wire-bytes per server side, delta
transfer, steady-state identical-set reconcile, convergence verified.
- Storage footprint after full ingest.
Corpora (all cached as NDJSON + manifest with a sha256 id fingerprint so
results are comparable across runs and machines):
- synthetic (default): deterministic to the byte — seeded keys, fixed
timestamps, seed-derived BIP-340 aux nonces — with a realistic social
shape (zipf authors, threads, reactions, reposts, zap request/receipt
pairs, hashtags);
- the checked-in real dump (quartz test fixture, ~31k unique 2024 events);
- external dumps (NDJSON or JSON array, gzip sniffed by magic bytes),
e.g. the 2.1M contact-list archive, with --max-event-bytes/--max-tags
raising both the corpus filter and the strfry config together;
- live download from public relays.
Every source runs through the same preparation: dedup, drop unsigned/
ephemeral/kind-5, enforce relay ingest caps, parallel Schnorr verify,
chronological sort.
relayBench/run.sh is the one-command entry point: builds geode + harness,
resolves strfry (STRFRY_BIN, PATH, or source build into .cache), runs the
suite and renders an ANSI report with per-metric bars and winners, plus
report.md and results.json under relayBench/results/<timestamp>/.
The harness client disables Nagle (TCP_NODELAY): with the JDK default, a
REQ following the previous round's CLOSE stalls a full delayed-ACK
interval and every latency floors at ~44 ms against both geode and strfry
(verified: ~0.3 ms with it off).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Raw key:value tokens like include:spam reached SQLite FTS MATCH, where
the colon is column-filter syntax — any REQ carrying an extension token
died with CLOSED "no such column: include" instead of matching.
Adds SearchQuery.stripExtensions() plus Filter/List<Filter>
.strippingSearchExtensions() so EventStore users can drop the tokens
before querying, and applies them in LiveEventStore (query, count,
negentropy snapshots). Per NIP-50, unsupported extensions are ignored:
an extensions-only search becomes unconstrained, not match-nothing.
EventSource-backed search relays still receive the raw string since a
real search backend wants the extensions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tujoyfc2kNLZNVgiLZAR7F
The 2026-06-25 translation import left U+FFFD replacement characters
in show_npub_as_a_qr_code, show_nprofile_as_a_qr_code and relay_reorder
in both values-ta and values-ta-rIN. Restore the accusative suffix
as npub-ஐ (independent vowel AI), matching the nsec-ஐ
precedent in the same files. Clears Sonar's file-encoding warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017iWg6SxLXteav6xJJz7gKS
Delete 19 strings.xml files that contain zero string resources:
- 12 exports of Crowdin targets with 0% translation (ca-rES, cy-rGB, da-rDK,
gu-rIN, hr-rHR, iw-rIL, kk-rKZ, ks-rIN, lt-rLT, ne-rNP, sa-rIN, pcm-rNG).
NOTE: the next Crowdin sync recreates these unless the corresponding target
languages (Catalan, Welsh, Danish, Gujarati, Croatian, Hebrew, Kazakh,
Kashmiri, Lithuanian, Nepali, Sanskrit, Nigerian Pidgin) are unchecked in
the Crowdin project settings; they hold zero translations there, so
unchecking loses nothing and a language can be re-enabled when a translator
volunteers.
- 6 orphans with no Crowdin target at all (et-rEE, fo-rFO, ku-rTR, so-rSO,
ss-rZA, ur-rIN).
- values-hu, emptied earlier to fix aapt2 duplicate resources; the real
Hungarian translation lives in values-hu-rHU (until Hungarian is
consolidated to base like Czech in #3461).
Also drop the matching locales_config entries so the per-app language picker
stops offering languages that render 100% in English, including the dangling
"ar" entry (values-ar has never existed; ar-SA remains) and the duplicate
"hu" entry (hu-HU remains).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KhEN93aKyQCFVpLUgRNXid
Czech was consolidated onto values-cs in PR #3461 (values-cs-rCZ deleted,
cs: cs languages_mapping). Update the reference-locale table, diff commands,
and prose accordingly, and note that the remaining locales keep their
region-qualified dirs until consolidated the same way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KhEN93aKyQCFVpLUgRNXid
Czech has a single Crowdin target (cs), but Crowdin's default android_code
for it is cs-rCZ, so the human/Crowdin translation was written to
values-cs-rCZ. Promote it to base values-cs, delete the regional dir, map
cs -> cs in crowdin.yml so future syncs land at the base (single target =>
no collision), and drop cs-CZ from locales_config so the per-app language
picker shows just "Czech".
Also removes the bulk-filled base values-cs from PR #3371 ("Add and update
translations across 50+ locales"), which had translated 38 strings marked
translatable="false" in the source (the *_search_keywords English index,
github/mastodon proper nouns, notification-channel names). The promoted
cs-rCZ is Crowdin-clean (0 translatable="false" leakage), so this also
fixes that contamination for Czech.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KhEN93aKyQCFVpLUgRNXid
Replaces the hand-rolled raw-WebSocket NIP-77 negotiate loop (single
un-windowed session — a strfry max_sync_events overflow was a hard
error) with quartz's negentropyReconcile: created_at window splitting
on overflow, keep-alive connection pinning, and streaming id batches.
Downloads and uploads now pipeline with the remaining reconcile
rounds: need-id batches feed 4 concurrent by-id drains, have-ids feed
an uploader (peak 7 subscriptions, under the common relay cap of 20).
Every downloaded event still funnels through the verify-and-store
path. Output field 'rounds' (protocol round-trips) is now 'windows'
(created_at splits).
Verified end-to-end against embedded geode relays: down-only 25/25,
up-only 5/5, and bidirectional re-runs converge to a zero diff.
Also records both adoptions in the perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Passes decoder = CachingEventDecoder() at all four NostrClient
construction sites: the Android app pool (AppModules), the Android
crawl client (buildCrawlClient — Event Sync / Cashu discovery, the
duplicate-heaviest path), the desktop RelayConnectionManager, and
amy's Context. Duplicate EVENT frames (14-57% of production traffic)
now skip the full JSON re-parse; dispatch semantics unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Same-JVM interleaved comparison against the resurrected spin-lock
implementation: single-threaded parity (the uncontended lock was ~free),
but 8 threads sharing one decoder run 3.0-4.5x faster lock-free — the
spin lock serialized the concurrent hit path just like the old global
PoolRequests lock did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The speedup assertion depends on AVAILABLE parallelism, which a
full-suite run (the pre-push hook) can eat — it flaked at 1.14x under
load. Now retries up to 3 measurement passes and enforces a hard 1.05x
floor that a genuine serialization regression (the per-event-async
version measured 0.94x) can never pass, warning instead of failing in
the noise band between floor and the 1.5x target.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Replaces the decoder's spin lock with lock-free concurrent maps so the
hot per-frame duplicate check never serializes across the pool's relay
consumer coroutines. New minimal ConcurrentHashCache expect/actual
(get/put/size/clear) following LargeCache's per-platform choices:
ConcurrentHashMap on JVM/Android, CacheMap on Apple, copy-on-write on
the CI-only Linux target. Counters become atomics; the generational
rotation keeps its deliberately-tolerated benign races, now documented
per failure mode (each is at worst a redundant re-parse, never a wrong
message).
CachingEventDecoderConcurrencyTest hammers one shared decoder from 8
threads with 80k duplicate-heavy frames and capacity 256 (rotations
fire constantly): zero wrong messages, exact parsed+reused accounting.
The 7 scan-safety tests and DedupDecodeBenchmark's enforced speedup
pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Code review confirmed each relay connection owns its own consumer
coroutine on Dispatchers.IO with no downstream funnel or shared lock
before justVerify (LargeCache is a ConcurrentSkipListMap; the
PoolRequests lock is per-subscription now), so multi-relay bursts
already verify in parallel across cores. Corrects the earlier follow-up
suggesting a CacheClientConnector integration: the accessory's scope is
single-connection bulk streams, plus a possible future dispatcher-
hygiene fix if on-device profiling shows CPU-bound verifies
oversubscribing the 64-thread IO pool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Deliberate design decision reversing the 4096-frame receive bound:
1. The remote infrastructure isn't ours — TCP backpressure parks the
backlog in the RELAY's outbound buffers. A client should release the
relay from its duties as fast as it can send and own the buffering
itself.
2. The app holds 2000+ simultaneous relay connections; a bounded buffer
under a slow consumer blocks OkHttp reader threads, and at that
connection count blocked readers are a thread-starvation hazard far
worse than the heap growth they prevent.
The UNLIMITED channels now carry an explicit do-not-bound comment with
this rationale, and the slow-consumer risk is addressed from the other
side: keep the consumer faster than any relay's send rate
(CachingEventDecoder, ParallelEventVerifier, PoolRequests sharding).
BoundedReceiveBufferTest removed with the bound it tested; plan doc
records the decision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Rework the Amethyst Desktop notification experience end-to-end.
**In-app inbox** (`desktopApp/…/ui/NotificationsScreen.kt`)
- Dedicated Notifications entry in the sidebar and a new
`DeckColumnType.NotificationSettings` overlay reachable from a ⚙ button
in the column header — back button renders automatically via
`navState.hasBackStack` in deck mode and via body Back in single-pane.
- Redesigned column: filter tabs (All / Mentions / Replies / Reactions /
Zaps / Reposts / DMs) with per-kind counts, grouped cards (reactions
and reposts collapse to "N reactions on your post" per day), unread
dots driven by a persisted `lastReadAt` per pubkey, freshest-first
ordering via `compareByDescending { timestamp }`.
- User metadata: avatars + display names on every row (including reactor
strip inside grouped cards) resolved from `LocalCache`, with
`metadataVersion` observation. Zap sender is the actual zapper (via
`NotificationItem.effectiveAuthorPubKey` unwrapping
`LnZapEvent.zapRequest.pubKey`), not the LNURL provider.
- Reaction/repost group cards are clickable → thread; DM cards click →
Messages column; expandable to show note preview + reactor list.
- `NotificationSettingsScreen`: master toggle, 7 per-kind toggles,
manual-DND dropdown, preview-privacy switch, per-platform status
card, "Send a test toast". Permission-aware button adapts across
NotRequested → Granted / Denied / BundleRequired with a macOS System
Settings deep-link. State syncs with OS-level changes via
`LocalWindowInfo.isWindowFocused` regain refresh.
**Native OS notifications** (`commons/…/moderation/notifications/`)
- `NotificationDispatcher` interface + `PermissionState` sealed
hierarchy in `commonMain`. JVM impl `NucleusNotificationDispatcher`
routes through Nucleus (three per-OS artifacts: macOS
`UNUserNotificationCenter` via Swift/JNI, Windows WinRT toast via
JNI, Linux libnotify via D-Bus). Falls back to `AwtTrayNotifier` when
native lib fails to load. Async `requestPermission` +
`refreshPermission` bridge Nucleus's callback API to `suspend`.
- `DesktopNotificationAutoDispatcher` subscribes to
`DesktopLocalCache.eventStream.newEventBundles` and fires OS toasts,
applying a 9-check suppression pipeline: kind allow-list, master
toggle, per-kind toggle, DND, window-focused, cold-boot
(event.createdAt < sessionStart or >30s stale), macOS permission,
semantic accept, 30s per-(kind,event-id) dedupe. Wired in Main.kt
with DisposableEffect(loggedIn.pubKeyHex); window focus tracked via
LocalWindowInfo → StateFlow.
- Adds `windows { menu = true; shortcut = true }` to
`desktopApp/build.gradle.kts` so AUMID persists and Windows toasts
survive reboot.
**Shared notification filter** (`commons/…/moderation/notifications/NotificationKinds.kt`)
- Extracted from Android's `NotificationFeedFilter`. Exposes
`SUBSCRIPTION_KINDS` (13 kinds: text, DMs kind 4 + 14 + 1059
gift-wrap, encrypted-file-header, comments 1111, reactions, reposts,
generic reposts, channel messages 42, nutzaps 9321, zap receipts
9735, onchain zaps 8333), `subscriptionFilter(pubKey, since, limit)`
builder that `FilterBuilders.notificationsForUser` delegates to, and
`tagsAnEventForUser(event, myPubKey, isTargetAuthoredByMe)` semantic
gate. Reactions/reposts require target-author-match; other kinds
require `p=me`. Fixes a bug where the helper defaulted to accept and
let cache-seed leak "mentioned you" notifications from unrelated
text notes.
- Android's `NotificationFeedFilter.NOTIFICATION_KINDS` now spreads
`SUBSCRIPTION_KINDS` + Android-only extras (badges, git, highlights,
polls, videos, voice, live-activities), so a change on either side
propagates. Downstream push consumers (`NotificationDispatcher.kt`,
`EventNotificationConsumer.kt`) read the resulting set transparently.
- Content sanitizer strips control chars, RTL overrides, zero-width
chars, and URLs from toast titles. DM cards never render ciphertext
body (decryption pipeline deferred).
**Tests** — `NotificationKindsTest` covers 17 scenarios: reactions
target-author-mismatch rejection, own-event rejection except zap kinds,
p-tag routing for text/DMs/zaps/nutzaps/gift-wraps/channel messages,
`SUBSCRIPTION_KINDS` sanity + `subscriptionFilter` shape.
**Testing constraint**: macOS OS notifications require a bundled
process. `gradle run` will always show BundleRequired — use
`gradle :desktopApp:runDistributable` and open the resulting
`Amethyst.app`. First permission grant surfaces the app in
System Settings → Notifications.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Two server bugs masking each other made a single giant REQ crawl and
then wedge:
LiveEventStore's historical-replay dedup used an immutable Set under an
AtomicReference with copy-on-add — set + id copies the whole set per
streamed event, so large replays were accidentally O(n²) (100k-event
REQ: ~700 events/s, degrading as the response grew). Replaced with a
spin-lock-guarded mutable HashSet (same threads, single contains/add
per critical section).
Fixing that unmasked WebSocketSessionPump's slow-client policy: a fast
replay instantly overflowed the 8192-frame cap — which conflated 'slow
client' with 'replay outruns the socket writer', normal for bulk — and
the 'drop' only closed the internal queue, leaving the socket half-dead
(no EOSE, no close frame, tail silently missing: the likely cause of
the benchmark's 99,998/100,000). Producers are now paced against a full
backlog (bounded blocking wait, consistent with the documented ingest
fanout behavior) and only a client still behind after 30s is dropped,
by actually cancelling the socket.
GiantReqStreamTest guards the regression: 20k-event REQ pre-fix 8.4s
(~2.4k events/s), post-fix 0.7s (~27k events/s), all events + EOSE
delivered. 96 geode tests and the quartz relay/server suites pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
The per-connection-wall test prints the negotiated
Sec-WebSocket-Extensions header; against strfry OkHttp negotiates
permessage-deflate (client_no_context_takeover) out of the box, closing
the 'is compression actually on?' question from the optimization list —
it is, no change needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
One reconcile feeds by-id download batches to N clients (one socket
each) x reqsPerClient workers; reconcile windows also round-robin
across the connections, since a single connection produced need-ids at
only ~9k/s on a 2.6M corpus and starved the downloads. Events funnel
through a bounded channel to a single consumer (exact maxEvents,
single-threaded onEvent); all stages backpressure; localEntries diffing
and have-counting match negentropyReconcile. reconcileWindows became
multi-client internally; single-client paths pass listOf(this).
Production shootout (same-run pairs, 100k cap): +18% to +64% over the
tuned single client, capped by the relay's server-side reconcile id
production rather than download parallelism (which the by-id matrix
shows scales 2.7x with connections). 4 in-process multi-client tests;
18 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Client-side mirror of IngestQueue.parallelVerify: submit() is a cheap
bounded-channel send from the relay consumer coroutine; a drain loop
batches greedily (up to 256) and fans each batch across
Dispatchers.Default in core-sized chunks, dispatching callbacks in
submission order. preVerified short-circuits already-trusted ids; the
bounded channel backpressures the socket instead of growing heap.
Batch/chunk sizing is measurement-driven: per-event async cost ~40us of
scheduling each (swallowing the gain), and the per-batch join barrier
at 64 still cost half (64 -> 1.2x, 256 -> 2.2x, 1024 -> 3.2x on 4
cores). ParallelVerifyBenchmark (fresh signed events per pass — Event
caches derived state after first verify, so passes must not share
instances) measures 1.8x vs sequential with an enforced >=1.5x
assertion. 4 correctness tests cover valid/tampered routing, ordering,
preVerified and callback-crash resilience.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Claude
David KasparandGitHub
vitorpamplonaandgithub-actions[bot]
Vitor PamplonaandGitHub
davotoulaandgithub-actions[bot]
nrobi144andClaude