- only parse commonName when the alt tag has the Birdstar prefix
- summary() on both Birdstar events now uses the canonical NIP-31
tags.alt() helper instead of a raw firstTagValue("alt") lookup
- speciesReference() only returns http(s) URLs since UIs render it as
a clickable link (rejects e.g. javascript: schemes), with a test
- Detection card: parse tags once into a single remember slot, drop
the near-dead '?: summary' title fallback, stop rebuilding the
italic TextStyle every recomposition
- Hoist the duplicated bird-emoji literal into a shared BIRD_PREFIX
- Trim the redundant factory test to the assertIs idiom
- fetch Birdex life lists in home and profile relay REQs
- richer Birdstar cards — common name title, Wikidata link, bird emoji
- surface Birdstar bird detections in home and profile feeds
On connect, syncFilters re-sends every desired REQ through
PoolRequests.syncState. It previously sent the frame and only recorded
the subscription as SENT afterward, in the post-send onSent callback. A
relay that answers faster than that callback runs — the in-process
transport used by the desktop launch-optimization tests, or any relay on
a fast path — can deliver the EOSE while the per-sub state still reads
"nothing in flight" (onConnecting cleared it, onSent hasn't recorded it).
The EOSE handler then sees empty filters, concludes it never sent a REQ,
and fires a duplicate, replaying the whole page a second time.
Pre-mark the sub as SENT under its lock before the frame leaves, mirroring
the decideCommandLocked pre-mark already used by sendToRelayIfChanged, so
a response can never race ahead of the record. The send stays
unconditional: this is a fresh-connection sync (onConnecting always
cleared the per-relay state first), so there is no in-flight REQ on the
new socket to dedupe against.
Fixes the flaky SubscribeBeforeConnectTest, which asserted a pre-connect
subscription delivers exactly its events once and intermittently saw them
doubled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4kmtZoNUSXxwG23wD2JwP
Fail-loud on config that would silently degrade the running relay:
- readers = 0 makes every query hang forever on an empty reader pool;
readers < 0 crashes with an unrelated message. optimize_interval_seconds
<= 0 busy-loops PRAGMA optimize under the writer mutex. StaticConfig
.validate() (called at boot) rejects both.
- A typo in [[mirror]].filter (e.g. `kindss`) parsed to an empty
match-everything filter through the tolerant deserializer — silently
widening a trusted upstream's skip-verify scope to the whole firehose.
MirrorFilterValidator strict-checks the filter JSON at boot: unknown
keys and non-array list fields fail startup.
- The self-mirror guard now compares scheme-insensitively (ws:// vs
wss:// for the same host is still us) via displayUrl().
- The maintenance loop rethrows CancellationException and no longer
swallows Errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Two mirror hardening fixes from the audit:
- Trust leak: the skip-verify decision was keyed on the subscription id,
but the client pool dispatches EVENTs by subscription id alone and
every [[mirror]] upstream shares one client. A hostile untrusted
upstream could answer with the trusted upstream's subscription id and
ride its skip-verify into the store. startDown now drops any event
whose delivering relay isn't the one that subscription dialed
(relay != up.url). New MirrorWorkerTrustOriginTest injects a foreign
subId frame from a hostile relay and asserts it never lands.
- Reconnect replay: `since` was frozen at boot, so a long-lived daemon
re-streamed the whole backfill window on every upstream flap. The down
path now tracks the newest ingested created_at and, on the
connected->disconnected edge, advances the REQ's since to
watermark - overlap before the reconnect re-sends it. Advancing only on
disconnect means a stable link never re-queries. The reconnect test now
asserts the watermark advanced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
TcpNoDelaySocketFactory's connecting overloads used
`socket().apply { connect(...) }`, which leaks the file descriptor if
bind/connect throws (the JDK's connecting Socket constructors close on
failure; ours didn't). Wrapped in a helper that closes on throw. OkHttp
only calls the no-arg overload, so this guards any other direct caller.
DesktopHttpClient's pre-init `simpleClient` (direct relay sockets opened
before setInstance) now gets the same TcpNoDelaySocketFactory as
directClient. failClosedClient is left as-is: it's a SOCKS client and
OkHttp bypasses the socket factory for SOCKS proxies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
ingest() bypasses the per-connection policy chain, which is where
VerifyPolicy lives. The IngestQueue verify hook only exists when
parallelVerify is true, so with parallelVerify = false and
skipVerify = false, ingest() previously verified nothing — an untrusted
mirror upstream on a relay running the legacy in-policy verify path could
inject forgeries. ingest() now verifies inline in that configuration
(same rejection reason as the queue), so the documented "default keeps
verify-everything semantics" holds regardless of parallelVerify. KDoc
also spells out that ingest() skips the entire policy chain (blacklists,
size limits), which callers must screen for themselves.
Test covers the parallelVerify = false server: forged rejected, trusted
skip and valid still land.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Audit follow-ups on the live NIP-77 index, all with the store/scan
equivalence test extended to cover them:
- Same-batch replaceable displacement left a dead id in the index.
applyAfterCommit applied all removes before all adds, so when a later
row in one transaction displaced an earlier row of the same batch (two
versions of one replaceable — the mirror-backfill hot path), the
displaced row's remove no-op'd against an index that hadn't taken its
add yet, then the add re-inserted it: the index advertised an id the
trigger had already deleted. recordAccepted now cancels the pending add
instead of queueing a remove (added is a LinkedHashSet for O(1)
cancel).
- A kind-5 that deleted nothing (a delete broadcast for events this relay
never stored — the common case) still invalidated the whole index,
forcing a full-scan rebuild under the writer mutex on the next
NEG-OPEN. DeletionRequestModule.insert now returns the rows it deleted;
recordAccepted only invalidates when that count is > 0, else records the
kind-5 as a plain row.
- The first NEG-OPEN over a corpus larger than the serve cap scanned the
whole table uncapped, built a full index that could never produce a
snapshot, and then maintained it forever for zero benefit.
liveNegentropySnapshot now caps the rebuild scan at maxEntries + 1 and
leaves the index unpopulated when the corpus is over-cap (the scan path
answers NEG-ERR, as before).
- delete/deleteExpired/clearDB invalidate() moved inside the writer mutex
so no NEG-OPEN can seal a snapshot of just-deleted rows, and no
concurrent rebuild can be discarded by a late invalidate.
Also corrects the ~40 B/event heap figure to ~140 B (IdAndTime keeps the
id as a 64-char hex string, not 32 bytes) in the strategy/plan docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Each [[mirror]] entry now takes strfry-router's dir:
- down (default): pull — subscribe to the upstream and ingest, exactly
as before.
- up: push — an in-process session on the LOCAL relay subscribes with
the same scoped filter (so backfill_seconds and filter behave
identically in both directions, and the relay's own policy chain
gates what leaves), and every matching event is handed to the
client's outbox for the upstream, which owns delivery and re-sends
across reconnects.
- both: pull and push, with echo suppression: a per-upstream LRU of
recently exchanged ids keeps an event pulled down from being pushed
straight back (and vice versa when the upstream fans our own publish
back). Eviction only costs a duplicate round trip — the stores'
unique-id constraints stay the correctness backstop.
trusted (verify skip) remains a down-only concept; the up direction
never verifies since the upstream does its own gatekeeping.
Tests: up pushes both the backfill window and the live tail; both
converges two stores with disjoint content and holds exact counts after
the echo settles (no ping-pong). Live-published test events are
genuinely signed — the local publish path verifies, which is also what
the debugging showed: the mechanism was fine, the first version of the
tests was pushing forged events into a verifying relay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog items 4-5 close. Both-variants-in-one-run A/B at 50k, repeated
with relay order reversed: every delta flipped with the order (the
second-running relay won queries and ingest latency in BOTH runs), so
readers=8 / mmap_size=256MiB / temp_store=MEMORY / periodic PRAGMA
optimize are all noise-level on container-class hardware. The config
plumbing stays (hardware-dependent, operators should measure their own
box); the example config now says so explicitly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog items 4-5 plumbing, config-gated and off by default so quartz
library defaults stay untouched for the app-side stores:
- quartz: SQLiteEventStore/EventStore accept extraPragmas (applied on
every pooled connection AFTER the built-in configuration, so they
can override it) and expose optimize() — an analysis_limit-bounded
PRAGMA optimize for incremental planner-stats refresh.
- geode: [database] readers / mmap_size / temp_store_memory map onto
the store; optimize_interval_seconds drives a maintenance coroutine
(cancelled first in the shutdown hook, before the store closes).
The A/B verdict on whether the example config should RECOMMEND any of
these on container-class hardware follows in the next commit — the
knobs themselves are operator tools worth having either way, since
mmap/temp-store value is hardware-dependent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Found while attributing the small-REQ wire floor (backlog item 6,
latency half): geode's new WireReqFloorBenchmark measured a flat
43.7 ms per REQ round trip that survived every server-side change —
store configs, dispatchers, the pump — and then vanished when the
round's preceding CLOSE was dropped. Root cause is client-side: OkHttp
does not set TCP_NODELAY, relays never answer a CLOSE (NIP-01), so its
bytes sit unACKed for the peer's ~40 ms delayed-ACK window and Nagle
holds the next REQ behind them. CLOSE-then-REQ is a Nostr client's
hottest pattern — every feed/filter switch.
relayBench's harness client already shipped a no-delay socket factory
(which is why benchmark numbers never showed the stall) but the
production clients did not. New TcpNoDelaySocketFactory (quartz
jvmAndroid, next to BasicOkHttpWebSocket) is now used by the Android
relay pool factory, the Desktop relay client, amy's relay connections,
and geode's mirror worker. Direct connections only — SOCKS/Tor paths
are untouched.
With the factory, the benchmark puts geode's ~21-row REQ at ~1.25 ms
on the wire (matching relayBench): ~0.6 ms Ktor CIO+OkHttp loopback
floor, ~0.5 ms per-REQ server work (already investigated). Per-frame
burst cost measured negligible and the pump adds ~nothing, so the
send-path latency angle of backlog item 6 is closed as not-a-problem;
its ingest-CPU share remains a separate throughput question. Findings
recorded in quartz/plans/2026-07-04-small-req-floor.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Reverts the queryRawInline fast path (fb29d655, 1b786f31) per the
keep-only-winners rule. Three relayBench runs at 50k (baseline, cap-256
where the path never engaged, cap-512 where author-archive/by-ids/
500-limit feeds genuinely took it) showed no movement outside the
container drift band — strfry's own numbers drifted ±30% between runs
and inline-eligible scenarios moved the same as ineligible ones.
The in-process win was real but small (~17%, 0.60 -> 0.50 ms per
~21-row REQ); the wire-level p50 is 1.2-1.7 ms, so the missing ~1 ms
per REQ sits in the transport (Ktor frame send path + client round
trip) — backlog item 6 territory, not dispatch. Findings, numbers, and
the do-not-retry note live in quartz/plans/2026-07-04-small-req-floor.md;
SmallReqFloorBenchmark stays as the measurement tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
The first cut's rule (every filter needs limit, sum <= 256) missed the
shapes relays actually receive: relayBench's author-archive carries
limit=500 and by-ids carries only an ids list, so the fast path never
engaged in the acceptance run. Bounds now come from limit OR the ids
count (ids are unique keys, the result cannot exceed the list), summed
against a 512 cap — a 500-row inline replay measures single-digit ms,
nothing a CLOSE could meaningfully preempt. Provably unbounded filters
(no limit, no ids — e.g. a bare tag filter) keep the launched path.
Pending: relayBench verdict decides whether the fast path stays at all,
per the keep-only-winners rule.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog item 2 (small-REQ dispatch floor), first cut. relayBench at 50k
shows geode ~2.5x slower than strfry when a REQ returns ~20 events
(author-archive 1.18 vs 0.48 ms EOSE p50) while WINNING the 500-event
scenarios — with tiny results the fixed per-REQ cost dominates. A new
stage benchmark (SmallReqFloorBenchmark) decomposes the floor: at ~21
rows, raw SQL is 0.18 ms and the remaining ~0.42 ms is live-subscription
machinery plus per-REQ coroutine dispatch.
A REQ whose filters all carry a limit summing to <= 256 now runs its
stored replay inline on the receive coroutine and only retains a
live-tail handle (SessionBackend.queryRawInline; LiveEventStore's
queryRaw is re-expressed on the same core) — no per-REQ launch, no Job,
no dispatcher handoffs, no awaitCancellation scaffolding. Unbounded
REQs keep the launched path so CLOSE can always interrupt a giant
replay. In-process time-to-EOSE for ~21-row REQs drops 0.60 -> 0.50 ms;
the bigger effect expected under concurrency (no per-REQ Job+dispatch
churn) is relayBench's to judge.
InlineReqFastPathTest pins wire-behavior parity: stored-then-EOSE
ordering, live tail delivery and CLOSE detachment, same-subId
replacement, launched fallback for unbounded and over-cap REQs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Same container, 50k corpus, geode --no-search, strfry built from
source: identical-set reconcile 41.4 ms (geode) vs 30.1 ms (strfry) —
down to ~1.4x from the campaign-opening full-scan-per-open; cold
reconcile 177 vs 112 ms (geode's first open pays the lazy rebuild);
ingest at parity in this container; storage 72.8 vs 106 MiB. Notes the
zero-copy IStorage follow-up that would close the remaining ~11 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Two relayBench runs (50k corpus, both geode variants side by side, then
order-reversed): identical-set reconcile 56/57 ms with the index vs
110/130 ms without in both orders (~2.2x, the post-write NEG-OPEN the
old cache always missed); ingest and first-ever reconcile deltas flip
with relay order, i.e. run-order noise — no regression. Also records
the run-order-bias protocol note for future A/Bs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestones 2-3 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
SQLiteEventStore now maintains the LiveNegentropyIndex when the strategy
opts in (geode's does by default; [negentropy].live_index = false turns
it off; app-side stores are untouched):
- Write paths collect a LiveIndexDelta and apply it after COMMIT while
still holding the writer mutex — rolled-back savepoint rows never
reach the index and updates land in exact commit order (also vs the
rebuild, which runs under the same mutex).
- Replaceable/addressable overwrites report the row their BEFORE-INSERT
trigger displaces via one indexed pre-SELECT that mirrors the trigger
predicate (including the NIP-01 lowest-id tie-break and the
d_tag-NULL case).
- Paths that can't itemize (kind-5, vanish, delete-by-filter/id,
expiration sweeps, clearDB) invalidate; the next NEG-OPEN rebuilds
from one scan on the writer connection.
- Until that first NEG-OPEN populates the index, ingest pays zero
bookkeeping — the populated check happens under the writer mutex so
it can't race the rebuild.
LiveEventStore serves a single unconstrained filter (the relay-relay
sync default; relayBench sends exactly this) from the index; everything
else keeps the scan+seal path and its single-slot cache.
Micro-benchmark at 50k events (LiveNegentropyBenchmark, in-container):
scan+seal cold path 80-100 ms; index post-write open 9-16 ms (~5-10x).
The relayBench A/B is the acceptance gate and comes next.
Correctness: LiveNegentropyIndexStoreTest asserts index content ==
snapshotIdsForNegentropy scan after every mutation pattern (overwrites,
losers, kind-5 rebuilds, filter deletes, mixed-outcome batches,
transactions); the full geode suite (NIP-77 + interop sync tests) runs
with the index on.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Milestone 1 of quartz/plans/2026-07-03-incremental-negentropy-storage.md.
Sorted array with binary-search insert (near-tail in the common case),
itemized remove for displaced rows, wholesale invalidate for delete
paths that can't itemize, and sealed snapshots memoized per mutation
generation — reconcile only reads, so one snapshot backs any number of
concurrent sessions and stays immutable under later writes. Over-cap
answers null so the caller keeps the strfry-parity NEG-ERR.
Not wired into any store yet; next milestones plumb displaced-row
deltas from the SQLite modules and serve index-total filters from it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Design for backlog item 3 of the relay performance campaign: an
always-current (created_at, id) index maintained from the store's write
path so cold NEG-OPENs stop paying the full scan + O(n log n) seal
(~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers
the snapshot/COW model, the removal-correctness split (RETURNING deltas
for replaceable overwrites, wholesale invalidation for rare delete
paths), IndexingStrategy gating so app-side stores are untouched, and
the micro + relayBench A/B measurement plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Measured over the real OkHttp transport (upstream killed mid-mirror and
restarted on the same port): NostrClient re-dials once on disconnect and
then falls back to its 60s keep-alive, which showed up as a 61s mirror
blackout when the immediate re-dial raced the port rebind.
- Retry pump: the worker nudges the pool every 5s. Cheap and safe —
reconnectIfNeedsTo skips connected relays and each relay's
exponential backoff (1s doubling, 5min cap) still gates real dial
attempts, so dead upstreams aren't hammered. Restart recovery drops
from 61s to ~6s in the new reconnect test.
- WebSocket pings (120s, same as the Android relay pool): without
them a half-open connection (network drop, no FIN) never fires
onDisconnected and the mirror would stall silently forever.
- MirrorWorkerReconnectTest: end-to-end over a real Ktor port — ride
through the drop, re-subscribe, drop the duplicate replay, pull the
event that only exists on the new instance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Each [[mirror]] entry takes an optional NIP-01 filter as a JSON object
string, like strfry-router's per-stream filter. It is applied twice,
matching strfry's design (cmd_router.cpp):
- it shapes the REQ sent upstream (the mirror still owns since via
backfill_seconds and strips limit — the subscription is unbounded);
- every delivered event is re-checked against it before ingest, so an
upstream answering outside its REQ — including a trusted one whose
events skip signature verification — can only inject events inside
the operator-declared scope. Out-of-scope deliveries surface on a
'filtered' counter.
Malformed filter JSON fails the boot, not the first delivery. Several
disjoint scopes for one upstream = repeat [[mirror]] with the same url.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog item 1 of the relay performance campaign: skip signature
verification for events ingested from explicitly configured trusted
upstream relays, strfry-router style.
geode now dials each [[mirror]] url from the config, subscribes to
everything newer than now - backfill_seconds, and feeds the stream
through NostrServer.ingest (same group-commit writer + live fanout as
client publishes). The NostrClient underneath owns reconnects, backoff
and REQ re-sync; duplicate replays after a reconnect are rejected by
the store's unique-id constraint and only surface as counters.
trusted = true is the per-upstream trust switch: events from that
connection skip Schnorr verify. The trusted identity is the URL this
relay dialed (TLS-authenticated for wss://), never anything an inbound
peer claims. Default is false — mirror-but-verify — and a relay with no
[[mirror]] entries behaves exactly as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Adds a public local-ingest entry to NostrServer for events that don't
arrive over a client connection (mirror/sync workers, import jobs). It
routes through the same group-commit IngestQueue and live fanout as a
client publish, and each Submission can opt out of the parallel
Schnorr-verify hook — the relay-to-relay trust model, for events
streamed from an upstream that already verified them (verify profiles
at ~8% of busy ingest CPU).
Library defaults are unchanged: skipVerify defaults to false everywhere
and nothing in the client-publish path can set it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
A single-sample wall-clock speedup assertion (>1.5x) flakes when the
shared CI machine is loaded. Apply the same pattern already used by
ParallelVerifyBenchmark: retry up to 3 measurement attempts taking the
best, keep a hard 1.05x floor that a real regression (~1.0x when dedup
saves no work) can never pass, and warn instead of fail in the noise
band. The deterministic parsed/reused-count correctness gate still
fails hard on every attempt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzMix4ZivxoYrcn88TMqtF
The worker's writer-mutex fast path can slip past its own scope
cancellation and run one batch against a store that close() already
freed. The resulting closed-connection exception escaped scope.launch
under a SupervisorJob with no handler — harmless in production, but the
kotlinx-coroutines-test global handler attributes it to whatever runTest
starts next (seen on CI as EventSourceServerTest.countUsesSource failing
with UncaughtExceptionsBeforeTest). Catch, log, and stop the pass:
nothing depends on it — the pre-search drain covers search correctness.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Two issues surfaced by rebasing onto main:
- SQLiteConnectionPool.close() freed the writer's native handle without
taking the writer mutex, so a block still running on another thread
(the deferred-FTS catch-up worker's current batch outlives its scope
cancellation) could call sqlite3_prepare on a freed sqlite3* — native
heap corruption, SIGSEGV in sqlite3DbMallocRawNN (reproduced twice in
:geode:test). close() now reclaims every reader from the channel and
acquires the writer mutex before closing handles, then releases so
stragglers get the managed closed-connection exception. Idempotent
via a closed flag.
- main's NIP-50 fix (7deda28d) strips search-extension tokens before
every store query, but the zero-decode queryRaw replay path added on
this branch predates it and passed raw filters through — an
extensions-only search would hit SQLite FTS as column syntax and
error. queryRaw now strips like query/count/snapshot do.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
FTS indexing ran inside every insert's transaction — a measurable slice
of write cost (relayBench: ~18% of ingest throughput) paid at publish
time for a feature only search queries read. It now runs as a watermark
catch-up:
- IndexingStrategy.deferFullTextSearchIndexing (default false; geode's
relay strategy enables it with search). Deferred inserts skip
tokenization entirely.
- FullTextSearchModule keeps a fts_catchup_state watermark (everything
<= last_row_id is indexed) and gains catchUpBatch(): scan past the
watermark, tokenize, advance — one write transaction per batch, so
publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at
MAX(row_id) for existing (synchronously indexed) databases.
- NostrServer runs the catch-up worker, poked by IngestQueue's new
onBatchCommitted hook, and *yields to publish traffic*: it only
drains while the queue has no backlog (IngestQueue.hasBacklog()), so
bursts ingest at no-FTS speed and tokenization fills the gaps.
- LiveEventStore drains the backlog synchronously before serving any
filter with a search term (query, queryRaw, count) — NIP-50 results
stay exactly as fresh as the synchronous path; the deferral is
invisible to correctness. Geode's existing search tests pass
unchanged through this path.
The first implementation reused reindexBatch and collapsed ingest 8x —
its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?'
matches on a plain FTS5 column, i.e. a full FTS-table scan per row
(O(n²) overall), and the worker competed with the replay for the writer
mutex. catchUpBatch therefore inserts without the delete (rows past the
watermark are never indexed; switching a DB between deferred and
synchronous strategies requires reindexAll, same rule as a
searchable-kinds change), and the worker backs off whenever publishes
are pending.
Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136
events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the
--no-search ceiling while keeping NIP-50 advertised and fresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
A NIP-77 server session rebuilt its reconciliation structure from
scratch on every NEG-OPEN: full id+created_at scan, per-entry hex
decode into a fresh StorageVector, O(n log n) seal. That cost grows
with the corpus and is paid even when nothing changed — the exact
shape of a periodic mirror's heartbeat, where N peers reconcile the
same broad filter over and over. relayBench measured 342 ms per
identical-set reconcile at 50k events vs strfry's 26 ms off its
always-current LMDB tree.
Reconciliation only *reads* the sealed storage, so one instance can
back any number of concurrent sessions:
- NegentropyServerSession now accepts a pre-sealed IStorage (the
List<IdAndTime> constructor remains and delegates).
- SessionBackend.sealedNegentropyStorage() builds + seals (null when
the set exceeds maxSyncEvents); LiveEventStore overrides it with a
single-slot cache keyed by (filter set, write generation) plus a 30s
TTL. The generation bumps on every accepted ingest; the TTL bounds
staleness from delete paths the counter can't see (expiration
sweeps, admin purges) — negentropy snapshots are point-in-time sets,
so seconds of staleness only means a peer briefly re-offers ids.
- NegSessionRegistry.open consumes the shared sealed storage;
over-cap NEG-ERR behavior unchanged (strfry parity).
relayBench gains a 'heartbeat' measurement — the identical-set
reconcile repeated immediately with no writes in between. At 50k
events: geode 342 ms -> 27.8 ms vs strfry 21.1 ms (near parity; was
13x). Cold reconciles (first open after a write) are unchanged.
Also fixes the GeodeVsStrfryNegentropySyncTest fixture to write
'nofiles = 0' so the opt-in interop test can boot strfry inside
containers with a low RLIMIT_NOFILE hard cap; the interop suite passes
against strfry v1-b80cda3 with the cache in place.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Reviewed strfry's LMDB indices (golpe.yaml) against geode's SQLite set:
the two are nearly isomorphic — time, id, kind+time, author+kind+time,
tag+time, plus conditional deletion/expiration/replaceable entries
(geode's are partial indexes, so ordinary events don't pay for them).
Nothing to drop. One real hole: strfry maintains a plain
pubkey(+created_at) index and geode had none, so an authors-only filter
(no kinds) — archive pulls, account-migration tools, 'everything by
these pubkeys' — degraded to a full walk of the time index. EXPLAIN
confirmed: SCAN query_by_created_at_id.
- quartz: IndexingStrategy.indexEventsByPubkeyAlone (default false —
clients query their supported kinds and can skip it) gates a new
query_by_pubkey_created index; DATABASE_VERSION 2→3 with an
idempotent migration that backfills it for opted-in strategies.
- geode: relayIndexingStrategy turns it on.
- relayBench: new 'author-archive' scenario — every kind by 3 *quiet*
pubkeys. Quiet is the point: prolific authors are dense in the time
index and a scan finds them quickly, which is why the suite never
caught this; sparse authors force the full walk.
Measured (50k corpus): author-archive EOSE p50 42.8 ms -> 3.6 ms (12x,
and the old path grows linearly with table size); ingest 5,337 -> 5,156
events/s (~3%, the one extra B-tree per event). strfry reference on the
same scenario: 0.52 ms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
The drain loop ran verify(batch N) and insert(batch N) strictly
back-to-back: the SQLite writer idled during every Schnorr verify and
the CPU cores idled during every commit. Split it into two stage
coroutines joined by a capacity-1 channel — a verifier that collects
and checks batch N+1 while the writer commits batch N. Same shape as
strfry's ingester/writer thread split.
Ordering (single verifier, single writer, FIFO handoff), OK-after-commit
semantics, per-row error isolation and submit() backpressure are all
unchanged; total in-flight grows by at most one batch.
Alternating A/B on the 10k corpus, 4-core host with the benchmark client
competing for the same cores: sequential 4,111/3,977/4,443 vs pipelined
4,050/5,020/5,092 events/s (~+13% mean). The overlap should widen on
dedicated relay hosts where verify has its own cores. Feature-parity
run after this change: geode --no-search 5,911 vs strfry 9,374 events/s
(1.59x, down from 2.2x at the start of the perf work).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
strfry implements no NIP-50 at all, so geode's default setup pays FTS
tokenization on every searchable event for a feature the other side of a
benchmark isn't providing. The new switch removes it cleanly:
- geode: --no-search CLI flag and [options].full_text_search TOML key
(default true). When off, the store skips the FTS index entirely,
NIP-11 stops advertising 50 (an explicit [info] nips list stays
operator-authoritative), and search filters match nothing via the
existing QueryBuilder guard. RelayIndexingStrategy becomes
relayIndexingStrategy(fullTextSearch) with the stock val kept.
- relayBench: --geode-no-search runs the geode entry with the flag (relay
named geode-nosearch in reports); README documents the apples-to-apples
rationale and a recipe for benchmarking both geode flavors side by side.
Alternating A/B on the 10k corpus: 4,767/4,792 ev/s without search vs
4,111/3,977 with — NIP-50 costs ~18% of ingest throughput at current
write-path speed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Main's giant-REQ fix (18bd3600) replaced query()'s copy-on-add immutable
dedupe set with a spin-locked HashSet, but the rebase left queryRaw —
now the default REQ path — on the old pattern, which would have
reintroduced the O(n²) crawl for large replays. Both paths now share
the mutable-set-under-spinlock shape.
relayBench's sync driver moves to NegentropySession.fromEvents(),
following the session's new List<IdAndTime> constructor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Three changes, each validated head-to-head against strfry with relayBench
(same 10k-event corpus a1cd3517a8296911, stock configs, sig verify on):
- geode: RelayIndexingStrategy turns on indexEventsByCreatedAtAlone for
the relay's stores (quartz's DefaultIndexingStrategy stays off for
client-side stores). A relay can't predict client filters, and the
cheapest REQ of all — {"limit":N} — was a full-table scan + top-N
sort: 40 ms and O(table) growth before; 11 ms and index-streamed
(first event 30 ms -> 1.9 ms) after.
- quartz: zero-decode REQ replay. Stored events now stream as RawEvent
(tags kept in serialized form) and are spliced directly into wire
frames — no tags parse, no EventFactory dispatch, no re-serialize per
row. Gated on the new IRelayPolicy.filtersOutgoingEvents capability:
policies that can veto per-event delivery (none today) keep the
materialized path; everyone else skips it. Live post-EOSE delivery is
unchanged (live matching needs Event objects).
- quartz: StatementCachingConnection wraps the pool's writer and reader
connections, replaying prepared statements instead of re-preparing per
event/REQ (eager reset on return keeps cursors from holding table
locks; a 256-statement cap bounds client-controlled filter-shape
variety). Ingest went 3,000 -> 4,700 events/s (+55%) — prepare
overhead was the single largest non-crypto write cost.
Net effect on the benchmark: ingest gap vs strfry narrowed from 2.2x to
1.8x, the firehose latency gap from 4x to 1.2x, and geode now wins 4 of
9 query-latency scenarios (notifications, hashtag, by-ids,
recent-window) plus most concurrent-throughput scenarios, while keeping
the smaller on-disk footprint. Result sets stayed byte-identical across
relays and NIP-77 sync still converges.
Measured but deliberately NOT taken: per-row SAVEPOINT elision (+4%,
within run noise — not worth weakening batch error isolation), FTS-off
(+25% ingest but drops NIP-50), --no-verify (+45% but unfair/unsafe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
New :relayBench module that boots relay implementations as real external
processes under equivalent setups (persistent storage, sig verification on,
no auth) and compares them on the same corpus:
- Ingest: receipt->queryable-by-REQ latency (publish on one connection,
hammer-poll REQ{ids} on another), OK-ack latency, and pipelined corpus
replay throughput over N connections.
- Queries: client-realistic filters derived from the corpus (home feed,
thread, notifications, profiles, hashtag, by-ids, ...), time-to-EOSE
percentiles plus aggregate events/sec under concurrency; result-set
counts are cross-checked between relays and mismatches flagged.
- NIP-77 negentropy sync between every relay pair: 80%/80% slices with 60%
overlap, reconcile timing/rounds/wire-bytes per server side, delta
transfer, steady-state identical-set reconcile, convergence verified.
- Storage footprint after full ingest.
Corpora (all cached as NDJSON + manifest with a sha256 id fingerprint so
results are comparable across runs and machines):
- synthetic (default): deterministic to the byte — seeded keys, fixed
timestamps, seed-derived BIP-340 aux nonces — with a realistic social
shape (zipf authors, threads, reactions, reposts, zap request/receipt
pairs, hashtags);
- the checked-in real dump (quartz test fixture, ~31k unique 2024 events);
- external dumps (NDJSON or JSON array, gzip sniffed by magic bytes),
e.g. the 2.1M contact-list archive, with --max-event-bytes/--max-tags
raising both the corpus filter and the strfry config together;
- live download from public relays.
Every source runs through the same preparation: dedup, drop unsigned/
ephemeral/kind-5, enforce relay ingest caps, parallel Schnorr verify,
chronological sort.
relayBench/run.sh is the one-command entry point: builds geode + harness,
resolves strfry (STRFRY_BIN, PATH, or source build into .cache), runs the
suite and renders an ANSI report with per-metric bars and winners, plus
report.md and results.json under relayBench/results/<timestamp>/.
The harness client disables Nagle (TCP_NODELAY): with the JDK default, a
REQ following the previous round's CLOSE stalls a full delayed-ACK
interval and every latency floors at ~44 ms against both geode and strfry
(verified: ~0.3 ms with it off).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46
Raw key:value tokens like include:spam reached SQLite FTS MATCH, where
the colon is column-filter syntax — any REQ carrying an extension token
died with CLOSED "no such column: include" instead of matching.
Adds SearchQuery.stripExtensions() plus Filter/List<Filter>
.strippingSearchExtensions() so EventStore users can drop the tokens
before querying, and applies them in LiveEventStore (query, count,
negentropy snapshots). Per NIP-50, unsupported extensions are ignored:
an extensions-only search becomes unconstrained, not match-nothing.
EventSource-backed search relays still receive the raw string since a
real search backend wants the extensions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Tujoyfc2kNLZNVgiLZAR7F
The 2026-06-25 translation import left U+FFFD replacement characters
in show_npub_as_a_qr_code, show_nprofile_as_a_qr_code and relay_reorder
in both values-ta and values-ta-rIN. Restore the accusative suffix
as npub-ஐ (independent vowel AI), matching the nsec-ஐ
precedent in the same files. Clears Sonar's file-encoding warning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017iWg6SxLXteav6xJJz7gKS
Delete 19 strings.xml files that contain zero string resources:
- 12 exports of Crowdin targets with 0% translation (ca-rES, cy-rGB, da-rDK,
gu-rIN, hr-rHR, iw-rIL, kk-rKZ, ks-rIN, lt-rLT, ne-rNP, sa-rIN, pcm-rNG).
NOTE: the next Crowdin sync recreates these unless the corresponding target
languages (Catalan, Welsh, Danish, Gujarati, Croatian, Hebrew, Kazakh,
Kashmiri, Lithuanian, Nepali, Sanskrit, Nigerian Pidgin) are unchecked in
the Crowdin project settings; they hold zero translations there, so
unchecking loses nothing and a language can be re-enabled when a translator
volunteers.
- 6 orphans with no Crowdin target at all (et-rEE, fo-rFO, ku-rTR, so-rSO,
ss-rZA, ur-rIN).
- values-hu, emptied earlier to fix aapt2 duplicate resources; the real
Hungarian translation lives in values-hu-rHU (until Hungarian is
consolidated to base like Czech in #3461).
Also drop the matching locales_config entries so the per-app language picker
stops offering languages that render 100% in English, including the dangling
"ar" entry (values-ar has never existed; ar-SA remains) and the duplicate
"hu" entry (hu-HU remains).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KhEN93aKyQCFVpLUgRNXid
davotoula
Vitor PamplonaandGitHub
Claude
vitorpamplonaandgithub-actions[bot]
davotoulaandgithub-actions[bot]