A relay sends its matching events before its EOSE, so both an event and
the relay's completion can sit buffered in their channels at the same
time. The select() over the two channels picks a ready clause at random,
so it could process the doneChannel completion first, empty `remaining`,
and exit the loop while the matching event was still unread — returning
null instead of the event.
On a relay completion, drain the event channel first and treat any
already-buffered event as the result before marking the relay done.
Clears real Kotlin compiler warnings surfaced across quartz, cli,
relayBench, amethyst, and desktopApp:
- quartz Sha256/EventHasher/ScratchLocal: ThreadLocal.get() is nullable
in Kotlin; assert non-null (withInitial never yields null).
- quartz GitHttpClient: PriorityQueue.poll() under isNotEmpty() is
non-null; assert it.
- relayBench CorpusDownloader: drop redundant !! on smart-cast Long;
Jackson fields() -> properties().
- cli GrapeRankCommand: drop redundant ?. where latest is smart-cast.
- PodcastRemoteContent: OkHttp body is non-null; drop dead elvis.
- Dead/redundant expressions: remove no-op when-branch values and a
redundant trailing Unit (HomeScreen, LocalCache, EmbeddedTabLayer,
ParticipantHostActionsSheet, NestActionBar, ControlWhenPlayerIsActive,
ShareNoteAsImageScreen exhaustive-when else).
- CalendarEventDetailScreen / SetPasswordDialog / ProfileClinkOfferResolver:
drop always-true conditions (reorder to keep smart-casts).
- WalletColumnScreen: OkHttp body non-null; drop unreachable null-guards.
- PcmTapRegistry: the @OptIn used androidx.annotation.OptIn, which does
not opt into Kotlin's ExperimentalCoroutinesApi; use kotlin.OptIn.
- GitRepositoryScreen: suppress the standard ViewModel-factory cast.
- PushNotificationReceiverService: suppress override-of-deprecated.
- Desktop GlobalScope call sites: @OptIn(DelicateCoroutinesApi::class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
- GrapeRankPublisher: dTag() is non-null (""), so the Elvis on the
grouped target was dead code; skip blank targets via ifBlank instead.
- amethyst: migrate deprecated resourceConfigurations to
androidResources.localeFilters (same locale qualifiers).
- desktopApp: replace deprecated compose.desktop.uiTestJUnit4 accessor
with the direct org.jetbrains.compose.ui:ui-test-junit4 dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
The two-pass deletion convergence is protocol logic, not CLI assembly, and the
geode mirror is a near-term second consumer — so move it out of SyncCommand into
a reusable accessory alongside the rest of the negentropy family.
quartz: negentropySettleDeletions(relay, filter, store, sendUp, applyDown, …) —
re-reconciles after a content settle and resolves only the residual: publishes
our covering deletions up (sendUp) and/or ingests the relay's kind-5 down
(applyDown, vanish never auto-applied), looping until a round resolves nothing.
Returns DeletionSettleResult(sentUp, appliedDown, rounds). Everything it needs is
already quartz (negentropyReconcileIds, fetchAll, deletionsCovering,
publishAndConfirm, Event.verify, IEventStore), so it carries no CLI dependency.
SyncCommand's pass 2 collapses to a single call; pass 1 (content) is unchanged.
Catalogued in the accessories README.
Tests: DeletionSyncTest drives the accessory end-to-end both ways (sendUp → relay
converges to gone; applyDown → local converges to gone), on top of the existing
deletionsCovering unit + manual-wiring cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Replace the per-need-event fetch (which pulled the whole need set just to read
metadata — an O(db) regression on large syncs) with a second reconcile pass over
the residual, per the "settle, then diff, then explain what didn't converge" idea.
Pass 1 is the plain content sync again (drain needs, publish haves) — zero
deletion overhead. Pass 2+ re-reconciles; the leftover diff is exactly the
deletion mismatches, and only that (tiny) set is fetched:
- residual need (relay has it, we still lack it after --down) = we deleted it →
publish our covering deletion up so the relay drops it;
- residual have (we have it, relay still lacks it after --up) = the relay deleted
it → pull the relay's covering kind-5 down and apply locally (vanish is NOT
auto-applied on pull — account-wide blast radius).
Loops until a round resolves nothing (converges + self-verifies).
So `amy sync` makes the relay honor our deletions; `--up` makes us honor the
relay's; `--up --down` converges both ways. Cost is one cheap reconcile + the
residual regardless of database size — the large-DB bottleneck is gone by
construction, not by heuristics.
quartz: deletionsCovering is now source-agnostic (takes a query lambda) so the
same coverage rule runs against the local store (up) or the relay (down); the
IEventStore overload is the local convenience.
Tests: DeletionSyncTest gains the down-direction end-to-end (relay deleted →
local removes) alongside the up-direction and the per-form unit cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
MlsGroupState reconstructed the SecretTree from encryption_secret alone, so
every restore rewound each sender's generation counter to 0. The restored
local member then re-emitted generation 0 within the same epoch — reusing the
AEAD key+nonce (a confidentiality break) and getting rejected by strict
receivers (openmls / MDK / Whitenoise) that forbid generation reuse, per
RFC 9420 §9.
Two parts:
- Persist per-sender ratchet positions. SecretTree gains export/importSenderStates;
MlsGroupState carries them as an optional field (STATE_VERSION 2, v1 blobs still
decode as empty = legacy behavior); saveState/restore wire them through.
- Persist after every send. MlsGroupManager.encrypt now saves group state, not
just commits — application sends advance the ratchet but previously never hit
the store, so a restart between two commits still reset it.
Regression tests: a peer that consumed generation 0 accepts the restored
sender's next message (single + multi-send), encrypt persists the ratchet
between commits, and a v1 blob still decodes/restores.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G6uT4xzjty1xosZBkb3sHA
Speeds up a from-scratch GrapeRank crawl ~25-30% at equal completeness on a
drift-controlled A/B, by:
- lowering the per-relay concurrent-sub cap 100→16 — the old 100 drowned popular
relays (damus/nos.lol) in concurrent giant REQs, driving them to time out; 16
restores their responsiveness (damus yield 0%→14%) and is still generous for
the single-user fetches other amy commands do,
- shedding proven-dead relays from the kind:10002 discovery sweep instead of
re-hammering refusing indexers every round,
- trimming the sharded backbone sweep 6→2 rotations (Phase A was ~36% of the
crawl at half Phase B's per-list efficiency; 2 clears the bulk with no
completeness loss).
Also adds relay observability under --diagnose to document how relays reply to
our queries: per-relay telemetry (outcome mix, yield, latency, worst time-sinks),
a LIVE / THROTTLED / UNREACHABLE classification table with the limits we settled
on per relay, and per-round Phase-A/Phase-B timing; plus contact_lists_by_hop in
the sync result for per-hop completeness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-shot/high-level relay ops (fetchAll, fetchFirst, fetchAllPages,
publishAndConfirm, count, negentropy sync/reconcile, …) are INostrClient
extension functions spread across ~8 files with no index, so they don't surface
under "usages of NostrClient" or in completion — easy to miss and re-implement
(as just happened with a bespoke fetchRaw duplicating fetchAll).
- Add accessories/README.md cataloging each public extension with a one-line
"use when".
- CLAUDE.md (Feature Workflow): point at that package/README before hand-rolling
a subscribe/REQ/publish loop.
- relay-client skill: add a Related note steering headless/one-shot callers to
the accessories instead of Subscribable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Refine the sync deletion rule to what was asked: for the events the relay HAS
that we LACK (the reconcile need set), publish only the local deletions that
would actually make the relay remove them — and nothing else, not other
deletions by the same author.
Determining coverage needs the need event's author/address/created_at, which we
don't have for an id we lack, so we fetch the need events (raw — no verify, no
store) purely for metadata. quartz gains IEventStore.deletionsCovering(events,
relay), which maps server-held events to the covering local deletions across all
three forms:
- NIP-09 id-based: a kind-5 with an `e` tag naming the event id;
- NIP-09 address-based: a kind-5 with an `a` tag naming the event's
addressable/replaceable coordinate, at/after it (created_at <= deletion);
- NIP-62 vanish: a kind-62 by the event's author, targeting this relay, issued
after it (created_at < vanish).
SyncCommand's need workers now fetch each need batch once (Context.fetchRaw),
publish its covering deletions (deduped across workers), and — when --down —
store the rest; anything we deleted is rejected by the store's own tombstone.
Nothing is pulled down or applied locally, so it cannot over-delete the store.
DeletionSyncTest covers each form (with cutoff and wrong-relay negatives) plus an
end-to-end reconcile → cover → publish that removes the note on the relay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Per the actual requirement, deletion propagation is exactly: for the ids the
relay HAS that we LACK (the negentropy need set), if we hold a kind-5 deletion
targeting one of them, publish that deletion up — so a note we deleted is
deleted on the relay too instead of being re-downloaded. Only the need ids,
only kind-5, up only.
This removes all the machinery the earlier approach accreted and that the audit
flagged as over-broad / data-loss-prone:
- deleted NostrClientDeletionSyncExt (the bidirectional side-channel, author
scoping, vanish gating, kind selection);
- reverted geode MirrorWorker to base (no deletion side-channel, live-sub
changes, catch-up ordering, or convergence changes);
- dropped the 3-phase SyncCommand flow (deletions-first pull, author-scope
derivation, reject-reaction backstop, --sync-vanish, deletions_* output).
The new path pulls nothing down and applies nothing locally, so it cannot
over-delete the store, and it needs no author scoping — the need set already
bounds it. Kind-62 is intentionally excluded: a vanish is not "of an id".
Emits deletions_sent. DeletionSyncTest now exercises the exact wiring
(reconcile → look up local kind-5 by its e tag for the need ids → publish),
including the negative case (a need id we never had sends nothing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
An audit (adversarial-verified) found the deletion side-channel over-deletes
and over-propagates. Root cause: deletionSideChannelFilter fell open to
authors=null for any non-author-scoped content sync, so `amy sync --kind 1`
reconciled the RELAY'S ENTIRE kind-5/62 history and applied it to the personal
FsEventStore — every kind-5 deleting its targets + installing an id-tombstone
for every target id, every ALL_RELAYS kind-62 wiping all of a pubkey's events
(all kinds), and pushing our whole local deletion history up. Data loss plus a
full-history reconcile on every scoped sync.
Fixes:
- Bound the side-channel to the authors we actually hold content for (filter
authors ∪ local matched-set authors), never the relay's population. Skip when
that scope is empty; Phase 3's reject-reaction covers the author-less case.
- Kind-5 (precise, owner-scoped) propagates by default; kind-62 vanish is opt-in
via --sync-vanish (its blast radius always exceeds a content sync's scope).
- excludesDeletionKinds() now checks each deletion kind independently
(`--kind 1,5` no longer silently drops kind-62); the side-channel reconciles
only the missing kinds.
- amy Phase 1 is best-effort: a deletion-reconcile failure records deletions_error
and falls through to content, never aborting the primary sync (matches geode).
- Mirror up-catch-up converges on whether a PUBLISHABLE event was pushed, not raw
haveCount — a vanish targeting another relay no longer burns all 8 rounds every
startup. Mirror keeps its (correct) global scope for relay-to-relay replication.
Helper API: negentropyPropagateDeletions gains scopeAuthors + deletionKinds;
deletionSideChannelFilter takes authors + deletionKinds and returns only the
missing kinds. Tests updated for the new semantics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
NIP-77 reconciles by event id over the content filter, so a scoped sync
(`--kind 1`) never carries the kind-5/62 that deletes one of those notes:
the deletion stays stuck on whichever side issued it while the target
lives on forever on the other. Add a deletion side-channel that reconciles
kinds 5 & 62 on their own, independent of the content filter.
quartz: NostrClientDeletionSyncExt — DELETION_PROPAGATION_KINDS,
Filter.excludesDeletionKinds()/deletionSideChannelFilter() (kinds 5/62 scoped
to the same authors, no time window since a deletion's created_at is not its
target's), shouldPropagateDeletionUp() (kind-5 always; kind-62 only to a relay
it targets, honoring the vanish's declared relays), and
negentropyPropagateDeletions() — one bidirectional reconcile that streams
have→upload and need→download.
amy sync: run the side-channel bidirectionally regardless of --up/--down
whenever the filter excludes 5/62; emits deletions_{need,have,downloaded,
uploaded}; --no-sync-deletions opts out.
geode MirrorWorker: thread a per-upstream deletionScope through both catch-up
phases and both live subs (down + up), in the mirror's configured direction;
relax down containment to accept in-scope deletions, gate kind-62 pushes by
target relay, and carry the deletion filter on re-subscribe so a reconnect
never drops it.
Tests: DeletionSyncTest (up/down propagation + filter/vanish-gate units) and
MirrorDeletionSyncTest (a kind-scoped down mirror still removes the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The crawler only logged once per round, so a deep hop (140k users, minutes of
work) went silent between lines. Add a heartbeat ticker on the background scope
that emits every few seconds with the current round's completion (a real X/Y %
against the round's known pending target), a rolling fetch rate + rough ETA for
it, and live counts (events stored, relays parked/dead) — and a "finishing"
line while draining the parked tail. Scoring stays sub-second, so it keeps its
per-sweep lines and needs no ticker.
Also make crawl()'s builder nullable: null runs a persist-only pass (every event
still lands in the store, the frontier still expands off each contact list) with
no in-memory graph — the basis for a `sync` that loads data without scoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-relay concurrency/rate throttle notices and the "Rejected <url>"
normalizer messages fire constantly during a large crawl (thousands of
rejected/throttled relays) and are operational detail, not warnings. Move
them from Log.w to Log.d so they stay available under debug logging without
flooding a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The park window's timeout was absolute from subscription open, so a relay
still actively streaming a large result set once it passed parkTimeoutMs was
unsubscribed and its untransmitted tail lost. Reset the window on every
incoming event (a conflated activity signal drives a select against the
terminal deferred), so a parked subscription is closed only after
parkTimeoutMs of actual silence — never while events are still arriving. The
fast window stays absolute: it only decides when to hand a slow relay to the
background park lane, which loses nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
- Replace unused Unit/null expressions in statement-position when branches
with empty blocks (CommandSerializer, QuicConnection, QuicConnectionParser,
Http3FrameReader, WtPeerStreamDemux).
- Suppress DEPRECATION on KindNames.names, which intentionally registers the
deprecated GitReplyEvent and TorrentCommentEvent kinds for display.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nqdy4VTLKidUWzGTJPja9
The crawl was round-synchronised: each hop drained all its relays and only
started the next hop after the slowest one reached EOSE or the timeout. That
made waiting for slow-but-alive relays expensive — every hop paid its slow
tail before the next hop's fast relays could begin — so a long timeout for
completeness cost ~2x wall-clock (measured), and a short one dropped the slow
relays' data.
Diagnostics on a ~190k-user crawl showed the genuinely-slow set is a stable
~30 relays that DO reach EOSE, just in 5-25s. So decouple the two concerns:
- drainGated now drains on the FAST `timeoutMs` that sets the round cadence. A
relay still streaming when it elapses is not cut but PARKED: it hands its
open subscription to a background scope (releasing its AdaptiveRelayLimiter
permit so the round moves on), keeps receiving for up to the new
`parkTimeoutMs`, and its late events are persisted + its late contact lists
pushed to a crawl-wide lateHarvest channel.
- The round loop folds late harvest into the graph between rounds and won't
converge until the frontier is empty AND no relay is still parked — so the
crawl waits for slow relays for completeness without paying that wait in each
round's wall-clock.
Graph state stays single-writer: parked coroutines only touch the store,
seenIds, and the channel — never hopOf/done/builder. Persistence moved from a
single per-drain consumer to a shared `persist()` that fast and parked units
both call; crawl-wide dedup is now race-safe via ConcurrentSet.add's atomic
test-and-set (an id is added only after a good signature, so no duplicate
reaches the store's UNIQUE constraint and a forged copy can't suppress the
genuine one).
Also carries the --diagnose slow-relay logging (relay + filter + elapsed for
every slow/parked REQ, so a human can replay it) and keeps --drain-concurrency
at the validated default of 24 (an A/B at 64 was ~2x slower with more dead
relays). New --park-timeout flag (default 40s; set <= --timeout to disable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Add a --diagnose slow-relay log: every content drain that reaches its terminal
(EOSE or timeout) slower than SLOW_DRAIN_LOG_MS, or times out entirely, is
recorded with the offending relay URL, the failure/EOSE reason, elapsed ms, and
the exact filter shape (kinds + author count + first authors). This lets a human
replay that precise REQ later to understand why the relay lags. Gated on
--diagnose so there is no per-group timing/collection overhead otherwise.
Make the content-drain fan-out configurable via a new --drain-concurrency flag
(Config.drainConcurrency), replacing the DRAIN_CONCURRENCY constant. Default
stays at the validated 24: an A/B at 64 ran ~2x slower with more dead relays
(a higher global fan-out re-floods busy hubs faster than the per-relay demotion
catches up), so the flag is a probe knob, not a speedup. Client WebSocket pings
were also tried and reverted — busy-but-alive relays don't reliably pong while
their query handler runs, so pinging just cut them as dead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-user retry counter was blunt: it bumped attempts whether an outbox was
dead, timed out, or cleanly EOSE'd with no event — so a straggler kept being
re-queried against a live relay that had already definitively answered it lacks
their kind:3. Distinguish the cases: drainGated now reports the relays that fully
EOSE'd (answeredOut); the consumer records, per user, the relays that answered but
did not return their contact list (askedEmpty); routeByOutbox excludes those from
the user's candidate relays. A timed-out relay is never added (it might just be
slow — still worth a retry), only a clean-EOSE-empty one; dead relays stay pruned
as before.
Measured on --max-hops 3: redundant fetching dropped ~8% (74k -> 68k events
stored). It does NOT move the wall-clock tail, though — that tail is dominated by
timeout/dead outboxes (the retryable case), not EOSE-empty relays. The wall-clock
lever remains the timeout retry budget (MAX_OUTBOX_ATTEMPTS / drain timeout).
Mirror the crawler extraction on the emit side: the NIP-85 kind:30382 card
reconcile + publish logic (existingCards read-back, rank-diff upsert, stale-card
kind:5 retraction batched under the 64KB event cap) moves out of GrapeRankCommand
into a reusable GrapeRankPublisher in quartz experimental/graperank. It takes an
IEventStore for the prior-card read-back and an injected publish function
(event + relays -> per-relay ack), so the store/relay wiring stays in the app
while the reconcile logic is reusable (e.g. by the Android app).
GrapeRankCommand is now a thin orchestrator: crawl (GrapeRankDataCrawler) ->
score (GrapeRank) -> publish (GrapeRankPublisher). The account-specific bits stay
in the CLI: operator-key derivation, the observer's kind:10040 discovery pointer,
and the operator/register/providers sub-verbs.
The crawl re-verified and re-inserted the same event many times: the outbox
model mirrors each event (especially kind:10002 relay lists) across relays,
indexers, and rounds, but dedup lived in a per-drain SeenIds, so only the copies
within one drain were caught. Add a crawl-wide seen-set (thread-safe ConcurrentSet
of event ids, shared across all 24 concurrent drains and every round), checked
before verify and added only after verify so a forged copy can't suppress the
genuine one. Group-commit the store writes via IEventStore.batchInsert instead of
one transaction per event.
Measured on a from-scratch --max-hops 3 crawl: events actually verified+stored
dropped ~34% (112k -> 74k) and verify time fell in lockstep. The write path now
also reports verify/insert timing + events_stored in Stats, exposed as verify_ms/
insert_ms/events_stored on the CLI, and takes an --insert-batch knob.
Finding: with the work reduced, inserts serialize on SQLite's single writer
mutex rather than transaction count, and the crawl's wall-clock ceiling is the
drain-timeout retry tail on dead outboxes, not the disk.
Cleanups from a reuse/simplification/efficiency/altitude review of the crawler
extraction:
- Collapse the redundant `discovered` set into `hopOf` — a user is discovered
iff it has a hop stamp, so the two always held the same key set. The frontier
is now `hopOf.keys`; one fewer collection to keep in sync.
- Drop the unused `Stats.discovered` / `Stats.deadRelays` fields (no reader —
the CLI reports rounds / relaysContacted / hopHistogram / downloadMs).
- Extract a single shared verify-then-store sink, `IEventStore.verifyAndInsert`,
and route both the crawler and `Context.verifyAndStore` through it instead of
each carrying its own verify + insert + UNIQUE-swallow copy.
- Fast-path the present-key hit in `ConcurrentMap.getOrPut` (jvmAndroid) so the
crawl's hot relay-hint accumulation stops allocating a mapping-function closure
on every call.
- Hoist the repeated `crawlStats?.hopHistogram` null-plumbing in GrapeRankCommand.
The web-of-trust crawl (~400 lines: outbox routing, sharded backbone sweep,
Phase-B worker pool, relay-list discovery, report-deletion fetch, warm pool)
was making the CLI's GrapeRankCommand unmaintainably large. Move it into a
reusable, KMP-portable GrapeRankDataCrawler in quartz commonMain.
The crawler takes a NostrClient + IEventStore + AdaptiveRelayLimiter, injected
relay policy (discovery + content-fallback sets, since those defaults live in
app code, not the protocol library), and a log callback; it streams contact
lists into a TrustGraphBuilder and returns crawl Stats. GrapeRankCommand shrinks
to arg-parsing + offline load + scoring + publish + sub-verbs, delegating the
online path to the crawler.
To reach commonMain (portable to every target, incl. iOS):
- Add ConcurrentMap / ConcurrentSet expect classes under utils/concurrent, with
jvmAndroid actuals (java.util.concurrent) and native actuals (copy-on-write
over kotlin.concurrent.atomics.AtomicReference, mirroring ConcurrentHashCache).
commonMain has no ConcurrentHashMap, and the crawl's producer/consumer/drain-
worker state needs atomic getOrPut/merge plus a concurrent set.
- Move AdaptiveRelayLimiter and DrainFailure/classifyDrainFailure from cli to
quartz commonMain (java atomics -> kotlin.concurrent.atomics, ConcurrentHashMap
-> ConcurrentMap, System.currentTimeMillis -> TimeUtils.nowMillis, stderr -> Log).
- The gated drain (REQ-size splitting, per-relay permits, verify+store) moves into
the crawler; Context.drain loses its now-unused gatePerRelay path.
Net: cli -1077 lines; the crawler + relay machinery are now reusable by the
Android app. Adds ConcurrentCollectionsTest; verified via JVM + commonMain
metadata compile, the wot/graperank suites, and a bounded live crawl.
BasicRelayClient collapsed a connection failure into a message string built
from the throwable's text alone. Message text is localized and inconsistent
across platforms, so a listener can't reliably tell a busy relay (a connect
timeout) from a dead one (bad domain / TLS misconfig) from it. Always append
the exception class name (SocketTimeoutException / UnknownHostException /
SSLHandshakeException / ConnectException …), which is stable, so listeners
can classify the failure by type. Message text is preserved; the type is
added in parentheses. Updated the one test that pinned the old format.
The GrapeRank engine, TrustGraph (compact int-CSR) and TrustGraphBuilder
are pure Nostr-social-graph computation over HexKeys — no UI, no Compose,
and no commons-only dependency. They're a utility for implementing the
NIP-85 rank assertions quartz already models, so they belong in quartz
rather than commons. Move commons/wot -> quartz experimental/graperank
(package com.vitorpamplona.quartz.experimental.graperank), including both
commonTest suites, and repoint the CLI import. TrustGraphBuilder was already
protocol-agnostic (takes HexKey lists; the caller does the event->edge
extraction), so nothing had to change but the package. Makes the algorithm
reusable by the Android app for spam/trust filtering without pulling in
commons.
Audit follow-ups before merge:
- amy fetch default limit is now the same on both paths: absent --limit → 100
for plain AND --paginate (previously --paginate silently meant "unbounded").
`--limit 0` is the explicit opt-in to drain everything (unbounded); negative
is rejected. The effective limit is carried on the filter so both paths agree.
- drainAllPages sizes its SeenIds for CLI-scale fetches (initialSlotsPow2 = 12,
~64 KB) instead of the large-walk default (~16 MB eagerly allocated per fetch);
it grows if an unbounded drain needs it.
- fetchAllPages clamps the inclusive advance to `min(pageMinTs, boundary)` so a
misbehaving relay that answers with an event past the requested `until` can't
push the cursor upward — the boundary dedup and termination rely on `until`
never increasing. No-op for honest relays (they only return events ≤ until).
Verified live: default and --paginate both cap at 100; --limit 50 → 50; --limit 0
--paginate drains the full window (>100); paging tests + SeenIds tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Two changes to the paginated fetch path:
- Cross-relay dedup before verify. drainAllPages' single consumer now runs a
SeenIds filter: the same widely-mirrored event arrives once per relay, and the
repeats are dropped BEFORE the expensive Schnorr verify + store instead of
after (they were only trimmed by FetchCommand's distinctBy). An id is marked
seen only once it verifies, so a forged copy (valid id, bad sig) delivered
first can't suppress the genuine one from another relay. Adds SeenIds.contains
(peek without recording) for that check-then-add.
- `amy fetch --paginate` no longer forces a --limit. With --limit N it still
pages up to N per relay; WITHOUT --limit it drains the whole filter unbounded
(the filter's null limit flows straight through). Plain (non-paginate) fetch
still trims to the default 100.
Verified live: unbounded --paginate over a ~20-min nos.lol firehose window
returns 406 (all unique, 3s) vs the old 100 cap; --limit 50 caps at 50; default
caps at 100; cross-relay fetch stays count==uniq.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Drop @Synchronized from add/reset/size: SeenIds is now documented as
single-writer (not thread-safe). Callers dedup across concurrent relay
producers by funneling events into one consumer that owns the instance — the
one-consumer ingest pattern used elsewhere — which keeps a single global set,
stays lock-free, and lets resize run without coordination.
With the JVM-only @Synchronized gone the class is pure common Kotlin
(LongArray + Hex.readLong), so it moves from the jvmAndroid source set to
commonMain and is now available on every target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
A run-scoped "already seen this id" filter for large, mostly-duplicate id
streams (a broad relay walk re-receiving the same event from many relays).
Keys on the first 128 bits of the id, sliced straight out of the hex with
Hex.readLong (table lookups, no parse, no allocation), in one open-addressed
LongArray — ~16 bytes/entry and the 64-char String is never retained, so tens
of millions of ids cost ~1 GB instead of a HashSet<String>'s ~6 GB. add() is
O(1) and synchronized.
Lives in the jvmAndroid source set (uses @Synchronized; a 40M-id walk is a
server-side concern). Ports the caller's implementation with the
parseUnsignedLong hot path swapped for Hex.readLong (~45-70 ns/op cheaper).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
fetchAllPages advanced with `until = oldest - 1` (exclusive) and no dedup. That
skips any event sharing the boundary second that didn't fit in the page — which
happens at *every* page boundary landing inside a second, not just pathological
dense ones — silently dropping events. An in-process probe with no second denser
than the relay's page cap still lost one event straddling the boundary.
Page inclusively now: `until = oldest created_at of the previous page`, and drop
the re-fetched boundary events by id. The dedup set is bounded to just the current
boundary second (`until` only decreases, so duplicates can only recur there), so
memory stays O(one second), never O(total).
A single second denser than the relay's page cap can't be drained (its tail is
unreachable — no client-side fix; raising the request limit is futile since we
already send one above the relay's cap). Once a page yields nothing new we step
strictly past that second so paging keeps progressing to older events instead of
stalling forever.
Tests: boundary-straddle retrieves all 6 (was 5); dense-second-beyond-cap steps
past without stalling and still delivers the neighbours. Verified on live relays
(strfry / nostr-rs-relay / khatru): ground-truthing each dense internal second
against the paginated set shows no gaps, incl. a 36-event second fully retrieved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
The search-single-page logic left two lists with different roles: the listener
counted matches over the full `pagedFilters` (including a search filter already
dropped from paging) while the subscription only sent `remainingFilters`. That
worked — the dropped filter's count was unused and `advancesCursor` kept its
hits off the cursor — but it read as if a non-subscribed filter still mattered.
Collapse to one `activeFilters` list (index + filter) that is both what we
subscribe and what the listener iterates, so counting can't drift from what was
asked. Behavior is identical; the multi-filter and search tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Amy's one-shot queries all go through Context.drain, a single REQ drained to
EOSE — so a relay that caps its REQ response (strfry's per-REQ limit, ~500)
silently truncates the result with no way to page past it.
Extract the per-relay fetchAllPages fan-out that already lived privately in
EventSync into a reusable quartz accessory, fetchAllPagesFromPool: a
sliding-window pool (maxConcurrentRelays) that paginates each relay on its own
`until` cursor, tags every event with its source relay, and does not dedup
across relays. EventSync now delegates to it (its private downloadPool/
downloadFromRelay are deleted — no behavior change: perRelayFilters is already
ordered by and complete over the relay list).
Add Context.drainAllPages, the paged sibling of drain: same verify+store and
per-relay tagging, but fully draining sets larger than one REQ. Wire it into
`amy fetch` behind --paginate/--all (filter mode only), pushing the limit into
the filter so paging stays bounded. sync (NIP-77) and fetch stay separate
interfaces.
Tests: fetchAllPagesFromPool fan-out/tagging/no-cross-relay-dedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
NIP-50 search results are ranked by relevance, not created_at, so paging a
search filter by an `until` cursor silently degrades a top-N search into a
full time-walk of the corpus — and never terminates against a relay that
runs FTS over its whole corpus regardless of `until`.
fetchAllPages now queries a `search` filter on its first page only: it is
dropped from every later page and its hits neither advance nor drag back the
`until` cursor that co-resident non-search filters page with. onNewPage also
moves below the empty-page break so it never announces a page that isn't
fetched. Adds a test proving a search filter returns a single relay page
while a plain filter over the same capped relay still pages through the set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Add Hex.toLong64/toLong128/toLong256 (plus the shared readLong helper) to
pack the first 64, 128 or 256 bits of a hex string into a single Long, two
Longs or four Longs. Big-endian, allocation-light, branch-free — 16 table
lookups and shifts per word. Useful as cheap map/set keys or bucket hashes
for 32-byte event ids and pubkeys without decoding to a ByteArray.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019CU1wR6NvQmdmNNsPe9GuN
- Use the app's cached stringRes helper instead of raw stringResource,
matching the dominant convention in ui/note/types
- Drive the icon animation from the Compose frame clock (withFrameMillis)
instead of a delay loop, so the ticker suspends whenever the composition
stops drawing rather than waking the main dispatcher 4x/sec from the
back stack
- Drop the unconsumed memoryCardId/blockState/blockHash accessors; the
tag schema stays documented in the class KDoc
- Document the frames arrays as frozen: mutating them in place would
silently break the @Immutable skip contract; build a new instance
to change pixels
- Replace the API-29-deprecated Bitmap.createBitmap(IntArray, ...)
overload with createBitmap(w, h, config) + setPixels
A relay that refuses negentropy with a NEG-ERR whose reason merely starts
with "blocked" (e.g. "blocked: Negentropy sync is disabled" from a relay
that has NIP-77 turned off, or an auth/ban refusal) was misclassified as a
strfry `max_sync_events` overflow by `isOverflow`. Overflow triggers
created_at window-splitting, so every split re-opened, was refused again,
and the splitter fanned out breadth-first across the whole created_at range
(~2^31 windows). The call therefore never threw NegentropySyncException (so
`negentropySyncOrFetch` never took its paging fallback) and never tripped
the idle watchdog (the relay answered every NEG-OPEN promptly), so it hung
indefinitely. A second relay whose refusal string did not start with
"blocked" fell through to `Failed` -> paging and completed, which is why the
two behaved differently despite advertising the same NIPs.
Narrow `isOverflow` to genuine "result set too large" signals only; a bare
`blocked:` refusal now maps to a hard failure and fails over to paging.
Adds a regression test driving an in-process relay that refuses every
NEG-OPEN with "blocked: Negentropy sync is disabled" while still serving
plain REQ: negentropySyncOrFetch now pages and delivers every event instead
of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015UF3eh76rRiwAuPg32rwiz
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):
geode import [--db …] [--no-verify] [FILE…] # files, or stdin when none
geode export [--db …] # NDJSON to stdout
Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.
This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.
Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.
Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:
- id tie-break: thread the IndexingStrategy through run()/prepareStreams
and append ", id ASC" to the per-stream ORDER BY when
useAndIndexIdOnOrderBy is set — matching every sibling query in
QueryBuilder. The id-indexed order comes straight off the index (no
extra sort, lazy cursor preserved), so the merge now matches the
single-SQL path byte-for-byte on same-second same-author events. Without
the id index the tie stays in rowid order (a valid NIP-01 newest-N);
documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
kinds, so a filter with a repeated pubkey can no longer open two
identical cursors and emit each matching event twice (the single-SQL
IN(…) path already dedups).
Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).
MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:
- Catch-up reconciles the local set against the upstream over the
[now - backfill_seconds, now] window and downloads only the diff via the
existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
overrun us) and it completes the pull. Reconcile-against-local means a warm
restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
the sync's job). The windows overlap at `now`; the store's unique-id
constraint dedups the seam.
Changes:
- quartz: add a backward-compatible `localEntries` param to the public
negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
trusted-scope re-check as the live path; failure is non-fatal). New `store` +
`negentropyBackfill` ctor params; default off so existing live-REQ tests are
unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
preloading historical events a live-only sub cannot deliver, then proves the
post-boot event still arrives (3000 catch-up + 1 live = 3001).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.
When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the full IngestQueue pipeline — parallel verify, greedy-drain group
commit, a concurrent deferred-FTS catch-up worker taking the pool writer, and
windowed concurrent submits via the trusted (skipVerify) mirror path — over the
clean 200k corpus, asserting every Accepted regular event is queryable after.
Passes (199,612 in/accepted/stored, 0 lost), together with BatchInsertLossTest
proving geode's ingest is lossless at every in-process layer. The geode↔geode
sync event-loss therefore lives above the store+queue — in the real Ktor
WebSocket path or the benchmark harness's hand-rolled delta transfer, which the
in-process paths bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the exact store path — batchInsertEvents with geode's indexing strategy
in 64-event batches — over the clean 200k corpus and asserts every Accepted
kind-1 (regular, never replaced/deleted here) is queryable afterward. Passes
(199,612 in, 199,612 stored), which is the point: it proves the sequential
store path is lossless and narrows the geode↔geode sync event-loss to the
concurrent IngestQueue pipeline (async verify + greedy-drain batching +
concurrent WS submits), not the store itself.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Claude
Vitor PamplonaandGitHub
davotoula