Drop the flaky publish-into-large-relay warmup from DeletionSettleBenchmark
(it timed out the measured reconcile at N=100k — the container noise the
docstring already warns against) and remove the throwaway ScratchSettleTiming
investigation tool. Record in the docstring what the phase breakdown proved:
the settle's extra time over a bare reconcile is O(K) relay-ingest of the K
residual deletions, dominated by one-time JVM/JIT warmup of the publish path
(consecutive K-note batches fell ~3100->570ms), not the deletion algorithm and
not O(N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
relayBench measures relay-to-relay reconcile, not the amy/quartz client feature,
so the deletion-settle perf claim belongs in an in-process benchmark of
negentropySettleDeletions itself.
Models the post-content-settle state: a relay with N notes, a local store with
the same N except K it deleted (keeping the K kind-5s). The reconcile residual is
exactly those K, so a sendUp settle fetches K — not N. Asserts residual==K,
sentUp==K, and relay convergence (correctness guard at the small default N),
and prints one-reconcile vs full-settle so the deletion overhead reads as
"a few reconciles + K", never "+ a content re-download". Measured:
N=2000 K=20: settle ~2x one reconcile, fetched K=20 not N
N=100000 K=20: settle ~5x one reconcile, fetched K=20 not N=100000
The growth is the relay rebuilding its negentropy index after the deletions
(O(N) once) — inherent to applying deletions, and still far cheaper than
re-fetching the need set, which the old per-need-fetch approach did.
Scale with -DdelBenchN / -DdelBenchK (forwarded by the geode test task).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Drives the built `amy` binary against a real `amy serve` relay to prove NIP-77
deletion propagation end-to-end — the running answer to "does it actually work",
on top of the in-process geode tests:
T1 (up) we deleted a note the relay still has → `amy sync` sends our kind-5
up and the relay drops it (checked by an isolated third account that
reads the relay only, so no local tombstone masks the result).
T2 (off) `--no-sync-deletions` sends nothing and the relay keeps the note.
T3 (down) the relay deleted a note we still hold → `amy sync --up` pulls the
relay's kind-5 down and applies it locally; a second sync converges.
Each amy account gets its own $HOME (accounts under one $HOME share the file
store). Follows the cli/tests/*-headless.sh pattern; state dir gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The two-pass deletion convergence is protocol logic, not CLI assembly, and the
geode mirror is a near-term second consumer — so move it out of SyncCommand into
a reusable accessory alongside the rest of the negentropy family.
quartz: negentropySettleDeletions(relay, filter, store, sendUp, applyDown, …) —
re-reconciles after a content settle and resolves only the residual: publishes
our covering deletions up (sendUp) and/or ingests the relay's kind-5 down
(applyDown, vanish never auto-applied), looping until a round resolves nothing.
Returns DeletionSettleResult(sentUp, appliedDown, rounds). Everything it needs is
already quartz (negentropyReconcileIds, fetchAll, deletionsCovering,
publishAndConfirm, Event.verify, IEventStore), so it carries no CLI dependency.
SyncCommand's pass 2 collapses to a single call; pass 1 (content) is unchanged.
Catalogued in the accessories README.
Tests: DeletionSyncTest drives the accessory end-to-end both ways (sendUp → relay
converges to gone; applyDown → local converges to gone), on top of the existing
deletionsCovering unit + manual-wiring cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Replace the per-need-event fetch (which pulled the whole need set just to read
metadata — an O(db) regression on large syncs) with a second reconcile pass over
the residual, per the "settle, then diff, then explain what didn't converge" idea.
Pass 1 is the plain content sync again (drain needs, publish haves) — zero
deletion overhead. Pass 2+ re-reconciles; the leftover diff is exactly the
deletion mismatches, and only that (tiny) set is fetched:
- residual need (relay has it, we still lack it after --down) = we deleted it →
publish our covering deletion up so the relay drops it;
- residual have (we have it, relay still lacks it after --up) = the relay deleted
it → pull the relay's covering kind-5 down and apply locally (vanish is NOT
auto-applied on pull — account-wide blast radius).
Loops until a round resolves nothing (converges + self-verifies).
So `amy sync` makes the relay honor our deletions; `--up` makes us honor the
relay's; `--up --down` converges both ways. Cost is one cheap reconcile + the
residual regardless of database size — the large-DB bottleneck is gone by
construction, not by heuristics.
quartz: deletionsCovering is now source-agnostic (takes a query lambda) so the
same coverage rule runs against the local store (up) or the relay (down); the
IEventStore overload is the local convenience.
Tests: DeletionSyncTest gains the down-direction end-to-end (relay deleted →
local removes) alongside the up-direction and the per-form unit cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
MlsGroupState reconstructed the SecretTree from encryption_secret alone, so
every restore rewound each sender's generation counter to 0. The restored
local member then re-emitted generation 0 within the same epoch — reusing the
AEAD key+nonce (a confidentiality break) and getting rejected by strict
receivers (openmls / MDK / Whitenoise) that forbid generation reuse, per
RFC 9420 §9.
Two parts:
- Persist per-sender ratchet positions. SecretTree gains export/importSenderStates;
MlsGroupState carries them as an optional field (STATE_VERSION 2, v1 blobs still
decode as empty = legacy behavior); saveState/restore wire them through.
- Persist after every send. MlsGroupManager.encrypt now saves group state, not
just commits — application sends advance the ratchet but previously never hit
the store, so a restart between two commits still reset it.
Regression tests: a peer that consumed generation 0 accepts the restored
sender's next message (single + multi-send), encrypt persists the ratchet
between commits, and a v1 blob still decodes/restores.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G6uT4xzjty1xosZBkb3sHA
Speeds up a from-scratch GrapeRank crawl ~25-30% at equal completeness on a
drift-controlled A/B, by:
- lowering the per-relay concurrent-sub cap 100→16 — the old 100 drowned popular
relays (damus/nos.lol) in concurrent giant REQs, driving them to time out; 16
restores their responsiveness (damus yield 0%→14%) and is still generous for
the single-user fetches other amy commands do,
- shedding proven-dead relays from the kind:10002 discovery sweep instead of
re-hammering refusing indexers every round,
- trimming the sharded backbone sweep 6→2 rotations (Phase A was ~36% of the
crawl at half Phase B's per-list efficiency; 2 clears the bulk with no
completeness loss).
Also adds relay observability under --diagnose to document how relays reply to
our queries: per-relay telemetry (outcome mix, yield, latency, worst time-sinks),
a LIVE / THROTTLED / UNREACHABLE classification table with the limits we settled
on per relay, and per-round Phase-A/Phase-B timing; plus contact_lists_by_hop in
the sync result for per-hop completeness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-shot/high-level relay ops (fetchAll, fetchFirst, fetchAllPages,
publishAndConfirm, count, negentropy sync/reconcile, …) are INostrClient
extension functions spread across ~8 files with no index, so they don't surface
under "usages of NostrClient" or in completion — easy to miss and re-implement
(as just happened with a bespoke fetchRaw duplicating fetchAll).
- Add accessories/README.md cataloging each public extension with a one-line
"use when".
- CLAUDE.md (Feature Workflow): point at that package/README before hand-rolling
a subscribe/REQ/publish loop.
- relay-client skill: add a Related note steering headless/one-shot callers to
the accessories instead of Subscribable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
fetchAll already does exactly what the need-metadata fetch needs — subscribe,
collect (deduped by id), return on EOSE/timeout, no verify, no store — so drop
the duplicated Context.fetchRaw and call the existing extension.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
`@Volatile` without `import kotlin.concurrent.Volatile` resolves to the
JVM-only `kotlin.jvm.Volatile`, which breaks `commonMain` on iOS/Native
targets. CI catches this on `:commons:compileKotlinIosSimulatorArm64`.
Two call sites needed the import:
- OutboxDispatcher.kt:468 (`@Volatile private var lastCount`)
- FeedMetadataCoordinator.kt:433 (`@Volatile private var lastCount`)
Verified: `./gradlew :commons:compileKotlinIosSimulatorArm64` now green.
Closes CI break on PR #3483.
Refine the sync deletion rule to what was asked: for the events the relay HAS
that we LACK (the reconcile need set), publish only the local deletions that
would actually make the relay remove them — and nothing else, not other
deletions by the same author.
Determining coverage needs the need event's author/address/created_at, which we
don't have for an id we lack, so we fetch the need events (raw — no verify, no
store) purely for metadata. quartz gains IEventStore.deletionsCovering(events,
relay), which maps server-held events to the covering local deletions across all
three forms:
- NIP-09 id-based: a kind-5 with an `e` tag naming the event id;
- NIP-09 address-based: a kind-5 with an `a` tag naming the event's
addressable/replaceable coordinate, at/after it (created_at <= deletion);
- NIP-62 vanish: a kind-62 by the event's author, targeting this relay, issued
after it (created_at < vanish).
SyncCommand's need workers now fetch each need batch once (Context.fetchRaw),
publish its covering deletions (deduped across workers), and — when --down —
store the rest; anything we deleted is rejected by the store's own tombstone.
Nothing is pulled down or applied locally, so it cannot over-delete the store.
DeletionSyncTest covers each form (with cutoff and wrong-relay negatives) plus an
end-to-end reconcile → cover → publish that removes the note on the relay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Per the actual requirement, deletion propagation is exactly: for the ids the
relay HAS that we LACK (the negentropy need set), if we hold a kind-5 deletion
targeting one of them, publish that deletion up — so a note we deleted is
deleted on the relay too instead of being re-downloaded. Only the need ids,
only kind-5, up only.
This removes all the machinery the earlier approach accreted and that the audit
flagged as over-broad / data-loss-prone:
- deleted NostrClientDeletionSyncExt (the bidirectional side-channel, author
scoping, vanish gating, kind selection);
- reverted geode MirrorWorker to base (no deletion side-channel, live-sub
changes, catch-up ordering, or convergence changes);
- dropped the 3-phase SyncCommand flow (deletions-first pull, author-scope
derivation, reject-reaction backstop, --sync-vanish, deletions_* output).
The new path pulls nothing down and applies nothing locally, so it cannot
over-delete the store, and it needs no author scoping — the need set already
bounds it. Kind-62 is intentionally excluded: a vanish is not "of an id".
Emits deletions_sent. DeletionSyncTest now exercises the exact wiring
(reconcile → look up local kind-5 by its e tag for the need ids → publish),
including the negative case (a need id we never had sends nothing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
An audit (adversarial-verified) found the deletion side-channel over-deletes
and over-propagates. Root cause: deletionSideChannelFilter fell open to
authors=null for any non-author-scoped content sync, so `amy sync --kind 1`
reconciled the RELAY'S ENTIRE kind-5/62 history and applied it to the personal
FsEventStore — every kind-5 deleting its targets + installing an id-tombstone
for every target id, every ALL_RELAYS kind-62 wiping all of a pubkey's events
(all kinds), and pushing our whole local deletion history up. Data loss plus a
full-history reconcile on every scoped sync.
Fixes:
- Bound the side-channel to the authors we actually hold content for (filter
authors ∪ local matched-set authors), never the relay's population. Skip when
that scope is empty; Phase 3's reject-reaction covers the author-less case.
- Kind-5 (precise, owner-scoped) propagates by default; kind-62 vanish is opt-in
via --sync-vanish (its blast radius always exceeds a content sync's scope).
- excludesDeletionKinds() now checks each deletion kind independently
(`--kind 1,5` no longer silently drops kind-62); the side-channel reconciles
only the missing kinds.
- amy Phase 1 is best-effort: a deletion-reconcile failure records deletions_error
and falls through to content, never aborting the primary sync (matches geode).
- Mirror up-catch-up converges on whether a PUBLISHABLE event was pushed, not raw
haveCount — a vanish targeting another relay no longer burns all 8 rounds every
startup. Mirror keeps its (correct) global scope for relay-to-relay replication.
Helper API: negentropyPropagateDeletions gains scopeAuthors + deletionKinds;
deletionSideChannelFilter takes authors + deletionKinds and returns only the
missing kinds. Tests updated for the new semantics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Reorder amy sync so both sides fully reflect each other, including deletions:
1. Deletion side-channel now runs FIRST, before content, in both directions.
The content snapshot is taken AFTER it, closing a resurrection bug: a
deletion pulled down mid-sync removes a local event, but the old top-of-run
snapshot still listed it and would re-offer it up — resurrecting it on a
relay that also lacked the deletion.
2. Content reconcile, over the post-deletion snapshot.
3. Reject-reaction backstop: when the relay blocks a content push (usually it
holds a deletion we lack), pull that author's kind-5/62 and ingest locally
so we stop re-offering the dead event. Verify-by-fetch — only a real
deletion the store accepts has any effect; fires only on an actual reject.
The up-push of deletions is already verified per-event: ctx.publish awaits the
relay's OK, and ingesting the kind-5 runs the delete synchronously, so OK=true
confirms the remote applied it.
Mirror catch-up gets the same deletions-first ordering (down and up), so a
deletion lands, or the reject-trigger is armed, before its target — no
add-then-delete churn.
Tests: MirrorDeletionSyncTest gains scopedUpMirrorPushesDeletion (authoritative
push — local holds the deletion, remote drops the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
NIP-77 reconciles by event id over the content filter, so a scoped sync
(`--kind 1`) never carries the kind-5/62 that deletes one of those notes:
the deletion stays stuck on whichever side issued it while the target
lives on forever on the other. Add a deletion side-channel that reconciles
kinds 5 & 62 on their own, independent of the content filter.
quartz: NostrClientDeletionSyncExt — DELETION_PROPAGATION_KINDS,
Filter.excludesDeletionKinds()/deletionSideChannelFilter() (kinds 5/62 scoped
to the same authors, no time window since a deletion's created_at is not its
target's), shouldPropagateDeletionUp() (kind-5 always; kind-62 only to a relay
it targets, honoring the vanish's declared relays), and
negentropyPropagateDeletions() — one bidirectional reconcile that streams
have→upload and need→download.
amy sync: run the side-channel bidirectionally regardless of --up/--down
whenever the filter excludes 5/62; emits deletions_{need,have,downloaded,
uploaded}; --no-sync-deletions opts out.
geode MirrorWorker: thread a per-upstream deletionScope through both catch-up
phases and both live subs (down + up), in the mirror's configured direction;
relax down containment to accept in-scope deletions, gate kind-62 pushes by
target relay, and carry the deletion filter on re-subscribe so a reconnect
never drops it.
Tests: DeletionSyncTest (up/down propagation + filter/vanish-gate units) and
MirrorDeletionSyncTest (a kind-scoped down mirror still removes the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The crawl persists everything to the store and the score is a pure function
over it, so separate them: `amy graperank sync` crawls the reachable graph into
the store (idempotent + cumulative — run it a few times to be sure it's loaded)
and reports what it loaded without scoring; `amy graperank score` builds from the
store and scores instantly, repeatable with different params and no re-crawl
(same as bare `--offline`). Bare `amy graperank` stays the sync+score combo.
Extract the shared crawler wiring into newCrawler(); score() just forces the
offline path, and sync() runs a persist-only crawl (null builder).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The crawler only logged once per round, so a deep hop (140k users, minutes of
work) went silent between lines. Add a heartbeat ticker on the background scope
that emits every few seconds with the current round's completion (a real X/Y %
against the round's known pending target), a rolling fetch rate + rough ETA for
it, and live counts (events stored, relays parked/dead) — and a "finishing"
line while draining the parked tail. Scoring stays sub-second, so it keeps its
per-sweep lines and needs no ticker.
Also make crawl()'s builder nullable: null runs a persist-only pass (every event
still lands in the store, the frontier still expands off each contact list) with
no in-memory graph — the basis for a `sync` that loads data without scoring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
amy gated every non-primitive verb behind a chosen account: `DataDir.resolve`
threw when `~/.amy/` had no unambiguous account, and every networked command
called `Context.open`, which requires an identity — even though queries only
read relays and the shared store and never sign. `store` maintenance and the
local `offer`/`debit info` decoders were caught by the same gate despite
touching no account state.
Reads now work anonymously; only signing needs an account:
- `DataDir.resolveOptional` hands back an accountless dir (`hasAccount = false`)
pointing only at the shared event store when there is no unambiguous account,
instead of throwing.
- `Context.openOrAnonymous` uses the resolved account when present, else an
ephemeral key-less `Identity.anonymous()` — can read, can't sign. Marmot
stores are now lazy and run-state isn't persisted for anonymous runs, so an
accountless read leaves `~/.amy/shared/` clean.
- `Context.open` (signing path) re-asserts the requirement with the "which
account?" hint, so ambiguous/no-account signing verbs still exit 2.
- Main resolves optionally for every verb except the identity-lifecycle ones
(`init`/`create`/`login`/`logoff`/`whoami`), which still need a concrete
account. `offer info` / `debit info` join the stateless primitive block.
- Read subverbs (fetch, subscribe, count, publish, outbox, search, sync,
store, profile/git/podcast/podcast20 reads, nsite/napplet fetch·serve·list,
blossom download·check) switch to `openOrAnonymous`.
No `--json` shapes change. Docs updated (help text, README, DEVELOPMENT).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRaoqGod5LUeSwq4GHRSNF
A cross-account, read-only snapshot of everything amy holds under
`~/.amy/`, built for the returning user: which accounts exist, which
one is pinned as current, each signer type (local keychain/ncryptsec/
plaintext, NIP-46 bunker, or read-only) and whether it can still sign,
the per-account local footprint (aliases, Marmot groups, published
KeyPackage bundle, Cashu wallet, sync cursors), and the shared event
store's size.
Like `use`, it dispatches before account resolution so it works with
zero, one, or many accounts. Strictly metadata-only: it never unlocks a
private key (no keychain prompt / NIP-49 passphrase) and never touches
the network.
Factors the on-disk event-store walk into a shared `StoreStats` helper
reused by both `status` and `store stat`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K3GJb11JkvopP61ETWAVyy
The CLI ran at the library's default Log.minLevel = DEBUG, so quartz internal
chatter (relay-auth init, MLS restore, URL-rejection, throttle notices) leaked
onto stderr around every command's real output. Set Log.minLevel = WARN at
startup, before dispatch, so a normal run shows only warnings/errors plus the
command's own progress. A new global --verbose / -v flag restores full DEBUG;
it's parsed with the other global flags so subcommands never see it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-relay concurrency/rate throttle notices and the "Rejected <url>"
normalizer messages fire constantly during a large crawl (thousands of
rejected/throttled relays) and are operational detail, not warnings. Move
them from Log.w to Log.d so they stay available under debug logging without
flooding a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Adds `amy logoff [--yes] [--keep-events]`, the CLI counterpart to logging
out: it removes everything an account left on the machine.
- the identity file and any backend-held secret (keychain / ncryptsec /
plaintext), via DataDir.deleteIdentity
- the rest of the per-account directory ~/.amy/<account>/ (run-state
cursors, aliases, cashu counters, all Marmot/MLS state)
- the ~/.amy/current pin, when it points at this account
- the account's events in the SHARED ~/.amy/shared/events-store/
The event store is shared across accounts, so logoff does not wipe it
wholesale — it deletes only the events that involve this account: those it
authored plus those addressed to it via a #p tag (gift wraps, nutzaps,
reactions, mentions). Other accounts' cached events are left untouched.
`--keep-events` skips the shared-cache purge entirely.
The public key is read straight from identity.json (never unlocking the
private key), so logoff needs no passphrase and pops no keychain prompt.
Destructive and irreversible, so it follows the `marmot reset` precedent:
`--yes` is required to execute; without it the command prints a dry run of
what would be deleted and exits 2.
Thin-assembly only — event deletion is quartz's FsEventStore.delete; this
just resolves the account, counts, and wires the filesystem teardown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PH3rqz5KaA7CYFPAtxgoz1
The park window's timeout was absolute from subscription open, so a relay
still actively streaming a large result set once it passed parkTimeoutMs was
unsubscribed and its untransmitted tail lost. Reset the window on every
incoming event (a conflated activity signal drives a select against the
terminal deferred), so a parked subscription is closed only after
parkTimeoutMs of actual silence — never while events are still arriving. The
fast window stays absolute: it only decides when to hand a slow relay to the
background park lane, which loses nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
- Suppress DEPRECATION on REASONABLE_SIGN_KINDS, which intentionally lists the
deprecated TorrentCommentEvent kind.
- Replace deprecated readLine() with readlnOrNull() in SecureKeyStorage.
- Drop unnecessary !! non-null assertions in KeyCommands and NostrConnect where
the receiver is already smart-cast to non-null.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nqdy4VTLKidUWzGTJPja9
- Replace unused Unit/null expressions in statement-position when branches
with empty blocks (CommandSerializer, QuicConnection, QuicConnectionParser,
Http3FrameReader, WtPeerStreamDemux).
- Suppress DEPRECATION on KindNames.names, which intentionally registers the
deprecated GitReplyEvent and TorrentCommentEvent kinds for display.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018nqdy4VTLKidUWzGTJPja9
The crawl was round-synchronised: each hop drained all its relays and only
started the next hop after the slowest one reached EOSE or the timeout. That
made waiting for slow-but-alive relays expensive — every hop paid its slow
tail before the next hop's fast relays could begin — so a long timeout for
completeness cost ~2x wall-clock (measured), and a short one dropped the slow
relays' data.
Diagnostics on a ~190k-user crawl showed the genuinely-slow set is a stable
~30 relays that DO reach EOSE, just in 5-25s. So decouple the two concerns:
- drainGated now drains on the FAST `timeoutMs` that sets the round cadence. A
relay still streaming when it elapses is not cut but PARKED: it hands its
open subscription to a background scope (releasing its AdaptiveRelayLimiter
permit so the round moves on), keeps receiving for up to the new
`parkTimeoutMs`, and its late events are persisted + its late contact lists
pushed to a crawl-wide lateHarvest channel.
- The round loop folds late harvest into the graph between rounds and won't
converge until the frontier is empty AND no relay is still parked — so the
crawl waits for slow relays for completeness without paying that wait in each
round's wall-clock.
Graph state stays single-writer: parked coroutines only touch the store,
seenIds, and the channel — never hopOf/done/builder. Persistence moved from a
single per-drain consumer to a shared `persist()` that fast and parked units
both call; crawl-wide dedup is now race-safe via ConcurrentSet.add's atomic
test-and-set (an id is added only after a good signature, so no duplicate
reaches the store's UNIQUE constraint and a forged copy can't suppress the
genuine one).
Also carries the --diagnose slow-relay logging (relay + filter + elapsed for
every slow/parked REQ, so a human can replay it) and keeps --drain-concurrency
at the validated default of 24 (an A/B at 64 was ~2x slower with more dead
relays). New --park-timeout flag (default 40s; set <= --timeout to disable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Add a --diagnose slow-relay log: every content drain that reaches its terminal
(EOSE or timeout) slower than SLOW_DRAIN_LOG_MS, or times out entirely, is
recorded with the offending relay URL, the failure/EOSE reason, elapsed ms, and
the exact filter shape (kinds + author count + first authors). This lets a human
replay that precise REQ later to understand why the relay lags. Gated on
--diagnose so there is no per-group timing/collection overhead otherwise.
Make the content-drain fan-out configurable via a new --drain-concurrency flag
(Config.drainConcurrency), replacing the DRAIN_CONCURRENCY constant. Default
stays at the validated 24: an A/B at 64 ran ~2x slower with more dead relays
(a higher global fan-out re-floods busy hubs faster than the per-relay demotion
catches up), so the flag is a probe knob, not a speedup. Client WebSocket pings
were also tried and reverted — busy-but-alive relays don't reliably pong while
their query handler runs, so pinging just cut them as dead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-user retry counter was blunt: it bumped attempts whether an outbox was
dead, timed out, or cleanly EOSE'd with no event — so a straggler kept being
re-queried against a live relay that had already definitively answered it lacks
their kind:3. Distinguish the cases: drainGated now reports the relays that fully
EOSE'd (answeredOut); the consumer records, per user, the relays that answered but
did not return their contact list (askedEmpty); routeByOutbox excludes those from
the user's candidate relays. A timed-out relay is never added (it might just be
slow — still worth a retry), only a clean-EOSE-empty one; dead relays stay pruned
as before.
Measured on --max-hops 3: redundant fetching dropped ~8% (74k -> 68k events
stored). It does NOT move the wall-clock tail, though — that tail is dominated by
timeout/dead outboxes (the retryable case), not EOSE-empty relays. The wall-clock
lever remains the timeout retry budget (MAX_OUTBOX_ATTEMPTS / drain timeout).
Mirror the crawler extraction on the emit side: the NIP-85 kind:30382 card
reconcile + publish logic (existingCards read-back, rank-diff upsert, stale-card
kind:5 retraction batched under the 64KB event cap) moves out of GrapeRankCommand
into a reusable GrapeRankPublisher in quartz experimental/graperank. It takes an
IEventStore for the prior-card read-back and an injected publish function
(event + relays -> per-relay ack), so the store/relay wiring stays in the app
while the reconcile logic is reusable (e.g. by the Android app).
GrapeRankCommand is now a thin orchestrator: crawl (GrapeRankDataCrawler) ->
score (GrapeRank) -> publish (GrapeRankPublisher). The account-specific bits stay
in the CLI: operator-key derivation, the observer's kind:10040 discovery pointer,
and the operator/register/providers sub-verbs.
The crawl re-verified and re-inserted the same event many times: the outbox
model mirrors each event (especially kind:10002 relay lists) across relays,
indexers, and rounds, but dedup lived in a per-drain SeenIds, so only the copies
within one drain were caught. Add a crawl-wide seen-set (thread-safe ConcurrentSet
of event ids, shared across all 24 concurrent drains and every round), checked
before verify and added only after verify so a forged copy can't suppress the
genuine one. Group-commit the store writes via IEventStore.batchInsert instead of
one transaction per event.
Measured on a from-scratch --max-hops 3 crawl: events actually verified+stored
dropped ~34% (112k -> 74k) and verify time fell in lockstep. The write path now
also reports verify/insert timing + events_stored in Stats, exposed as verify_ms/
insert_ms/events_stored on the CLI, and takes an --insert-batch knob.
Finding: with the work reduced, inserts serialize on SQLite's single writer
mutex rather than transaction count, and the crawl's wall-clock ceiling is the
drain-timeout retry tail on dead outboxes, not the disk.
Cleanups from a reuse/simplification/efficiency/altitude review of the crawler
extraction:
- Collapse the redundant `discovered` set into `hopOf` — a user is discovered
iff it has a hop stamp, so the two always held the same key set. The frontier
is now `hopOf.keys`; one fewer collection to keep in sync.
- Drop the unused `Stats.discovered` / `Stats.deadRelays` fields (no reader —
the CLI reports rounds / relaysContacted / hopHistogram / downloadMs).
- Extract a single shared verify-then-store sink, `IEventStore.verifyAndInsert`,
and route both the crawler and `Context.verifyAndStore` through it instead of
each carrying its own verify + insert + UNIQUE-swallow copy.
- Fast-path the present-key hit in `ConcurrentMap.getOrPut` (jvmAndroid) so the
crawl's hot relay-hint accumulation stops allocating a mapping-function closure
on every call.
- Hoist the repeated `crawlStats?.hopHistogram` null-plumbing in GrapeRankCommand.
The web-of-trust crawl (~400 lines: outbox routing, sharded backbone sweep,
Phase-B worker pool, relay-list discovery, report-deletion fetch, warm pool)
was making the CLI's GrapeRankCommand unmaintainably large. Move it into a
reusable, KMP-portable GrapeRankDataCrawler in quartz commonMain.
The crawler takes a NostrClient + IEventStore + AdaptiveRelayLimiter, injected
relay policy (discovery + content-fallback sets, since those defaults live in
app code, not the protocol library), and a log callback; it streams contact
lists into a TrustGraphBuilder and returns crawl Stats. GrapeRankCommand shrinks
to arg-parsing + offline load + scoring + publish + sub-verbs, delegating the
online path to the crawler.
To reach commonMain (portable to every target, incl. iOS):
- Add ConcurrentMap / ConcurrentSet expect classes under utils/concurrent, with
jvmAndroid actuals (java.util.concurrent) and native actuals (copy-on-write
over kotlin.concurrent.atomics.AtomicReference, mirroring ConcurrentHashCache).
commonMain has no ConcurrentHashMap, and the crawl's producer/consumer/drain-
worker state needs atomic getOrPut/merge plus a concurrent set.
- Move AdaptiveRelayLimiter and DrainFailure/classifyDrainFailure from cli to
quartz commonMain (java atomics -> kotlin.concurrent.atomics, ConcurrentHashMap
-> ConcurrentMap, System.currentTimeMillis -> TimeUtils.nowMillis, stderr -> Log).
- The gated drain (REQ-size splitting, per-relay permits, verify+store) moves into
the crawler; Context.drain loses its now-unused gatePerRelay path.
Net: cli -1077 lines; the crawler + relay machinery are now reusable by the
Android app. Adds ConcurrentCollectionsTest; verified via JVM + commonMain
metadata compile, the wot/graperank suites, and a bounded live crawl.
Each addressable coordinate is ~130 bytes, so 500 pushed the deletion event to
~65KB — over the 64KB event-size cap many relays enforce (stricter than the
256KB message cap). Drop DELETE_PER_EVENT to 400 (~52KB).
README + amy usage: add the `graperank operator [status|relay|providers]`
sub-verb, update the `graperank --publish` description to the per-observer
service-key model (sign with a derived key, publish to the operator relay,
reconcile new/changed/skip/retract, cutoff rank>=2, NIP-09-drop retracted
reports), and add a 'Publishing GrapeRank scores' section explaining the
operator master, deterministic per-observer key derivation, and the kind:10040
discovery wiring.
Rewire `graperank --publish` onto the operator-key model:
- Sign each observer's kind:30382 cards with the dedicated service key derived
for that observer (OperatorKeys), not the account key — a stable per-observer
identity so re-signing replaces the addressable prior card.
- Publish to the operator's configured relay(s) (new `graperank operator relay
<url>` sub-verb; --publish-relay still overrides). Errors clearly if unset.
- Three-way reconciliation against what the provider key already published:
upsert cards whose rank tag string changed (or are new), skip unchanged, and
RETRACT (kind:5, same service key, addressable `a`-tag, chunked under the
message cap) any existing card whose target is no longer publishable — dropped
from the graph, or below the cutoff.
- Raise the default publish cutoff to rank >= 2 (drops the barely-trusted tail);
the retract rule removes any now-sub-cutoff cards.
- When we hold the observer's key (observer == active account), publish/refresh
their kind:10040 pointing 30382:rank -> providerPubkey at the operator relay,
to their outbox — the pointer clients follow to find the cards.
Adds `operator [status|relay|providers]` for managing the machine's operator.
A machine holds one operator master seed, independent of any amy account, stored
under ~/.amy/operator/ via the same SecretStore backend the accounts use. From it
OperatorKeys deterministically derives one service key per observer —
serviceKey(observer) = sha256(masterPriv || "graperank-provider:" || observerHex)
— which will sign that observer's kind:30382 rank cards and their retractions.
Deterministic derivation gives a stable per-observer identity (so re-signing a
card replaces the addressable prior one instead of orphaning it) and one-secret
backup (every service key re-derives from the master alone). The manifest
(operator.json) records the master pubkey, operator relay(s), and observer ->
provider-pubkey mapping — public data; only the master rides the SecretStore.
Exposed via DataDir.operatorKeys(). Wiring into publish comes next.
A REQ carries all of a subscription's filters in one frame, so a popular relay
routed thousands of authors produced a multi-MB frame that most relays reject
outright ("message too large (2MB > 256KB)"), silently dropping every author
in it. The gated drain now splits each relay's filters into REQ-sized groups by
total entry count (authors + ids + tag values), MAX_REQ_ENTRIES=2500 (~167KB,
under the common 256KB cap), and opens one gated subscription per group. A relay
with more authors simply gets several smaller REQs instead of one rejected huge
one. Each group carries its own subId/listener/terminal signal; per-relay
failure classification takes HARD over TRANSIENT across a relay's groups.
Manual test showed the previous sequential loop over recommendations
timed out the overall budget when a well-connected account produced 23
outbox-relay recommendations (23 x 4s per-relay = 92s worst case).
Refactor: build one filterMap keyed by recommendation.relay and issue a
single client.subscribe. All relays fan out in parallel; the per-relay
EOSE gate bounds the wait regardless of set size. Phase 3 fallback
gets the same shape for consistency.
Also:
- Bump overallTimeoutMs default from 8s -> 20s (belt only; parallel
Phase 2 makes it unlikely to trip).
- Log OVERALL TIMEOUT when withTimeoutOrNull returns null so
reviewers can distinguish 'ran fine, no data' from 'timed out'.
- Add per-phase debug logs (start / phase1 done / phase2 recommendations
and done / phase3 fallback and done) so the outbox pipeline is
auditable without a debugger.
Existing 7 OutboxDispatcher tests still pass.
vitorpamplonaandgithub-actions[bot]
Vitor PamplonaandGitHub
Claude
nrobi144