Round-8 profiling showed the crawl re-querying the same never-had-a-10002
users' outboxes every round they recirculated — ~144k slow kind:10002 drains
(p50 17.4s) against a static discovery set, dragging the round to ~18 users/s.
1. ensureRelayLists guards with `relayListDiscoverySwept`: each user's outbox
discovery runs once. The discovery relay set is static, so a second sweep of
a user still lacking a 10002 cannot find one the first missed.
2. The discovery REQ to the bounded INDEXER set co-fetches [10002, 3]: the
outbox lookup already pays the round-trip and an indexer holding a user's
10002 often holds their kind:3, so we harvest the contact list as a cheap
byproduct. The wide "every live relay" completeness sweep stays 10002-ONLY —
co-fetching kind:3 across thousands of relays downloaded the same big contact
lists repeatedly and inflated the fire-and-forget bgScope sweep the finishing
drain waits on (measured +260s at hop-3; the indexer-only co-fetch keeps
coverage flat at baseline speed).
3. harvestFromStore folds any already-stored kind:3 into the graph at Phase-A
time so Phase B never re-drains a list we hold (also speeds re-runs).
Verified same-session hop-3: pre-fix 685s / narrowed 690s / wide-co-fetch 945s,
coverage 91.74% across all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The network-only WoT data traversal is a crawl, not a sync — and main now
ships negentropy sync (`amy sync`, `graperank update`), so the old verb name
was ambiguous. Rename the subcommand and its handler to `crawl`, keeping
`sync` as a back-compat alias so existing scripts keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Audit follow-ups on the sync engine:
- Perf: syncGroup reconciled against a full store.query<Event>(filter),
decoding the entire local matched set (~1 KB/event) just to read ids +
created_at and to index events for a small residual upload. Reconcile now
uses store.snapshotIdsForNegentropy (id + created_at only, ~40 B/entry) and
the uploader fetches only the residual haves by id. Peak memory drops from
O(all local matches) to O(residual) — matters when a relay hosts a large set.
- Bug: sync() promised best-effort ("one bad relay can't abort the set") but
syncGroup only caught NegentropySyncException, so any other failure (store
I/O, an unexpected throw) escaped async and cancelled every other relay via
awaitAll. Each group now runs under a guard that records the failure instead.
- Bug: the page-fallback catch (Exception) swallowed CancellationException,
breaking cooperative cancellation. Both new catch sites rethrow it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Extracts GrapeRankUpdater's per-relay sync engine into a standalone
NegentropyStoreSync in the relay-client accessories, so any caller can
two-pass sync an arbitrary `relay -> filters` set against a local store.
Given an INostrClient + IEventStore it syncs each (relay, filter) group:
a bidirectional NIP-77 reconcile into/from the store (down/up), a deletion
settle over the residual (applyDown downloads the relay's kind:5 when an
uploaded record was rejected), and a paged-download fallback when a relay
can't reconcile. sync() runs many groups with relays concurrent and each
relay's own filters sequential (so one relay never exceeds its subscription
budget). Directions and bounds are a Config; every group is best-effort and
its outcome is a GroupResult. This is also the reusable engine `amy sync`
open-codes today.
GrapeRankUpdater now only owns the GrapeRank specifics: it reads kind:10002,
inverts to write-relay -> authors (the outbox model), fans that into one
filter per (relay, author chunk), hands the set to NegentropyStoreSync, and
folds the per-group results back up per relay. Its public Config/Result and
the CLI wrapper are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Moves the `amy graperank update` logic out of the CLI and into quartz as
GrapeRankUpdater, alongside GrapeRankDataCrawler in experimental/graperank,
so Android and any other quartz consumer can run the same refresh.
Given an INostrClient + IEventStore it reads every kind:10002 in the store,
inverts them into a write-relay -> authors map (the outbox model), then runs
one NIP-77 negentropy reconcile per write relay scoped to its authors:
bidirectional content sync into/from the store, deletion settle over the
residual (applyDown downloads the relay's kind:5 when an uploaded record was
rejected because the author retracted it), and a full paged-download fallback
when a relay can't reconcile. Bounds and directions are a Config; per-relay
and aggregate outcomes are returned as a Result.
The CLI `graperank update` is now a thin wrapper: it parses flags, builds the
Config, and renders GrapeRankUpdater.Result as text/JSON — no sync logic left
in cli/ (all reconcile/window/back-pressure/deletion logic lives in quartz's
relay-client accessories, which GrapeRankUpdater composes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a store-driven refresh of the record kinds a GrapeRank score is a
function of (0 profiles / 3 follows / 10002 outbox lists / 1984 reports).
It reads every kind:10002 already in the local store, inverts them into a
write-relay -> authors map (the outbox model), then runs one NIP-77
negentropy reconcile per write relay scoped to exactly the authors who
publish there. Bidirectional by default; each group then settles deletions
over the reconcile residual via quartz's negentropySettleDeletions, whose
applyDown direction downloads the relay's covering kind:5 when an uploaded
record was rejected because the author retracted it.
When negentropy can't reconcile a relay (no NIP-77, an over-cap minimal
window, a mid-sync disconnect), the group falls back to a full paged
download (Context.drainAllPages) of the same authors+kinds so those
records are still refreshed.
Thin assembly only: reconcile, windowing, back-pressure, and deletion
settle all live in the quartz relay-client accessories, mirroring
SyncCommand; this only routes ids to Context.drain / drainAllPages /
publish and inverts the relay list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
The probe does blocking DNS + TCP connect, and dead-domain DNS lookups hang well
past the connect timeout. On the shared Dispatchers.IO those hanging lookups
starved the crawl's own IO: an A/B at hop-3 showed probe-on 981s vs probe-off
517s, the entire +464s landing on the finishing drain (rounds were identical).
Coverage was unchanged (91.84% vs 91.74%), so the probe classification is correct
— it was purely IO contention.
Give the probe its own fixed daemon pool (128 threads) so its blocking work can
never touch the crawl's IO, and align the culler's concurrency to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The culler filtered candidates by !isDead and not-yet-probed, but not by
liveRelays — so it probed relays the WS path had already proven live, wasting a
probe and opening a needless TCP connection to the hot relays the crawl depends
on. Skip any authority already in liveRelays up front. liveRelays becomes a
ConcurrentSet so the background culler can read it while the crawl writes it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
At hop-8 the crawl dials into thousands of dead relay hints from old accounts.
Most fail slowly: a silently-dropping host has no RST to receive, so the WS
connect just hangs to the 7s connectTimeout. First-strike eviction pays that once
per host, but with ~3,000 dead hosts that's ~80s of connect-setup serialized
through the dispatcher.
Add a background reachability culler: a cheap raw TCP connect (one round trip, 2s
timeout) over the learned relays COLD-TAIL FIRST, dropping the unreachable ones
into deadHosts before the WS path pays its 7s. The key property is that a tight
TCP timeout is safe where a tight WS timeout is not — a busy-but-alive relay
accepts the SYN instantly at the kernel level and only stalls at the app layer, so
the probe separates "unreachable" from "slow" and never false-kills the busy. It
only ever marks dead and probes each authority once; a host the WS path already
resolved (isDead) is skipped, and a live host passes the probe, so the WS verdict
always wins. Injected as an optional Config.reachabilityProbe (JVM: java.net.Socket
in the CLI; --no-probe disables); writeRelayFreq becomes concurrent so the culler
can read it while routeByOutbox writes.
Also: when there's no Tor transport (Config.torEnabled=false), isDead skips every
.onion relay on sight — no socket, no wasted connect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Adds a `sonar` Gradle target that activates only when `sonar.host.url`
is present in local.properties (gitignored). Developers who don't opt
in are unaffected: the scanner plugin is neither resolved nor applied,
so no dependency downloads, no extra tasks, no config-time cost
Dropping the per-batch awaitAll (previous commit) made the rounds faster but a
hop-3 A/B regressed total wall (727s vs 532s): the Semaphore(1024) throttled
concurrent relay drains to ~249 parked at peak vs the old batch model's ~4,438,
so slow-relay park windows that the old model absorbed during the rounds spilled
into a long serial finishing drain. Raise the default so the parked work drains
inside the rounds again. Value under validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Since API 34, ComponentCallbacks2 no longer notifies apps of the foreground
RUNNING_* levels or the deeper MODERATE/COMPLETE background tiers — those
constants are deprecated and the OS only ever delivers UI_HIDDEN (20) and
BACKGROUND (40). The tiered trim logic keyed on the deprecated levels was
therefore dead on any Android 14+ device.
Rebuild the whole trim chain around the two levels still delivered:
- UI_HIDDEN (every app switch): light trim — release image bitmaps, keep the
CPU-heavy rich-text/Robohash caches warm so resuming is instant.
- BACKGROUND (process on the LRU list, real reclaim pressure): aggressive —
free every rebuildable cache, run the heavy LocalCache prune, release the
ExoPlayer warm pool, trim feeds, and evict warm embedded tabs.
Folds the old COMPLETE/MODERATE "free everything" behavior into BACKGROUND and
removes all deprecated TRIM_MEMORY_* references across AppModules,
MemoryTrimmingService, Amethyst, PlaybackService and AccountFeedContentStates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016FnhMdPu7XaCu8DwW8FmLf
Phase B drained users in 256-user batches: a worker called drainGated for the
whole batch and awaitAll'd every relay in it, so one slow relay held the worker
(and the batch's already-finished fast relays' contact lists) for the full 10s
fast window before anything was ingested. With 24 workers all waiting out their
batches' slowest relay at once, progress dropped to 0 lists/sec in waves.
Restructure to drain each relay independently and stream its result the instant
it resolves — no per-batch join. A per-user counter (relaysLeft) tracks how many
of a user's relays are still outstanding; the single-writer consumer finalizes a
user (ingest, or count a failed outbox attempt) only when the last of its relays
resolves, so correctness is unchanged. Concurrency is now a semaphore over
relay-units rather than an implicit batches×fan-out product; drainConcurrency
becomes "concurrent relay drains" (default 1024, ~the old 24-batch fan-out).
A straggler the outbox model routes nowhere is finalized directly as a miss.
Fast relays' lists are now ingested immediately instead of behind a batch's
slowest relay, removing the 0/s stalls on slow-relay-heavy rounds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The drain classifier had two "act on it" verdicts, HARD (drop now) and TRANSIENT
(strike a few times, might clear). Re-probing hop-8's failed relays fresh showed
the TRANSIENT bucket almost never clears: 503 Service Unavailable 0/12 reachable,
502 Bad Gateway 3/15, connection-establishment failures 0/30; the codes that were
alive (402/403) are gated and will never serve us, and 200 isn't a relay. So the
extra dials TRANSIENT bought were spent on hosts that stay dead for the run.
Collapse to a single DEAD verdict, dropped on the first strike, and carve out the
only two connect failures that genuinely recover so they stay retryable (null):
a READ timeout (relay answered the handshake, slow — 67% reachable fresh, kept on
the clear-on-success authority-strike path) and an HTTP 429 rate-limit (alive,
4/4 reachable — retrying spaced by the limiter is how we get its data). Removes
the now-unused relayStrikes map, MAX_DEAD_STRIKES, and the HARD/TRANSIENT merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A relay sends its matching events before its EOSE, so both an event and
the relay's completion can sit buffered in their channels at the same
time. The select() over the two channels picks a ready clause at random,
so it could process the doneChannel completion first, empty `remaining`,
and exit the loop while the matching event was still unread — returning
null instead of the event.
On a relay completion, drain the event channel first and treat any
already-buffered event as the result before marking the relay done.
A connect failure and a read timeout were both treated as "busy, retry" and
took three strikes to drop. Re-probing hop-8's failed relays fresh, outside the
crawl, showed the two are not alike: relays that failed to ESTABLISH a
connection (connect timed out, refused, unroutable, or the proxy couldn't tunnel
the CONNECT) were 0/30 reachable — genuinely dead — while relays that hit a READ
timeout were 12/18 (67%) reachable, alive but overloaded by the crawl's fan-out
(user.kindpag.es among them).
So classifyDrainFailure now returns HARD for connection-establishment failures
(one strike drops them instead of burning two more dials on a dead host), while
a read/generic timeout still returns null and stays on the patient,
clear-on-success timeout-strike path so live-but-slow relays we need are not
wrongly evicted. Mid-stream resets stay TRANSIENT. Adds DrainFailureTest, which
the classifier previously had none of.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Clears real Kotlin compiler warnings surfaced across quartz, cli,
relayBench, amethyst, and desktopApp:
- quartz Sha256/EventHasher/ScratchLocal: ThreadLocal.get() is nullable
in Kotlin; assert non-null (withInitial never yields null).
- quartz GitHttpClient: PriorityQueue.poll() under isNotEmpty() is
non-null; assert it.
- relayBench CorpusDownloader: drop redundant !! on smart-cast Long;
Jackson fields() -> properties().
- cli GrapeRankCommand: drop redundant ?. where latest is smart-cast.
- PodcastRemoteContent: OkHttp body is non-null; drop dead elvis.
- Dead/redundant expressions: remove no-op when-branch values and a
redundant trailing Unit (HomeScreen, LocalCache, EmbeddedTabLayer,
ParticipantHostActionsSheet, NestActionBar, ControlWhenPlayerIsActive,
ShareNoteAsImageScreen exhaustive-when else).
- CalendarEventDetailScreen / SetPasswordDialog / ProfileClinkOfferResolver:
drop always-true conditions (reorder to keep smart-casts).
- WalletColumnScreen: OkHttp body non-null; drop unreachable null-guards.
- PcmTapRegistry: the @OptIn used androidx.annotation.OptIn, which does
not opt into Kotlin's ExperimentalCoroutinesApi; use kotlin.OptIn.
- GitRepositoryScreen: suppress the standard ViewModel-factory cast.
- PushNotificationReceiverService: suppress override-of-deprecated.
- Desktop GlobalScope call sites: @OptIn(DelicateCoroutinesApi::class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
- GrapeRankPublisher: dTag() is non-null (""), so the Elvis on the
grouped target was dead code; skip blank targets via ifBlank instead.
- amethyst: migrate deprecated resourceConfigurations to
androidResources.localeFilters (same locale qualifiers).
- desktopApp: replace deprecated compose.desktop.uiTestJUnit4 accessor
with the direct org.jetbrains.compose.ui:ui-test-junit4 dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
A single REQ can match up to authors×kinds events; a relay that caps its
response below that silently drops the tail. Measured: user.kindpag.es returns
at most ~100 events per REQ and ignores our limit, so a dense chunk -- 300
authors × the 4 FETCH_KINDS, or a popular-author kind:3 sweep -- loses
everything past the newest 100 on the first (and only) page drainGated fetched.
On a dense set kindpag returned 100 events single-shot vs 238 paginated; nos.lol
and damus (higher caps) matched at 246 and 127.
drainGated never paginated -- it took one page and moved on -- so this bit every
sweep and outbox query, not just the aggregator recovery. Truncated users became
stragglers that the multi-round retry mostly (not always) recovered elsewhere,
which is why it stayed hidden.
Now any page that comes back at FULL_PAGE_THRESHOLD (100, the smallest cap
observed) is treated as possibly-capped and its remainder is drained in the
background with fetchAllPages `until` cursors, streamed to lateHarvest exactly
like a parked slow relay (tracked by parkedInFlight so the round waits for it,
gated by the limiter). The boundary second is re-fetched and de-duplicated by
persist's crawl-wide seen-set, so nothing double-counts. Only dense pages pay
the extra REQs; the common under-cap page is untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The recovery pass drained each aggregator with the crawl's single-shot path
(drainGated: one REQ, collect until EOSE). Against an indexer that caps a page
at ~100 events and ignores our limit, every straggler beyond the newest 100 was
silently dropped -- and drainGated additionally merged all chunks into one giant
REQ, which the big indexers answer with nothing at all.
Query each aggregator with fetchAllPages instead, walking `until` cursors to
exhaustion, one AUTHORS_PER_FILTER chunk per request so no request carries the
whole straggler set. Relays paginate concurrently; each relay's chunks run
sequentially to keep one subscription live per connection, gated by the same
limiter. Delivered events land on a channel off the reader threads, then are
verified/persisted and folded once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The comments said a multi-kind filter makes the big indexers "time out returning
nothing." Reproduced against user.kindpag.es, the real mechanism is a per-REQ
result cap: it returns ~100 events regardless of the requested limit, and a
kinds=[3,10000,1984,10002] query fills that cap entirely with the far more
abundant kind:10002, returning 0 kind:3. Asked kind:3-only it returns the
contact lists in a few seconds. Same conclusion (query kind:3 alone), accurate
reason.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Per-relay attribution on observer 460c25e6 showed two big general relays hold
kind:3 for a chunk of the missing authors that no profile indexer has:
nostr.oxtr.dev (76 distinct) and nos.lol (72). Add both to the aggregator set
so the patient kind:3-only recovery pass sweeps them alongside the indexers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The dedicated straggler-recovery pass was skipping any content aggregator the
main crawl had timeout-evicted, so it recovered ~1 contact list instead of the
hundreds those indexers actually hold.
Root cause: during the competitive crawl an indexer like user.kindpag.es is
only ever asked for kind:10002 in bulk and kind:[3,10000,1984,10002] one author
at a time. The latter parks and times out (60-80s each), striking the host until
its authority is timeout-evicted. It is never asked for a clean bulk kind:3 --
the one thing it serves fast (~19 lists per 300 authors in seconds; ~369 of the
run's missing authors live there). So by the time recovery runs, kindpag.es is
dead and dropped from the aggregator set (8 configured -> 6 used), and the
biggest single source of missing lists is never queried.
Fix: the recovery pass now queries every configured aggregator regardless of
eviction (drainGated doesn't re-check isDead, and a genuinely dead endpoint only
costs one shared park window since units run concurrently), and clears any
timeout strikes first so a partially-struck host starts clean. Also stops
folding aggregators into routeByOutbox's multi-kind fan-out (they time out
there) and asks them kind:3-only, matching what they serve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The outbox model fetches a user's kind:3 only from their own kind:10002 write
relays (and the write-frequency backbone). But a large tail of reachable users
have no kind:3 on their own advertised outbox at all — it's dead, or they never
published one there — while a network-wide aggregator (user.kindpag.es, …) that
scrapes the whole network holds it. Those aggregators were queried only for
kind:10002 relay lists in ensureRelayLists, never for kind:3 content, so the
crawl structurally could not find these lists no matter how many rounds it ran.
Add Config.contentAggregatorRelays and fold it into routeByOutbox for stragglers
— users whose own outbox already failed (attempts > 0) or is unknown. The CLI
wires the profile indexers (kindpag/purplepag/coracle/yabu/nostr1) plus the
ActivityPub bridges (ditto/momostr/mostr, which host bridged users' lists);
--no-aggregators disables it.
Measured offline on observer 460c25e6 (max-hops 3): of ~2.2k users the crawl
left without a contact list, querying the aggregators for kind:3 recovers ~500
(user.kindpag.es alone ~180) — lifting coverage from ~89% toward ~92%. The
remainder have no kind:3 retrievable on any relay we know (bridged / inactive /
never-published) — a data-absence floor, not a crawl deficiency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Drop the flaky publish-into-large-relay warmup from DeletionSettleBenchmark
(it timed out the measured reconcile at N=100k — the container noise the
docstring already warns against) and remove the throwaway ScratchSettleTiming
investigation tool. Record in the docstring what the phase breakdown proved:
the settle's extra time over a bare reconcile is O(K) relay-ingest of the K
residual deletions, dominated by one-time JVM/JIT warmup of the publish path
(consecutive K-note batches fell ~3100->570ms), not the deletion algorithm and
not O(N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
relayBench measures relay-to-relay reconcile, not the amy/quartz client feature,
so the deletion-settle perf claim belongs in an in-process benchmark of
negentropySettleDeletions itself.
Models the post-content-settle state: a relay with N notes, a local store with
the same N except K it deleted (keeping the K kind-5s). The reconcile residual is
exactly those K, so a sendUp settle fetches K — not N. Asserts residual==K,
sentUp==K, and relay convergence (correctness guard at the small default N),
and prints one-reconcile vs full-settle so the deletion overhead reads as
"a few reconciles + K", never "+ a content re-download". Measured:
N=2000 K=20: settle ~2x one reconcile, fetched K=20 not N
N=100000 K=20: settle ~5x one reconcile, fetched K=20 not N=100000
The growth is the relay rebuilding its negentropy index after the deletions
(O(N) once) — inherent to applying deletions, and still far cheaper than
re-fetching the need set, which the old per-need-fetch approach did.
Scale with -DdelBenchN / -DdelBenchK (forwarded by the geode test task).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Drives the built `amy` binary against a real `amy serve` relay to prove NIP-77
deletion propagation end-to-end — the running answer to "does it actually work",
on top of the in-process geode tests:
T1 (up) we deleted a note the relay still has → `amy sync` sends our kind-5
up and the relay drops it (checked by an isolated third account that
reads the relay only, so no local tombstone masks the result).
T2 (off) `--no-sync-deletions` sends nothing and the relay keeps the note.
T3 (down) the relay deleted a note we still hold → `amy sync --up` pulls the
relay's kind-5 down and applies it locally; a second sync converges.
Each amy account gets its own $HOME (accounts under one $HOME share the file
store). Follows the cli/tests/*-headless.sh pattern; state dir gitignored.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The two-pass deletion convergence is protocol logic, not CLI assembly, and the
geode mirror is a near-term second consumer — so move it out of SyncCommand into
a reusable accessory alongside the rest of the negentropy family.
quartz: negentropySettleDeletions(relay, filter, store, sendUp, applyDown, …) —
re-reconciles after a content settle and resolves only the residual: publishes
our covering deletions up (sendUp) and/or ingests the relay's kind-5 down
(applyDown, vanish never auto-applied), looping until a round resolves nothing.
Returns DeletionSettleResult(sentUp, appliedDown, rounds). Everything it needs is
already quartz (negentropyReconcileIds, fetchAll, deletionsCovering,
publishAndConfirm, Event.verify, IEventStore), so it carries no CLI dependency.
SyncCommand's pass 2 collapses to a single call; pass 1 (content) is unchanged.
Catalogued in the accessories README.
Tests: DeletionSyncTest drives the accessory end-to-end both ways (sendUp → relay
converges to gone; applyDown → local converges to gone), on top of the existing
deletionsCovering unit + manual-wiring cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
classifyDrainFailure deliberately treats every timeout — connect timeout or
park idle-cut — as "busy, retry" and never dead, so a relay that connects but
never answers a REQ gets re-routed through every straggler's outbox, every
round, each visit burning the full timeout + park window for zero data. The
outbox model makes this worse: one dead server (e.g. filter.nostr.wine) is
advertised as hundreds of distinct per-user path URLs, so a per-URL counter
never reaches a threshold on any single one.
Count unproductive-timeout strikes per relay AUTHORITY (host[:port]) and evict
the whole host after Config.timeoutEvictStrikes (default 3; CLI --timeout-evict,
0 disables). Any clean EOSE or delivered event clears the authority, so only
never-productive hosts are evicted; a multi-path relay where some paths are
slow but others deliver stays live. Authority is host-only and never folds a
filter. subdomain into its parent, so an open bare host is untouched when its
sibling filter host is shed. Purely behavior-driven — no NIP-11.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Replace the per-need-event fetch (which pulled the whole need set just to read
metadata — an O(db) regression on large syncs) with a second reconcile pass over
the residual, per the "settle, then diff, then explain what didn't converge" idea.
Pass 1 is the plain content sync again (drain needs, publish haves) — zero
deletion overhead. Pass 2+ re-reconciles; the leftover diff is exactly the
deletion mismatches, and only that (tiny) set is fetched:
- residual need (relay has it, we still lack it after --down) = we deleted it →
publish our covering deletion up so the relay drops it;
- residual have (we have it, relay still lacks it after --up) = the relay deleted
it → pull the relay's covering kind-5 down and apply locally (vanish is NOT
auto-applied on pull — account-wide blast radius).
Loops until a round resolves nothing (converges + self-verifies).
So `amy sync` makes the relay honor our deletions; `--up` makes us honor the
relay's; `--up --down` converges both ways. Cost is one cheap reconcile + the
residual regardless of database size — the large-DB bottleneck is gone by
construction, not by heuristics.
quartz: deletionsCovering is now source-agnostic (takes a query lambda) so the
same coverage rule runs against the local store (up) or the relay (down); the
IEventStore overload is the local convenience.
Tests: DeletionSyncTest gains the down-direction end-to-end (relay deleted →
local removes) alongside the up-direction and the per-form unit cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Claude
Vitor PamplonaandGitHub
davotoula
vitorpamplonaandgithub-actions[bot]