CachingDns duplicated 20% of what the app's SurgeDns already did better
(stale-while-revalidate, single-flight, jittered 24-48h positive TTL,
persistence, poison filtering). Consolidate on SurgeDns and fix what the
verification pass found along the way:
- move SurgeDns + SurgeDnsStore (+ their 41 tests) to quartz jvmAndroid so
the CLI can share them; delete CachingDns
- negative TTL 10s -> 10min (class default): a dead domain burns 10-30s of
getaddrinfo per re-dial and both the relay pool and a crawl re-dial dead
hosts continuously; the relay pool's own backoff already reaches 5min
- stop the call-failure listeners from erasing negative entries they were
just written from: callFailed caused by UnknownHostException must not
invalidate (it deleted every negative entry milliseconds after creation,
silently defeating the negative TTL entirely). The invalidation remains
for its real purpose - stale positives with rotated IPs
- new SurgeDns.staleAll(): soft-expire everything WITHOUT discarding -
positives serve stale and revalidate in the background on next use,
negatives re-try on first touch. Wired to network-identity changes in
AppModules (same trigger as Tor's onNetworkChange), replacing nothing:
the full-clear invalidate() was never actually called on network change
- amy: SurgeDns wired into the CLI OkHttp client, snapshot persisted at
~/.amy/shared/dns-cache.bin across runs (best-effort load/save)
Verified: all SurgeDns/SurgeDnsStore tests pass in the new location incl.
two new staleAll tests; :amethyst compiles (main + unit tests); cli suite
green. Caveat recorded in the plan doc: behind an HTTP CONNECT proxy OkHttp
never consults the client resolver (hostname goes to the proxy), so this
layer pays off on direct-connect deployments only - which also means the
branch's A/B numbers never depended on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
NIP-29 metadata/roster events (39000-39003) "are addressable events signed by
the relay keypair directly ... as stated by the NIP-11 `self` pubkey", and
"relays shouldn't accept these events if they're signed by anyone else". So the
authoritative test for a genuine group is `39000.author == relay.self` — which
also rejects a stray user-published 39000 even on a real NIP-29 relay, something
the earlier supported_nips heuristic could not.
Add `isRelaySignedRelayGroup(channel)`: strict `author == self` when the relay
publishes `self`, falling back to `supported_nips ∋ 29` when it omits `self`, and
false when it has neither. Apply it at the surfaces that show unsolicited groups:
- Discovery feed: replace the relay-level supported_nips filter with the
per-channel self-key check in matches(); the screen now warms each candidate
relay's NIP-11 and re-invalidates the feed as each doc resolves.
- On-relay group list: filter to relay-signed groups, warming that relay's
NIP-11 so genuine groups fill in and fakes stay hidden.
- CLI `relaygroup browse`/`info`: fetch the relay's NIP-11 (new
Context.relayInfo) and drop 39xxx not signed by `self`; browse reports the
dropped count.
Explicit user actions (a received invite link, opening an naddr) are left
untouched — hiding those would be user-hostile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Phase-B throughput plateaus at ~100-150 users/s at every crawl scale while
CPU/FDs/network sit idle, pointing at the per-relay 16-permit gates on the
hot backbone/fallback relays every batch touches. The 16-vs-100 benchmark
that chose the default predates the multithreaded-crawl fix, so expose the
starting cap for A/B runs; the NOTICE/CLOSED demotion ladder still protects
relays that push back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Cold hop-3 A/B (fresh store per leg, same observer): 48 workers beat 24
twice at identical contact-list counts (579s->559s and 474s->396s). The
old result that made 64 look 2x slower predated the multithreaded
dispatcher fix - more coroutines on one starved event-loop thread. Plan
doc gains the full A/B table (kept: background-park degating, sweep
dedup, tail overlap, dc48; rejected: 5s fast window).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
A saved crawl log couldn't attribute wall time to the finishing tail
(aggregator recovery vs report deletions vs parked drains) without
external timestamps. Prefix every crawl progress line with [t+SSSs].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Any graperank crawl/probe now creates ~/.amy/operator/ (the reachability
cache signs with the operator monitor key), and listAccounts counted it as
an account - so a single-account user's very first crawl left every later
command failing with "multiple accounts (operator, <name>)". Add it to
RESERVED_NAMES alongside shared/current. Hit for real while network-testing
the connect-storm changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Most of a from-scratch crawl's wall clock was connection setup, re-paid
serially: the relay pool tears down a relay's socket ~300ms after its last
drain unsubscribes, OkHttp allowed only 256 concurrent WS handshakes, and
every dial re-ran an uncached blocking getaddrinfo (10-30s per dead domain,
once per per-user path URL of the same host). Open the connections once, in
parallel, and keep them:
- GrapeRankCrawler: the warm-pool trick (never-matching REQ that only holds
the socket) now covers the whole candidate universe instead of the top 20 -
seeded at crawl start from the reachability cache's live set (one parallel
connection storm, Config.knownLiveRelays) and refreshed each round with
newly learned outbox relays, capped by Config.preconnectCap (FD-budget
aware, --preconnect-cap / --no-preconnect).
- CachingDns (quartz jvmAndroid): 10-min positive + negative DNS cache with
in-flight per-host dedup; dead domains fail in microseconds instead of
re-burning resolver timeouts, path URLs of one host resolve once.
- cli Context: dispatcher and pre-connect caps derived from the process's
open-files limit (UnixOperatingSystemMXBean), warning when ulimit is low.
- amy graperank probe: relay census - mass-connects every relay the store
knows (kind:10002 universe deduped per authority + cached verdicts) in
waves, records live/dead with real measured rtt-open into the NIP-66
reachability cache (RelayProber + RelayReachabilityStore.recordProbed),
so the next crawl skips dead relays and waits once for the slow-but-alive.
Single-server limits (FDs, ephemeral ports, DNS, threads, conntrack) and the
design are documented in quartz/plans/2026-07-10-graperank-connect-storm.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Measured: relays deliver their events in ~0.6s then sit ~4.6s (86% of drain wall)
before sending EOSE — mostly relay-side (our pipeline adds only ~200ms). So instead
of waiting the full 10s fast window then parking, close a drain that has delivered
>=1 event and then gone silent for eoseIdleMs, treating it as complete ("eose-idle").
awaitTerminalOrQuiescent: the idle timer arms only AFTER the first event, so a relay
merely slow to answer still gets the full timeoutMs and is never cut prematurely; a
still-streaming relay keeps resetting the window. eose-idle paginates if the page was
capped and clears timeout strikes (it delivered), but joins notAnswered (no clean
EOSE, so its missing authors are retried elsewhere). Off by default (eoseIdleMs=0),
CLI --eose-idle-ms, so it can be A/B'd against the plain fast window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Two changes:
1. The reachability flush re-wrote the SEEDED known-dead relays with a fresh
created_at every run, refreshing their TTL without a re-probe — so a relay
marked dead once (and thereafter skipped, never re-dialed) would stay
blacklisted forever as long as crawls kept running, defeating the TTL's
re-probe. Stats.deadRelays now reports only relays actually dialed this run
(deadRelays - knownDeadRelays); seeded records keep their original timestamp
and age out on schedule so the next run re-probes them.
2. Rename GrapeRankDataCrawler -> GrapeRankCrawler (file + all references).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Wire RelayReachabilityStore into the crawler and the WoT updater so liveness is
shared across procedures and runs instead of each rediscovering dead relays.
- OperatorKeys.monitorKey(): a dedicated machine monitor identity derived from the
operator master (domain "relay-monitor:"), independent of any account — the
30166 records are published under this, not the observer key.
- Context.reachability: a RelayReachabilityStore over the shared store, signed by
the monitor key.
- Crawler: Config.knownDeadRelays seeds deadRelays before the run; Stats now
returns the final dead/live sets. GrapeRankCommand seeds from snapshot().dead
and flushes the crawl's verdicts back via reachability.record().
- Updater: Config.knownDead skips proven-dead relays from the reconcile plan — a
dead relay cannot serve its authors, so reconciling it only burns a timeout.
Live author-advertised relays are always synced.
All behind --no-reachability-cache. TTL'd (24h), so a recovered relay is retried
once its record ages out — a "skip for now", never a permanent ignore, keeping
the outbox rule that every live advertised relay is tried.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The network-only WoT data traversal is a crawl, not a sync — and main now
ships negentropy sync (`amy sync`, `graperank update`), so the old verb name
was ambiguous. Rename the subcommand and its handler to `crawl`, keeping
`sync` as a back-compat alias so existing scripts keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Moves the `amy graperank update` logic out of the CLI and into quartz as
GrapeRankUpdater, alongside GrapeRankDataCrawler in experimental/graperank,
so Android and any other quartz consumer can run the same refresh.
Given an INostrClient + IEventStore it reads every kind:10002 in the store,
inverts them into a write-relay -> authors map (the outbox model), then runs
one NIP-77 negentropy reconcile per write relay scoped to its authors:
bidirectional content sync into/from the store, deletion settle over the
residual (applyDown downloads the relay's kind:5 when an uploaded record was
rejected because the author retracted it), and a full paged-download fallback
when a relay can't reconcile. Bounds and directions are a Config; per-relay
and aggregate outcomes are returned as a Result.
The CLI `graperank update` is now a thin wrapper: it parses flags, builds the
Config, and renders GrapeRankUpdater.Result as text/JSON — no sync logic left
in cli/ (all reconcile/window/back-pressure/deletion logic lives in quartz's
relay-client accessories, which GrapeRankUpdater composes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a store-driven refresh of the record kinds a GrapeRank score is a
function of (0 profiles / 3 follows / 10002 outbox lists / 1984 reports).
It reads every kind:10002 already in the local store, inverts them into a
write-relay -> authors map (the outbox model), then runs one NIP-77
negentropy reconcile per write relay scoped to exactly the authors who
publish there. Bidirectional by default; each group then settles deletions
over the reconcile residual via quartz's negentropySettleDeletions, whose
applyDown direction downloads the relay's covering kind:5 when an uploaded
record was rejected because the author retracted it.
When negentropy can't reconcile a relay (no NIP-77, an over-cap minimal
window, a mid-sync disconnect), the group falls back to a full paged
download (Context.drainAllPages) of the same authors+kinds so those
records are still refreshed.
Thin assembly only: reconcile, windowing, back-pressure, and deletion
settle all live in the quartz relay-client accessories, mirroring
SyncCommand; this only routes ids to Context.drain / drainAllPages /
publish and inverts the relay list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Correctness:
- Discovery sort no longer reads createdAt live in the comparator (TimSort
"contract violated" crash risk); orders by member count then the shared
sortedByDefaultFeedOrder snapshot.
- One shared resolver (relayGroupDiscoveryChannelFor) is used by the feed
match, sort AND the row, so a 39000 seen on >1 relay always binds one
channel — no more "sorted by relay B, rendered with relay A's empty roster".
- Roster (39001/39002) arrivals now re-inject the group's 39000 into the feed
and re-invalidate the datasource, so a group where a follow is an admin/
member surfaces instead of staying hidden / frozen at 0 members.
- Group replies route to the group's host (resolved from the channel), never
falling through to signAndComputeBroadcast — fixes the outbox leak when the
parent note had no relay provenance.
- Messages list (inline mode) updates group rows incrementally: the additive
path now handles group-scoped messages, not just public/ephemeral/DM.
- Optimistic null-relay group sends attach only to an unambiguous single
channel, so a message to the "_" group on relay A no longer bleeds into "_"
on relay B.
Consistency:
- AllFollows discovery also REQs #t/#g (not just authors), matching the local
AnyOf constraint; muted-authors maps to ByPeople; community stays AllGroups
— fetch and display now agree.
- Global drops the 1-week `since` floor so long-lived group metadata is fetched.
Performance:
- #d metadata backfill scans the group cache once per filter assembly (grouped
by relay), not once per relay.
- Discovery rows warm content only; the directory subscription already streams
metadata/rosters for the relay.
- memberCount is memoized (members ∪ admins recomputed on roster change, not per
read); thread re-sort extracted.
- Discovery list gets rememberFeedContentPadding, contentType and animateItem.
CLI:
- `relaygroup edit` preserves current name/about/tags when only a flag changes.
Cleanup: dropped dead relayKeys() branches and the redundant AnyOf single-lens
collapse in the discovery constraint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
The probe does blocking DNS + TCP connect, and dead-domain DNS lookups hang well
past the connect timeout. On the shared Dispatchers.IO those hanging lookups
starved the crawl's own IO: an A/B at hop-3 showed probe-on 981s vs probe-off
517s, the entire +464s landing on the finishing drain (rounds were identical).
Coverage was unchanged (91.84% vs 91.74%), so the probe classification is correct
— it was purely IO contention.
Give the probe its own fixed daemon pool (128 threads) so its blocking work can
never touch the crawl's IO, and align the culler's concurrency to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
At hop-8 the crawl dials into thousands of dead relay hints from old accounts.
Most fail slowly: a silently-dropping host has no RST to receive, so the WS
connect just hangs to the 7s connectTimeout. First-strike eviction pays that once
per host, but with ~3,000 dead hosts that's ~80s of connect-setup serialized
through the dispatcher.
Add a background reachability culler: a cheap raw TCP connect (one round trip, 2s
timeout) over the learned relays COLD-TAIL FIRST, dropping the unreachable ones
into deadHosts before the WS path pays its 7s. The key property is that a tight
TCP timeout is safe where a tight WS timeout is not — a busy-but-alive relay
accepts the SYN instantly at the kernel level and only stalls at the app layer, so
the probe separates "unreachable" from "slow" and never false-kills the busy. It
only ever marks dead and probes each authority once; a host the WS path already
resolved (isDead) is skipped, and a live host passes the probe, so the WS verdict
always wins. Injected as an optional Config.reachabilityProbe (JVM: java.net.Socket
in the CLI; --no-probe disables); writeRelayFreq becomes concurrent so the culler
can read it while routeByOutbox writes.
Also: when there's no Tor transport (Config.torEnabled=false), isDead skips every
.onion relay on sight — no socket, no wasted connect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Dropping the per-batch awaitAll (previous commit) made the rounds faster but a
hop-3 A/B regressed total wall (727s vs 532s): the Semaphore(1024) throttled
concurrent relay drains to ~249 parked at peak vs the old batch model's ~4,438,
so slow-relay park windows that the old model absorbed during the rounds spilled
into a long serial finishing drain. Raise the default so the parked work drains
inside the rounds again. Value under validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Phase B drained users in 256-user batches: a worker called drainGated for the
whole batch and awaitAll'd every relay in it, so one slow relay held the worker
(and the batch's already-finished fast relays' contact lists) for the full 10s
fast window before anything was ingested. With 24 workers all waiting out their
batches' slowest relay at once, progress dropped to 0 lists/sec in waves.
Restructure to drain each relay independently and stream its result the instant
it resolves — no per-batch join. A per-user counter (relaysLeft) tracks how many
of a user's relays are still outstanding; the single-writer consumer finalizes a
user (ingest, or count a failed outbox attempt) only when the last of its relays
resolves, so correctness is unchanged. Concurrency is now a semaphore over
relay-units rather than an implicit batches×fan-out product; drainConcurrency
becomes "concurrent relay drains" (default 1024, ~the old 24-batch fan-out).
A straggler the outbox model routes nowhere is finalized directly as a miss.
Fast relays' lists are now ingested immediately instead of behind a batch's
slowest relay, removing the 0/s stalls on slow-relay-heavy rounds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Clears real Kotlin compiler warnings surfaced across quartz, cli,
relayBench, amethyst, and desktopApp:
- quartz Sha256/EventHasher/ScratchLocal: ThreadLocal.get() is nullable
in Kotlin; assert non-null (withInitial never yields null).
- quartz GitHttpClient: PriorityQueue.poll() under isNotEmpty() is
non-null; assert it.
- relayBench CorpusDownloader: drop redundant !! on smart-cast Long;
Jackson fields() -> properties().
- cli GrapeRankCommand: drop redundant ?. where latest is smart-cast.
- PodcastRemoteContent: OkHttp body is non-null; drop dead elvis.
- Dead/redundant expressions: remove no-op when-branch values and a
redundant trailing Unit (HomeScreen, LocalCache, EmbeddedTabLayer,
ParticipantHostActionsSheet, NestActionBar, ControlWhenPlayerIsActive,
ShareNoteAsImageScreen exhaustive-when else).
- CalendarEventDetailScreen / SetPasswordDialog / ProfileClinkOfferResolver:
drop always-true conditions (reorder to keep smart-casts).
- WalletColumnScreen: OkHttp body non-null; drop unreachable null-guards.
- PcmTapRegistry: the @OptIn used androidx.annotation.OptIn, which does
not opt into Kotlin's ExperimentalCoroutinesApi; use kotlin.OptIn.
- GitRepositoryScreen: suppress the standard ViewModel-factory cast.
- PushNotificationReceiverService: suppress override-of-deprecated.
- Desktop GlobalScope call sites: @OptIn(DelicateCoroutinesApi::class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
Per-relay attribution on observer 460c25e6 showed two big general relays hold
kind:3 for a chunk of the missing authors that no profile indexer has:
nostr.oxtr.dev (76 distinct) and nos.lol (72). Add both to the aggregator set
so the patient kind:3-only recovery pass sweeps them alongside the indexers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The outbox model fetches a user's kind:3 only from their own kind:10002 write
relays (and the write-frequency backbone). But a large tail of reachable users
have no kind:3 on their own advertised outbox at all — it's dead, or they never
published one there — while a network-wide aggregator (user.kindpag.es, …) that
scrapes the whole network holds it. Those aggregators were queried only for
kind:10002 relay lists in ensureRelayLists, never for kind:3 content, so the
crawl structurally could not find these lists no matter how many rounds it ran.
Add Config.contentAggregatorRelays and fold it into routeByOutbox for stragglers
— users whose own outbox already failed (attempts > 0) or is unknown. The CLI
wires the profile indexers (kindpag/purplepag/coracle/yabu/nostr1) plus the
ActivityPub bridges (ditto/momostr/mostr, which host bridged users' lists);
--no-aggregators disables it.
Measured offline on observer 460c25e6 (max-hops 3): of ~2.2k users the crawl
left without a contact list, querying the aggregators for kind:3 recovers ~500
(user.kindpag.es alone ~180) — lifting coverage from ~89% toward ~92%. The
remainder have no kind:3 retrievable on any relay we know (bridged / inactive /
never-published) — a data-absence floor, not a crawl deficiency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The two-pass deletion convergence is protocol logic, not CLI assembly, and the
geode mirror is a near-term second consumer — so move it out of SyncCommand into
a reusable accessory alongside the rest of the negentropy family.
quartz: negentropySettleDeletions(relay, filter, store, sendUp, applyDown, …) —
re-reconciles after a content settle and resolves only the residual: publishes
our covering deletions up (sendUp) and/or ingests the relay's kind-5 down
(applyDown, vanish never auto-applied), looping until a round resolves nothing.
Returns DeletionSettleResult(sentUp, appliedDown, rounds). Everything it needs is
already quartz (negentropyReconcileIds, fetchAll, deletionsCovering,
publishAndConfirm, Event.verify, IEventStore), so it carries no CLI dependency.
SyncCommand's pass 2 collapses to a single call; pass 1 (content) is unchanged.
Catalogued in the accessories README.
Tests: DeletionSyncTest drives the accessory end-to-end both ways (sendUp → relay
converges to gone; applyDown → local converges to gone), on top of the existing
deletionsCovering unit + manual-wiring cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
classifyDrainFailure deliberately treats every timeout — connect timeout or
park idle-cut — as "busy, retry" and never dead, so a relay that connects but
never answers a REQ gets re-routed through every straggler's outbox, every
round, each visit burning the full timeout + park window for zero data. The
outbox model makes this worse: one dead server (e.g. filter.nostr.wine) is
advertised as hundreds of distinct per-user path URLs, so a per-URL counter
never reaches a threshold on any single one.
Count unproductive-timeout strikes per relay AUTHORITY (host[:port]) and evict
the whole host after Config.timeoutEvictStrikes (default 3; CLI --timeout-evict,
0 disables). Any clean EOSE or delivered event clears the authority, so only
never-productive hosts are evicted; a multi-path relay where some paths are
slow but others deliver stays live. Authority is host-only and never folds a
filter. subdomain into its parent, so an open bare host is untouched when its
sibling filter host is shed. Purely behavior-driven — no NIP-11.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Replace the per-need-event fetch (which pulled the whole need set just to read
metadata — an O(db) regression on large syncs) with a second reconcile pass over
the residual, per the "settle, then diff, then explain what didn't converge" idea.
Pass 1 is the plain content sync again (drain needs, publish haves) — zero
deletion overhead. Pass 2+ re-reconciles; the leftover diff is exactly the
deletion mismatches, and only that (tiny) set is fetched:
- residual need (relay has it, we still lack it after --down) = we deleted it →
publish our covering deletion up so the relay drops it;
- residual have (we have it, relay still lacks it after --up) = the relay deleted
it → pull the relay's covering kind-5 down and apply locally (vanish is NOT
auto-applied on pull — account-wide blast radius).
Loops until a round resolves nothing (converges + self-verifies).
So `amy sync` makes the relay honor our deletions; `--up` makes us honor the
relay's; `--up --down` converges both ways. Cost is one cheap reconcile + the
residual regardless of database size — the large-DB bottleneck is gone by
construction, not by heuristics.
quartz: deletionsCovering is now source-agnostic (takes a query lambda) so the
same coverage rule runs against the local store (up) or the relay (down); the
IEventStore overload is the local convenience.
Tests: DeletionSyncTest gains the down-direction end-to-end (relay deleted →
local removes) alongside the up-direction and the per-form unit cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Two compatibility gaps found analyzing nostrord (a NIP-29 client):
- Thread titles: NIP-7D (and Amethyst) use a `title` tag, but nostrord
writes/reads `subject`, so neither showed the other's thread titles.
ThreadEvent.title() now reads `title` OR `subject`; we still emit only the
spec-correct `title`.
- Joined-groups list (kind 10009): Amethyst wrote memberships as NIP-44
private items, but both reference clients (Flotilla, nostrord) store — and
nostrord only READS — public `["group", id, relay]` tags, so an Amethyst
user's groups were invisible to them. follow() (and the amy CLI) now write
public tags. NIP-29 membership is already public via the relay's kind-39002
list, so this loses no real privacy; reads still merge any legacy private
items so existing lists keep working.
Tests: read title from title/subject (title wins; we emit title only); public
group is a plain tag and still read through the cache; a mixed public+private
list reads as both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Speeds up a from-scratch GrapeRank crawl ~25-30% at equal completeness on a
drift-controlled A/B, by:
- lowering the per-relay concurrent-sub cap 100→16 — the old 100 drowned popular
relays (damus/nos.lol) in concurrent giant REQs, driving them to time out; 16
restores their responsiveness (damus yield 0%→14%) and is still generous for
the single-user fetches other amy commands do,
- shedding proven-dead relays from the kind:10002 discovery sweep instead of
re-hammering refusing indexers every round,
- trimming the sharded backbone sweep 6→2 rotations (Phase A was ~36% of the
crawl at half Phase B's per-list efficiency; 2 clears the bulk with no
completeness loss).
Also adds relay observability under --diagnose to document how relays reply to
our queries: per-relay telemetry (outcome mix, yield, latency, worst time-sinks),
a LIVE / THROTTLED / UNREACHABLE classification table with the limits we settled
on per relay, and per-round Phase-A/Phase-B timing; plus contact_lists_by_hop in
the sync result for per-hop completeness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fetchAll already does exactly what the need-metadata fetch needs — subscribe,
collect (deduped by id), return on EOSE/timeout, no verify, no store — so drop
the duplicated Context.fetchRaw and call the existing extension.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Refine the sync deletion rule to what was asked: for the events the relay HAS
that we LACK (the reconcile need set), publish only the local deletions that
would actually make the relay remove them — and nothing else, not other
deletions by the same author.
Determining coverage needs the need event's author/address/created_at, which we
don't have for an id we lack, so we fetch the need events (raw — no verify, no
store) purely for metadata. quartz gains IEventStore.deletionsCovering(events,
relay), which maps server-held events to the covering local deletions across all
three forms:
- NIP-09 id-based: a kind-5 with an `e` tag naming the event id;
- NIP-09 address-based: a kind-5 with an `a` tag naming the event's
addressable/replaceable coordinate, at/after it (created_at <= deletion);
- NIP-62 vanish: a kind-62 by the event's author, targeting this relay, issued
after it (created_at < vanish).
SyncCommand's need workers now fetch each need batch once (Context.fetchRaw),
publish its covering deletions (deduped across workers), and — when --down —
store the rest; anything we deleted is rejected by the store's own tombstone.
Nothing is pulled down or applied locally, so it cannot over-delete the store.
DeletionSyncTest covers each form (with cutoff and wrong-relay negatives) plus an
end-to-end reconcile → cover → publish that removes the note on the relay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Per the actual requirement, deletion propagation is exactly: for the ids the
relay HAS that we LACK (the negentropy need set), if we hold a kind-5 deletion
targeting one of them, publish that deletion up — so a note we deleted is
deleted on the relay too instead of being re-downloaded. Only the need ids,
only kind-5, up only.
This removes all the machinery the earlier approach accreted and that the audit
flagged as over-broad / data-loss-prone:
- deleted NostrClientDeletionSyncExt (the bidirectional side-channel, author
scoping, vanish gating, kind selection);
- reverted geode MirrorWorker to base (no deletion side-channel, live-sub
changes, catch-up ordering, or convergence changes);
- dropped the 3-phase SyncCommand flow (deletions-first pull, author-scope
derivation, reject-reaction backstop, --sync-vanish, deletions_* output).
The new path pulls nothing down and applies nothing locally, so it cannot
over-delete the store, and it needs no author scoping — the need set already
bounds it. Kind-62 is intentionally excluded: a vanish is not "of an id".
Emits deletions_sent. DeletionSyncTest now exercises the exact wiring
(reconcile → look up local kind-5 by its e tag for the need ids → publish),
including the negative case (a need id we never had sends nothing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Fixes found in a full audit of the NIP-29 relay-groups feature across
quartz/commons/amethyst/cli.
Correctness (app):
- Own group messages never appeared in the timeline until an app restart:
the optimistic send is consumed with a null relay, so attachToRelayGroup
bailed on the relay==null guard, and the host relay's echo (new==false)
was skipped by the "only attach when newly consumed" gate. Attach now runs
on every arrival, gated on the note being loaded, and the null-relay case
attaches to the already-open channel(s) for that group id. Also avoids the
wrong-relay phantom by only fabricating a channel from real provenance.
- Roster subscription was frozen after an in-place join/leave (state keyed
on the stable account, never re-derived); it now invalidates on every
liveRelayGroupList change, so a fresh join's 39002 admission is fetched.
- membershipOf demoted a 39001 admin with an empty/unknown role to MEMBER,
hiding moderation; presence in the admins list now means at least MODERATOR.
- Members roster showed permanent truncated-hex names (one-shot
getUserIfExists cached null); uses checkGetOrCreateUser so UsernameDisplay
fills in when kind:0 arrives.
Protocol / data:
- GroupTag had no value equality → joined-group Sets never deduped and the
StateFlow re-emitted on every identical re-arrival. Equality is now the
(id, relay) pair, excluding the cosmetic name.
- create/edit emitted non-canonical ["public"]/["open"] status tags; NIP-29
flags are presence-only, so only private/closed are emitted when set.
- Metadata/member/admin supersede guards use <= so an equal-createdAt
duplicate isn't reprocessed (first-arrival wins); updatedMetadataAt is now
private-set. Relay-group channels are now included in the prune loops.
CLI:
- join/leave/create updated the kind:10009 list from a network-only drain;
a slow/empty fetch could publish a fresh list containing ONLY the new
group, wiping the rest. Now reads the local store (source of truth) too.
- edit re-asserted both visibility axes from flag presence, so --closed on a
private group leaked it public. It now reads current 39000 and merges,
with --public/--open counter-flags; only the specified axis changes.
- create now tracks the new group in kind:10009 (parity with join/Android).
UI polish:
- Invite dialog no longer mints a 9009 for open groups and won't copy a code
it never displayed. Browse "popular" list normalizes URLs before filtering.
Tests: GroupTag identity, unknown-role-admin-moderates, equal-createdAt
no-resupersede added; all quartz+commons NIP-29 suites and the amy
relaygroup harness pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
An audit (adversarial-verified) found the deletion side-channel over-deletes
and over-propagates. Root cause: deletionSideChannelFilter fell open to
authors=null for any non-author-scoped content sync, so `amy sync --kind 1`
reconciled the RELAY'S ENTIRE kind-5/62 history and applied it to the personal
FsEventStore — every kind-5 deleting its targets + installing an id-tombstone
for every target id, every ALL_RELAYS kind-62 wiping all of a pubkey's events
(all kinds), and pushing our whole local deletion history up. Data loss plus a
full-history reconcile on every scoped sync.
Fixes:
- Bound the side-channel to the authors we actually hold content for (filter
authors ∪ local matched-set authors), never the relay's population. Skip when
that scope is empty; Phase 3's reject-reaction covers the author-less case.
- Kind-5 (precise, owner-scoped) propagates by default; kind-62 vanish is opt-in
via --sync-vanish (its blast radius always exceeds a content sync's scope).
- excludesDeletionKinds() now checks each deletion kind independently
(`--kind 1,5` no longer silently drops kind-62); the side-channel reconciles
only the missing kinds.
- amy Phase 1 is best-effort: a deletion-reconcile failure records deletions_error
and falls through to content, never aborting the primary sync (matches geode).
- Mirror up-catch-up converges on whether a PUBLISHABLE event was pushed, not raw
haveCount — a vanish targeting another relay no longer burns all 8 rounds every
startup. Mirror keeps its (correct) global scope for relay-to-relay replication.
Helper API: negentropyPropagateDeletions gains scopeAuthors + deletionKinds;
deletionSideChannelFilter takes authors + deletionKinds and returns only the
missing kinds. Tests updated for the new semantics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Adds a first-class `amy relaygroup` verb group so the CLI can drive the
same NIP-29 relay groups as the app. Thin assembly over quartz builders +
Context.publish/drain — no protocol logic in cli/.
Verbs:
- list / browse RELAY / info RELAY GID — reads (joined kind:10009 list with
private-item decryption; a relay's 39000-39003 directory; one group's
metadata + roster).
- create / join / leave / message — lifecycle. join/leave also maintain the
caller's kind:10009 list (add/remove private item) so `list` reflects them,
mirroring the app's follow/unfollow.
- edit / invite / put-user / remove-user — moderation (9002/9009/9000/9001).
All writes pin to the group's single host relay. Output follows amy's
text/--json contract with snake_case keys.
Verified end-to-end against an embedded relay (amy serve) with a new
self-contained harness, cli/tests/relaygroup/relaygroup-headless.sh:
create/message/join/list/browse all pass (5/5). browse/info return empty
against geode since it doesn't sign 39000-39003 — a relay capability, not a
client issue. README command table + ROADMAP updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Reorder amy sync so both sides fully reflect each other, including deletions:
1. Deletion side-channel now runs FIRST, before content, in both directions.
The content snapshot is taken AFTER it, closing a resurrection bug: a
deletion pulled down mid-sync removes a local event, but the old top-of-run
snapshot still listed it and would re-offer it up — resurrecting it on a
relay that also lacked the deletion.
2. Content reconcile, over the post-deletion snapshot.
3. Reject-reaction backstop: when the relay blocks a content push (usually it
holds a deletion we lack), pull that author's kind-5/62 and ingest locally
so we stop re-offering the dead event. Verify-by-fetch — only a real
deletion the store accepts has any effect; fires only on an actual reject.
The up-push of deletions is already verified per-event: ctx.publish awaits the
relay's OK, and ingesting the kind-5 runs the delete synchronously, so OK=true
confirms the remote applied it.
Mirror catch-up gets the same deletions-first ordering (down and up), so a
deletion lands, or the reject-trigger is armed, before its target — no
add-then-delete churn.
Tests: MirrorDeletionSyncTest gains scopedUpMirrorPushesDeletion (authoritative
push — local holds the deletion, remote drops the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
NIP-77 reconciles by event id over the content filter, so a scoped sync
(`--kind 1`) never carries the kind-5/62 that deletes one of those notes:
the deletion stays stuck on whichever side issued it while the target
lives on forever on the other. Add a deletion side-channel that reconciles
kinds 5 & 62 on their own, independent of the content filter.
quartz: NostrClientDeletionSyncExt — DELETION_PROPAGATION_KINDS,
Filter.excludesDeletionKinds()/deletionSideChannelFilter() (kinds 5/62 scoped
to the same authors, no time window since a deletion's created_at is not its
target's), shouldPropagateDeletionUp() (kind-5 always; kind-62 only to a relay
it targets, honoring the vanish's declared relays), and
negentropyPropagateDeletions() — one bidirectional reconcile that streams
have→upload and need→download.
amy sync: run the side-channel bidirectionally regardless of --up/--down
whenever the filter excludes 5/62; emits deletions_{need,have,downloaded,
uploaded}; --no-sync-deletions opts out.
geode MirrorWorker: thread a per-upstream deletionScope through both catch-up
phases and both live subs (down + up), in the mirror's configured direction;
relax down containment to accept in-scope deletions, gate kind-62 pushes by
target relay, and carry the deletion filter on re-subscribe so a reconnect
never drops it.
Tests: DeletionSyncTest (up/down propagation + filter/vanish-gate units) and
MirrorDeletionSyncTest (a kind-scoped down mirror still removes the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The crawl persists everything to the store and the score is a pure function
over it, so separate them: `amy graperank sync` crawls the reachable graph into
the store (idempotent + cumulative — run it a few times to be sure it's loaded)
and reports what it loaded without scoring; `amy graperank score` builds from the
store and scores instantly, repeatable with different params and no re-crawl
(same as bare `--offline`). Bare `amy graperank` stays the sync+score combo.
Extract the shared crawler wiring into newCrawler(); score() just forces the
offline path, and sync() runs a persist-only crawl (null builder).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
amy gated every non-primitive verb behind a chosen account: `DataDir.resolve`
threw when `~/.amy/` had no unambiguous account, and every networked command
called `Context.open`, which requires an identity — even though queries only
read relays and the shared store and never sign. `store` maintenance and the
local `offer`/`debit info` decoders were caught by the same gate despite
touching no account state.
Reads now work anonymously; only signing needs an account:
- `DataDir.resolveOptional` hands back an accountless dir (`hasAccount = false`)
pointing only at the shared event store when there is no unambiguous account,
instead of throwing.
- `Context.openOrAnonymous` uses the resolved account when present, else an
ephemeral key-less `Identity.anonymous()` — can read, can't sign. Marmot
stores are now lazy and run-state isn't persisted for anonymous runs, so an
accountless read leaves `~/.amy/shared/` clean.
- `Context.open` (signing path) re-asserts the requirement with the "which
account?" hint, so ambiguous/no-account signing verbs still exit 2.
- Main resolves optionally for every verb except the identity-lifecycle ones
(`init`/`create`/`login`/`logoff`/`whoami`), which still need a concrete
account. `offer info` / `debit info` join the stateless primitive block.
- Read subverbs (fetch, subscribe, count, publish, outbox, search, sync,
store, profile/git/podcast/podcast20 reads, nsite/napplet fetch·serve·list,
blossom download·check) switch to `openOrAnonymous`.
No `--json` shapes change. Docs updated (help text, README, DEVELOPMENT).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TRaoqGod5LUeSwq4GHRSNF
A cross-account, read-only snapshot of everything amy holds under
`~/.amy/`, built for the returning user: which accounts exist, which
one is pinned as current, each signer type (local keychain/ncryptsec/
plaintext, NIP-46 bunker, or read-only) and whether it can still sign,
the per-account local footprint (aliases, Marmot groups, published
KeyPackage bundle, Cashu wallet, sync cursors), and the shared event
store's size.
Like `use`, it dispatches before account resolution so it works with
zero, one, or many accounts. Strictly metadata-only: it never unlocks a
private key (no keychain prompt / NIP-49 passphrase) and never touches
the network.
Factors the on-disk event-store walk into a shared `StoreStats` helper
reused by both `status` and `store stat`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K3GJb11JkvopP61ETWAVyy
The CLI ran at the library's default Log.minLevel = DEBUG, so quartz internal
chatter (relay-auth init, MLS restore, URL-rejection, throttle notices) leaked
onto stderr around every command's real output. Set Log.minLevel = WARN at
startup, before dispatch, so a normal run shows only warnings/errors plus the
command's own progress. A new global --verbose / -v flag restores full DEBUG;
it's parsed with the other global flags so subcommands never see it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Claude
davotoula
Vitor PamplonaandGitHub