The CLI ran at the library's default Log.minLevel = DEBUG, so quartz internal
chatter (relay-auth init, MLS restore, URL-rejection, throttle notices) leaked
onto stderr around every command's real output. Set Log.minLevel = WARN at
startup, before dispatch, so a normal run shows only warnings/errors plus the
command's own progress. A new global --verbose / -v flag restores full DEBUG;
it's parsed with the other global flags so subcommands never see it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-relay concurrency/rate throttle notices and the "Rejected <url>"
normalizer messages fire constantly during a large crawl (thousands of
rejected/throttled relays) and are operational detail, not warnings. Move
them from Log.w to Log.d so they stay available under debug logging without
flooding a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The park window's timeout was absolute from subscription open, so a relay
still actively streaming a large result set once it passed parkTimeoutMs was
unsubscribed and its untransmitted tail lost. Reset the window on every
incoming event (a conflated activity signal drives a select against the
terminal deferred), so a parked subscription is closed only after
parkTimeoutMs of actual silence — never while events are still arriving. The
fast window stays absolute: it only decides when to hand a slow relay to the
background park lane, which loses nothing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The crawl was round-synchronised: each hop drained all its relays and only
started the next hop after the slowest one reached EOSE or the timeout. That
made waiting for slow-but-alive relays expensive — every hop paid its slow
tail before the next hop's fast relays could begin — so a long timeout for
completeness cost ~2x wall-clock (measured), and a short one dropped the slow
relays' data.
Diagnostics on a ~190k-user crawl showed the genuinely-slow set is a stable
~30 relays that DO reach EOSE, just in 5-25s. So decouple the two concerns:
- drainGated now drains on the FAST `timeoutMs` that sets the round cadence. A
relay still streaming when it elapses is not cut but PARKED: it hands its
open subscription to a background scope (releasing its AdaptiveRelayLimiter
permit so the round moves on), keeps receiving for up to the new
`parkTimeoutMs`, and its late events are persisted + its late contact lists
pushed to a crawl-wide lateHarvest channel.
- The round loop folds late harvest into the graph between rounds and won't
converge until the frontier is empty AND no relay is still parked — so the
crawl waits for slow relays for completeness without paying that wait in each
round's wall-clock.
Graph state stays single-writer: parked coroutines only touch the store,
seenIds, and the channel — never hopOf/done/builder. Persistence moved from a
single per-drain consumer to a shared `persist()` that fast and parked units
both call; crawl-wide dedup is now race-safe via ConcurrentSet.add's atomic
test-and-set (an id is added only after a good signature, so no duplicate
reaches the store's UNIQUE constraint and a forged copy can't suppress the
genuine one).
Also carries the --diagnose slow-relay logging (relay + filter + elapsed for
every slow/parked REQ, so a human can replay it) and keeps --drain-concurrency
at the validated default of 24 (an A/B at 64 was ~2x slower with more dead
relays). New --park-timeout flag (default 40s; set <= --timeout to disable).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Add a --diagnose slow-relay log: every content drain that reaches its terminal
(EOSE or timeout) slower than SLOW_DRAIN_LOG_MS, or times out entirely, is
recorded with the offending relay URL, the failure/EOSE reason, elapsed ms, and
the exact filter shape (kinds + author count + first authors). This lets a human
replay that precise REQ later to understand why the relay lags. Gated on
--diagnose so there is no per-group timing/collection overhead otherwise.
Make the content-drain fan-out configurable via a new --drain-concurrency flag
(Config.drainConcurrency), replacing the DRAIN_CONCURRENCY constant. Default
stays at the validated 24: an A/B at 64 ran ~2x slower with more dead relays
(a higher global fan-out re-floods busy hubs faster than the per-relay demotion
catches up), so the flag is a probe knob, not a speedup. Client WebSocket pings
were also tried and reverted — busy-but-alive relays don't reliably pong while
their query handler runs, so pinging just cut them as dead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-user retry counter was blunt: it bumped attempts whether an outbox was
dead, timed out, or cleanly EOSE'd with no event — so a straggler kept being
re-queried against a live relay that had already definitively answered it lacks
their kind:3. Distinguish the cases: drainGated now reports the relays that fully
EOSE'd (answeredOut); the consumer records, per user, the relays that answered but
did not return their contact list (askedEmpty); routeByOutbox excludes those from
the user's candidate relays. A timed-out relay is never added (it might just be
slow — still worth a retry), only a clean-EOSE-empty one; dead relays stay pruned
as before.
Measured on --max-hops 3: redundant fetching dropped ~8% (74k -> 68k events
stored). It does NOT move the wall-clock tail, though — that tail is dominated by
timeout/dead outboxes (the retryable case), not EOSE-empty relays. The wall-clock
lever remains the timeout retry budget (MAX_OUTBOX_ATTEMPTS / drain timeout).
Mirror the crawler extraction on the emit side: the NIP-85 kind:30382 card
reconcile + publish logic (existingCards read-back, rank-diff upsert, stale-card
kind:5 retraction batched under the 64KB event cap) moves out of GrapeRankCommand
into a reusable GrapeRankPublisher in quartz experimental/graperank. It takes an
IEventStore for the prior-card read-back and an injected publish function
(event + relays -> per-relay ack), so the store/relay wiring stays in the app
while the reconcile logic is reusable (e.g. by the Android app).
GrapeRankCommand is now a thin orchestrator: crawl (GrapeRankDataCrawler) ->
score (GrapeRank) -> publish (GrapeRankPublisher). The account-specific bits stay
in the CLI: operator-key derivation, the observer's kind:10040 discovery pointer,
and the operator/register/providers sub-verbs.
The crawl re-verified and re-inserted the same event many times: the outbox
model mirrors each event (especially kind:10002 relay lists) across relays,
indexers, and rounds, but dedup lived in a per-drain SeenIds, so only the copies
within one drain were caught. Add a crawl-wide seen-set (thread-safe ConcurrentSet
of event ids, shared across all 24 concurrent drains and every round), checked
before verify and added only after verify so a forged copy can't suppress the
genuine one. Group-commit the store writes via IEventStore.batchInsert instead of
one transaction per event.
Measured on a from-scratch --max-hops 3 crawl: events actually verified+stored
dropped ~34% (112k -> 74k) and verify time fell in lockstep. The write path now
also reports verify/insert timing + events_stored in Stats, exposed as verify_ms/
insert_ms/events_stored on the CLI, and takes an --insert-batch knob.
Finding: with the work reduced, inserts serialize on SQLite's single writer
mutex rather than transaction count, and the crawl's wall-clock ceiling is the
drain-timeout retry tail on dead outboxes, not the disk.
Cleanups from a reuse/simplification/efficiency/altitude review of the crawler
extraction:
- Collapse the redundant `discovered` set into `hopOf` — a user is discovered
iff it has a hop stamp, so the two always held the same key set. The frontier
is now `hopOf.keys`; one fewer collection to keep in sync.
- Drop the unused `Stats.discovered` / `Stats.deadRelays` fields (no reader —
the CLI reports rounds / relaysContacted / hopHistogram / downloadMs).
- Extract a single shared verify-then-store sink, `IEventStore.verifyAndInsert`,
and route both the crawler and `Context.verifyAndStore` through it instead of
each carrying its own verify + insert + UNIQUE-swallow copy.
- Fast-path the present-key hit in `ConcurrentMap.getOrPut` (jvmAndroid) so the
crawl's hot relay-hint accumulation stops allocating a mapping-function closure
on every call.
- Hoist the repeated `crawlStats?.hopHistogram` null-plumbing in GrapeRankCommand.
The web-of-trust crawl (~400 lines: outbox routing, sharded backbone sweep,
Phase-B worker pool, relay-list discovery, report-deletion fetch, warm pool)
was making the CLI's GrapeRankCommand unmaintainably large. Move it into a
reusable, KMP-portable GrapeRankDataCrawler in quartz commonMain.
The crawler takes a NostrClient + IEventStore + AdaptiveRelayLimiter, injected
relay policy (discovery + content-fallback sets, since those defaults live in
app code, not the protocol library), and a log callback; it streams contact
lists into a TrustGraphBuilder and returns crawl Stats. GrapeRankCommand shrinks
to arg-parsing + offline load + scoring + publish + sub-verbs, delegating the
online path to the crawler.
To reach commonMain (portable to every target, incl. iOS):
- Add ConcurrentMap / ConcurrentSet expect classes under utils/concurrent, with
jvmAndroid actuals (java.util.concurrent) and native actuals (copy-on-write
over kotlin.concurrent.atomics.AtomicReference, mirroring ConcurrentHashCache).
commonMain has no ConcurrentHashMap, and the crawl's producer/consumer/drain-
worker state needs atomic getOrPut/merge plus a concurrent set.
- Move AdaptiveRelayLimiter and DrainFailure/classifyDrainFailure from cli to
quartz commonMain (java atomics -> kotlin.concurrent.atomics, ConcurrentHashMap
-> ConcurrentMap, System.currentTimeMillis -> TimeUtils.nowMillis, stderr -> Log).
- The gated drain (REQ-size splitting, per-relay permits, verify+store) moves into
the crawler; Context.drain loses its now-unused gatePerRelay path.
Net: cli -1077 lines; the crawler + relay machinery are now reusable by the
Android app. Adds ConcurrentCollectionsTest; verified via JVM + commonMain
metadata compile, the wot/graperank suites, and a bounded live crawl.
Each addressable coordinate is ~130 bytes, so 500 pushed the deletion event to
~65KB — over the 64KB event-size cap many relays enforce (stricter than the
256KB message cap). Drop DELETE_PER_EVENT to 400 (~52KB).
README + amy usage: add the `graperank operator [status|relay|providers]`
sub-verb, update the `graperank --publish` description to the per-observer
service-key model (sign with a derived key, publish to the operator relay,
reconcile new/changed/skip/retract, cutoff rank>=2, NIP-09-drop retracted
reports), and add a 'Publishing GrapeRank scores' section explaining the
operator master, deterministic per-observer key derivation, and the kind:10040
discovery wiring.
Rewire `graperank --publish` onto the operator-key model:
- Sign each observer's kind:30382 cards with the dedicated service key derived
for that observer (OperatorKeys), not the account key — a stable per-observer
identity so re-signing replaces the addressable prior card.
- Publish to the operator's configured relay(s) (new `graperank operator relay
<url>` sub-verb; --publish-relay still overrides). Errors clearly if unset.
- Three-way reconciliation against what the provider key already published:
upsert cards whose rank tag string changed (or are new), skip unchanged, and
RETRACT (kind:5, same service key, addressable `a`-tag, chunked under the
message cap) any existing card whose target is no longer publishable — dropped
from the graph, or below the cutoff.
- Raise the default publish cutoff to rank >= 2 (drops the barely-trusted tail);
the retract rule removes any now-sub-cutoff cards.
- When we hold the observer's key (observer == active account), publish/refresh
their kind:10040 pointing 30382:rank -> providerPubkey at the operator relay,
to their outbox — the pointer clients follow to find the cards.
Adds `operator [status|relay|providers]` for managing the machine's operator.
A machine holds one operator master seed, independent of any amy account, stored
under ~/.amy/operator/ via the same SecretStore backend the accounts use. From it
OperatorKeys deterministically derives one service key per observer —
serviceKey(observer) = sha256(masterPriv || "graperank-provider:" || observerHex)
— which will sign that observer's kind:30382 rank cards and their retractions.
Deterministic derivation gives a stable per-observer identity (so re-signing a
card replaces the addressable prior one instead of orphaning it) and one-secret
backup (every service key re-derives from the master alone). The manifest
(operator.json) records the master pubkey, operator relay(s), and observer ->
provider-pubkey mapping — public data; only the master rides the SecretStore.
Exposed via DataDir.operatorKeys(). Wiring into publish comes next.
A REQ carries all of a subscription's filters in one frame, so a popular relay
routed thousands of authors produced a multi-MB frame that most relays reject
outright ("message too large (2MB > 256KB)"), silently dropping every author
in it. The gated drain now splits each relay's filters into REQ-sized groups by
total entry count (authors + ids + tag values), MAX_REQ_ENTRIES=2500 (~167KB,
under the common 256KB cap), and opens one gated subscription per group. A relay
with more authors simply gets several smaller REQs instead of one rejected huge
one. Each group carries its own subId/listener/terminal signal; per-relay
failure classification takes HARD over TRANSIENT across a relay's groups.
The outbox model — and especially the wide relay-list broadcast — delivers the
same event from many relays at once, and the gated drain ran a Schnorr verify
(and a store insert) on every copy before the store's UNIQUE constraint dropped
it. On a fan-out that asks hundreds of relays for the same kind:10002s, that is
hundreds of redundant verifications per event and pegged a core.
Add a per-drain SeenIds skip-before-verify to the consumer, mirroring
drainAllPages: an id is marked seen only after it verifies, so a forged copy
(valid id, bad signature) delivered first can't suppress the genuine one. Cuts
the redundant verification across the whole crawl, not just the wide sweep.
Replace the hand-rolled report-id/author matching with quartz's DeletionIndex —
the same NIP-09 indexer the Android app's LocalCache uses. It keys each deletion
under the deleter's pubkey, so hasBeenDeleted(report) is authoritative only when
the report's own author deleted it, and it also handles created_at ordering (and
addressable events, for free). Deletions come from the store, which already
verified them, so they're added as pre-verified.
A report the author has since deleted should not count as a negative trust
edge. After the crawl, ask each reporter's outbox for kind:5 deletion requests
that cite the reports we gathered — #e-filtered to those report ids, so we pull
only the deletions that affect our reports, not every deletion the user ever
made. When building the graph, a report is dropped iff a kind:5 in the store
cites its id AND is signed by the report's own author (NIP-09: a deletion is
authoritative only from the event's author). Reports the reporter never
retracted are unaffected. The run reports reports_deleted.
Gate publishing on the exact rank TAG VALUE STRING, not a re-parsed Int. A
card carries only a `rank` tag (plus the d-tag target), and RankTag.assemble
writes `rank.toString()`, so we diff that string against the one on the newest
kind:30382 card the signing key already published (read back from the store).
An unchanged score is skipped — no new signature, no new event id — so a client
that syncs the provider's cards by id only ever downloads the ranks that
actually moved. Replaces the prior Int comparison with a faithful
what-would-be-written string diff.
Three crawl fixes:
1. Fetch kind:10002 alongside content. The content query asked only for
3/10000/1984, so a user's freshest relay list — which lives on their own
outbox — was never pulled from there; we trusted a possibly-stale indexer
copy. Fold 10002 into the same fetch. The store keeps newest-by-created_at,
so pulling it from popular relays too can't stale it.
2. Widen ensureRelayLists Tier 2. It only asked the top-30 backbone for a
still-missing 10002. A stray relay list can sit on any one relay, so Tier 2
now asks EVERY relay we've seen work — fired fire-and-forget on a background
scope so the large fan-out never blocks the round; results enrich routing
for later rounds.
3. Classify dead relays instead of striking everything the same. A connect
TIMEOUT is a busy relay — retried, never marked dead. A HARD failure (bad
domain, TLS misconfig, dead HTTP code — see DrainFailure/classifyDrainFailure,
keyed on the exception type now in the failure message) is dropped on the
first strike. Transient failures (refused/reset/unreachable, 429/5xx) keep
the multi-strike leniency. Connect timeout raised 5s -> 7s so slow-but-alive
relays finish the handshake.
A relay pushes back for two different reasons that need two different fixes,
and treating them the same mishandles the relay:
- a subscription-COUNT cap ("too many subscriptions", "maximum concurrent
subscription count") is fixed by fewer CONCURRENT subs — demote the
per-relay concurrency cap (100 -> 20 -> 10), as before;
- a RATE limit ("rate-limited: too many messages", "burst exhausted") is
too many subscription CHANGES per second — fewer concurrent subs don't
help; the fix is to SPACE the REQs out in time.
AdaptiveRelayLimiter now routes each complaint to its own actuator by matching
the notice text, and adds a per-relay rate gate: a growing minimum interval
between subscription opens (250ms -> 500ms -> 1s -> 2s), enforced in withPermit
before the concurrency permit. A relay can be under both controls at once. The
snapshot reports each dimension separately.
BasicRelayClient collapsed a connection failure into a message string built
from the throwable's text alone. Message text is localized and inconsistent
across platforms, so a listener can't reliably tell a busy relay (a connect
timeout) from a dead one (bad domain / TLS misconfig) from it. Always append
the exception class name (SocketTimeoutException / UnknownHostException /
SSLHandshakeException / ConnectException …), which is stable, so listeners
can classify the failure by type. Message text is preserved; the type is
added in parentheses. Updated the one test that pinned the old format.
Phase B drained DRAIN_CONCURRENCY batches, waited for the SLOWEST (a dead
relay's full timeout), ingested, then started the next group — so every
batch's long tail idled the whole pool, and connections were torn down and
rebuilt between groups. Replace the chunked awaitAll barriers with a
continuous producer -> workers -> consumer pipeline:
- Producer (1 coroutine) routes each author-batch by outbox and feeds a
bounded queue (keeps writeRelayFreq single-writer, backpressured so we
don't precompute every filter map at once).
- DRAIN_CONCURRENCY workers pull a batch, drain it, and grab the next the
instant the drain returns — no worker waits on a slow sibling, and hot
relays stay connected because some worker is always subscribed to them.
- Consumer (1 coroutine) ingests serially (discovered/done/builder/hopOf
stay single-writer), now overlapped with draining instead of blocked
behind each batch.
The four structures now crossed between producer/worker/consumer
(relayHints, attempts, deadRelays, relayStrikes) become concurrent; all
graph mutation stays single-writer on the consumer. Removes the dead-relay
timeout stalls that were serializing the crawl and cuts reconnect churn.
The crawl's client ran on OkHttp defaults: Dispatcher.maxRequests=64 and a
10s connectTimeout. Every relay WS-upgrade handshake is an async call through
that shared dispatcher, so 64 caps the connection-ramp width — and a dead
relay squats on a slot for the full connectTimeout, starving live relays
queued behind it (observed: only ~150 sockets open at once during an active
wave touching hundreds of relays). Raise maxRequests to 256 /
maxRequestsPerHost to 16 and drop connectTimeout to 5s so unreachable relays
release their slot fast.
This is orthogonal to REQ concurrency (bounded per-relay by
AdaptiveRelayLimiter on already-open sockets), so it can't trip a relay's
REQ rate-limit — it only speeds connection setup. The dispatcher's executor
pool grows threads on demand, and FD headroom is ample (4096 limit vs ~150
in use), so the wider cap just lets more short-lived handshakes run at once.
Measured 48 against the per-relay adaptive limiter (hop-4 A/B): it demoted
the right hubs and cut rate-limited CLOSEDs further (1433 -> 501), but the
higher fan-out re-floods busy relays faster than demotion catches up — a
new dominant complaint ("max concurrent subscription count reached")
appeared and download_ms regressed ~11% vs the 24 baseline. Keep the
adaptive per-relay cap (it targets the misbehaving relays precisely) but
return the global fan-out to 24, where the 20/10 ladder still bites below
the global bound and wall-time stays at its best-observed value.
The GrapeRank engine, TrustGraph (compact int-CSR) and TrustGraphBuilder
are pure Nostr-social-graph computation over HexKeys — no UI, no Compose,
and no commons-only dependency. They're a utility for implementing the
NIP-85 rank assertions quartz already models, so they belong in quartz
rather than commons. Move commons/wot -> quartz experimental/graperank
(package com.vitorpamplona.quartz.experimental.graperank), including both
commonTest suites, and repoint the CLI import. TrustGraphBuilder was already
protocol-agnostic (takes HexKey lists; the caller does the event->edge
extraction), so nothing had to change but the package. Makes the algorithm
reusable by the Android app for spam/trust filtering without pulling in
commons.
Replace the blunt global concurrency number with per-relay back-pressure.
Every relay starts generous (100 concurrent subscriptions) and is demoted
down a ladder (100 -> 20 -> 10) only when it complains about concurrency —
a CLOSED rate-limited, or a NOTICE like "too many concurrent REQs" /
"too many subscriptions" / "burst exhausted". Well-behaved relays keep
the full cap; only the busy hubs that push back get throttled, and only as
far as they keep pushing.
AdaptiveRelayLimiter registers as a RelayConnectionListener so demotions
are driven straight off the same NOTICE/CLOSED frames RelayDiagnostics
already observes, keyed by relay.url. Context.drain gains a gatePerRelay
path that opens one subscription per relay, each held behind that relay's
gate, so our concurrent subs on it never exceed its current cap. The gate
is a fair FIFO bounded semaphore whose limit can only be lowered; shrinking
below the in-use count admits no new subs until enough finish, so
concurrency converges down to the new cap.
Because a hot relay can no longer be flooded, the global content-drain
fan-out is raised (18 -> 48) to crawl the many well-behaved relays faster.
The crawl emits a relay_throttling summary (which relays were capped, and
to what) alongside relay_feedback.
Each concurrent content drain opens exactly one subscription per relay it
touches, so DRAIN_CONCURRENCY is effectively the per-relay concurrent-sub
cap. RelayDiagnostics showed the previous value (24) blew past typical
relay limits — rate-limited=1433, "too many concurrent REQs"=1286,
"too many subscriptions"=710 — causing dropped fetches and retry churn.
Lower it to 18 so the peak (18 drain subs + 1 persistent warm-pool sub)
stays ~19, just under the common ~20 cap.
The client's relay pool reconciles open sockets to the relays that active
subscriptions currently need, so between-round gaps (routing + contactsOf scans
> ~300ms) and niche-relay churn dropped connections we reuse every round, then
reconnected them — a TCP+TLS+WS handshake each time.
Hold a persistent do-nothing subscription (WARM_SUB_ID) open to the busiest
WARM_POOL_SIZE(20) live relays, refreshed to the current top set at each round
start (same subId → just updates the desired-relay set) and closed when the
crawl finishes. Its filter matches an impossible event id, so the relay EOSEs
immediately and streams nothing — it only keeps the socket warm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Two blind spots in the crawl's relay I/O:
- No NIP-42 auth. Auth-gated relays sent AUTH, the client never answered, and
their sub CLOSed 'auth-required' — so those outboxes served us nothing. Wire a
RelayAuthenticator into Context that signs the AUTH challenge with the account
key (local signer only; a remote bunker is skipped to avoid a per-relay
round-trip storm mid-crawl). Signing with any key still unlocks relays that
just want some auth.
- No visibility into REQ failures. Add RelayDiagnostics, a connection listener
that tallies NOTICE frames, CLOSED reasons by NIP-01 prefix (auth-required /
rate-limited / restricted / …), and AUTH challenges. Surfaced in the crawl's
stderr summary and as relay_feedback in the JSON, so a failed fetch can be
explained instead of guessed at.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Review of the sharded-sweep restructure flagged two recall regressions vs the
removed last-mile:
- the sweep + broadcast only ever hit the top SHARD_RELAYS (10), while the old
last-mile reached busy relays ranked 11-80 where a user's kind:3 is often
mirrored. Broadcast the small remainder to BROADCAST_RELAYS (60) top live
relays instead of just the rotation's 10, restoring that reach (indexers are
intentionally excluded — they don't serve kind:3).
- MAX_DEAD_STRIKES was 2 with no recovery, so two transient connect blips
evicted a relay for the whole run. Raise to 3 for a safety margin; drain still
only counts hard connect failures, not slow relays.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The download phase was ~90% idle, blocked on the per-wave drain timeout waiting
on dead/stalled outbox relays. Three changes:
1. Dead-relay pruning. Context.drain now reports relays that failed to CONNECT
via a deadOut set; the crawl strikes them (MAX_DEAD_STRIKES=2) into a
deadRelays set and excludes them from routeByOutbox and the sweep, so a wave
stops re-paying the timeout on the same dead outboxes.
2. DRAIN_CONCURRENCY 8 -> 24, safe now that dead relays are pruned rather than
piling up as stalled connections.
3. Sharded backbone sweep (Phase A each round): split the pending authors across
the top-SHARD_RELAYS(10) live relays — one shard per relay, no relay gets the
same list twice — drain all concurrently, rotate the still-missing onto
different relays for up to SHARD_ROTATIONS(6) passes, then broadcast the
remainder to all top relays only once it drops below SHARD_BROADCAST_THRESHOLD
(2000). Phase B then outbox-routes only whoever the popular relays lacked. The
old broadcast-everyone last-mile is removed (the sweep subsumes it).
Each concurrent drain uses its own dead-set (no shared-HashSet race); harvest
feeds from the drain's returned events instead of re-scanning contactsOf over
the whole missing set. Adds download_ms so the crawl phase is timed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The online crawl phase — the network-bound fetch of the whole graph off the
relays (rounds + last-mile sweep) — was untimed; only the offline store-load
had a phase timer. Add download_ms around the crawl block and fold it into the
"crawl complete" log and the JSON, so a from-scratch run reports every phase:
download -> graph build -> scoring (and, with --bench-sign, card signing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Adds a benchmark path to `amy graperank`: after scoring, build + sign one
NIP-85 kind:30382 ContactCardEvent per scored user (rank >= --min-rank) with a
throwaway keypair, fanned out across CPU cores, and report bench_signed +
bench_sign_ms. The signed events are discarded — this measures the id-hash +
Schnorr-sign cost of emitting the full card set without touching any relay or
real identity. Complements the existing store_load_ms / graph_build_ms /
scoring_ms phase timings so the whole pipeline (load → build → score → sign) is
measured end to end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The worklist re-enqueued a node's dependents on every >convergence nudge, so on
a dense graph its total work scaled with the in-degree of the churning core —
an offline score over the full ~419k-node / 19.3M-edge Vitor graph ran 640M+
node-visits and still had not converged after 32 minutes.
Replace it with synchronous Gauss-Seidel sweeps over all nodes (in-place
updates, so trust flows outward within a sweep), ending when no node moves more
than the convergence delta. Same per-node formula, same 0.0001 threshold, same
unique fixed point as NosFabrica's Brainstorm reference (which iterates the same
way) — verified byte-identical by the existing GrapeRankTest suite — but each
node is touched once per sweep instead of once per churning rater, cutting the
work by roughly the average in-degree. Progress now reports per-sweep
(node-updates + nodes-still-moving) and the CLI surfaces the sweep count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The graph_build_ms timer covered only builder.build() (the in-memory int-CSR
pack, a few hundred ms). It excluded the real pre-scoring cost: reading and
deserializing every kind:3 contact list out of the store. Add store_load_ms
(offline path) so the three phases — store load, CSR build, scoring — are each
measured and reported (stderr + JSON), instead of the load hiding behind a
misleadingly small build number.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Outbox resolution used only the fixed indexer/aggregator set to find a user's
kind:10002. Users the aggregators don't carry got no relay list, so their
content couldn't be routed to their own outbox (falling back to hints / the
broad last-mile content sweep).
Add a tier-2 pass in ensureRelayLists: any pubkey still without a relay list
after the indexer sweep is retried for kind:10002 against the known-good
backbone — the busiest live relays learned from the `r` tags in everyone
else's 10002s. A user publishes their own 10002 to their own write relays,
which overlap heavily with that pool, so this recovers relay lists the
aggregators miss. Bounded to the backbone (not an unbounded fan-out to every
working relay) to avoid connection saturation; early rounds no-op until the
backbone is learned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Measure and surface the two compute phases of `amy graperank`: building the
int-CSR trust graph and running GrapeRank to convergence. Both are logged to
stderr ("graph built … in N ms", "scored N users in M ms") and added to the
--json result as graph_build_ms / scoring_ms, so the pure scoring cost over a
given dataset is measurable without eyeballing logs — e.g. an `--offline` pass
over a fully-crawled store times score generation with no network in the loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The SQLite backend raises a catchable UNIQUE-constraint exception when an
event is a duplicate id or an older/duplicate replaceable (kind 0/3/10000-
19999) — which the outbox model produces constantly, since each user's
replaceable is fetched from several of their write relays. verifyAndStore
was logging every one as `[cli] store insert failed`, so a full-network
GrapeRank crawl emitted ~294k spurious error lines. The store is behaving
correctly (its partial unique index + trigger keep the newest version and
reject stale copies); the FS backend simply no-ops on the same duplicates.
Suppress UNIQUE-constraint rejections (normal dedup) while still surfacing
genuine persistence failures (I/O, full disk, corruption).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Audit follow-ups before merge:
- amy fetch default limit is now the same on both paths: absent --limit → 100
for plain AND --paginate (previously --paginate silently meant "unbounded").
`--limit 0` is the explicit opt-in to drain everything (unbounded); negative
is rejected. The effective limit is carried on the filter so both paths agree.
- drainAllPages sizes its SeenIds for CLI-scale fetches (initialSlotsPow2 = 12,
~64 KB) instead of the large-walk default (~16 MB eagerly allocated per fetch);
it grows if an unbounded drain needs it.
- fetchAllPages clamps the inclusive advance to `min(pageMinTs, boundary)` so a
misbehaving relay that answers with an event past the requested `until` can't
push the cursor upward — the boundary dedup and termination rely on `until`
never increasing. No-op for honest relays (they only return events ≤ until).
Verified live: default and --paginate both cap at 100; --limit 50 → 50; --limit 0
--paginate drains the full window (>100); paging tests + SeenIds tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Two changes to the paginated fetch path:
- Cross-relay dedup before verify. drainAllPages' single consumer now runs a
SeenIds filter: the same widely-mirrored event arrives once per relay, and the
repeats are dropped BEFORE the expensive Schnorr verify + store instead of
after (they were only trimmed by FetchCommand's distinctBy). An id is marked
seen only once it verifies, so a forged copy (valid id, bad sig) delivered
first can't suppress the genuine one from another relay. Adds SeenIds.contains
(peek without recording) for that check-then-add.
- `amy fetch --paginate` no longer forces a --limit. With --limit N it still
pages up to N per relay; WITHOUT --limit it drains the whole filter unbounded
(the filter's null limit flows straight through). Plain (non-paginate) fetch
still trims to the default 100.
Verified live: unbounded --paginate over a ~20-min nos.lol firehose window
returns 406 (all unique, 3s) vs the old 100 cap; --limit 50 caps at 50; default
caps at 100; cross-relay fetch stays count==uniq.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
The FS event store writes one pretty-printed JSON file per event plus one
file per index posting (kind, author, every p-tag value). At crawl scale
this explodes: a 96k-event GrapeRank crawl produced 5.6M tiny index files
rounding up to 2.8GB on disk — only 457MB of which was actual event data.
An 8-hop crawl would blow past available disk.
Wire amy's shared store through a new StoreFactory that selects the backend
from AMY_STORE (default `sqlite`, opt into the legacy tree with `fs`). Both
implement IEventStore, so every command works unchanged. SQLite packs the
same postings into shared B-tree pages — several times smaller on disk and
the natural fit for large crawls. The two stores live side by side under
`<data-dir>/shared/` (events.db vs events-store/) so switching never
clobbers the other's data.
`amy store` maintenance verbs are now backend-aware: stat reports the total
+ disk bytes for both (kind histogram/mtime stay fs-only); scrub is a no-op
on sqlite (indexes are transactional); compact runs VACUUM on sqlite.
Verified end-to-end via the built amy image on both backends: init, notes
post round-trip (event persisted + read back), and every store verb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Drop @Synchronized from add/reset/size: SeenIds is now documented as
single-writer (not thread-safe). Callers dedup across concurrent relay
producers by funneling events into one consumer that owns the instance — the
one-consumer ingest pattern used elsewhere — which keeps a single global set,
stays lock-free, and lets resize run without coordination.
With the JVM-only @Synchronized gone the class is pure common Kotlin
(LongArray + Hex.readLong), so it moves from the jvmAndroid source set to
commonMain and is now available on every target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
A run-scoped "already seen this id" filter for large, mostly-duplicate id
streams (a broad relay walk re-receiving the same event from many relays).
Keys on the first 128 bits of the id, sliced straight out of the hex with
Hex.readLong (table lookups, no parse, no allocation), in one open-addressed
LongArray — ~16 bytes/entry and the 64-char String is never retained, so tens
of millions of ids cost ~1 GB instead of a HashSet<String>'s ~6 GB. add() is
O(1) and synchronized.
Lives in the jvmAndroid source set (uses @Synchronized; a 40M-id walk is a
server-side concern). Ports the caller's implementation with the
parseUnsignedLong hot path swapped for Hex.readLong (~45-70 ns/op cheaper).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
After the outbox crawl gives up on users whose own kind:10002 relays never
answered (dead/misconfigured outboxes), take one more pass at every still-missing
user within the hop budget against the WHOLE known-good relay pool — the busiest
live relays learned from the crawl (which include the big aggregators) plus the
discovery set — instead of re-asking each straggler's broken outbox. Recovered
contact lists feed the graph and can reveal a few more reachable users, so the
sweep repeats up to LAST_MILE_PASSES times until it stops recovering.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Answers "do we try relays we know work from other people's lists when a user
isn't in the first set?" — now yes.
- Drop dead relays I'd added unchecked (relay.nostr.band, relayable.org,
relay.nostr.bg); keep only ones that reply to a limit:1 query. NIP-11 presence
is not used for liveness (it's optional).
- Learn a backbone dynamically from the crawl: tally how often each relay appears
as someone's kind:10002 write relay, and mark relays that actually delivered
events as live. The most-used live relays form the backbone.
- Route retried users (whose own outbox already failed) and outbox-less users to
outbox + backbone, since popular relays usually hold a copy of their kind:3.
Effect for Vitor (--max-hops 3): hop-3 coverage rose from ~112,600 to 146,413
(95% of Brainstorm's 153,409), as round-3 contact-list fetches more than doubled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Fixes the hop-2 (and general) completeness undercount, and makes the cause
observable.
- Add a broad set of aggregator + big general relays (relay.nostr.band,
relay.damus.io, snort, offchain.pub, relayable.org, …) to the kind:10002
discovery set. Effect: for Vitor, hop-2 discovery went from ~16,980 to 19,886
(~83% -> ~98% of Brainstorm's 20,332).
- `Context.drain` gains a `diagnoseSlow` flag: on a timeout it logs which relays
stalled and why — slow (no EOSE, with the event count they did send) vs
cannot-connect (with the failure reason) vs closed. `amy graperank --diagnose`
turns it on. This showed the remaining misses are relay-side: users advertise
dead/misconfigured write relays (HTTP 404/530/503, "Unexpected response",
read/connect timeouts), not anything blocking on our end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
fetchAllPages advanced with `until = oldest - 1` (exclusive) and no dedup. That
skips any event sharing the boundary second that didn't fit in the page — which
happens at *every* page boundary landing inside a second, not just pathological
dense ones — silently dropping events. An in-process probe with no second denser
than the relay's page cap still lost one event straddling the boundary.
Page inclusively now: `until = oldest created_at of the previous page`, and drop
the re-fetched boundary events by id. The dedup set is bounded to just the current
boundary second (`until` only decreases, so duplicates can only recur there), so
memory stays O(one second), never O(total).
A single second denser than the relay's page cap can't be drained (its tail is
unreachable — no client-side fix; raising the request limit is futile since we
already send one above the relay's cap). Once a page yields nothing new we step
strictly past that second so paging keeps progressing to older events instead of
stalling forever.
Tests: boundary-straddle retrieves all 6 (was 5); dense-second-beyond-cap steps
past without stalling and still delivers the neighbours. Verified on live relays
(strfry / nostr-rs-relay / khatru): ground-truthing each dense internal second
against the paginated set shows no gaps, incl. a 36-event second fully retrieved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Claude
Vitor PamplonaandGitHub