A relay pushes back for two different reasons that need two different fixes,
and treating them the same mishandles the relay:
- a subscription-COUNT cap ("too many subscriptions", "maximum concurrent
subscription count") is fixed by fewer CONCURRENT subs — demote the
per-relay concurrency cap (100 -> 20 -> 10), as before;
- a RATE limit ("rate-limited: too many messages", "burst exhausted") is
too many subscription CHANGES per second — fewer concurrent subs don't
help; the fix is to SPACE the REQs out in time.
AdaptiveRelayLimiter now routes each complaint to its own actuator by matching
the notice text, and adds a per-relay rate gate: a growing minimum interval
between subscription opens (250ms -> 500ms -> 1s -> 2s), enforced in withPermit
before the concurrency permit. A relay can be under both controls at once. The
snapshot reports each dimension separately.
Phase B drained DRAIN_CONCURRENCY batches, waited for the SLOWEST (a dead
relay's full timeout), ingested, then started the next group — so every
batch's long tail idled the whole pool, and connections were torn down and
rebuilt between groups. Replace the chunked awaitAll barriers with a
continuous producer -> workers -> consumer pipeline:
- Producer (1 coroutine) routes each author-batch by outbox and feeds a
bounded queue (keeps writeRelayFreq single-writer, backpressured so we
don't precompute every filter map at once).
- DRAIN_CONCURRENCY workers pull a batch, drain it, and grab the next the
instant the drain returns — no worker waits on a slow sibling, and hot
relays stay connected because some worker is always subscribed to them.
- Consumer (1 coroutine) ingests serially (discovered/done/builder/hopOf
stay single-writer), now overlapped with draining instead of blocked
behind each batch.
The four structures now crossed between producer/worker/consumer
(relayHints, attempts, deadRelays, relayStrikes) become concurrent; all
graph mutation stays single-writer on the consumer. Removes the dead-relay
timeout stalls that were serializing the crawl and cuts reconnect churn.
The crawl's client ran on OkHttp defaults: Dispatcher.maxRequests=64 and a
10s connectTimeout. Every relay WS-upgrade handshake is an async call through
that shared dispatcher, so 64 caps the connection-ramp width — and a dead
relay squats on a slot for the full connectTimeout, starving live relays
queued behind it (observed: only ~150 sockets open at once during an active
wave touching hundreds of relays). Raise maxRequests to 256 /
maxRequestsPerHost to 16 and drop connectTimeout to 5s so unreachable relays
release their slot fast.
This is orthogonal to REQ concurrency (bounded per-relay by
AdaptiveRelayLimiter on already-open sockets), so it can't trip a relay's
REQ rate-limit — it only speeds connection setup. The dispatcher's executor
pool grows threads on demand, and FD headroom is ample (4096 limit vs ~150
in use), so the wider cap just lets more short-lived handshakes run at once.
Measured 48 against the per-relay adaptive limiter (hop-4 A/B): it demoted
the right hubs and cut rate-limited CLOSEDs further (1433 -> 501), but the
higher fan-out re-floods busy relays faster than demotion catches up — a
new dominant complaint ("max concurrent subscription count reached")
appeared and download_ms regressed ~11% vs the 24 baseline. Keep the
adaptive per-relay cap (it targets the misbehaving relays precisely) but
return the global fan-out to 24, where the 20/10 ladder still bites below
the global bound and wall-time stays at its best-observed value.
The GrapeRank engine, TrustGraph (compact int-CSR) and TrustGraphBuilder
are pure Nostr-social-graph computation over HexKeys — no UI, no Compose,
and no commons-only dependency. They're a utility for implementing the
NIP-85 rank assertions quartz already models, so they belong in quartz
rather than commons. Move commons/wot -> quartz experimental/graperank
(package com.vitorpamplona.quartz.experimental.graperank), including both
commonTest suites, and repoint the CLI import. TrustGraphBuilder was already
protocol-agnostic (takes HexKey lists; the caller does the event->edge
extraction), so nothing had to change but the package. Makes the algorithm
reusable by the Android app for spam/trust filtering without pulling in
commons.
Replace the blunt global concurrency number with per-relay back-pressure.
Every relay starts generous (100 concurrent subscriptions) and is demoted
down a ladder (100 -> 20 -> 10) only when it complains about concurrency —
a CLOSED rate-limited, or a NOTICE like "too many concurrent REQs" /
"too many subscriptions" / "burst exhausted". Well-behaved relays keep
the full cap; only the busy hubs that push back get throttled, and only as
far as they keep pushing.
AdaptiveRelayLimiter registers as a RelayConnectionListener so demotions
are driven straight off the same NOTICE/CLOSED frames RelayDiagnostics
already observes, keyed by relay.url. Context.drain gains a gatePerRelay
path that opens one subscription per relay, each held behind that relay's
gate, so our concurrent subs on it never exceed its current cap. The gate
is a fair FIFO bounded semaphore whose limit can only be lowered; shrinking
below the in-use count admits no new subs until enough finish, so
concurrency converges down to the new cap.
Because a hot relay can no longer be flooded, the global content-drain
fan-out is raised (18 -> 48) to crawl the many well-behaved relays faster.
The crawl emits a relay_throttling summary (which relays were capped, and
to what) alongside relay_feedback.
Each concurrent content drain opens exactly one subscription per relay it
touches, so DRAIN_CONCURRENCY is effectively the per-relay concurrent-sub
cap. RelayDiagnostics showed the previous value (24) blew past typical
relay limits — rate-limited=1433, "too many concurrent REQs"=1286,
"too many subscriptions"=710 — causing dropped fetches and retry churn.
Lower it to 18 so the peak (18 drain subs + 1 persistent warm-pool sub)
stays ~19, just under the common ~20 cap.
The client's relay pool reconciles open sockets to the relays that active
subscriptions currently need, so between-round gaps (routing + contactsOf scans
> ~300ms) and niche-relay churn dropped connections we reuse every round, then
reconnected them — a TCP+TLS+WS handshake each time.
Hold a persistent do-nothing subscription (WARM_SUB_ID) open to the busiest
WARM_POOL_SIZE(20) live relays, refreshed to the current top set at each round
start (same subId → just updates the desired-relay set) and closed when the
crawl finishes. Its filter matches an impossible event id, so the relay EOSEs
immediately and streams nothing — it only keeps the socket warm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Two blind spots in the crawl's relay I/O:
- No NIP-42 auth. Auth-gated relays sent AUTH, the client never answered, and
their sub CLOSed 'auth-required' — so those outboxes served us nothing. Wire a
RelayAuthenticator into Context that signs the AUTH challenge with the account
key (local signer only; a remote bunker is skipped to avoid a per-relay
round-trip storm mid-crawl). Signing with any key still unlocks relays that
just want some auth.
- No visibility into REQ failures. Add RelayDiagnostics, a connection listener
that tallies NOTICE frames, CLOSED reasons by NIP-01 prefix (auth-required /
rate-limited / restricted / …), and AUTH challenges. Surfaced in the crawl's
stderr summary and as relay_feedback in the JSON, so a failed fetch can be
explained instead of guessed at.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Review of the sharded-sweep restructure flagged two recall regressions vs the
removed last-mile:
- the sweep + broadcast only ever hit the top SHARD_RELAYS (10), while the old
last-mile reached busy relays ranked 11-80 where a user's kind:3 is often
mirrored. Broadcast the small remainder to BROADCAST_RELAYS (60) top live
relays instead of just the rotation's 10, restoring that reach (indexers are
intentionally excluded — they don't serve kind:3).
- MAX_DEAD_STRIKES was 2 with no recovery, so two transient connect blips
evicted a relay for the whole run. Raise to 3 for a safety margin; drain still
only counts hard connect failures, not slow relays.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The download phase was ~90% idle, blocked on the per-wave drain timeout waiting
on dead/stalled outbox relays. Three changes:
1. Dead-relay pruning. Context.drain now reports relays that failed to CONNECT
via a deadOut set; the crawl strikes them (MAX_DEAD_STRIKES=2) into a
deadRelays set and excludes them from routeByOutbox and the sweep, so a wave
stops re-paying the timeout on the same dead outboxes.
2. DRAIN_CONCURRENCY 8 -> 24, safe now that dead relays are pruned rather than
piling up as stalled connections.
3. Sharded backbone sweep (Phase A each round): split the pending authors across
the top-SHARD_RELAYS(10) live relays — one shard per relay, no relay gets the
same list twice — drain all concurrently, rotate the still-missing onto
different relays for up to SHARD_ROTATIONS(6) passes, then broadcast the
remainder to all top relays only once it drops below SHARD_BROADCAST_THRESHOLD
(2000). Phase B then outbox-routes only whoever the popular relays lacked. The
old broadcast-everyone last-mile is removed (the sweep subsumes it).
Each concurrent drain uses its own dead-set (no shared-HashSet race); harvest
feeds from the drain's returned events instead of re-scanning contactsOf over
the whole missing set. Adds download_ms so the crawl phase is timed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The online crawl phase — the network-bound fetch of the whole graph off the
relays (rounds + last-mile sweep) — was untimed; only the offline store-load
had a phase timer. Add download_ms around the crawl block and fold it into the
"crawl complete" log and the JSON, so a from-scratch run reports every phase:
download -> graph build -> scoring (and, with --bench-sign, card signing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Adds a benchmark path to `amy graperank`: after scoring, build + sign one
NIP-85 kind:30382 ContactCardEvent per scored user (rank >= --min-rank) with a
throwaway keypair, fanned out across CPU cores, and report bench_signed +
bench_sign_ms. The signed events are discarded — this measures the id-hash +
Schnorr-sign cost of emitting the full card set without touching any relay or
real identity. Complements the existing store_load_ms / graph_build_ms /
scoring_ms phase timings so the whole pipeline (load → build → score → sign) is
measured end to end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The worklist re-enqueued a node's dependents on every >convergence nudge, so on
a dense graph its total work scaled with the in-degree of the churning core —
an offline score over the full ~419k-node / 19.3M-edge Vitor graph ran 640M+
node-visits and still had not converged after 32 minutes.
Replace it with synchronous Gauss-Seidel sweeps over all nodes (in-place
updates, so trust flows outward within a sweep), ending when no node moves more
than the convergence delta. Same per-node formula, same 0.0001 threshold, same
unique fixed point as NosFabrica's Brainstorm reference (which iterates the same
way) — verified byte-identical by the existing GrapeRankTest suite — but each
node is touched once per sweep instead of once per churning rater, cutting the
work by roughly the average in-degree. Progress now reports per-sweep
(node-updates + nodes-still-moving) and the CLI surfaces the sweep count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The graph_build_ms timer covered only builder.build() (the in-memory int-CSR
pack, a few hundred ms). It excluded the real pre-scoring cost: reading and
deserializing every kind:3 contact list out of the store. Add store_load_ms
(offline path) so the three phases — store load, CSR build, scoring — are each
measured and reported (stderr + JSON), instead of the load hiding behind a
misleadingly small build number.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Outbox resolution used only the fixed indexer/aggregator set to find a user's
kind:10002. Users the aggregators don't carry got no relay list, so their
content couldn't be routed to their own outbox (falling back to hints / the
broad last-mile content sweep).
Add a tier-2 pass in ensureRelayLists: any pubkey still without a relay list
after the indexer sweep is retried for kind:10002 against the known-good
backbone — the busiest live relays learned from the `r` tags in everyone
else's 10002s. A user publishes their own 10002 to their own write relays,
which overlap heavily with that pool, so this recovers relay lists the
aggregators miss. Bounded to the backbone (not an unbounded fan-out to every
working relay) to avoid connection saturation; early rounds no-op until the
backbone is learned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Measure and surface the two compute phases of `amy graperank`: building the
int-CSR trust graph and running GrapeRank to convergence. Both are logged to
stderr ("graph built … in N ms", "scored N users in M ms") and added to the
--json result as graph_build_ms / scoring_ms, so the pure scoring cost over a
given dataset is measurable without eyeballing logs — e.g. an `--offline` pass
over a fully-crawled store times score generation with no network in the loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The SQLite backend raises a catchable UNIQUE-constraint exception when an
event is a duplicate id or an older/duplicate replaceable (kind 0/3/10000-
19999) — which the outbox model produces constantly, since each user's
replaceable is fetched from several of their write relays. verifyAndStore
was logging every one as `[cli] store insert failed`, so a full-network
GrapeRank crawl emitted ~294k spurious error lines. The store is behaving
correctly (its partial unique index + trigger keep the newest version and
reject stale copies); the FS backend simply no-ops on the same duplicates.
Suppress UNIQUE-constraint rejections (normal dedup) while still surfacing
genuine persistence failures (I/O, full disk, corruption).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Audit follow-ups before merge:
- amy fetch default limit is now the same on both paths: absent --limit → 100
for plain AND --paginate (previously --paginate silently meant "unbounded").
`--limit 0` is the explicit opt-in to drain everything (unbounded); negative
is rejected. The effective limit is carried on the filter so both paths agree.
- drainAllPages sizes its SeenIds for CLI-scale fetches (initialSlotsPow2 = 12,
~64 KB) instead of the large-walk default (~16 MB eagerly allocated per fetch);
it grows if an unbounded drain needs it.
- fetchAllPages clamps the inclusive advance to `min(pageMinTs, boundary)` so a
misbehaving relay that answers with an event past the requested `until` can't
push the cursor upward — the boundary dedup and termination rely on `until`
never increasing. No-op for honest relays (they only return events ≤ until).
Verified live: default and --paginate both cap at 100; --limit 50 → 50; --limit 0
--paginate drains the full window (>100); paging tests + SeenIds tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
Two changes to the paginated fetch path:
- Cross-relay dedup before verify. drainAllPages' single consumer now runs a
SeenIds filter: the same widely-mirrored event arrives once per relay, and the
repeats are dropped BEFORE the expensive Schnorr verify + store instead of
after (they were only trimmed by FetchCommand's distinctBy). An id is marked
seen only once it verifies, so a forged copy (valid id, bad sig) delivered
first can't suppress the genuine one from another relay. Adds SeenIds.contains
(peek without recording) for that check-then-add.
- `amy fetch --paginate` no longer forces a --limit. With --limit N it still
pages up to N per relay; WITHOUT --limit it drains the whole filter unbounded
(the filter's null limit flows straight through). Plain (non-paginate) fetch
still trims to the default 100.
Verified live: unbounded --paginate over a ~20-min nos.lol firehose window
returns 406 (all unique, 3s) vs the old 100 cap; --limit 50 caps at 50; default
caps at 100; cross-relay fetch stays count==uniq.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
The FS event store writes one pretty-printed JSON file per event plus one
file per index posting (kind, author, every p-tag value). At crawl scale
this explodes: a 96k-event GrapeRank crawl produced 5.6M tiny index files
rounding up to 2.8GB on disk — only 457MB of which was actual event data.
An 8-hop crawl would blow past available disk.
Wire amy's shared store through a new StoreFactory that selects the backend
from AMY_STORE (default `sqlite`, opt into the legacy tree with `fs`). Both
implement IEventStore, so every command works unchanged. SQLite packs the
same postings into shared B-tree pages — several times smaller on disk and
the natural fit for large crawls. The two stores live side by side under
`<data-dir>/shared/` (events.db vs events-store/) so switching never
clobbers the other's data.
`amy store` maintenance verbs are now backend-aware: stat reports the total
+ disk bytes for both (kind histogram/mtime stay fs-only); scrub is a no-op
on sqlite (indexes are transactional); compact runs VACUUM on sqlite.
Verified end-to-end via the built amy image on both backends: init, notes
post round-trip (event persisted + read back), and every store verb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
After the outbox crawl gives up on users whose own kind:10002 relays never
answered (dead/misconfigured outboxes), take one more pass at every still-missing
user within the hop budget against the WHOLE known-good relay pool — the busiest
live relays learned from the crawl (which include the big aggregators) plus the
discovery set — instead of re-asking each straggler's broken outbox. Recovered
contact lists feed the graph and can reveal a few more reachable users, so the
sweep repeats up to LAST_MILE_PASSES times until it stops recovering.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Answers "do we try relays we know work from other people's lists when a user
isn't in the first set?" — now yes.
- Drop dead relays I'd added unchecked (relay.nostr.band, relayable.org,
relay.nostr.bg); keep only ones that reply to a limit:1 query. NIP-11 presence
is not used for liveness (it's optional).
- Learn a backbone dynamically from the crawl: tally how often each relay appears
as someone's kind:10002 write relay, and mark relays that actually delivered
events as live. The most-used live relays form the backbone.
- Route retried users (whose own outbox already failed) and outbox-less users to
outbox + backbone, since popular relays usually hold a copy of their kind:3.
Effect for Vitor (--max-hops 3): hop-3 coverage rose from ~112,600 to 146,413
(95% of Brainstorm's 153,409), as round-3 contact-list fetches more than doubled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Fixes the hop-2 (and general) completeness undercount, and makes the cause
observable.
- Add a broad set of aggregator + big general relays (relay.nostr.band,
relay.damus.io, snort, offchain.pub, relayable.org, …) to the kind:10002
discovery set. Effect: for Vitor, hop-2 discovery went from ~16,980 to 19,886
(~83% -> ~98% of Brainstorm's 20,332).
- `Context.drain` gains a `diagnoseSlow` flag: on a timeout it logs which relays
stalled and why — slow (no EOSE, with the event count they did send) vs
cannot-connect (with the failure reason) vs closed. `amy graperank --diagnose`
turns it on. This showed the remaining misses are relay-side: users advertise
dead/misconfigured write relays (HTTP 404/530/503, "Unexpected response",
read/connect timeouts), not anything blocking on our end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Stamp each discovered user with its follow-graph hop distance from the observer
(observer=0, a user's fresh follows = its hop+1). Report the per-hop histogram
on the crawl-complete line and as `max_hop_reached` / `users_by_hop` in --json,
and add `--max-hops N` to bound how deep the crawl fetches (deeper users still
appear as follow targets). Brainstorm's graph for an observer saturates within
~8 hops, so `--max-hops 8` matches its scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Point #1 (freshness): each run now fetches every discovered user's LATEST
kind:3/10000/1984 once from their outbox (grouped by write relay in
routeByOutbox; empty-tagged relays already count as write), instead of skipping
users whose list is already cached. `done` still guarantees once-per-run and
that a fresh download isn't repeated.
Point #2 (memory): replace the HexKey-keyed, TrustEdge-object graph with a
compact representation that scales to the whole network:
- commons/wot: pubkeys interned to dense Int ids; edges stored in two CSR
IntArray layouts (by target for scoring, by source for the worklist), each
incoming entry packing source id + relation into one int. GrapeRank.compute
returns a DoubleArray by node id (no boxed map at millions of nodes).
TrustGraphBuilder is now stateful/streaming (addFollows/addMutes/addReports).
Tests rewritten; the full-sweep cross-check still passes.
- cli: contact lists stream straight into the builder as they arrive and the
Event is discarded — the crawl never holds millions of kind:3 objects. Mutes
and reports (far fewer) are fed from the store. Output de-interns the top-N.
Validated: a fresh online run built a 108,961-user / 1.67M-edge graph and
scored 83,613 users in a 4 GB heap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Two related fixes so a warm/shared store isn't re-downloaded every run:
- Only DOWNLOAD contact lists we don't already have. Each round now splits
pending users into "already in the store" (expanded from disk with zero
network) vs "need to fetch" (routed to their outbox). Verified on a warm
store: round 2 pending=250 -> cached=236, downloaded=7; round 3 pending=19491
-> cached=15827, downloaded=87.
- Build the trust graph from the store (kind:3/10000/1984 query) instead of the
in-run `collected` list, so cached-and-skipped lists still contribute their
edges. Online and offline paths now share the same graph source.
No behavioural change on a cold store (nothing cached -> download everything
once), and the crawl still runs to full graph depth with no user cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Amy's one-shot queries all go through Context.drain, a single REQ drained to
EOSE — so a relay that caps its REQ response (strfry's per-REQ limit, ~500)
silently truncates the result with no way to page past it.
Extract the per-relay fetchAllPages fan-out that already lived privately in
EventSync into a reusable quartz accessory, fetchAllPagesFromPool: a
sliding-window pool (maxConcurrentRelays) that paginates each relay on its own
`until` cursor, tags every event with its source relay, and does not dedup
across relays. EventSync now delegates to it (its private downloadPool/
downloadFromRelay are deleted — no behavior change: perRelayFilters is already
ordered by and complete over the relay list).
Add Context.drainAllPages, the paged sibling of drain: same verify+store and
per-relay tagging, but fully draining sets larger than one REQ. Wire it into
`amy fetch` behind --paginate/--all (filter mode only), pushing the limit into
the filter so paging stays bounded. sync (NIP-77) and fetch stay separate
interfaces.
Tests: fetchAllPagesFromPool fan-out/tagging/no-cross-relay-dedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
A live run against Vitor's ~110k-user WoT exposed the crawl's real bottleneck:
draining every pending user's outbox in one subscription saturates connections
and times out. Round-by-round evidence — 250 users queried downloaded 205
contact lists (82%), but 17,055 queried downloaded only 137 (0.8%). Net: 93k
outboxes found but only ~14k contact lists pulled, so most users had no
outgoing edges and scores came out far below Brainstorm's.
Fix: fetch content in bounded batches (USER_BATCH=256) drained a few at a time
(DRAIN_CONCURRENCY=8). Routing (store reads) runs serially; only the drains run
concurrently — inserts serialize on the store write lock, so that is safe.
kind:10002 discovery stays a bulk indexer query (they aggregate 10002 and
handle bulk author filters fine); only the per-user outbox fan-out is batched.
Effect on the same graph: round 3 went from 137 → 4,157 contact lists
downloaded (+36k events), and users scored after 3 rounds jumped 34,976 →
109,760. Full-run parity verification against the live Brainstorm set is in
progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-user outbox retry bound doesn't need to be tunable — replace the
--max-attempts flag with a MAX_OUTBOX_ATTEMPTS = 3 constant. Same behaviour,
one fewer knob. Updates usage text, README, and the parity doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Replace the depth-limited, user-capped BFS with a completeness loop that runs
until every discovered user's kind:10002 outbox has been checked and their
latest kind:3/10000/1984 pulled from it:
- Delete the --max-users cap entirely.
- Crawl round by round until the pending set (discovered minus done) is empty.
A user is "done" once we download its contact list, or after --max-attempts
(default 3) failed tries of its outbox — so an unreachable outbox can't stall
the crawl, and it still terminates on a finite graph.
- --max-rounds replaces --max-depth as an (unbounded by default) safety backstop.
- Track and report the pool of relays actually contacted (relays_contacted),
the "running relays" we connect to as more outboxes are discovered.
JSON: `depth_reached` -> `crawl_rounds`, add `relays_contacted`. Per-round and
final crawl-summary progress on stderr. Local regression: scores unchanged
(rank 26); the crawl retries contact-list-less users then terminates cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Correct the injector's relay model: indexer relays (purplepag.es, coracle, …)
aggregate kind:10002 (and kind:0) for the whole network but do NOT serve
kind:3/10000/1984. Those live only on each user's own outbox.
- Split the relay sets: `relayListDiscoveryRelays` (bootstrap + event-finder +
indexers) is used only to locate kind:10002; `contentFallbackRelays`
(bootstrap + event-finder, no indexers) is the best-effort fallback for
content when a user's outbox is unknown/down.
- Content is fetched from each user's outbox write relays, with harvested
relay hints and general relays as fallback — never indexers.
Also add progress status (all on stderr, stdout stays the JSON contract):
- loading already logs per-hop frontier/recovered/new/total counts;
- "graph built: N users, E edges; scoring…" and "scored N users" bracket the
calculation;
- GrapeRank.compute gains an optional (visited, scored, queued) progress
callback, wired to emit a scoring line every 5000 worklist visits so a large
graph shows movement instead of hanging silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
To match Brainstorm's full-graph ingest, the crawl now discovers data through
three tiers (mirroring the app's pickRelaysToLoadUsers) instead of just the
outbox + a small fallback:
- Indexer relays (purplepag.es, coracle, …) join the discovery set. They serve
kind:0/3/10002 for the whole network and are where a stranger's relay list
and contact list are actually found — the biggest completeness lever.
- Per-follow relay hints are harvested from the `p`-tag hints in every contact
list we crawl and used as a discovery tier below each user's kind:10002.
- A per-hop retry pass re-queries any frontier member whose contact list still
didn't arrive (no kind:10002, or its outbox was unreachable) against the
indexer + hint set, recovering users the outbox model alone would miss.
This tightens the only real source of score divergence from Brainstorm — data
completeness — since a signal's weight scales by the rater's influence, so the
users that matter are exactly the in-graph ones this crawl now reaches more
reliably. Local regression check: scores unchanged (rank 26 for direct follows).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Analysis of NosFabrica/brainstorm_graperank_algorithm (Java scoring worker)
and NosFabrica/brainstorm_server (Python orchestration) to confirm amy's
scores match the reference GrapeRank service.
Finding: our commons/wot formula and every scoring parameter are already
identical to Brainstorm's DEFAULT preset (attenuation 0.85, rigor 0.5,
follow 1.0/0.03, from-observer 0.5, mute/report -0.1/0.5, delta 0.0001). Our
`score` is exactly their ScoreCard `influence`. Remaining divergence is data
completeness, not math — and because a signal's weight scales by the rater's
influence, only in-graph raters move a score, which our outbox crawl already
captures. Documents the pipeline, side-by-side params, divergence sources,
and follow-ups (presets, influence/verified fields).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Publishing kind:30382 rank cards is only half of NIP-85 — clients also need
the kind:10040 TrustProviderListEvent to discover which key provides which
assertion, and where. Add the discovery layer as two sub-verbs:
- `amy graperank register [PROVIDER]` — append a ServiceProviderTag
(default `30382:rank`, self, first outbox relay) to the account's kind:10040,
fetching the freshest list first so existing providers are preserved.
Idempotent, supports `--service KIND:TAG`, `--relay`, and `--private`.
- `amy graperank providers [USER]` — list a user's declared providers
(cache-first; own private entries are decrypted and included).
Bare `amy graperank [OBSERVER]` still computes scores; the dispatcher only
peels off the `register` / `providers` words. All built on quartz's existing
`TrustProviderListEvent` / `ServiceProviderTag` / `ProviderTypes`.
Verified against a local geode relay: register creates the 10040 and is
idempotent on re-run; providers lists both a public 30382:rank entry and a
private 30382:followers entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Previously `graperank --publish` rebuilt and rebroadcast a NIP-85 kind:30382
ContactCard for every scored user on every run, minting a new event id and
created_at even when the rank was identical — pure churn for a parameterized-
replaceable event.
Read back the ranks we last published from the account's own kind:30382 cards
in the local store (ctx.publish already persists them) and publish only the
targets whose rank is new or changed. Report the count left alone as
`skipped_unchanged`.
Verified against a local geode relay: first run publishes N cards
(skipped_unchanged=0); an immediate re-run with identical ranks publishes 0
(skipped_unchanged=N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
- Remove `--target USER`: the command already emits the full ranking, and a
single-user lookup is a trivial slice of it.
- Remove `--no-mutes` / `--no-reports` and the include* parameters on
TrustGraphBuilder.build. GrapeRank is defined over follows, mutes and
reports together; scoring with a signal disabled isn't a meaningful WoT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Bring the GrapeRank algorithm into Amethyst as a WoT service calculator
on the CLI.
commons/wot (protocol-agnostic, CLI-safe, reusable by the apps):
- TrustGraph / TrustEdge / TrustRelation — pubkey-keyed graph model.
- GrapeRank — single-observer scoring engine, a faithful port of the
reference v3 TargetedBFS variant using a worklist that reaches the same
fixed point a full sweep would while only touching reachable users.
- TrustGraphBuilder — pure kind:3 / kind:10000 / kind:1984 events -> graph
(latest-replaceable-per-author, dedup, self-edge drop).
- Unit tests: hand-computed values plus an adversarial full-sweep
cross-check over 50 random graphs.
cli: `amy graperank [OBSERVER]` crawls the follow/mute/report graph via the
outbox model (locate each user's kind:10002 write relays, then fetch their
lists from their own relays, with a broad event-finder fallback) until no
new users appear, scores it, and prints a ranked list (text / --json).
--target queries one user, --offline scores from the local store, and
--publish writes NIP-85 kind:30382 ContactCard assertions
(rank = round(score*100)) per user at or above --min-rank.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Restructure `amy relay` from verb-first `relay add URL --type T` to noun-first
`relay <noun> <verb>`, matching amy's `marmot group …` / `cashu mint …`
convention. The relay-list type is now a required path segment (no implicit
default), and a bare noun lists that bucket.
NIP-65 (kind:10002) is fronted by two facet-nouns, `outbox` (write) and `inbox`
(read), replacing the `--marker` flag. They edit the single 10002 event and
apply the spec's merge rules:
- outbox add R on a read-only R → both
- inbox add R on a write-only R → both
- outbox remove R on a both-R → read (stays in inbox)
- inbox remove R on a both-R → write (stays in outbox)
- dropping the last facet removes R entirely
`relay nip65` shows the combined view; `nip65 remove`/`clear` edit the whole
event.
Other buckets are noun+verb: `relay dm|key-package|search|private|blocked|
trusted|proxy|indexer|broadcast|feeds <add|remove|set|clear|list>`. `set` needs
≥1 URL; `clear` empties. `relay add|remove URL` (no noun) stays as the
transport fan-out (nip65 both + dm + key-package).
BREAKING (cli --json/args): removes `relay add/remove/set --type T` and
`--marker`; `relay list` overview now keys nip65 as `outbox`/`inbox`/`nip65`
and the DM bucket as `dm` (was `inbox`). In-repo harnesses updated
(cache/dm/marmot setup drop `--type all`; cache T5 asserts `.dm`).
Verified end-to-end: merge semantics, encrypted NIP-51 round-trips, facet
set/clear, fan-out, aliases, and error paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
`relay set --type T` with no URLs is now rejected (bad_args, exit 2) instead
of silently wiping the list — a bare empty `set` is almost always a shell
variable that expanded to nothing. Emptying a bucket is explicit: pass
`--clear` (mutually exclusive with URLs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
Expand `amy relay` from the 3 transport lists (nip65/inbox/key_package) to
every relay-list bucket Amethyst's relay-settings screen manages, and add
remove/set verbs alongside add/list.
Buckets (kind): nip65 (10002, read/write markers), inbox/dm (10050),
key_package (10051), search (10007), private (10013), blocked (10006),
trusted (10089), proxy (10087), indexer (10086), broadcast (10088),
feeds/favorites (10012). The private NIP-51 lists are signed NIP-44-encrypted
via the quartz event factories, exactly like the app. Local relays (device
pref, no event) and named relay sets (30002) are intentionally out of scope.
New/changed commands:
- `relay add URL --type T [--marker read|write|both]` — `--marker` sets the
nip65 role; `all` still means nip65+inbox+key_package.
- `relay remove URL --type T` — new.
- `relay set --type T [URL…] [--marker …]` — new; replace a whole bucket
(no URLs clears it).
- `relay list [--type T]` — lists every bucket, or one.
- `relay publish-lists` — now broadcasts every configured list.
Thin-assembly only: buckets are a small registry over the existing quartz
`create`/`relays` factories; adds one generic `Context.latestReplaceable`
helper. `--json` is additive — legacy keys (`nip65`/`inbox`/`key_package`,
`nip65_event_id`/…) are unchanged, so the existing test harnesses keep passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
Found while attributing the small-REQ wire floor (backlog item 6,
latency half): geode's new WireReqFloorBenchmark measured a flat
43.7 ms per REQ round trip that survived every server-side change —
store configs, dispatchers, the pump — and then vanished when the
round's preceding CLOSE was dropped. Root cause is client-side: OkHttp
does not set TCP_NODELAY, relays never answer a CLOSE (NIP-01), so its
bytes sit unACKed for the peer's ~40 ms delayed-ACK window and Nagle
holds the next REQ behind them. CLOSE-then-REQ is a Nostr client's
hottest pattern — every feed/filter switch.
relayBench's harness client already shipped a no-delay socket factory
(which is why benchmark numbers never showed the stall) but the
production clients did not. New TcpNoDelaySocketFactory (quartz
jvmAndroid, next to BasicOkHttpWebSocket) is now used by the Android
relay pool factory, the Desktop relay client, amy's relay connections,
and geode's mirror worker. Direct connections only — SOCKS/Tor paths
are untouched.
With the factory, the benchmark puts geode's ~21-row REQ at ~1.25 ms
on the wire (matching relayBench): ~0.6 ms Ktor CIO+OkHttp loopback
floor, ~0.5 ms per-REQ server work (already investigated). Per-frame
burst cost measured negligible and the pump adds ~nothing, so the
send-path latency angle of backlog item 6 is closed as not-a-problem;
its ingest-CPU share remains a separate throughput question. Findings
recorded in quartz/plans/2026-07-04-small-req-floor.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Replaces the hand-rolled raw-WebSocket NIP-77 negotiate loop (single
un-windowed session — a strfry max_sync_events overflow was a hard
error) with quartz's negentropyReconcile: created_at window splitting
on overflow, keep-alive connection pinning, and streaming id batches.
Downloads and uploads now pipeline with the remaining reconcile
rounds: need-id batches feed 4 concurrent by-id drains, have-ids feed
an uploader (peak 7 subscriptions, under the common relay cap of 20).
Every downloaded event still funnels through the verify-and-store
path. Output field 'rounds' (protocol round-trips) is now 'windows'
(created_at splits).
Verified end-to-end against embedded geode relays: down-only 25/25,
up-only 5/5, and bidirectional re-runs converge to a zero diff.
Also records both adoptions in the perf plan doc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Passes decoder = CachingEventDecoder() at all four NostrClient
construction sites: the Android app pool (AppModules), the Android
crawl client (buildCrawlClient — Event Sync / Cashu discovery, the
duplicate-heaviest path), the desktop RelayConnectionManager, and
amy's Context. Duplicate EVENT frames (14-57% of production traffic)
now skip the full JSON re-parse; dispatch semantics unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
Splits the reconcile out of negentropySync so callers decide how to
load: negentropyReconcile streams needIds (relay has, local lacks —
download) and haveIds (local has, relay lacks — publish) in batchSize
chunks with back-pressure, taking local state as List<IdAndTime> and
slicing it per created_at window on overflow splits; the accumulating
negentropyReconcileIds convenience returns both lists. negentropySync
now delegates to the same window engine.
NegentropySession's primary constructor takes List<IdAndTime> (JVM
erasure forbids a List<Event> overload); the event-list form moved to
NegentropySession.fromEvents, mirroring NegentropyServerSession, with
all call sites migrated.
Adds NostrClientNegentropyReconcileTest (empty local set, partial
overlap both directions, identical sets, batch streaming, since/until
window slicing) — 49 negentropy tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018saXqYfAa3RvSJoDXK591R
relay.nostr.band has been decommissioned. Remove it from every runtime
relay list and route search-relay defaults through the shared
AmethystDefaults.DefaultSearchRelayList in commons:
- amy NipCommand: SEARCH_RELAYS now = DefaultSearchRelayList (drops the
hardcoded relay.nostr.band/nostr.wine pair; RelayUrlNormalizer import
no longer needed).
- desktop DesktopRelayCategories: DEFAULT_SEARCH_RELAYS now =
DefaultSearchRelayList instead of a single relay.nostr.band entry
(which would otherwise be empty after removal).
- desktop DefaultRelays and FollowPacks DISCOVERY_RELAYS: drop
relay.nostr.band.
- Update NIP-50 example hostnames in desktop comments, the search-relay
editor help text, and the localized search_relays_not_found_examples
string across all locales to nostr.wine.
Preview sample data, captured sample-event JSON, and quartz test
fixtures that mention relay.nostr.band are left untouched (no runtime
effect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Ttcqa3V78bugGraGhtehj
relay.damus.io is being decommissioned, so remove it from every runtime
default/fallback relay set to stop the app and amy from wasting connection
slots on a dead host:
- commons Constants: remove `damus`; it dropped out of `bootstrapInbox`
(default NIP-65 inbox) and `eventFinderRelays` (default outbox/fallback),
both still carrying 6 healthy relays.
- ChessConfig: remove damus from CHESS_RELAYS / CHESS_RELAY_NAMES, leaving
the 3 relays the FETCH_TIMEOUT comment already assumes.
- desktop DefaultRelays: remove damus and the also-dead relay.snort.social.
- desktop FollowPacks DISCOVERY_RELAYS: remove damus.
- amy NipCommand SEARCH_RELAYS: swap damus for the NIP-50-capable nostr.wine.
Comments, @Preview sample data, and test fixtures that mention damus.io are
left untouched — they have no runtime effect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Ttcqa3V78bugGraGhtehj
Claude
davotoula