Addresses correctness/perf issues found in the crawler + reachability audit:
- deadHosts permanent eviction (#1): an authority that accrued timeoutEvictStrikes
before its first EOSE was evicted forever — clearTimeoutStrikes only zeroed the
counter and could not un-evict, contradicting the "a host that ever produces is
never evicted" invariant. Add a producedHosts set that isDead() consults, so a
proven-productive authority is never treated as dead even if a concurrent strike
from the 24-worker fan-out raced it into deadHosts.
- Parking-disabled event loss (#2): when parking is off (no bgScope, or
parkTimeoutMs <= timeoutMs), a relay that streamed events but didn't EOSE in the
fast window had its buffer dropped without persist() and reported count 0. Drain,
persist, and return those events like the other two branches; strike only when
nothing was delivered.
- Wide-sweep over-narrowing (#4): relayListDiscoverySwept excluded an already-swept
straggler from the wide pass even though the wide net grows each round, so a 10002
hosted only on a later-learned relay was never fetched. Gate the wide pass on the
asked-relay set (wideRelaysSwept) instead: new users get the full net, older
stragglers get only newly-appeared relays, no (user, relay) pair asked twice.
- Onion detection (#10): replace loose relay.url.contains(".onion") with
RelayUrlNormalizer.isOnion() in isDead() and networkTypeOf(), fixing the
foo.onionfake.com false positive and the store/crawler disagreement.
- rtt-open=0 semantics (#9): document that the crawler's reachable records use
rtt-open purely as a liveness flag (0 = latency not probed), not a real 0 ms
measurement, and must not be published as authoritative latency data.
deadHosts is deliberately still NOT persisted to the 24h reachability cache (#8):
a timeout eviction means "too slow under our fan-out this run", not "proven
unreachable", so persisting it would blacklist slow-but-live hubs across runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
FrameDispatchStats stamped a ValueTimeMark on every relay frame and recorded a
contended atomic per frame in BasicOkHttpWebSocket — the WebSocket layer used by
the whole app, unconditionally, forever — to answer a one-time question that only
graperank --diagnose read. It served its purpose (proved the our-side dispatch
lag is ~200ms mean and the EOSE-wait is dominantly relay-side, so the crawler is
network-bound), but the ongoing per-frame Pair allocation + atomic contention on
every client's relay traffic isn't worth carrying. Revert the channel back to
Channel<String> and delete the stats holder. The diagnose-gated saturation ticker
and per-drain latency breakdown stay — they're crawler-local, off the hot path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured: relays deliver their events in ~0.6s then sit ~4.6s (86% of drain wall)
before sending EOSE — mostly relay-side (our pipeline adds only ~200ms). So instead
of waiting the full 10s fast window then parking, close a drain that has delivered
>=1 event and then gone silent for eoseIdleMs, treating it as complete ("eose-idle").
awaitTerminalOrQuiescent: the idle timer arms only AFTER the first event, so a relay
merely slow to answer still gets the full timeoutMs and is never cut prematurely; a
still-streaming relay keeps resetting the window. eose-idle paginates if the page was
capped and clears timeout strikes (it delivered), but joins notAnswered (no clean
EOSE, so its missing authors are retried elsewhere). Off by default (eoseIdleMs=0),
CLI --eose-idle-ms, so it can be A/B'd against the plain fast window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The dedicated frame-dispatch pool (ea1093ad) made dispatch lag WORSE, not better:
mean 200ms→460ms, max 3.5s→5.5s, frames>1s 43k→76k. The pool was sized cores*2
(=8 here) vs Dispatchers.IO's 64 threads, so it cut frame-processing parallelism
~8x. Lesson: the our-side lag is dominated by per-connection serial decode
throughput / thread count, NOT cross-contention with the store's IO writes — the
experiment ruled that hypothesis out. Reverting to shared IO; keep FrameDispatchStats.
EOSE-wait is confirmed dominantly relay-side (200ms our-mean vs ~5s eose-wait).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured on a GrapeRank crawl, frame decode/dispatch (per-connection consumer
coroutines) ran on the shared Dispatchers.IO — the same pool that runs the store's
blocking SQLite inserts. During event floods, frame coroutines queued behind those
inserts: mean 200ms and up to 3.5s of dispatch lag, with 43k frames waiting >1s in
our pipeline. That lag also skews the relay-idle/EOSE timing the crawler reads.
Give frame processing its own daemon thread pool (sized to a small multiple of
cores; decode is light + CPU-bound), shared across all connections. Frame delivery
stays prompt regardless of what the IO pool is doing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Adds FrameDispatchStats: the lag between a relay frame arriving on the OkHttp
reader thread and our per-connection consumer coroutine (on shared Dispatchers.IO)
pulling it off the channel — pure our-side pipeline delay, relay send-timing
excluded. BasicOkHttpWebSocket stamps arrival before enqueue and records the lag
on dequeue; the crawler resets it at start and dumps it in the --diagnose summary.
Answers whether a drain's 5s gap between the relay's last event and its EOSE is
the relay being slow to SEND eose (low dispatch-lag) or our IO pipeline backing up
so the already-arrived eose frame sits queued (high dispatch-lag).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A Nostr pubkey is x-only, exactly 32 bytes, but NPub.parse/NProfile.parse never
checked the length — they hex-encoded whatever bytes the bech32/TLV carried. A
malformed npub/nprofile that some clients encode with the full 33-byte COMPRESSED
secp256k1 key (0x02/0x03 prefix) therefore round-tripped its 66-char hex straight
into a `p`/`q` tag via the quote/mention path, and a strict relay (relay29 /
pyramid.fiatjaf.com) rejected the whole group message:
blocked: schema validation failed: tag[..]: invalid pubkey value
'02977dcf…c3402' ... pubkey should be 64-char hex
We never generate compressed keys ourselves (Nip01Crypto.pubKeyCreate strips the
prefix byte); this is purely inbound malformed input. Enforce the 32-byte length
at the decode boundary so the bad entity never becomes a mention/quote tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Answers "are we resource-bound or waiting on relays" without a profiler:
- progress ticker gains "Nw/CAPw" (drain workers busy vs drainConcurrency) and
"N rl" (rate-limit responses so far) — a rarely-full pool means the producer or
the relays are the limit, not concurrency; a climbing rl count is the external
ceiling that made concurrency 60 backfire.
- crawl-end "latency breakdown": splits each drain's wall into time-to-first-event
vs EOSE-wait-AFTER-the-relay's-last-event, and reports the % of drain wall spent
waiting for EOSE after the relay was already done, how many drains blew the fast
window and parked, and total rate-limit hits. A high EOSE-wait % is the direct
case for a shorter/adaptive fast window over more concurrency.
All gated on config.diagnose; zero cost on a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Two changes:
1. The reachability flush re-wrote the SEEDED known-dead relays with a fresh
created_at every run, refreshing their TTL without a re-probe — so a relay
marked dead once (and thereafter skipped, never re-dialed) would stay
blacklisted forever as long as crawls kept running, defeating the TTL's
re-probe. Stats.deadRelays now reports only relays actually dialed this run
(deadRelays - knownDeadRelays); seeded records keep their original timestamp
and age out on schedule so the next run re-probes them.
2. Rename GrapeRankDataCrawler -> GrapeRankCrawler (file + all references).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Wire RelayReachabilityStore into the crawler and the WoT updater so liveness is
shared across procedures and runs instead of each rediscovering dead relays.
- OperatorKeys.monitorKey(): a dedicated machine monitor identity derived from the
operator master (domain "relay-monitor:"), independent of any account — the
30166 records are published under this, not the observer key.
- Context.reachability: a RelayReachabilityStore over the shared store, signed by
the monitor key.
- Crawler: Config.knownDeadRelays seeds deadRelays before the run; Stats now
returns the final dead/live sets. GrapeRankCommand seeds from snapshot().dead
and flushes the crawl's verdicts back via reachability.record().
- Updater: Config.knownDead skips proven-dead relays from the reconcile plan — a
dead relay cannot serve its authors, so reconciling it only burns a timeout.
Live author-advertised relays are always synced.
All behind --no-reachability-cache. TTL'd (24h), so a recovered relay is retried
once its record ages out — a "skip for now", never a permanent ignore, keeping
the outbox rule that every live advertised relay is tried.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A durable, shareable relay-reachability cache backed by the EventStore as NIP-66
kind:30166 Relay Discovery events, so the crawler, the WoT updater, and future
runs share liveness knowledge instead of each rediscovering dead relays from an
in-memory set wiped at process exit.
- 30166 is addressable by its d-tag (relay URL) → one replaceable status slot per
(monitor, relay), with created_at giving a free TTL.
- Reachable → 30166 with rtt-open; dead → 30166 without (NIP-66 has no explicit
offline field; liveness is inferred from a fresh successful open). Live wins
over dead within the TTL, so third-party monitors' 30166 can be ingested.
- snapshot() loads the fresh set once (not a per-request hot-path query); record()
flushes a run's findings. A relay is only skipped for the TTL, never permanently
— consistent with the outbox rule that every advertised write relay is tried.
Reuses the existing RelayDiscoveryEvent. jvmTest covers record/reload,
live-overrides-dead, TTL expiry, and .onion→Tor network tagging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Round-8 profiling showed the crawl re-querying the same never-had-a-10002
users' outboxes every round they recirculated — ~144k slow kind:10002 drains
(p50 17.4s) against a static discovery set, dragging the round to ~18 users/s.
1. ensureRelayLists guards with `relayListDiscoverySwept`: each user's outbox
discovery runs once. The discovery relay set is static, so a second sweep of
a user still lacking a 10002 cannot find one the first missed.
2. The discovery REQ to the bounded INDEXER set co-fetches [10002, 3]: the
outbox lookup already pays the round-trip and an indexer holding a user's
10002 often holds their kind:3, so we harvest the contact list as a cheap
byproduct. The wide "every live relay" completeness sweep stays 10002-ONLY —
co-fetching kind:3 across thousands of relays downloaded the same big contact
lists repeatedly and inflated the fire-and-forget bgScope sweep the finishing
drain waits on (measured +260s at hop-3; the indexer-only co-fetch keeps
coverage flat at baseline speed).
3. harvestFromStore folds any already-stored kind:3 into the graph at Phase-A
time so Phase B never re-drains a list we hold (also speeds re-runs).
Verified same-session hop-3: pre-fix 685s / narrowed 690s / wide-co-fetch 945s,
coverage 91.74% across all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
NIP17Factory.createWraps launches all per-recipient seal builds via
mapNotNullAsync, which today runs them fully parallel. Each seal
needs nip44_encrypt + sign — for a NIP-46 (bunker) signer that means
two round-trips per recipient. A 5-recipient group send launches 10
concurrent in-flight requests against the bunker socket, and nsec.app
/ Amber / Keychat typically serialize past ~10 in-flight, so some
requests queue past the 65s timeout and silently fail.
Cap at 4 concurrent when signer is NostrSignerRemote. Local signers
(NostrSignerInternal, NostrSignerSync) bypass the semaphore and stay
fully parallel — no overhead, no behaviour change for nsec users.
The real fix is the batched nip44_get_conversation_keys NIP-46 RPC
(separate spec PR + plan) which collapses N×2 round-trips into ~2.
This commit is the interim throttle until that lands.
Three regression tests covering the NIP-17 relay-hint contract just
introduced on GiftWrapEvent.create:
- default (no hint) emits the historical two-element ["p", pubkey]
shape — guards every existing caller against a wire-format
regression.
- with-hint emits ["p", pubkey, relay-url] — the canonical NIP-17
shape with the hint on the public wrap (NOT inside the seal, which
is the encrypted envelope and would hide routing info).
- null-hint must NOT produce ["p", pubkey, ""] — that would broadcast
"this user has no canonical inbox" as a metadata leak.
Per NIP-17 §Publishing, a gift wrap (kind 1059) MAY carry the
recipient's primary DM inbox relay as a third element of the p tag.
Other clients the recipient runs (or relays acting as inbox routers)
can then locate the wrap without performing their own kind:10050
lookup — handy when the recipient is multi-device and the second
device's 10050 cache is cold.
GiftWrapEvent.create gains an optional `recipientRelayHint:
NormalizedRelayUrl?` parameter that flows into PTag.assemble (which
already accepts a relay hint). NIP17Factory.createWraps and the four
public createMessageNIP17 / createEncryptedFileNIP17 /
createReactionWithinGroup entry points gain a matching
`recipientRelayHints: (HexKey) -> NormalizedRelayUrl?` lambda so
multi-recipient sends can pass per-recipient hints in one shot.
All new parameters default to null / { null }, so every existing
caller compiles unchanged and still emits the historical
two-element ["p", recipientPubKey] shape. Callers that resolve
kind:10050 via the (forthcoming) DmInboxRelayResolver can wire the
result through to populate the hint.
While here, document the existing — but undocumented — invariant
that shared rumor created_at falls out naturally because the
rumor is signed once before the per-recipient mapNotNullAsync loop.
This is what anchors cross-recipient reaction/receipt dedupe.
RelayAuthStatus has to stay mutable — it holds LruCaches addressable from
the per-relay OkHttp dispatcher thread, and replacing the whole holder
on every mutation would be wasteful. But its mutability also makes it
useless as a StateFlow value: mutating an entry doesn't change map
identity, so distinct-until-changed downstream swallows the update and
Compose never recomposes.
Add an immutable view alongside: RelayAuthSnapshot (phase +
lastAuthSuccessAt). RelayAuthStatus.snapshot() derives it from the LRU.
RelayAuthenticator publishes a PersistentMap<NormalizedRelayUrl,
RelayAuthSnapshot> via authStateFlow on every mutation (connect,
disconnect, AUTH-submitted, AUTH-OK, AUTH-fail). PersistentMap gives
O(log32 n) updates and a fresh identity per put, so both StateFlow
equality and Compose strong-skipping work.
This is the substrate for downstream consumers — the AUTH approval
banner, the retry-queue wake on authCompleted, the indexer-fan-out gate
— none of which are wired yet. They will read authStateFlow rather than
querying RelayAuthStatus directly.
NIP-42 AUTH challenges arrive as `auth-required:` OK responses. Today
they accumulate via PoolEventOutboxState.newResponse → Tries.addResponse,
and after three of them the relay is silently dropped from the outbox on
the next newTry — even though RelayAuthenticator is concurrently signing
the AUTH event and the relay would have accepted the original publish
once authenticated.
Carve `auth-required:` out of the failure path: it's a "wait, AUTH in
flight" signal, not a rejection. The existing
RelayAuthenticator.checkAuthResults → client.syncFilters hook re-pumps
the outbox after AUTH-OK, so the original event is retried naturally.
Adds PoolEventOutboxStateTest covering the carve-out plus regressions
for regular rejections, terminal rejections, and success.
Audit of the authorsMissingOutbox anti-join surfaced one correctness bug and
one performance win:
- Bug (semantic): kind-1059 giftwraps store a random one-time key in
event_headers.pubkey (the real recipient is only a hash), so the query
returned an unbounded set of ephemeral keys that can never own a 10002 —
junk for the outbox model this feeds. Both the SQLite path and the generic
default now exclude kind 1059 from the "authors" set.
- Performance: replaced the DISTINCT + correlated NOT EXISTS scan with an
index-only EXCEPT (all authors minus 10002 owners). Both sides ride the
unconditional query_by_kind_pubkey_created covering index — so it does NOT
depend on the optional pubkey-alone index — and measured ~3x faster
(44ms vs 137ms at 152k events / 20k authors); the gap widens with author
count, since the old form paid one seek per distinct author. A loose-index
skip-scan was rejected: it needs the pubkey-alone index and degrades to a
full scan per author without it.
Also: corrected the KDocs (the old text implied an efficient index-only
distinct that wasn't guaranteed), added a giftwrap-exclusion test, and added
FsAuthorsMissingOutboxTest — the only coverage of the IEventStore DEFAULT
implementation, which EventStore always overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Audit follow-ups on the sync engine:
- Perf: syncGroup reconciled against a full store.query<Event>(filter),
decoding the entire local matched set (~1 KB/event) just to read ids +
created_at and to index events for a small residual upload. Reconcile now
uses store.snapshotIdsForNegentropy (id + created_at only, ~40 B/entry) and
the uploader fetches only the residual haves by id. Peak memory drops from
O(all local matches) to O(residual) — matters when a relay hosts a large set.
- Bug: sync() promised best-effort ("one bad relay can't abort the set") but
syncGroup only caught NegentropySyncException, so any other failure (store
I/O, an unexpected throw) escaped async and cancelled every other relay via
awaitAll. Each group now runs under a guard that records the failure instead.
- Bug: the page-fallback catch (Exception) swallowed CancellationException,
breaking cooperative cancellation. Both new catch sites rethrow it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds AuthorsMissingOutboxBenchmark (gated behind -PprodRelayBench=1, like the
other prod benches). It syncs a real sample from relay.damus.io (kind 1 notes +
kind 10002 relay lists), replicates it to 1,000,000 stored rows while preserving
the real author set and outbox-owner set, then times the two shipping
implementations of authorsMissingOutbox() on the same store:
- generic: the IEventStore interface default (decodes every event via
query(Filter()))
- sqlite: EventStore's SELECT DISTINCT pubkey ... NOT EXISTS
Both are asserted to return the same set, matching the seeded ground truth.
Measured on a 4-core container, 1,000,000 events (best of 3):
generic 138,880 ms
sqlite 2,623 ms → ~53x faster
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Extracts GrapeRankUpdater's per-relay sync engine into a standalone
NegentropyStoreSync in the relay-client accessories, so any caller can
two-pass sync an arbitrary `relay -> filters` set against a local store.
Given an INostrClient + IEventStore it syncs each (relay, filter) group:
a bidirectional NIP-77 reconcile into/from the store (down/up), a deletion
settle over the residual (applyDown downloads the relay's kind:5 when an
uploaded record was rejected), and a paged-download fallback when a relay
can't reconcile. sync() runs many groups with relays concurrent and each
relay's own filters sequential (so one relay never exceeds its subscription
budget). Directions and bounds are a Config; every group is best-effort and
its outcome is a GroupResult. This is also the reusable engine `amy sync`
open-codes today.
GrapeRankUpdater now only owns the GrapeRank specifics: it reads kind:10002,
inverts to write-relay -> authors (the outbox model), fans that into one
filter per (relay, author chunk), hands the set to NegentropyStoreSync, and
folds the per-group results back up per relay. Its public Config/Result and
the CLI wrapper are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a whole-store query returning every distinct author with at least
one stored event that has NO NIP-65 relay list (kind 10002 / outbox).
This is a set-difference the positive-only nostr Filter grammar can't
express (there is no "NOT kind 10002"), so it lives as a dedicated
IEventStore method rather than a query(Filter). The interface carries a
correct default (collect authors-with-outbox, then stream events keeping
the rest — O(events)); SQLiteEventStore overrides it with a single
SELECT DISTINCT ... NOT EXISTS that seeks the outbox check on the
(kind, pubkey, created_at) index.
"Missing" is relative to what the store holds: an author whose only
10002 was deleted (NIP-09) or expired (NIP-40) is reported as missing
again, since no row remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Moves the `amy graperank update` logic out of the CLI and into quartz as
GrapeRankUpdater, alongside GrapeRankDataCrawler in experimental/graperank,
so Android and any other quartz consumer can run the same refresh.
Given an INostrClient + IEventStore it reads every kind:10002 in the store,
inverts them into a write-relay -> authors map (the outbox model), then runs
one NIP-77 negentropy reconcile per write relay scoped to its authors:
bidirectional content sync into/from the store, deletion settle over the
residual (applyDown downloads the relay's kind:5 when an uploaded record was
rejected because the author retracted it), and a full paged-download fallback
when a relay can't reconcile. Bounds and directions are a Config; per-relay
and aggregate outcomes are returned as a Result.
The CLI `graperank update` is now a thin wrapper: it parses flags, builds the
Config, and renders GrapeRankUpdater.Result as text/JSON — no sync logic left
in cli/ (all reconcile/window/back-pressure/deletion logic lives in quartz's
relay-client accessories, which GrapeRankUpdater composes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
The probe does blocking DNS + TCP connect, and dead-domain DNS lookups hang well
past the connect timeout. On the shared Dispatchers.IO those hanging lookups
starved the crawl's own IO: an A/B at hop-3 showed probe-on 981s vs probe-off
517s, the entire +464s landing on the finishing drain (rounds were identical).
Coverage was unchanged (91.84% vs 91.74%), so the probe classification is correct
— it was purely IO contention.
Give the probe its own fixed daemon pool (128 threads) so its blocking work can
never touch the crawl's IO, and align the culler's concurrency to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The culler filtered candidates by !isDead and not-yet-probed, but not by
liveRelays — so it probed relays the WS path had already proven live, wasting a
probe and opening a needless TCP connection to the hot relays the crawl depends
on. Skip any authority already in liveRelays up front. liveRelays becomes a
ConcurrentSet so the background culler can read it while the crawl writes it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
At hop-8 the crawl dials into thousands of dead relay hints from old accounts.
Most fail slowly: a silently-dropping host has no RST to receive, so the WS
connect just hangs to the 7s connectTimeout. First-strike eviction pays that once
per host, but with ~3,000 dead hosts that's ~80s of connect-setup serialized
through the dispatcher.
Add a background reachability culler: a cheap raw TCP connect (one round trip, 2s
timeout) over the learned relays COLD-TAIL FIRST, dropping the unreachable ones
into deadHosts before the WS path pays its 7s. The key property is that a tight
TCP timeout is safe where a tight WS timeout is not — a busy-but-alive relay
accepts the SYN instantly at the kernel level and only stalls at the app layer, so
the probe separates "unreachable" from "slow" and never false-kills the busy. It
only ever marks dead and probes each authority once; a host the WS path already
resolved (isDead) is skipped, and a live host passes the probe, so the WS verdict
always wins. Injected as an optional Config.reachabilityProbe (JVM: java.net.Socket
in the CLI; --no-probe disables); writeRelayFreq becomes concurrent so the culler
can read it while routeByOutbox writes.
Also: when there's no Tor transport (Config.torEnabled=false), isDead skips every
.onion relay on sight — no socket, no wasted connect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Relay-signed kind-39000 has no author-of-a-follow, but the people dimension
still exists: a follow may be the relay signing key, a group admin (39001), or
a member (39002). Discovery now resolves each top-nav filter into a per-relay
GroupDiscoveryConstraint instead of collapsing every filter to the same REQ.
quartz:
- GroupMetadataEvent / EditMetadataEvent: build + read #t (topics) and #g
(geohash, mip-mapped so a coarser followed geohash still matches). Interop
tests for parse/build round-trips.
amethyst:
- dal/RelayGroupDiscoveryFeedFilter: sealed GroupDiscoveryConstraint
(AllGroups / ByPeople / ByHashtags / ByGeohashes / AnyOf) + toGroupConstraints()
mapping each IFeedTopNavPerRelayFilterSet to per-relay constraints, with
matches() covering the relay-key/admin/member people paths and topic/geo tags.
Unit tests.
- Directory REQ narrows to 39000 #t/#g for topic/geo filters, broad directory
otherwise (people match needs the rosters).
- ViewModel keys the feed on the constraint map and re-scans on any directory
event (metadata OR roster) so late-arriving admins/members surface groups.
- Create/edit form gains a Discovery section (topics + geohash) threaded through
Account.createRelayGroup/editRelayGroupMetadata and EditMetadata.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Dropping the per-batch awaitAll (previous commit) made the rounds faster but a
hop-3 A/B regressed total wall (727s vs 532s): the Semaphore(1024) throttled
concurrent relay drains to ~249 parked at peak vs the old batch model's ~4,438,
so slow-relay park windows that the old model absorbed during the rounds spilled
into a long serial finishing drain. Raise the default so the parked work drains
inside the rounds again. Value under validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Phase B drained users in 256-user batches: a worker called drainGated for the
whole batch and awaitAll'd every relay in it, so one slow relay held the worker
(and the batch's already-finished fast relays' contact lists) for the full 10s
fast window before anything was ingested. With 24 workers all waiting out their
batches' slowest relay at once, progress dropped to 0 lists/sec in waves.
Restructure to drain each relay independently and stream its result the instant
it resolves — no per-batch join. A per-user counter (relaysLeft) tracks how many
of a user's relays are still outstanding; the single-writer consumer finalizes a
user (ingest, or count a failed outbox attempt) only when the last of its relays
resolves, so correctness is unchanged. Concurrency is now a semaphore over
relay-units rather than an implicit batches×fan-out product; drainConcurrency
becomes "concurrent relay drains" (default 1024, ~the old 24-batch fan-out).
A straggler the outbox model routes nowhere is finalized directly as a miss.
Fast relays' lists are now ingested immediately instead of behind a batch's
slowest relay, removing the 0/s stalls on slow-relay-heavy rounds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The drain classifier had two "act on it" verdicts, HARD (drop now) and TRANSIENT
(strike a few times, might clear). Re-probing hop-8's failed relays fresh showed
the TRANSIENT bucket almost never clears: 503 Service Unavailable 0/12 reachable,
502 Bad Gateway 3/15, connection-establishment failures 0/30; the codes that were
alive (402/403) are gated and will never serve us, and 200 isn't a relay. So the
extra dials TRANSIENT bought were spent on hosts that stay dead for the run.
Collapse to a single DEAD verdict, dropped on the first strike, and carve out the
only two connect failures that genuinely recover so they stay retryable (null):
a READ timeout (relay answered the handshake, slow — 67% reachable fresh, kept on
the clear-on-success authority-strike path) and an HTTP 429 rate-limit (alive,
4/4 reachable — retrying spaced by the limiter is how we get its data). Removes
the now-unused relayStrikes map, MAX_DEAD_STRIKES, and the HARD/TRANSIENT merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A relay sends its matching events before its EOSE, so both an event and
the relay's completion can sit buffered in their channels at the same
time. The select() over the two channels picks a ready clause at random,
so it could process the doneChannel completion first, empty `remaining`,
and exit the loop while the matching event was still unread — returning
null instead of the event.
On a relay completion, drain the event channel first and treat any
already-buffered event as the result before marking the relay done.
A connect failure and a read timeout were both treated as "busy, retry" and
took three strikes to drop. Re-probing hop-8's failed relays fresh, outside the
crawl, showed the two are not alike: relays that failed to ESTABLISH a
connection (connect timed out, refused, unroutable, or the proxy couldn't tunnel
the CONNECT) were 0/30 reachable — genuinely dead — while relays that hit a READ
timeout were 12/18 (67%) reachable, alive but overloaded by the crawl's fan-out
(user.kindpag.es among them).
So classifyDrainFailure now returns HARD for connection-establishment failures
(one strike drops them instead of burning two more dials on a dead host), while
a read/generic timeout still returns null and stays on the patient,
clear-on-success timeout-strike path so live-but-slow relays we need are not
wrongly evicted. Mid-stream resets stay TRANSIENT. Adds DrainFailureTest, which
the classifier previously had none of.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Clears real Kotlin compiler warnings surfaced across quartz, cli,
relayBench, amethyst, and desktopApp:
- quartz Sha256/EventHasher/ScratchLocal: ThreadLocal.get() is nullable
in Kotlin; assert non-null (withInitial never yields null).
- quartz GitHttpClient: PriorityQueue.poll() under isNotEmpty() is
non-null; assert it.
- relayBench CorpusDownloader: drop redundant !! on smart-cast Long;
Jackson fields() -> properties().
- cli GrapeRankCommand: drop redundant ?. where latest is smart-cast.
- PodcastRemoteContent: OkHttp body is non-null; drop dead elvis.
- Dead/redundant expressions: remove no-op when-branch values and a
redundant trailing Unit (HomeScreen, LocalCache, EmbeddedTabLayer,
ParticipantHostActionsSheet, NestActionBar, ControlWhenPlayerIsActive,
ShareNoteAsImageScreen exhaustive-when else).
- CalendarEventDetailScreen / SetPasswordDialog / ProfileClinkOfferResolver:
drop always-true conditions (reorder to keep smart-casts).
- WalletColumnScreen: OkHttp body non-null; drop unreachable null-guards.
- PcmTapRegistry: the @OptIn used androidx.annotation.OptIn, which does
not opt into Kotlin's ExperimentalCoroutinesApi; use kotlin.OptIn.
- GitRepositoryScreen: suppress the standard ViewModel-factory cast.
- PushNotificationReceiverService: suppress override-of-deprecated.
- Desktop GlobalScope call sites: @OptIn(DelicateCoroutinesApi::class).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
- GrapeRankPublisher: dTag() is non-null (""), so the Elvis on the
grouped target was dead code; skip blank targets via ifBlank instead.
- amethyst: migrate deprecated resourceConfigurations to
androidResources.localeFilters (same locale qualifiers).
- desktopApp: replace deprecated compose.desktop.uiTestJUnit4 accessor
with the direct org.jetbrains.compose.ui:ui-test-junit4 dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GMqkg1ndvFihEwZcENiRs
A single REQ can match up to authors×kinds events; a relay that caps its
response below that silently drops the tail. Measured: user.kindpag.es returns
at most ~100 events per REQ and ignores our limit, so a dense chunk -- 300
authors × the 4 FETCH_KINDS, or a popular-author kind:3 sweep -- loses
everything past the newest 100 on the first (and only) page drainGated fetched.
On a dense set kindpag returned 100 events single-shot vs 238 paginated; nos.lol
and damus (higher caps) matched at 246 and 127.
drainGated never paginated -- it took one page and moved on -- so this bit every
sweep and outbox query, not just the aggregator recovery. Truncated users became
stragglers that the multi-round retry mostly (not always) recovered elsewhere,
which is why it stayed hidden.
Now any page that comes back at FULL_PAGE_THRESHOLD (100, the smallest cap
observed) is treated as possibly-capped and its remainder is drained in the
background with fetchAllPages `until` cursors, streamed to lateHarvest exactly
like a parked slow relay (tracked by parkedInFlight so the round waits for it,
gated by the limiter). The boundary second is re-fetched and de-duplicated by
persist's crawl-wide seen-set, so nothing double-counts. Only dense pages pay
the extra REQs; the common under-cap page is untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The recovery pass drained each aggregator with the crawl's single-shot path
(drainGated: one REQ, collect until EOSE). Against an indexer that caps a page
at ~100 events and ignores our limit, every straggler beyond the newest 100 was
silently dropped -- and drainGated additionally merged all chunks into one giant
REQ, which the big indexers answer with nothing at all.
Query each aggregator with fetchAllPages instead, walking `until` cursors to
exhaustion, one AUTHORS_PER_FILTER chunk per request so no request carries the
whole straggler set. Relays paginate concurrently; each relay's chunks run
sequentially to keep one subscription live per connection, gated by the same
limiter. Delivered events land on a channel off the reader threads, then are
verified/persisted and folded once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Reject a single-character group id (except the default `_`) so a possessive
glued to a bare relay URL — `wss://relay.damus.io's uptime` — no longer
linkifies group "s". Real ids (relay29/Wisp/0xchat) are all longer.
Adds coverage proving only genuine ws/wss relay URLs are peeked: apostrophes
after http, nostr:, blossom:, email and bech32 tokens never become group links;
plus ws:// (insecure), second-apostrophe boundary, and multi-link cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
The comments said a multi-kind filter makes the big indexers "time out returning
nothing." Reproduced against user.kindpag.es, the real mechanism is a per-REQ
result cap: it returns ~100 events regardless of the requested limit, and a
kinds=[3,10000,1984,10002] query fills that cap entirely with the far more
abundant kind:10002, returning 0 kind:3. Asked kind:3-only it returns the
contact lists in a few seconds. Same conclusion (query kind:3 alone), accurate
reason.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The dedicated straggler-recovery pass was skipping any content aggregator the
main crawl had timeout-evicted, so it recovered ~1 contact list instead of the
hundreds those indexers actually hold.
Root cause: during the competitive crawl an indexer like user.kindpag.es is
only ever asked for kind:10002 in bulk and kind:[3,10000,1984,10002] one author
at a time. The latter parks and times out (60-80s each), striking the host until
its authority is timeout-evicted. It is never asked for a clean bulk kind:3 --
the one thing it serves fast (~19 lists per 300 authors in seconds; ~369 of the
run's missing authors live there). So by the time recovery runs, kindpag.es is
dead and dropped from the aggregator set (8 configured -> 6 used), and the
biggest single source of missing lists is never queried.
Fix: the recovery pass now queries every configured aggregator regardless of
eviction (drainGated doesn't re-check isDead, and a genuinely dead endpoint only
costs one shared park window since units run concurrently), and clears any
timeout strikes first so a partially-struck host starts clean. Also stops
folding aggregators into routeByOutbox's multi-kind fan-out (they time out
there) and asks them kind:3-only, matching what they serve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Claude
Vitor PamplonaandGitHub
davotoula
nrobi144