docs(graperank): correct why aggregator recovery is kind:3-only

The comments said a multi-kind filter makes the big indexers "time out returning
nothing." Reproduced against user.kindpag.es, the real mechanism is a per-REQ
result cap: it returns ~100 events regardless of the requested limit, and a
kinds=[3,10000,1984,10002] query fills that cap entirely with the far more
abundant kind:10002, returning 0 kind:3. Asked kind:3-only it returns the
contact lists in a few seconds. Same conclusion (query kind:3 alone), accurate
reason.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
This commit is contained in:
Claude
2026-07-08 17:12:33 +00:00
parent eef2832bf4
commit 041e6c83b8
@@ -603,10 +603,11 @@ class GrapeRankDataCrawler(
* [backbone] — the known-good relays other people write to;
* - no outbox at all: harvested hints + backbone + the general fallback.
*
* The content aggregators are deliberately NOT mixed in here: they only serve
* kind:3 to a kind:3-only filter and time out on this path's multi-kind
* [FETCH_KINDS] query, so recovering from them is done separately, once and
* patiently, in [recoverStragglersFromAggregators].
* The content aggregators are deliberately NOT mixed in here: this path's
* multi-kind [FETCH_KINDS] query loses their kind:3 to their per-REQ result
* cap (a big indexer fills the response with the abundant kind:10002 and
* returns no kind:3), so recovering from them is done separately — kind:3-only,
* once and patiently — in [recoverStragglersFromAggregators].
*
* Also tallies each user's write relays into [writeRelayFreq] so the
* backbone can be learned from the crawl. Authors are chunked per relay.
@@ -687,10 +688,13 @@ class GrapeRankDataCrawler(
// Build the query against the full straggler set BEFORE any folding (the
// filter lists are materialized here, so later mutation of `stragglers` is
// safe). Ask ONLY for kind:3 — the contact list we're missing. A multi-kind
// filter breaks the big aggregators: user.kindpag.es serves kind:3 in a few
// seconds when asked for it alone, but times out returning nothing when the
// same authors are requested with kinds=[3,10000,1984,10002]. Mutes/reports
// still come from the outbox model; the aggregator's job here is the lists.
// filter is useless against the big indexers: user.kindpag.es caps its
// response at ~100 events per REQ (it ignores our limit), so a
// kinds=[3,10000,1984,10002] query comes back 100× kind:10002 and 0×
// kind:3 — the abundant relay lists crowd the contact lists out entirely.
// Asked for kind:3 alone it returns them in a few seconds. Their kind:10002
// is already fetched in bulk by [ensureRelayLists]; mutes/reports still come
// from the outbox model. The aggregator's job here is only the lists.
val filters =
aggregators.associateWith {
stragglers.chunked(AUTHORS_PER_FILTER).map { chunk -> Filter(kinds = listOf(ContactListEvent.KIND), authors = chunk) }