mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-12 09:13:23 +00:00
docs(graperank): correct why aggregator recovery is kind:3-only
The comments said a multi-kind filter makes the big indexers "time out returning nothing." Reproduced against user.kindpag.es, the real mechanism is a per-REQ result cap: it returns ~100 events regardless of the requested limit, and a kinds=[3,10000,1984,10002] query fills that cap entirely with the far more abundant kind:10002, returning 0 kind:3. Asked kind:3-only it returns the contact lists in a few seconds. Same conclusion (query kind:3 alone), accurate reason. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
This commit is contained in:
+12
-8
@@ -603,10 +603,11 @@ class GrapeRankDataCrawler(
|
||||
* [backbone] — the known-good relays other people write to;
|
||||
* - no outbox at all: harvested hints + backbone + the general fallback.
|
||||
*
|
||||
* The content aggregators are deliberately NOT mixed in here: they only serve
|
||||
* kind:3 to a kind:3-only filter and time out on this path's multi-kind
|
||||
* [FETCH_KINDS] query, so recovering from them is done separately, once and
|
||||
* patiently, in [recoverStragglersFromAggregators].
|
||||
* The content aggregators are deliberately NOT mixed in here: this path's
|
||||
* multi-kind [FETCH_KINDS] query loses their kind:3 to their per-REQ result
|
||||
* cap (a big indexer fills the response with the abundant kind:10002 and
|
||||
* returns no kind:3), so recovering from them is done separately — kind:3-only,
|
||||
* once and patiently — in [recoverStragglersFromAggregators].
|
||||
*
|
||||
* Also tallies each user's write relays into [writeRelayFreq] so the
|
||||
* backbone can be learned from the crawl. Authors are chunked per relay.
|
||||
@@ -687,10 +688,13 @@ class GrapeRankDataCrawler(
|
||||
// Build the query against the full straggler set BEFORE any folding (the
|
||||
// filter lists are materialized here, so later mutation of `stragglers` is
|
||||
// safe). Ask ONLY for kind:3 — the contact list we're missing. A multi-kind
|
||||
// filter breaks the big aggregators: user.kindpag.es serves kind:3 in a few
|
||||
// seconds when asked for it alone, but times out returning nothing when the
|
||||
// same authors are requested with kinds=[3,10000,1984,10002]. Mutes/reports
|
||||
// still come from the outbox model; the aggregator's job here is the lists.
|
||||
// filter is useless against the big indexers: user.kindpag.es caps its
|
||||
// response at ~100 events per REQ (it ignores our limit), so a
|
||||
// kinds=[3,10000,1984,10002] query comes back 100× kind:10002 and 0×
|
||||
// kind:3 — the abundant relay lists crowd the contact lists out entirely.
|
||||
// Asked for kind:3 alone it returns them in a few seconds. Their kind:10002
|
||||
// is already fetched in bulk by [ensureRelayLists]; mutes/reports still come
|
||||
// from the outbox model. The aggregator's job here is only the lists.
|
||||
val filters =
|
||||
aggregators.associateWith {
|
||||
stragglers.chunked(AUTHORS_PER_FILTER).map { chunk -> Filter(kinds = listOf(ContactListEvent.KIND), authors = chunk) }
|
||||
|
||||
Reference in New Issue
Block a user