docs: record accepted adaptive pagination-threshold design

Replaces the vetoed explicit-limit plan with the design settled in
review: keep omitting limit so generous relays can serve unbounded
pages, count raw delivered events instead of only Saved/Duplicate ones
(the counting gap is the sole reason thresholds need safety margin),
and adapt the threshold per relay as max(90, floor(0.9 x estimated
cap)) where the estimate combines the largest observed page (ground
truth, never unsafe) with NIP-11 default_limit as a verified hint.

Corrects two spec facts against the NIP-11 source and live documents
(2026-08-06): default_limit is a standard limitation field - the
earlier audit labelled nostream's copy nonstandard - and it is exactly
the omitted-limit page cap pagination needs, but it is rarely
advertised (neither nos.lol nor relay.ditto.pub emits it). max_limit
must never raise the threshold while requests omit limit: Ditto
advertises 1000 and serves 100; nostream advertises 5000 and serves
500.

The PAGINATION_THRESHOLD comment now points at the accepted design
instead of the withdrawn explicit-limit mitigation. Comment-only
change in src/sync/mod.rs; no behavioural change.
This commit is contained in:
DanConwayDev
2026-08-06 07:52:35 +00:00
parent a1a1b1884c
commit dfb054a5d0
2 changed files with 48 additions and 18 deletions
+42 -14
View File
@@ -71,12 +71,16 @@ The five reachable relays advertise their result cap as
### Discoverability gap (NIP-11)
NIP-11 `limitation` has **no field for tag values per filter**. It does define
`max_filters` (filters per subscription) and `max_limit` (query results), in
addition to `max_subscriptions` and `max_message_length`, but implementations
and operators advertise these unevenly. `max_limit` also cannot express
whether the allowance is per filter or aggregate across a multi-filter REQ.
Consequently neither filter sizing nor the pagination model can be negotiated
reliably; both need conservative defaults and reactive fallback.
`max_filters` (filters per subscription), `max_limit` (clamp applied to a
filter's explicit `limit`), and `default_limit` (maximum returned events when
`limit` is omitted — the field pagination actually needs), in addition to
`max_subscriptions` and `max_message_length`, but implementations and
operators advertise these unevenly: `default_limit` in particular is rarely
present (neither nos.lol nor relay.ditto.pub advertises it, checked live
2026-08-06). `max_limit` also cannot express whether the allowance is per
filter or aggregate across a multi-filter REQ. Consequently neither filter
sizing nor the pagination model can be negotiated reliably; both need
conservative defaults, observation, and reactive fallback.
### Our own embedded relay (nostr-sdk `LocalRelay`, 0.45.0)
@@ -215,14 +219,38 @@ Consequences:
raised from its original ultra-conservative 75 once the audit
established the real floor; filters with 75–199 results no longer pay
the extra page.
- Planned lever (not implemented): send an explicit `limit` on historic
filters — `min(page size, advertised max_limit)` where NIP-11 provides
one (strfry, nostream, rnostr, Ditto do) — so implicit defaults like
Ditto's 100 never apply, and use the advertised value as a per-relay
threshold with 200 as the static floor. Caveat: nostream *rejects* a
REQ whose requested limit exceeds its configured `maxLimit` rather than
clamping it, so explicit limits must respect advertised values where
present and stay modest otherwise.
- Planned design (accepted 2026-08-06, not implemented): keep omitting
`limit` — an explicit limit would cap the relays that serve unbounded
pages — fix the counting, and adapt the threshold per relay:
1. **Count raw delivered events.** Today only events processed as
Saved or Duplicate count toward the threshold and the `until`
cursor, so purgatory-routed and rejected events consume relay
allowance invisibly; this is the sole reason thresholds need
margin. Counting every delivered event that matches the filter
(and cursoring on them) makes a truncated page count exactly the
relay's page size.
2. **Adaptive per-relay threshold:**
`estimated_cap = max(largest observed page, advertised
default_limit if present)`;
`threshold = max(90, floor(0.9 × estimated_cap))`. Observed pages
are ground truth (always ≤ the true cap, so never unsafe, and
converging upward to eliminate redundant pages); the 0.9 slack
absorbs relay-side shrinkage such as expired-event skipping; the
floor of 90 stays below Ditto's 100, the smallest default found.
Learned state is per connection session and NIP-11 is refetched on
reconnect, so an operator lowering their cap cannot strand a stale
threshold.
3. **NIP-11 fields:** `default_limit` ("maximum returned events if
you send a filter without a limit") is the standard field for
exactly this and is used as a hint when advertised — though
rarely: neither nos.lol nor relay.ditto.pub advertises it (checked
live 2026-08-06). Being self-reported, a wrong-high value is
unsafe, so the first page that the hint would declare exhausted
triggers one verification page; if it yields new events the hint
is discarded in favour of learned-only. `max_limit` must never
raise the threshold while requests omit `limit`: it bounds
accepted explicit requests, not the omitted-limit page size
(Ditto: 1000 advertised vs 100 served; nostream: 5000 vs 500).
### Working floors
+6 -4
View File
@@ -598,10 +598,12 @@ const MAX_CONCURRENT_CONNECT_ATTEMPTS: usize = 8;
/// results the redundant final page. A relay capped below this threshold
/// silently truncates history. Known live exception (2026-08-06): Ditto
/// Relay applies a 100-event default to filters that omit `limit` — which
/// ours currently do — while accepting explicit limits up to its
/// advertised NIP-11 `max_limit`; sending explicit limits on historic
/// filters is the planned mitigation. See
/// docs/explanation/sync-scaling-constraints.md.
/// ours do — while advertising only its larger explicit-request cap in
/// NIP-11. The accepted mitigation (not yet implemented) is to count raw
/// delivered events instead of only Saved/Duplicate ones and adapt the
/// threshold per relay from observed page sizes and advertised
/// `default_limit`, with a floor of 90. See "Per-query result limits and
/// the pagination model" in docs/explanation/sync-scaling-constraints.md.
const PAGINATION_THRESHOLD: usize = 200;
/// Conservative number of OR filters carried by one NIP-01 REQ.