mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-06 07:28:23 +00:00
docs: record accepted adaptive pagination-threshold design
Replaces the vetoed explicit-limit plan with the design settled in review: keep omitting limit so generous relays can serve unbounded pages, count raw delivered events instead of only Saved/Duplicate ones (the counting gap is the sole reason thresholds need safety margin), and adapt the threshold per relay as max(90, floor(0.9 x estimated cap)) where the estimate combines the largest observed page (ground truth, never unsafe) with NIP-11 default_limit as a verified hint. Corrects two spec facts against the NIP-11 source and live documents (2026-08-06): default_limit is a standard limitation field - the earlier audit labelled nostream's copy nonstandard - and it is exactly the omitted-limit page cap pagination needs, but it is rarely advertised (neither nos.lol nor relay.ditto.pub emits it). max_limit must never raise the threshold while requests omit limit: Ditto advertises 1000 and serves 100; nostream advertises 5000 and serves 500. The PAGINATION_THRESHOLD comment now points at the accepted design instead of the withdrawn explicit-limit mitigation. Comment-only change in src/sync/mod.rs; no behavioural change.
This commit is contained in:
@@ -71,12 +71,16 @@ The five reachable relays advertise their result cap as
|
||||
### Discoverability gap (NIP-11)
|
||||
|
||||
NIP-11 `limitation` has **no field for tag values per filter**. It does define
|
||||
`max_filters` (filters per subscription) and `max_limit` (query results), in
|
||||
addition to `max_subscriptions` and `max_message_length`, but implementations
|
||||
and operators advertise these unevenly. `max_limit` also cannot express
|
||||
whether the allowance is per filter or aggregate across a multi-filter REQ.
|
||||
Consequently neither filter sizing nor the pagination model can be negotiated
|
||||
reliably; both need conservative defaults and reactive fallback.
|
||||
`max_filters` (filters per subscription), `max_limit` (clamp applied to a
|
||||
filter's explicit `limit`), and `default_limit` (maximum returned events when
|
||||
`limit` is omitted — the field pagination actually needs), in addition to
|
||||
`max_subscriptions` and `max_message_length`, but implementations and
|
||||
operators advertise these unevenly: `default_limit` in particular is rarely
|
||||
present (neither nos.lol nor relay.ditto.pub advertises it, checked live
|
||||
2026-08-06). `max_limit` also cannot express whether the allowance is per
|
||||
filter or aggregate across a multi-filter REQ. Consequently neither filter
|
||||
sizing nor the pagination model can be negotiated reliably; both need
|
||||
conservative defaults, observation, and reactive fallback.
|
||||
|
||||
### Our own embedded relay (nostr-sdk `LocalRelay`, 0.45.0)
|
||||
|
||||
@@ -215,14 +219,38 @@ Consequences:
|
||||
raised from its original ultra-conservative 75 once the audit
|
||||
established the real floor; filters with 75–199 results no longer pay
|
||||
the extra page.
|
||||
- Planned lever (not implemented): send an explicit `limit` on historic
|
||||
filters — `min(page size, advertised max_limit)` where NIP-11 provides
|
||||
one (strfry, nostream, rnostr, Ditto do) — so implicit defaults like
|
||||
Ditto's 100 never apply, and use the advertised value as a per-relay
|
||||
threshold with 200 as the static floor. Caveat: nostream *rejects* a
|
||||
REQ whose requested limit exceeds its configured `maxLimit` rather than
|
||||
clamping it, so explicit limits must respect advertised values where
|
||||
present and stay modest otherwise.
|
||||
- Planned design (accepted 2026-08-06, not implemented): keep omitting
|
||||
`limit` — an explicit limit would cap the relays that serve unbounded
|
||||
pages — fix the counting, and adapt the threshold per relay:
|
||||
1. **Count raw delivered events.** Today only events processed as
|
||||
Saved or Duplicate count toward the threshold and the `until`
|
||||
cursor, so purgatory-routed and rejected events consume relay
|
||||
allowance invisibly; this is the sole reason thresholds need
|
||||
margin. Counting every delivered event that matches the filter
|
||||
(and cursoring on them) makes a truncated page count exactly the
|
||||
relay's page size.
|
||||
2. **Adaptive per-relay threshold:**
|
||||
`estimated_cap = max(largest observed page, advertised
|
||||
default_limit if present)`;
|
||||
`threshold = max(90, floor(0.9 × estimated_cap))`. Observed pages
|
||||
are ground truth (always ≤ the true cap, so never unsafe, and
|
||||
converging upward to eliminate redundant pages); the 0.9 slack
|
||||
absorbs relay-side shrinkage such as expired-event skipping; the
|
||||
floor of 90 stays below Ditto's 100, the smallest default found.
|
||||
Learned state is per connection session and NIP-11 is refetched on
|
||||
reconnect, so an operator lowering their cap cannot strand a stale
|
||||
threshold.
|
||||
3. **NIP-11 fields:** `default_limit` ("maximum returned events if
|
||||
you send a filter without a limit") is the standard field for
|
||||
exactly this and is used as a hint when advertised — though
|
||||
rarely: neither nos.lol nor relay.ditto.pub advertises it (checked
|
||||
live 2026-08-06). Being self-reported, a wrong-high value is
|
||||
unsafe, so the first page that the hint would declare exhausted
|
||||
triggers one verification page; if it yields new events the hint
|
||||
is discarded in favour of learned-only. `max_limit` must never
|
||||
raise the threshold while requests omit `limit`: it bounds
|
||||
accepted explicit requests, not the omitted-limit page size
|
||||
(Ditto: 1000 advertised vs 100 served; nostream: 5000 vs 500).
|
||||
|
||||
### Working floors
|
||||
|
||||
|
||||
+6
-4
@@ -598,10 +598,12 @@ const MAX_CONCURRENT_CONNECT_ATTEMPTS: usize = 8;
|
||||
/// results the redundant final page. A relay capped below this threshold
|
||||
/// silently truncates history. Known live exception (2026-08-06): Ditto
|
||||
/// Relay applies a 100-event default to filters that omit `limit` — which
|
||||
/// ours currently do — while accepting explicit limits up to its
|
||||
/// advertised NIP-11 `max_limit`; sending explicit limits on historic
|
||||
/// filters is the planned mitigation. See
|
||||
/// docs/explanation/sync-scaling-constraints.md.
|
||||
/// ours do — while advertising only its larger explicit-request cap in
|
||||
/// NIP-11. The accepted mitigation (not yet implemented) is to count raw
|
||||
/// delivered events instead of only Saved/Duplicate ones and adapt the
|
||||
/// threshold per relay from observed page sizes and advertised
|
||||
/// `default_limit`, with a floor of 90. See "Per-query result limits and
|
||||
/// the pagination model" in docs/explanation/sync-scaling-constraints.md.
|
||||
const PAGINATION_THRESHOLD: usize = 200;
|
||||
|
||||
/// Conservative number of OR filters carried by one NIP-01 REQ.
|
||||
|
||||
Reference in New Issue
Block a user