mirror of
https://relay.ngit.dev/npub15qydau2hjma6ngxkl2cyar74wzyjshvl65za5k5rl69264ar2exs5cyejr/ngit-grasp.git
synced 2026-10-05 15:08:24 +00:00
DanConwayDev
c0a023c7f2
The earlier commit in this PR made every accepted repository owner's and declared maintainer's read/unmarked inboxes ordinary persistent live repository sources. Combined with unbounded per-author relay lists this drove the soak's connection growth: merely owning a repository let an author's advertised inventory imply long-lived live subscriptions. The 2026-08-19 investigation showed one bot-owned repository contributing 983 relay URLs through exactly this path. Owner/maintainer repository scope is now historical-only. The live inbox_repositories tier and its merge into derive_targets are removed entirely, and the repository-scoped overlay moves onto the existing paced, byte-bounded mailbox history workers using each author's bounded, sanitized read/unmarked inbox selection. Repository coordinates and known roots are still probed there, so a root or status stored only on a maintainer mailbox is still discovered - on the recurring history cycle instead of a persistent subscription. Correctness assumptions: inbox_roots is the pre-existing root-author live tier and is deliberately untouched, so accepted root authors keep live coverage on their own bounded inboxes. Participant mailboxes were already history-only. public_repository_mailbox_scope still empties repository scope in private mode before any overlay is built, and the per-relay independent mailbox worker behavior is unchanged. Excluded scope: no changes to instance-own-relay or repository-advertised relay coverage, no event-triggered drains, negentropy routing, or scheduling changes; those remain follow-up work. Validation: cargo fmt, cargo clippy --all-targets (clean), cargo test --lib (884 passed), cargo test --test sync proactive_sync (4 passed; the owner scenario now proves the inbox is probed through the history path, that a later root still arrives via historical refresh, and that the inbox never enters ordinary live sync).
186 lines
11 KiB
Markdown
186 lines
11 KiB
Markdown
# GRASP-03 proactive sync plus
|
|
|
|
GRASP-03 extends repository-declared GRASP-02 coverage to the Nostr outbox
|
|
model. An accepted issue, patch or pull request may have replies, reactions or
|
|
zaps in a conversation participant's inbox or outbox even when those events
|
|
are absent from repository relays.
|
|
|
|
The overlay is enabled by default and can be disabled with
|
|
`NGIT_SYNC_PLUS_ENABLED=false`. Because it extends the proactive GRASP-02
|
|
manager, it is effective only while that manager is running. NIP-11 advertises
|
|
`GRASP-03` only when the overlay is enabled.
|
|
|
|
## Minimal approach
|
|
|
|
The implementation adds a narrow discovery control-plane, not a second sync
|
|
engine: derive accepted root IDs and participant authors locally; ask a small
|
|
existing index set for each author's latest profile kind `0` and NIP-65 kind
|
|
`10002`; retain those replaceable events locally; keep accepted root authors'
|
|
`read` and unmarked inboxes in ordinary GRASP-02 coverage; and probe accepted
|
|
participants' `read`, `write` and unmarked conversation mailboxes with
|
|
repository and thread-reference filters. On public instances, accepted
|
|
repository owners' and declared maintainers' bounded `read` and unmarked
|
|
inboxes additionally receive repository-scoped historical mailbox probes.
|
|
Repository coordinates on those inboxes can therefore discover a root which is
|
|
not yet present locally on the recurring history cycle; the inboxes do not
|
|
enter ordinary GRASP-02 live repository coverage.
|
|
|
|
Kind `10002` lists are untrusted peer input, so every relay URL they advertise
|
|
passes single-URL target hygiene before it can occupy any selection slot,
|
|
overlay, queue, or metric (see `src/sync/target_hygiene.rs` for the rules and
|
|
their provenance). Accepted URLs are then canonicalized with the same relay-key
|
|
semantics as the connection map and deduplicated, so trailing-root-slash and
|
|
default-port variants of one relay consume one slot and reach the matching
|
|
connection. Each author contributes at most four relays per purpose, taken in
|
|
published tag order: up to four `read`/unmarked relays for live inbox coverage,
|
|
up to four `write`/unmarked relays as identity-refresh sources, up to four
|
|
history mailboxes preferring `write`/unmarked relays (read-only relays fill any
|
|
remaining slots), and up to four `read`/unmarked inboxes for owner/maintainer
|
|
repository coverage. The four-relay ceiling is a hard internal constant, not a
|
|
configuration option: NIP-65 already guides users toward two to four relays per
|
|
category, and the bound keeps one hostile or machine-generated list from
|
|
inflating mailbox inventory. Non-root paths and semantic query strings remain
|
|
distinct targets; operator-configured fallback relays are trusted input and are
|
|
not subject to peer-input hygiene.
|
|
|
|
Mailbox work reuses GRASP-02's connection safety, filter byte packing,
|
|
pagination, subscription ledger, request pacing, event pipeline and write
|
|
policy. The participant and owner/maintainer expansions are history-only: a
|
|
relay never becomes a permanent live source merely because an accepted author
|
|
advertised it or because its author owns or maintains a repository.
|
|
Root-author inboxes retain the pre-existing live/rotating Sync+ behavior.
|
|
|
|
Authors without an accepted stored relay list are queried through the configured
|
|
user-index set plus the bootstrap relay. Once a list is retained, discovery
|
|
follows its `write` and unmarked outboxes instead, allowing newer replacements
|
|
to converge without making arbitrary repository relays identity sources.
|
|
Discovery remains best effort and bounded to this operator-visible source graph.
|
|
|
|
If a successful user-index query returns no accepted kind `10002`, accepted
|
|
roots associated with that author temporarily use the operator-configured
|
|
Sync+ fallback relay set as mailboxes. Root-author fallback roots continue
|
|
through ordinary GRASP-02 historic, live and rotating coverage; non-root
|
|
participant roots use the paced history probe. An accepted relay list removes
|
|
the author from fallback coverage and schedules its declared relays instead.
|
|
|
|
The self-subscriber builds a compact root-candidate inventory during its
|
|
existing startup load and maintains it incrementally as roots arrive. StateOnly
|
|
candidates remain inert; promotion of their repository to Full makes them
|
|
eligible without rescanning retained root events. A bounded recursive scan of
|
|
locally accepted root threads adds the authors of replies, reactions, zaps and
|
|
other descendants. Root provenance is carried through event-ID and address
|
|
references, so each participant is associated only with accepted threads in
|
|
which their events occur, including indirect descendants that expose only an
|
|
immediate parent. Eligible identity authors are
|
|
the owners and declared maintainers of accepted Full announcements plus
|
|
accepted root and descendant authors—not every author retained by the relay. A
|
|
once-per-minute reconciliation derives sources from this index. Remote queries
|
|
contain at most 100 authors and only one discovery batch may be in flight
|
|
globally. Admission samples immediate
|
|
transient capacity; if historic work wins the small race before the permit is
|
|
acquired, that single batch may wait but discovery can never build a waiter
|
|
queue. Its SDK-owned REQ draws from the same pacer and subscription ledger as
|
|
historic work. Discovery sources use the managed connection safety and
|
|
reconnection machinery, but are a distinct control-plane role: connecting to a
|
|
user index or outbox does not start ordinary announcement, repository, or
|
|
descendant sync against it. At most one new discovery source is dialled per
|
|
maintenance pass, and an exclusively discovery connection retires after its
|
|
currently due identity or mailbox work drains. A relay that independently becomes a
|
|
repository source is promoted to the ordinary lifecycle without opening a
|
|
duplicate connection. Authors returned by a successful query refresh after 24 hours;
|
|
missing authors and failed queries retry after five minutes. Configured user
|
|
index relays discover authors with no accepted local relay list, requesting the
|
|
profile alongside it. Accepted NIP-65
|
|
write/unmarked outboxes are then followed additively for newer replacements;
|
|
new outboxes discovered by a replacement join the same bounded round until no
|
|
unvisited source remains. Identity events
|
|
pass through the ordinary write policy, persistence and broadcast path. Only a
|
|
stored, accepted kind `10002` can change mailbox ownership. On startup, retained
|
|
relay lists for eligible authors rebuild mailbox ownership from the local
|
|
database before any network refresh, so serving established coverage does not
|
|
depend on an external index remaining available. Remote discovery then refreshes
|
|
that retained state on the normal cadence.
|
|
|
|
Root-author read/unmarked inboxes use ordinary GRASP-02 coverage. Participant
|
|
mailboxes and owner/maintainer repository-scoped inboxes use history-only
|
|
workers. A maintenance pass admits at most one new
|
|
due history relay, while each admitted relay may have at most one worker in
|
|
flight and drains its own filter cursor without returning to a global
|
|
round-robin between successful groups. Up to 32 relay workers may run at once
|
|
as a process-safety ceiling. Relays do not share a mailbox lane or terminal
|
|
state: a slow or unavailable relay consumes only its own worker slot.
|
|
|
|
Each worker selects one stable-sorted, byte-bounded filter and delegates its
|
|
REQ lifecycle to the existing `RelayConnection::fetch_events` path. That path
|
|
owns per-relay request pacing, background priority, subscription-ledger
|
|
capacity, EOSE/CLOSED handling and a 30-second timeout. The worker reuses the
|
|
ordinary pagination state to continue through full historic pages until the
|
|
filter is exhausted or a page makes no new progress. It then passes events
|
|
through the normal write policy and persistence pipeline. A successful group
|
|
immediately starts that relay's next group until the cycle completes; a failed
|
|
group advances the cursor but releases the worker for the five-minute retry
|
|
delay. No mailbox-specific pending-batch kind, EOSE hook, close API, watchdog
|
|
or cross-relay coordinator is added.
|
|
|
|
Coverage uses accepted repository coordinates, root IDs, bounded recursive
|
|
descendant IDs, and descendant address coordinates with `a`/`A`/`q` and
|
|
`e`/`E`/`q`. A repository-scoped owner or maintainer inbox receives the exact
|
|
repository and every currently known root in it as historical probe work. This
|
|
permits a status or other descendant stored only on that inbox to be found on
|
|
the recurring history cycle even when the root author's current relay list
|
|
does not name it. A participant-only mailbox remains constrained to its
|
|
derived roots. A numeric in-memory cursor
|
|
gives each byte-bounded history filter group a turn. A successful complete
|
|
rotation refreshes after 24 hours; a failed group advances the cursor after a
|
|
five-minute delay and is retried on a later rotation. A relay already needed
|
|
for ordinary repository sync shares its connection; an exclusively
|
|
history/control-plane connection retires when its identity and mailbox work is
|
|
idle.
|
|
|
|
Owner/maintainer expansion adds history rotation proportional to the
|
|
byte-packed repository coordinates and known roots assigned to each of their
|
|
bounded read/unmarked inboxes; it adds no persistent live filters or
|
|
subscriptions. Shared relays and duplicate repository scopes are unioned
|
|
before filter construction. History still admits at most one new relay per
|
|
maintenance pass and remains single-flight per relay, but an admitted relay
|
|
keeps its worker while its successful filter groups drain. With hygiene and
|
|
the four-relay per-author bound applied before selection, distinct maintainer
|
|
relays increase only active history workers up to the process-wide safety
|
|
ceiling; the fixed-cardinality relay, cursor, worker, connection, and
|
|
subscription metrics must be watched during a production soak.
|
|
|
|
On restart, accepted roots and retained kind `10002` events rebuild participant
|
|
and mailbox ownership from LMDB. The in-memory group cursor is intentionally
|
|
not restored, so historic mailbox probing starts again from the first current
|
|
filter and remains safe to repeat. Inventory changes keep the cursor modulo the
|
|
new stable-sorted filter set; they do not coordinate with a worker on another
|
|
relay.
|
|
|
|
The fixed-cardinality retained-state metric reports eligible authors, desired
|
|
history-probe relays, allocated filter cursors, and active per-relay workers.
|
|
This makes a stuck worker or unexpected inventory expansion visible without
|
|
putting peer URLs or public keys into metric labels.
|
|
|
|
Mailbox replacement/removal changes desired ownership immediately. Root-author
|
|
inbox additions use ordinary coverage, while removal-only changes let existing
|
|
shared live subscriptions drain naturally.
|
|
History-probe additions become due promptly; removals prevent future groups
|
|
while an already in-flight `fetch_events` worker may finish naturally. Root
|
|
deletion follows the existing GRASP-02 root-index lifecycle and is
|
|
reconstructed from retained accepted events on restart; this change does not
|
|
add a second deletion graph.
|
|
|
|
## Deliberately excluded
|
|
|
|
- mailbox expansion for authors who have not produced locally accepted
|
|
repository-thread events and are not an accepted repository owner or
|
|
maintainer;
|
|
- repository mailbox expansion beyond an owner's or declared maintainer's
|
|
exact accepted Full repositories;
|
|
- repository-coordinate mailbox expansion in private mode;
|
|
- identity storage for authors outside accepted repositories and their
|
|
locally accepted threads;
|
|
- per-user fallback configuration knobs; and
|
|
- permanent live subscriptions on non-root participant mailboxes or on
|
|
owner/maintainer inboxes discovered only through repository ownership.
|