From 175cac3e467a77d083c5d337a4fac5ad9a448e75 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 3 Jul 2026 22:36:06 +0000 Subject: [PATCH] =?UTF-8?q?docs(quartz):=20plan=20=E2=80=94=20incremental?= =?UTF-8?q?=20live=20negentropy=20storage?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for backlog item 3 of the relay performance campaign: an always-current (created_at, id) index maintained from the store's write path so cold NEG-OPENs stop paying the full scan + O(n log n) seal (~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers the snapshot/COW model, the removal-correctness split (RETURNING deltas for replaceable overwrites, wholesale invalidation for rare delete paths), IndexingStrategy gating so app-side stores are untouched, and the micro + relayBench A/B measurement plan. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A --- PLANS.md | 3 +- ...26-07-03-incremental-negentropy-storage.md | 89 +++++++++++++++++++ quartz/plans/README.md | 3 +- 3 files changed, 93 insertions(+), 2 deletions(-) create mode 100644 quartz/plans/2026-07-03-incremental-negentropy-storage.md diff --git a/PLANS.md b/PLANS.md index e65a25495d..e3474caec3 100644 --- a/PLANS.md +++ b/PLANS.md @@ -33,7 +33,7 @@ listing (including archived plans), open that folder's `README.md`. | amethyst | 21 | 19 | 1 | 1 | 0 | [amethyst/plans](amethyst/plans/README.md) | | nestsClient | 26 | 23 | 1 | 2 | 0 | [nestsClient/plans](nestsClient/plans/README.md) | | desktopApp | 13 | 10 | 2 | 1 | 0 | [desktopApp/plans](desktopApp/plans/README.md) | -| quartz | 9 | 7 | 0 | 2 | 0 | [quartz/plans](quartz/plans/README.md) | +| quartz | 10 | 7 | 0 | 3 | 0 | [quartz/plans](quartz/plans/README.md) | | commons | 6 | 2 | 2 | 2 | 0 | [commons/plans](commons/plans/README.md) | | cli | 6 | 5 | 1 | 0 | 0 | [cli/plans](cli/plans/README.md) | | quic | 4 | 3 | 0 | 0 | 1 | [quic/plans](quic/plans/README.md) | @@ -67,6 +67,7 @@ them under each folder's `archive/` via the per-module index above. | amethyst | [napplet-inter-applet](amethyst/plans/2026-06-20-napplet-inter-applet.md) | NAP-INC / NAP-INTENT inter-applet messaging; prerequisites (multi-applet hosting, archetype registry, `MESSAGING` capability) not built. | | quartz | [local-headers-explorer](quartz/plans/2026-05-08-local-headers-explorer.md) | Headers-only Bitcoin P2P client to verify NIP-03 OTS attestations without a trusted block explorer. | | quartz | [giftwrap-deletion-requests](quartz/plans/2026-06-12-giftwrap-deletion-requests.md) | Let a recipient-authored kind-5 delete/block a gift wrap (kind 1059) addressed to them. | +| quartz | [incremental-negentropy-storage](quartz/plans/2026-07-03-incremental-negentropy-storage.md) | Always-current (created_at, id) index so cold NEG-OPENs stop paying a full scan + seal. | | commons | [event-renderer](commons/plans/2026-04-21-event-renderer.md) | Cross-platform UI-agnostic `RenderedEvent` subsystem shared by Amy, Desktop, Android; not started. | | commons | [amethyst-to-commons-migration](commons/plans/2026-05-30-amethyst-to-commons-migration.md) | Roadmap to move shared `amethyst` Android code into `commons`; keystone `Account`/`LocalCache` extraction not begun. | | desktopApp | [embedded-wallet-phase2-research](desktopApp/plans/2026-05-21-embedded-wallet-phase2-research.md) | Research for an embedded self-custodial Lightning wallet (Breez/ldk-node/lightning-kmp); parked, no code. | diff --git a/quartz/plans/2026-07-03-incremental-negentropy-storage.md b/quartz/plans/2026-07-03-incremental-negentropy-storage.md new file mode 100644 index 0000000000..6121a3c7c6 --- /dev/null +++ b/quartz/plans/2026-07-03-incremental-negentropy-storage.md @@ -0,0 +1,89 @@ +# Incremental live negentropy storage + +**Status: planned** — backlog item 3 of the relay performance campaign +(PR #3466 follow-up). + +## Problem + +A cold NEG-OPEN pays a full scan + O(n log n) seal: `snapshotIdsForNegentropy` +(SQL scan of every matching row) → `NegentropyServerSession.sealVector` +(sort + seal). relayBench measured ~340 ms per reconcile at 50k events before +the single-slot snapshot cache; strfry answers ~21 ms off its always-current +in-memory view. + +The cache added in #3466 (keyed on filter JSON + write generation + 30 s TTL) +only helps the *identical-filter, zero-writes-in-between* repeat. On a relay +ingesting continuously the generation moves constantly, so mirror heartbeats +are effectively always cold; any new filter is cold by definition. + +## Design + +### 1. `LiveNegentropyIndex` (quartz, server-only, opt-in) + +An always-current sorted set of `(createdAt, id₃₂)` maintained from the +store's write path — strfry's `MemoryView` equivalent, ~40 B/entry +(1M events ≈ 40 MB; capped by `negentropy.max_sync_events`). + +- **Structure**: single sorted array with binary-search insert. Nostr inserts + are near-tail (created_at ≈ now), so the memmove is tiny in the common + case; measure the out-of-order (backfill) worst case and only move to a + chunked layout if it shows. +- **Snapshot**: NEG-OPEN copies the array into a sealed `IStorage` — O(n) + arraycopy of already-sorted data (~2 MB at 50k, sub-ms), reusing the + existing single-slot cache so back-to-back opens share one snapshot. + Reconcile only reads (`size/getItem/iterate/indexAtOrBeforeBound`), so one + sealed snapshot serves any number of concurrent sessions. +- **Serves index-total filters only**: no `ids/authors/kinds/tags/search` + constraints. `since/until` ARE served — the structure is time-sorted, so a + time window is an index sub-range. Constrained filters keep the scan+seal + path (+ cache). The mirror-heartbeat pattern this optimizes is a broad + time-window filter, so this covers the case that matters. + +### 2. Removal correctness (the interesting part) + +The index must never advertise ids the store no longer has, or peers fetch +dead ids. Removal paths differ in frequency and get different treatment: + +- **Replaceable/addressable overwrite** (frequent — every kind 0/3/1xxxx/ + 3xxxx update): `ReplaceableModule`/`AddressableModule` run + `DELETE FROM event_headers WHERE …` inside the insert transaction. Add + `RETURNING created_at, id` (bundled SQLite ≥ 3.35) and report displaced + rows to the index alongside the insert. +- **Wholesale/rare paths** (kind-5 NIP-09, expiration sweep, right-to-vanish, + NIP-86 `delete(filter)`, FTS reindex/clear): invalidate the whole index; + the next NEG-OPEN rebuilds it lazily from one scan and incremental + maintenance resumes. Deletes are rare enough that occasional rebuilds beat + threading deltas through every module. + +### 3. Gating + +`IndexingStrategy.maintainLiveNegentropyIndex`, default **false** — library +defaults unchanged for app-side stores (campaign ground rule). geode's +`RelayIndexingStrategy` turns it on; config kill-switch under +`[negentropy]`. + +### 4. Concurrency + +Mutations happen only on the writer path (single-writer mutex — same +discipline as the FTS worker). Snapshots swap in via copy-on-write so a +reconcile never observes a mid-insert array. Shutdown: the index is memory +only, rebuilt on boot from the first NEG-OPEN's scan; no lifecycle beyond +the store's own close (ground rule 4: no worker, no uncaught exceptions). + +## Measurement plan + +1. Micro: quartz jvmTest benchmark, cold NEG-OPEN time at 50k events, + before/after (expect ~340 ms → single-digit ms). +2. Headline: relayBench pairwise sync, alternating A/B runs on the 50k + corpus (container noise ±10–30%; never trust a single run). Keep only + if it wins end-to-end; document either way. + +## Milestones + +1. `LiveNegentropyIndex` + unit tests (ordering, tail/backfill inserts, + removal, snapshot immutability under concurrent insert, cap behavior). +2. Displaced-row `RETURNING` plumbing in Replaceable/Addressable modules + + wholesale-invalidation hooks on the rare paths. +3. `LiveEventStore.sealedNegentropyStorage` wiring: index-total filters + from the index; everything else keeps scan+seal+cache. +4. Benchmarks (micro then relayBench A/B); revert if not a real win. diff --git a/quartz/plans/README.md b/quartz/plans/README.md index 5c0c538031..577d380717 100644 --- a/quartz/plans/README.md +++ b/quartz/plans/README.md @@ -1,12 +1,13 @@ # quartz plans -_Audited 2026-06-30. 9 plans: 7 shipped (archived), 0 in-progress, 2 queued, 0 abandoned._ +_Audited 2026-06-30. 10 plans: 7 shipped (archived), 0 in-progress, 3 queued, 0 abandoned._ ## Queued | Plan | Summary | | ---- | ------- | | [2026-05-08-local-headers-explorer.md](2026-05-08-local-headers-explorer.md) | Headers-only Bitcoin P2P client to verify NIP-03 OTS attestations without a trusted block explorer. | | [2026-06-12-giftwrap-deletion-requests.md](2026-06-12-giftwrap-deletion-requests.md) | Let a recipient-authored kind-5 delete/block a gift wrap (kind 1059) addressed to them. | +| [2026-07-03-incremental-negentropy-storage.md](2026-07-03-incremental-negentropy-storage.md) | Always-current (created_at, id) index so cold NEG-OPENs stop paying a full scan + seal (~340 ms at 50k vs strfry's ~21 ms). | ## Archived (shipped) | Plan | Summary |