mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-09 08:04:45 +00:00
docs(quartz): plan — incremental live negentropy storage
Design for backlog item 3 of the relay performance campaign: an always-current (created_at, id) index maintained from the store's write path so cold NEG-OPENs stop paying the full scan + O(n log n) seal (~340 ms at 50k events vs strfry's ~21 ms off its live tree). Covers the snapshot/COW model, the removal-correctness split (RETURNING deltas for replaceable overwrites, wholesale invalidation for rare delete paths), IndexingStrategy gating so app-side stores are untouched, and the micro + relayBench A/B measurement plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
This commit is contained in:
Claude
@@ -33,7 +33,7 @@ listing (including archived plans), open that folder's `README.md`.
|
||||
| amethyst | 21 | 19 | 1 | 1 | 0 | [amethyst/plans](amethyst/plans/README.md) |
|
||||
| nestsClient | 26 | 23 | 1 | 2 | 0 | [nestsClient/plans](nestsClient/plans/README.md) |
|
||||
| desktopApp | 13 | 10 | 2 | 1 | 0 | [desktopApp/plans](desktopApp/plans/README.md) |
|
||||
| quartz | 9 | 7 | 0 | 2 | 0 | [quartz/plans](quartz/plans/README.md) |
|
||||
| quartz | 10 | 7 | 0 | 3 | 0 | [quartz/plans](quartz/plans/README.md) |
|
||||
| commons | 6 | 2 | 2 | 2 | 0 | [commons/plans](commons/plans/README.md) |
|
||||
| cli | 6 | 5 | 1 | 0 | 0 | [cli/plans](cli/plans/README.md) |
|
||||
| quic | 4 | 3 | 0 | 0 | 1 | [quic/plans](quic/plans/README.md) |
|
||||
@@ -67,6 +67,7 @@ them under each folder's `archive/` via the per-module index above.
|
||||
| amethyst | [napplet-inter-applet](amethyst/plans/2026-06-20-napplet-inter-applet.md) | NAP-INC / NAP-INTENT inter-applet messaging; prerequisites (multi-applet hosting, archetype registry, `MESSAGING` capability) not built. |
|
||||
| quartz | [local-headers-explorer](quartz/plans/2026-05-08-local-headers-explorer.md) | Headers-only Bitcoin P2P client to verify NIP-03 OTS attestations without a trusted block explorer. |
|
||||
| quartz | [giftwrap-deletion-requests](quartz/plans/2026-06-12-giftwrap-deletion-requests.md) | Let a recipient-authored kind-5 delete/block a gift wrap (kind 1059) addressed to them. |
|
||||
| quartz | [incremental-negentropy-storage](quartz/plans/2026-07-03-incremental-negentropy-storage.md) | Always-current (created_at, id) index so cold NEG-OPENs stop paying a full scan + seal. |
|
||||
| commons | [event-renderer](commons/plans/2026-04-21-event-renderer.md) | Cross-platform UI-agnostic `RenderedEvent` subsystem shared by Amy, Desktop, Android; not started. |
|
||||
| commons | [amethyst-to-commons-migration](commons/plans/2026-05-30-amethyst-to-commons-migration.md) | Roadmap to move shared `amethyst` Android code into `commons`; keystone `Account`/`LocalCache` extraction not begun. |
|
||||
| desktopApp | [embedded-wallet-phase2-research](desktopApp/plans/2026-05-21-embedded-wallet-phase2-research.md) | Research for an embedded self-custodial Lightning wallet (Breez/ldk-node/lightning-kmp); parked, no code. |
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# Incremental live negentropy storage
|
||||
|
||||
**Status: planned** — backlog item 3 of the relay performance campaign
|
||||
(PR #3466 follow-up).
|
||||
|
||||
## Problem
|
||||
|
||||
A cold NEG-OPEN pays a full scan + O(n log n) seal: `snapshotIdsForNegentropy`
|
||||
(SQL scan of every matching row) → `NegentropyServerSession.sealVector`
|
||||
(sort + seal). relayBench measured ~340 ms per reconcile at 50k events before
|
||||
the single-slot snapshot cache; strfry answers ~21 ms off its always-current
|
||||
in-memory view.
|
||||
|
||||
The cache added in #3466 (keyed on filter JSON + write generation + 30 s TTL)
|
||||
only helps the *identical-filter, zero-writes-in-between* repeat. On a relay
|
||||
ingesting continuously the generation moves constantly, so mirror heartbeats
|
||||
are effectively always cold; any new filter is cold by definition.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. `LiveNegentropyIndex` (quartz, server-only, opt-in)
|
||||
|
||||
An always-current sorted set of `(createdAt, id₃₂)` maintained from the
|
||||
store's write path — strfry's `MemoryView` equivalent, ~40 B/entry
|
||||
(1M events ≈ 40 MB; capped by `negentropy.max_sync_events`).
|
||||
|
||||
- **Structure**: single sorted array with binary-search insert. Nostr inserts
|
||||
are near-tail (created_at ≈ now), so the memmove is tiny in the common
|
||||
case; measure the out-of-order (backfill) worst case and only move to a
|
||||
chunked layout if it shows.
|
||||
- **Snapshot**: NEG-OPEN copies the array into a sealed `IStorage` — O(n)
|
||||
arraycopy of already-sorted data (~2 MB at 50k, sub-ms), reusing the
|
||||
existing single-slot cache so back-to-back opens share one snapshot.
|
||||
Reconcile only reads (`size/getItem/iterate/indexAtOrBeforeBound`), so one
|
||||
sealed snapshot serves any number of concurrent sessions.
|
||||
- **Serves index-total filters only**: no `ids/authors/kinds/tags/search`
|
||||
constraints. `since/until` ARE served — the structure is time-sorted, so a
|
||||
time window is an index sub-range. Constrained filters keep the scan+seal
|
||||
path (+ cache). The mirror-heartbeat pattern this optimizes is a broad
|
||||
time-window filter, so this covers the case that matters.
|
||||
|
||||
### 2. Removal correctness (the interesting part)
|
||||
|
||||
The index must never advertise ids the store no longer has, or peers fetch
|
||||
dead ids. Removal paths differ in frequency and get different treatment:
|
||||
|
||||
- **Replaceable/addressable overwrite** (frequent — every kind 0/3/1xxxx/
|
||||
3xxxx update): `ReplaceableModule`/`AddressableModule` run
|
||||
`DELETE FROM event_headers WHERE …` inside the insert transaction. Add
|
||||
`RETURNING created_at, id` (bundled SQLite ≥ 3.35) and report displaced
|
||||
rows to the index alongside the insert.
|
||||
- **Wholesale/rare paths** (kind-5 NIP-09, expiration sweep, right-to-vanish,
|
||||
NIP-86 `delete(filter)`, FTS reindex/clear): invalidate the whole index;
|
||||
the next NEG-OPEN rebuilds it lazily from one scan and incremental
|
||||
maintenance resumes. Deletes are rare enough that occasional rebuilds beat
|
||||
threading deltas through every module.
|
||||
|
||||
### 3. Gating
|
||||
|
||||
`IndexingStrategy.maintainLiveNegentropyIndex`, default **false** — library
|
||||
defaults unchanged for app-side stores (campaign ground rule). geode's
|
||||
`RelayIndexingStrategy` turns it on; config kill-switch under
|
||||
`[negentropy]`.
|
||||
|
||||
### 4. Concurrency
|
||||
|
||||
Mutations happen only on the writer path (single-writer mutex — same
|
||||
discipline as the FTS worker). Snapshots swap in via copy-on-write so a
|
||||
reconcile never observes a mid-insert array. Shutdown: the index is memory
|
||||
only, rebuilt on boot from the first NEG-OPEN's scan; no lifecycle beyond
|
||||
the store's own close (ground rule 4: no worker, no uncaught exceptions).
|
||||
|
||||
## Measurement plan
|
||||
|
||||
1. Micro: quartz jvmTest benchmark, cold NEG-OPEN time at 50k events,
|
||||
before/after (expect ~340 ms → single-digit ms).
|
||||
2. Headline: relayBench pairwise sync, alternating A/B runs on the 50k
|
||||
corpus (container noise ±10–30%; never trust a single run). Keep only
|
||||
if it wins end-to-end; document either way.
|
||||
|
||||
## Milestones
|
||||
|
||||
1. `LiveNegentropyIndex` + unit tests (ordering, tail/backfill inserts,
|
||||
removal, snapshot immutability under concurrent insert, cap behavior).
|
||||
2. Displaced-row `RETURNING` plumbing in Replaceable/Addressable modules +
|
||||
wholesale-invalidation hooks on the rare paths.
|
||||
3. `LiveEventStore.sealedNegentropyStorage` wiring: index-total filters
|
||||
from the index; everything else keeps scan+seal+cache.
|
||||
4. Benchmarks (micro then relayBench A/B); revert if not a real win.
|
||||
@@ -1,12 +1,13 @@
|
||||
# quartz plans
|
||||
|
||||
_Audited 2026-06-30. 9 plans: 7 shipped (archived), 0 in-progress, 2 queued, 0 abandoned._
|
||||
_Audited 2026-06-30. 10 plans: 7 shipped (archived), 0 in-progress, 3 queued, 0 abandoned._
|
||||
|
||||
## Queued
|
||||
| Plan | Summary |
|
||||
| ---- | ------- |
|
||||
| [2026-05-08-local-headers-explorer.md](2026-05-08-local-headers-explorer.md) | Headers-only Bitcoin P2P client to verify NIP-03 OTS attestations without a trusted block explorer. |
|
||||
| [2026-06-12-giftwrap-deletion-requests.md](2026-06-12-giftwrap-deletion-requests.md) | Let a recipient-authored kind-5 delete/block a gift wrap (kind 1059) addressed to them. |
|
||||
| [2026-07-03-incremental-negentropy-storage.md](2026-07-03-incremental-negentropy-storage.md) | Always-current (created_at, id) index so cold NEG-OPENs stop paying a full scan + seal (~340 ms at 50k vs strfry's ~21 ms). |
|
||||
|
||||
## Archived (shipped)
|
||||
| Plan | Summary |
|
||||
|
||||
Reference in New Issue
Block a user