mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-07-30 19:46:17 +00:00
docs: sync comments/plans with the audit fixes
Follow-up to the audit fixes so nothing describes the pre-fix behavior: - CorpusServerMain: drop the leftover "reuses an already loaded DB … skips the reload" comment above `val dbFile` — the sentinel-gated reuse it described is now spelled out in the block just below it. - sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile` (publishing each onHaveIds batch) instead of materializing the full diff via negentropyReconcileIds; note the O(batch) memory win at 1M. - follow-feed plan: the k-way merge dedups repeated authors/kinds, and its id-ASC tie-break is byte-exact vs the single-SQL path only when the store indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
This commit is contained in:
@@ -54,8 +54,7 @@ fun main(args: Array<String>) {
|
||||
val maxCount = args.getOrNull(2)?.toInt() ?: Int.MAX_VALUE
|
||||
|
||||
// File-backed so a 1M in-memory corpus here doesn't compete for RAM with an
|
||||
// in-memory sink in the same box during the comparison. Reuses an already
|
||||
// loaded DB (serve-only) so a re-run skips the multi-minute reload.
|
||||
// in-memory sink in the same box during the comparison.
|
||||
val dbFile = "/tmp/geode-source-$port.sqlite"
|
||||
// Reuse an already-loaded DB (serve-only, skipping the multi-minute reload)
|
||||
// ONLY when a completion sentinel proves it holds *this* corpus, fully loaded.
|
||||
|
||||
@@ -77,11 +77,16 @@ built as `MergeQueryExecutor` and wired into both `query` and the zero-decode
|
||||
|
||||
It opens one lazy newest-first cursor per `(kind, pubkey)` stream off the
|
||||
existing `query_by_kind_pubkey_created` index (or `query_by_pubkey_created` for
|
||||
authors-only), merges their heads `(created_at DESC, id ASC)`, and stops at the
|
||||
limit — reading **O(limit + streams)** rows regardless of follow activity or
|
||||
corpus size, which is exactly strfry's algorithm. Eligibility is narrow
|
||||
(2..2048 streams, simple filter, explicit limit, no ids/d-tags); everything
|
||||
else falls through to the single-SQL plan, so no other query shape changes.
|
||||
authors-only), merges their heads newest-first (`created_at DESC`, tie-broken by
|
||||
`id ASC`), and stops at the limit — reading **O(limit + streams)** rows
|
||||
regardless of follow activity or corpus size, which is exactly strfry's
|
||||
algorithm. Eligibility is narrow (2..2048 streams, simple filter, explicit
|
||||
limit, no ids/d-tags); everything else falls through to the single-SQL plan, so
|
||||
no other query shape changes. Duplicate authors/kinds are collapsed before
|
||||
opening cursors (a repeat would double-emit), and the `id ASC` tie-break is
|
||||
byte-exact against the single-SQL path only when the store indexes id
|
||||
(`useAndIndexIdOnOrderBy`) — otherwise same-second ties fall in rowid order, a
|
||||
valid newest-N either way.
|
||||
|
||||
Correctness is pinned by `MergeQueryCorrectnessTest` (merge == single-SQL top-N
|
||||
vs both an independent reference and the SQL path, across ties/windows/raw).
|
||||
|
||||
@@ -214,14 +214,17 @@ confirms is bidirectional negentropy — `doUp = both||up`, `doDown = both||down
|
||||
- **down** catch-up pulls what the upstream has and we lack
|
||||
(`negentropySyncOrFetch`, with paged fallback), then the live REQ sub tails.
|
||||
- **up** catch-up reconciles and pushes what we have and the upstream lacks
|
||||
(`negentropyReconcileIds`'s `have` ids → `client.publish`), then the live
|
||||
up-session tails. Negentropy-only (no paged fallback needed — the live
|
||||
up-session covers a non-NIP-77 upstream). The push runs as a **reconcile→push
|
||||
convergence loop**: `client.publish`'s outbox is best-effort under a bulk
|
||||
burst (each publish also churns a reconnect; measured ~1–2% dropped per pass),
|
||||
so each round re-reconciles — the reconcile *is* the delivery check — and
|
||||
re-pushes only the stragglers until the `have` diff is empty. Observed on the
|
||||
3000-event up test: 3000 → 69 → 2 → 0 across 3 rounds, lossless.
|
||||
(streaming `negentropyReconcile`, publishing each `onHaveIds` batch straight
|
||||
to `client.publish`), then the live up-session tails. Negentropy-only (no
|
||||
paged fallback needed — the live up-session covers a non-NIP-77 upstream). The
|
||||
streaming reconcile keeps peak memory at one id batch rather than materializing
|
||||
the whole `have`/`need` diff (an important distinction at 1M — the full diff is
|
||||
~100 MB of id strings). The push runs as a **reconcile→push convergence loop**:
|
||||
`client.publish`'s outbox is best-effort under a bulk burst (each publish also
|
||||
churns a reconnect; measured ~1–2% dropped per pass), so each round
|
||||
re-reconciles — the reconcile *is* the delivery check — and re-pushes only the
|
||||
stragglers until the `have` diff is empty. Observed on the 3000-event up test:
|
||||
3000 → 69 → 2 → 0 across 3 rounds, lossless.
|
||||
|
||||
Same vocabulary as strfry throughout — one `[[mirror]]` entry, one `dir`
|
||||
(down/up/both) driving both phases; negentropy-vs-REQ is an internal transport
|
||||
|
||||
Reference in New Issue
Block a user