From cece5b6e04cb7ff82aa70802fce7858f1e5b8256 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 5 Jul 2026 12:20:11 +0000 Subject: [PATCH] docs: sync comments/plans with the audit fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the audit fixes so nothing describes the pre-fix behavior: - CorpusServerMain: drop the leftover "reuses an already loaded DB … skips the reload" comment above `val dbFile` — the sentinel-gated reuse it described is now spelled out in the block just below it. - sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile` (publishing each onHaveIds batch) instead of materializing the full diff via negentropyReconcileIds; note the O(batch) memory win at 1M. - follow-feed plan: the k-way merge dedups repeated authors/kinds, and its id-ASC tie-break is byte-exact vs the single-SQL path only when the store indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU --- .../geode/tools/CorpusServerMain.kt | 3 +-- .../2026-07-04-follow-feed-read-tradeoff.md | 15 ++++++++++----- .../plans/2026-07-04-sync-throughput-1m.md | 19 +++++++++++-------- 3 files changed, 22 insertions(+), 15 deletions(-) diff --git a/geode/src/main/kotlin/com/vitorpamplona/geode/tools/CorpusServerMain.kt b/geode/src/main/kotlin/com/vitorpamplona/geode/tools/CorpusServerMain.kt index ad7226c3ca..0f92f60a62 100644 --- a/geode/src/main/kotlin/com/vitorpamplona/geode/tools/CorpusServerMain.kt +++ b/geode/src/main/kotlin/com/vitorpamplona/geode/tools/CorpusServerMain.kt @@ -54,8 +54,7 @@ fun main(args: Array) { val maxCount = args.getOrNull(2)?.toInt() ?: Int.MAX_VALUE // File-backed so a 1M in-memory corpus here doesn't compete for RAM with an - // in-memory sink in the same box during the comparison. Reuses an already - // loaded DB (serve-only) so a re-run skips the multi-minute reload. + // in-memory sink in the same box during the comparison. val dbFile = "/tmp/geode-source-$port.sqlite" // Reuse an already-loaded DB (serve-only, skipping the multi-minute reload) // ONLY when a completion sentinel proves it holds *this* corpus, fully loaded. diff --git a/quartz/plans/2026-07-04-follow-feed-read-tradeoff.md b/quartz/plans/2026-07-04-follow-feed-read-tradeoff.md index 0aa7c63df8..ead19a80c9 100644 --- a/quartz/plans/2026-07-04-follow-feed-read-tradeoff.md +++ b/quartz/plans/2026-07-04-follow-feed-read-tradeoff.md @@ -77,11 +77,16 @@ built as `MergeQueryExecutor` and wired into both `query` and the zero-decode It opens one lazy newest-first cursor per `(kind, pubkey)` stream off the existing `query_by_kind_pubkey_created` index (or `query_by_pubkey_created` for -authors-only), merges their heads `(created_at DESC, id ASC)`, and stops at the -limit — reading **O(limit + streams)** rows regardless of follow activity or -corpus size, which is exactly strfry's algorithm. Eligibility is narrow -(2..2048 streams, simple filter, explicit limit, no ids/d-tags); everything -else falls through to the single-SQL plan, so no other query shape changes. +authors-only), merges their heads newest-first (`created_at DESC`, tie-broken by +`id ASC`), and stops at the limit — reading **O(limit + streams)** rows +regardless of follow activity or corpus size, which is exactly strfry's +algorithm. Eligibility is narrow (2..2048 streams, simple filter, explicit +limit, no ids/d-tags); everything else falls through to the single-SQL plan, so +no other query shape changes. Duplicate authors/kinds are collapsed before +opening cursors (a repeat would double-emit), and the `id ASC` tie-break is +byte-exact against the single-SQL path only when the store indexes id +(`useAndIndexIdOnOrderBy`) — otherwise same-second ties fall in rowid order, a +valid newest-N either way. Correctness is pinned by `MergeQueryCorrectnessTest` (merge == single-SQL top-N vs both an independent reference and the SQL path, across ties/windows/raw). diff --git a/relayBench/plans/2026-07-04-sync-throughput-1m.md b/relayBench/plans/2026-07-04-sync-throughput-1m.md index 53cf05a7b4..5230a2f4fe 100644 --- a/relayBench/plans/2026-07-04-sync-throughput-1m.md +++ b/relayBench/plans/2026-07-04-sync-throughput-1m.md @@ -214,14 +214,17 @@ confirms is bidirectional negentropy — `doUp = both||up`, `doDown = both||down - **down** catch-up pulls what the upstream has and we lack (`negentropySyncOrFetch`, with paged fallback), then the live REQ sub tails. - **up** catch-up reconciles and pushes what we have and the upstream lacks - (`negentropyReconcileIds`'s `have` ids → `client.publish`), then the live - up-session tails. Negentropy-only (no paged fallback needed — the live - up-session covers a non-NIP-77 upstream). The push runs as a **reconcile→push - convergence loop**: `client.publish`'s outbox is best-effort under a bulk - burst (each publish also churns a reconnect; measured ~1–2% dropped per pass), - so each round re-reconciles — the reconcile *is* the delivery check — and - re-pushes only the stragglers until the `have` diff is empty. Observed on the - 3000-event up test: 3000 → 69 → 2 → 0 across 3 rounds, lossless. + (streaming `negentropyReconcile`, publishing each `onHaveIds` batch straight + to `client.publish`), then the live up-session tails. Negentropy-only (no + paged fallback needed — the live up-session covers a non-NIP-77 upstream). The + streaming reconcile keeps peak memory at one id batch rather than materializing + the whole `have`/`need` diff (an important distinction at 1M — the full diff is + ~100 MB of id strings). The push runs as a **reconcile→push convergence loop**: + `client.publish`'s outbox is best-effort under a bulk burst (each publish also + churns a reconnect; measured ~1–2% dropped per pass), so each round + re-reconciles — the reconcile *is* the delivery check — and re-pushes only the + stragglers until the `have` diff is empty. Observed on the 3000-event up test: + 3000 → 69 → 2 → 0 across 3 rounds, lossless. Same vocabulary as strfry throughout — one `[[mirror]]` entry, one `dir` (down/up/both) driving both phases; negentropy-vs-REQ is an internal transport