docs: sync comments/plans with the audit fixes

Follow-up to the audit fixes so nothing describes the pre-fix behavior:

- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
  the reload" comment above `val dbFile` — the sentinel-gated reuse it
  described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
  (publishing each onHaveIds batch) instead of materializing the full diff
  via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
  id-ASC tie-break is byte-exact vs the single-SQL path only when the store
  indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
This commit is contained in:
Claude
2026-07-05 12:20:11 +00:00
parent 2ce3e2bf5c
commit cece5b6e04
3 changed files with 22 additions and 15 deletions
@@ -54,8 +54,7 @@ fun main(args: Array<String>) {
val maxCount = args.getOrNull(2)?.toInt() ?: Int.MAX_VALUE
// File-backed so a 1M in-memory corpus here doesn't compete for RAM with an
// in-memory sink in the same box during the comparison. Reuses an already
// loaded DB (serve-only) so a re-run skips the multi-minute reload.
// in-memory sink in the same box during the comparison.
val dbFile = "/tmp/geode-source-$port.sqlite"
// Reuse an already-loaded DB (serve-only, skipping the multi-minute reload)
// ONLY when a completion sentinel proves it holds *this* corpus, fully loaded.
@@ -77,11 +77,16 @@ built as `MergeQueryExecutor` and wired into both `query` and the zero-decode
It opens one lazy newest-first cursor per `(kind, pubkey)` stream off the
existing `query_by_kind_pubkey_created` index (or `query_by_pubkey_created` for
authors-only), merges their heads `(created_at DESC, id ASC)`, and stops at the
limit — reading **O(limit + streams)** rows regardless of follow activity or
corpus size, which is exactly strfry's algorithm. Eligibility is narrow
(2..2048 streams, simple filter, explicit limit, no ids/d-tags); everything
else falls through to the single-SQL plan, so no other query shape changes.
authors-only), merges their heads newest-first (`created_at DESC`, tie-broken by
`id ASC`), and stops at the limit — reading **O(limit + streams)** rows
regardless of follow activity or corpus size, which is exactly strfry's
algorithm. Eligibility is narrow (2..2048 streams, simple filter, explicit
limit, no ids/d-tags); everything else falls through to the single-SQL plan, so
no other query shape changes. Duplicate authors/kinds are collapsed before
opening cursors (a repeat would double-emit), and the `id ASC` tie-break is
byte-exact against the single-SQL path only when the store indexes id
(`useAndIndexIdOnOrderBy`) — otherwise same-second ties fall in rowid order, a
valid newest-N either way.
Correctness is pinned by `MergeQueryCorrectnessTest` (merge == single-SQL top-N
vs both an independent reference and the SQL path, across ties/windows/raw).
@@ -214,14 +214,17 @@ confirms is bidirectional negentropy — `doUp = both||up`, `doDown = both||down
- **down** catch-up pulls what the upstream has and we lack
(`negentropySyncOrFetch`, with paged fallback), then the live REQ sub tails.
- **up** catch-up reconciles and pushes what we have and the upstream lacks
(`negentropyReconcileIds`'s `have` ids → `client.publish`), then the live
up-session tails. Negentropy-only (no paged fallback needed — the live
up-session covers a non-NIP-77 upstream). The push runs as a **reconcile→push
convergence loop**: `client.publish`'s outbox is best-effort under a bulk
burst (each publish also churns a reconnect; measured ~12% dropped per pass),
so each round re-reconciles — the reconcile *is* the delivery check — and
re-pushes only the stragglers until the `have` diff is empty. Observed on the
3000-event up test: 3000 → 69 → 2 → 0 across 3 rounds, lossless.
(streaming `negentropyReconcile`, publishing each `onHaveIds` batch straight
to `client.publish`), then the live up-session tails. Negentropy-only (no
paged fallback needed — the live up-session covers a non-NIP-77 upstream). The
streaming reconcile keeps peak memory at one id batch rather than materializing
the whole `have`/`need` diff (an important distinction at 1M — the full diff is
~100 MB of id strings). The push runs as a **reconcile→push convergence loop**:
`client.publish`'s outbox is best-effort under a bulk burst (each publish also
churns a reconnect; measured ~12% dropped per pass), so each round
re-reconciles — the reconcile *is* the delivery check — and re-pushes only the
stragglers until the `have` diff is empty. Observed on the 3000-event up test:
3000 → 69 → 2 → 0 across 3 rounds, lossless.
Same vocabulary as strfry throughout — one `[[mirror]]` entry, one `dir`
(down/up/both) driving both phases; negentropy-vs-REQ is an internal transport