From 41240eeab9e9d32470379c0d15d89ba1f816ba21 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 14:39:21 +0000 Subject: [PATCH] docs(store): record measured index/planner gaps as TODOs in both stores Real numbers from the two new benchmarks, so the trade-offs are on the decision points instead of in a chat log: - IndexingStrategy.indexTagsWithKindAndPubkey: the KDoc called the kinds+authors+tags shape "rarely used", but the client assembler survey found 65 call sites. TagAuthorIndexBenchmark @ 200k events: DM-room query 9.4 ms -> 0.6 ms (~15x) with the flag on, insert cost +14% (41.5 -> 47.3 us/event). TODO: re-evaluate defaults (geode). - MergeQueryExecutor: tag-path analogue of the follow-feed collect-all sort (kinds + #e IN [hundreds] + limit never merges). Measured 12.8 ms cold / 6.0 ms since-bounded at 200k events; revisit if relayBench shows it at relay scale. - FsQueryPlanner: fixed driver order sends authors+kinds+limit (the most common CLI shape) through the kind tree. FsDriverSelection- Benchmark @ 30k events: 149 ms -> 3.4 ms (~44x) driving from the author tree; TODO: cost-based pick by directory entry counts. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01RzkoN3SJHCZWRAiadXXG4w --- .../nip01Core/store/sqlite/IndexingStrategy.kt | 15 ++++++++++++--- .../nip01Core/store/sqlite/MergeQueryExecutor.kt | 10 ++++++++++ .../quartz/nip01Core/store/fs/FsQueryPlanner.kt | 11 +++++++++++ 3 files changed, 33 insertions(+), 3 deletions(-) diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt index 816f50f0f5..4f2e848c12 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt @@ -74,9 +74,18 @@ interface IndexingStrategy { * Activate this if you see too many Tag-centric Filters without * kind AND pubkey at the same time. * - * This is a rarely used index (reports by your follows or - * NIP-04 DMs for instance) that becomes quite large without - * major gains. + * This shape (reports by your follows, NIP-04 DM rooms, follows-scoped + * community feeds) is not rare on the client side: the 2026-07 filter + * assembler survey counted 65 call sites building + * `kinds + authors + tags`. Without this index the plan seeks + * `(tag_hash, kind)` and reads every row for that tag/kind before + * filtering the author. + * + * TODO: re-evaluate the off-by-default choice (especially for geode) + * with `TagAuthorIndexBenchmark` (jvmTest prodbench). At 200k events: + * DM-room query 9.4 ms → 0.6 ms (~15×) with the flag on, for a batch + * insert cost of 41.5 → 47.3 µs/event (+14%) — measure at target + * corpus size before flipping, since the index competes for page cache. * * Keep in mind that activating too many indexes increases the size of the * DB so much that the indexes themselves won't fit in memory, requiring diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/MergeQueryExecutor.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/MergeQueryExecutor.kt index 62660454db..24b62c56d5 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/MergeQueryExecutor.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/MergeQueryExecutor.kt @@ -52,6 +52,16 @@ import androidx.sqlite.SQLiteStatement * exactly at a same-second boundary. */ internal object MergeQueryExecutor { + // TODO: the same collect-all + TEMP-B-TREE-sort pattern exists one index + // over, on the tag path: `kinds + tags(#e IN [hundreds]) + limit` (the + // reactions/replies watcher archetype) unions per-value streams that are + // each sorted off `(tag_hash, kind, created_at)` and sorts the union. + // `streamCount` currently rejects any filter with tags, so those queries + // never merge. Measured by `TagAuthorIndexBenchmark` at 200k events: + // `#e IN 300, limit 500` costs 12.8 ms cold / 6.0 ms since-bounded — + // tolerable client-side, but it scales with matching history like the + // follow-feed shape did; extend the merge to per-tag-value streams if + // relay-scale runs (relayBench) show it in the profile. const val COLS = "id, pubkey, created_at, kind, tags, content, sig" /** diff --git a/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsQueryPlanner.kt b/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsQueryPlanner.kt index 1c64f50360..f6958d8d80 100644 --- a/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsQueryPlanner.kt +++ b/quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsQueryPlanner.kt @@ -46,6 +46,17 @@ import kotlin.io.path.exists * later without changing callers. All FilterMatcher semantics (tag * AND/OR, since/until, id, author, kind cross-checks) are enforced in * the orchestrator, so picking a loose driver is correctness-safe. + * + * TODO: add the cost-based pick. The fixed tags → kinds → authors order + * makes `authors + kinds + limit` — the most common CLI shape (27 + * assembler call sites, every `amy feed`-style author timeline) — drive + * from the kind tree and post-filter the author. Measured by + * `FsDriverSelectionBenchmark` at 30k events: 149 ms via `idx/kind/1/` + * vs 3.4 ms via `idx/author//` with kind post-filtered (~44×), and + * the gap grows with the kind tree, not the result. Comparing candidate + * directory entry counts before walking (kind dirs vs author dirs vs tag + * dirs) is enough; the slot shortcut already rescues the + * replaceable/addressable subset. */ internal class FsQueryPlanner( private val layout: FsLayout,