docs(store): record measured index/planner gaps as TODOs in both stores

Real numbers from the two new benchmarks, so the trade-offs are on the
decision points instead of in a chat log:

- IndexingStrategy.indexTagsWithKindAndPubkey: the KDoc called the
  kinds+authors+tags shape "rarely used", but the client assembler
  survey found 65 call sites. TagAuthorIndexBenchmark @ 200k events:
  DM-room query 9.4 ms -> 0.6 ms (~15x) with the flag on, insert cost
  +14% (41.5 -> 47.3 us/event). TODO: re-evaluate defaults (geode).

- MergeQueryExecutor: tag-path analogue of the follow-feed collect-all
  sort (kinds + #e IN [hundreds] + limit never merges). Measured
  12.8 ms cold / 6.0 ms since-bounded at 200k events; revisit if
  relayBench shows it at relay scale.

- FsQueryPlanner: fixed driver order sends authors+kinds+limit (the
  most common CLI shape) through the kind tree. FsDriverSelection-
  Benchmark @ 30k events: 149 ms -> 3.4 ms (~44x) driving from the
  author tree; TODO: cost-based pick by directory entry counts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RzkoN3SJHCZWRAiadXXG4w
This commit is contained in:
Claude
2026-07-21 14:39:21 +00:00
parent 5988db010d
commit 41240eeab9
3 changed files with 33 additions and 3 deletions
@@ -74,9 +74,18 @@ interface IndexingStrategy {
* Activate this if you see too many Tag-centric Filters without
* kind AND pubkey at the same time.
*
* This is a rarely used index (reports by your follows or
* NIP-04 DMs for instance) that becomes quite large without
* major gains.
* This shape (reports by your follows, NIP-04 DM rooms, follows-scoped
* community feeds) is not rare on the client side: the 2026-07 filter
* assembler survey counted 65 call sites building
* `kinds + authors + tags`. Without this index the plan seeks
* `(tag_hash, kind)` and reads every row for that tag/kind before
* filtering the author.
*
* TODO: re-evaluate the off-by-default choice (especially for geode)
* with `TagAuthorIndexBenchmark` (jvmTest prodbench). At 200k events:
* DM-room query 9.4 ms → 0.6 ms (~15×) with the flag on, for a batch
* insert cost of 41.5 → 47.3 µs/event (+14%) — measure at target
* corpus size before flipping, since the index competes for page cache.
*
* Keep in mind that activating too many indexes increases the size of the
* DB so much that the indexes themselves won't fit in memory, requiring
@@ -52,6 +52,16 @@ import androidx.sqlite.SQLiteStatement
* exactly at a same-second boundary.
*/
internal object MergeQueryExecutor {
// TODO: the same collect-all + TEMP-B-TREE-sort pattern exists one index
// over, on the tag path: `kinds + tags(#e IN [hundreds]) + limit` (the
// reactions/replies watcher archetype) unions per-value streams that are
// each sorted off `(tag_hash, kind, created_at)` and sorts the union.
// `streamCount` currently rejects any filter with tags, so those queries
// never merge. Measured by `TagAuthorIndexBenchmark` at 200k events:
// `#e IN 300, limit 500` costs 12.8 ms cold / 6.0 ms since-bounded —
// tolerable client-side, but it scales with matching history like the
// follow-feed shape did; extend the merge to per-tag-value streams if
// relay-scale runs (relayBench) show it in the profile.
const val COLS = "id, pubkey, created_at, kind, tags, content, sig"
/**
@@ -46,6 +46,17 @@ import kotlin.io.path.exists
* later without changing callers. All FilterMatcher semantics (tag
* AND/OR, since/until, id, author, kind cross-checks) are enforced in
* the orchestrator, so picking a loose driver is correctness-safe.
*
* TODO: add the cost-based pick. The fixed tags → kinds → authors order
* makes `authors + kinds + limit` — the most common CLI shape (27
* assembler call sites, every `amy feed`-style author timeline) — drive
* from the kind tree and post-filter the author. Measured by
* `FsDriverSelectionBenchmark` at 30k events: 149 ms via `idx/kind/1/`
* vs 3.4 ms via `idx/author/<pk>/` with kind post-filtered (~44×), and
* the gap grows with the kind tree, not the result. Comparing candidate
* directory entry counts before walking (kind dirs vs author dirs vs tag
* dirs) is enough; the slot shortcut already rescues the
* replaceable/addressable subset.
*/
internal class FsQueryPlanner(
private val layout: FsLayout,