diff --git a/quartz/plans/2026-07-21-sqlite-query-scaling.md b/quartz/plans/2026-07-21-sqlite-query-scaling.md index d8a6860bf9..704843cb92 100644 --- a/quartz/plans/2026-07-21-sqlite-query-scaling.md +++ b/quartz/plans/2026-07-21-sqlite-query-scaling.md @@ -7,63 +7,69 @@ ingest 16×, author-timeline 13×, follow-feed 2.4× — while point reads staye flat. Two of those (author-timeline, follow-feed) were already addressed (the pubkey-alone index and `MergeQueryExecutor`) and mostly reflect running the report under the *client* `DefaultIndexingStrategy` rather than -`relayIndexingStrategy()`. This change takes the two structural read shapes -that were still corpus-bound: **NIP-50 search** and the **large-IN tag -watcher**, plus a statement-cache fix the merge paths needed. +`relayIndexingStrategy()`. This change adds the **large-IN tag watcher** to the +merge executor, fixes the **FTS delete path** (which degraded with corpus +size) and shrinks the FTS index, and fixes a **statement-cache** miss the merge +paths hit. + +It does **not** fix NIP-50 *search* latency: that is bounded by the +`created_at`-ordered sort over all matches, which FTS5 can't early-terminate +while staying NIP-01-compliant (see §1). Corpus-independent search is an +external-engine job. Everything here is read/size work; the write path and on-disk index set are -unchanged except the FTS table, which gets *smaller*. +unchanged except the FTS table, which gets *smaller* and deletes faster. -## 1. NIP-50 search — contentless FTS5 + optional rowid ordering + segment merge +## 1. FTS — contentless index (fast deletes + smaller), NOT a search-scaling fix The old index was `fts5(event_header_row_id, content)`: it stored a second -copy of the tokenized text, and its rowid was auto-assigned (unrelated to the -event). Search ran `… event_fts MATCH ? ORDER BY event_headers.created_at DESC -LIMIT n`, which must materialize **every** document matching the term and sort -it — cost grows with the whole corpus. That is the 18× curve. +copy of the tokenized text, its rowid was auto-assigned (unrelated to the +event), and — the real problem — the `fts_foreign_key` delete trigger deleted +by the `event_header_row_id` *column*, which FTS5 cannot seek, so it **scanned +the whole index per delete**. -Three changes (`FullTextSearchModule`, `IndexingStrategy`, `QueryBuilder`, -DB version 4→5): +Changes (`FullTextSearchModule`, `QueryBuilder`, DB version 4→5): +- **rowid = `event_headers.row_id`.** Set explicitly on every insert; the + delete trigger, reindex, and catch-up paths key off it. Deletes become an + O(log n) primary-key seek instead of an O(n) column scan. This is the win: + every event removal fires the trigger — replaceable rotation, kind-5, + expiration, right-to-vanish — so on the old schema deletion throughput + degraded with corpus size. - **Contentless** (`fts5(content, content='', contentless_delete=1)`). The indexed text is *derived* (`SearchableEvent.indexableContent()`, not a raw column), so FTS5 **external-content** — which re-reads the source column from the base table — cannot express it; **contentless** is the correct primitive. - It drops the stored content copy (index shrinks) and, with - `contentless_delete=1`, still supports the `fts_foreign_key` delete trigger. -- **rowid = `event_headers.row_id`.** Set explicitly on every insert. The - delete trigger and reindex/catch-up paths all key off it, so the join is - `event_headers.row_id = event_fts.rowid` with no stored linkage column. -- **`searchOrderByRowId`** (new `IndexingStrategy` flag, **default off**). When - on, simple search orders by `event_fts.rowid DESC LIMIT n`, which FTS5 - early-terminates off the doclist — **O(limit)**, corpus-independent. rowid is - *ingestion* order, so this ≈ recency only while events arrive in time order; - a relay that bulk-syncs history (NIP-77) diverges, which is why it is off by - default and left off for geode. Only the simple-search shape changes; tag∩ - search and the negentropy snapshot keep `created_at`. -- **Segment compaction.** `reindexAll` finishes with a full `'optimize'`, and - the periodic `SQLiteEventStore.optimize()` (geode's maintenance tick) folds - in a bounded `'merge'`, so incremental / deferred-catch-up inserts don't - leave the index as many small segments. + It drops the stored content copy (index shrinks) and `contentless_delete=1` + keeps the delete trigger working. +- **Segment compaction.** `reindexAll` finishes with `'optimize'`, and the + periodic `SQLiteEventStore.optimize()` (geode's maintenance tick) folds in a + bounded `'merge'`, so incremental / deferred-catch-up inserts don't leave the + index as many small segments. -Measured — `FtsSearchScalingBenchmark` (jvmTest prodbench), search a term in -~1% of events, `limit=50`, in-memory: +Delete cost — `FtsSearchScalingBenchmark.deleteByColumnVsByRowid`, 500 deletes, +in-memory: -| corpus | created_at, fragmented | created_at, optimized | **rowid, optimized** | -|---|---:|---:|---:| -| 50k | 1.92 ms | 1.16 ms | **0.26 ms** | -| 100k | 2.16 ms | 2.23 ms | **0.22 ms** | -| 200k | 4.33 ms | 4.26 ms | **0.22 ms** | +| rows | by column (old) | by rowid (new) | +|---|---:|---:| +| 2k | 91.8 ms | 6.6 ms | +| 8k | 362.0 ms | 4.7 ms | -- **rowid ordering is the corpus-independence lever**: flat ~0.22 ms vs - created_at's 1.16 → 4.26 ms (grows with the match set) — ~19× at 200k, and - the gap keeps widening. This is the direct answer to the search curve, for - deployments that accept ingestion-order results. -- **segment `optimize`** helps most at small sizes / right after bulk - incremental inserts (1.92 → 1.16 ms at 50k); FTS5 `automerge` already caps - fragmentation in steady state, so at 100k/200k here it is within noise. It is - a cheap, unconditional safety net (biggest for the deferred relay path, whose - catch-up commits in 1k batches), not the main lever. +By-column grows ~linearly with the table (O(n)/delete); by-rowid is flat — +~78× at 8k rows and widening. + +**Search is deliberately unchanged and still `created_at`-ordered.** NIP-01's +`limit` requires the newest events *by `created_at`*, and FTS5 only +early-terminates on its own rowid — so `MATCH … ORDER BY created_at DESC LIMIT +n` still materializes and sorts every match, and search latency still grows +with the match set (the report's 18× curve). An earlier draft added a +`searchOrderByRowId` flag that ordered by the FTS rowid to get O(limit) search; +that is *ingestion* order, which returns the wrong events under a limit once +ingestion diverges from `created_at` (any historical sync) — a NIP-01 +violation — so it was removed. `optimize` compacts the index but does not +change the asymptotics (measured within noise at 100k/200k). Corpus-independent +search is genuinely an external-engine job (the Vespa side of the report), not +this index. Migration (v4→v5): the old rowids can't be remapped, so `event_fts` is dropped and rebuilt — synchronous stores rebuild in the upgrade transaction (client @@ -118,6 +124,6 @@ reuse, and cap overflow → uncached fallback. whose asserted EXPLAIN output updated for the new join column (`event_fts.rowid`) and the contentless table's virtual-index marker (`0:M2`→`0:M1`). -- Defaults preserved: `searchOrderByRowId` off, so existing deployments keep - exact `created_at DESC` search ordering; the size + optimize wins are - unconditional. +- Search ordering unchanged: still exact `created_at DESC` (NIP-01 `limit` + semantics); the delete + size + optimize wins are unconditional and + spec-neutral. diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/FullTextSearchModule.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/FullTextSearchModule.kt index c913e1f9eb..e61f04e1ca 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/FullTextSearchModule.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/FullTextSearchModule.kt @@ -35,17 +35,26 @@ import com.vitorpamplona.quartz.utils.EventFactory * `contentless_delete=1`) whose `rowid` is the event's * `event_headers.row_id`. Two consequences: * + * - **Deletes.** The `fts_foreign_key` trigger deletes by `rowid` (an FTS5 + * primary-key seek, O(log n)). The old `fts5(event_header_row_id, + * content)` schema deleted by a *regular* column, which FTS5 cannot seek — + * it scans the whole index per delete (O(n)), so deletion throughput + * degraded with corpus size. Every event removal fires this trigger + * (replaceable rotation, kind-5, expiration, right-to-vanish), so the + * seek matters. Measured `Fts5CapabilityProbe`/`FtsSearchScalingBenchmark`: + * ~78× at 8k rows and widening with the table. * - **Size.** A contentless table stores only the inverted index, not a - * second copy of the tokenized text (the old `fts5(event_header_row_id, - * content)` schema kept the content verbatim). The indexed text is - * *derived* ([SearchableEvent.indexableContent], not any raw column), so - * FTS5 external-content — which reads the source column from the base - * table — cannot express it; contentless is the correct primitive. - * - **Ordering.** With `rowid = event_headers.row_id`, - * `ORDER BY event_fts.rowid DESC LIMIT n` early-terminates off the FTS - * doclist ([IndexingStrategy.searchOrderByRowId]); the default - * `created_at DESC` ordering still works via the join back to - * `event_headers` but must sort all matches. + * second copy of the tokenized text. The indexed text is *derived* + * ([SearchableEvent.indexableContent], not any raw column), so FTS5 + * external-content — which reads the source column from the base table — + * cannot express it; contentless is the correct primitive. + * + * Search still orders by `event_headers.created_at DESC` (NIP-01 `limit` + * semantics: the newest events *by created_at*), joining back to + * `event_headers` on `row_id = event_fts.rowid`. FTS5 cannot early-terminate + * that — it materializes and sorts all matches — so search latency still + * grows with the match set; corpus-independent search needs an external + * engine, not this index. * * When [enabled] is `false` the module becomes an inert no-op: no * `event_fts` virtual table and no `fts_foreign_key` delete trigger are diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt index ea76cf181d..01362e03fb 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt @@ -131,32 +131,6 @@ interface IndexingStrategy { */ val indexFullTextSearch: Boolean - /** - * Order NIP-50 search results by the FTS index rowid (= event row_id, - * i.e. **ingestion order**) instead of `created_at DESC`. - * - * The `event_fts` rowid is the event's `event_headers.row_id`, so - * `ORDER BY event_fts.rowid DESC LIMIT n` walks the FTS doclist - * newest-rowid-first and stops at the limit — **O(limit)** — where the - * default `ORDER BY created_at DESC` must materialize *every* document - * matching the term and sort it (cost grows with the whole corpus; the - * NIP-50 curve that falls ~18× from 25k→400k events). The unconditional - * contentless-index + segment-merge changes already shrink and speed the - * default path; this flag is the one that makes search cost independent - * of corpus size. - * - * The trade-off is semantic: ingestion order ≈ `created_at` order only - * while events arrive roughly in time order. A relay that bulk-syncs - * historical events (NIP-77) ingests old events late, so "newest rowid" - * and "newest created_at" diverge during/after a sync. Leave it **off** - * (the default) where strict recency ordering matters; turn it **on** - * for stores that want corpus-independent search latency and accept - * recently-ingested-first ordering. Only affects the simple-search query - * shape (`search [+ kinds/authors] + limit`); tag∩search and the - * negentropy snapshot path keep `created_at` ordering. - */ - val searchOrderByRowId: Boolean get() = false - /** * Maintain an always-current in-memory `(created_at, id)` set (a * [com.vitorpamplona.quartz.nip77Negentropy.LiveNegentropyIndex]) so @@ -186,7 +160,6 @@ class DefaultIndexingStrategy( override val useAndIndexIdOnOrderBy: Boolean = false, override val indexFullTextSearch: Boolean = true, override val deferFullTextSearchIndexing: Boolean = false, - override val searchOrderByRowId: Boolean = false, override val maintainLiveNegentropyIndex: Boolean = false, ) : IndexingStrategy { override fun shouldIndex( diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryBuilder.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryBuilder.kt index 5af41c7684..8ad16c5d4a 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryBuilder.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryBuilder.kt @@ -1005,18 +1005,9 @@ class QueryBuilder( if (clause.conditions.isNotEmpty()) { append("\nWHERE ${clause.conditions}") } - if (indexStrategy.searchOrderByRowId) { - // The FTS rowid is event_headers.row_id (ingestion order). - // Ordering by it lets FTS5 walk the doclist newest-first - // and stop at LIMIT — O(limit) — instead of materializing - // and sorting every match by created_at. See - // IndexingStrategy.searchOrderByRowId for the trade-off. - append("\nORDER BY ${fts.tableName}.rowid DESC") - } else { - append("\nORDER BY event_headers.created_at DESC") - if (indexStrategy.useAndIndexIdOnOrderBy) { - append(", event_headers.id ASC") - } + append("\nORDER BY event_headers.created_at DESC") + if (indexStrategy.useAndIndexIdOnOrderBy) { + append(", event_headers.id ASC") } if (limit != null) { append("\nLIMIT ") diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SearchOrderByRowIdTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SearchOrderByRowIdTest.kt deleted file mode 100644 index 104630782c..0000000000 --- a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SearchOrderByRowIdTest.kt +++ /dev/null @@ -1,109 +0,0 @@ -/* - * Copyright (c) 2025 Vitor Pamplona - * - * Permission is hereby granted, free of charge, to any person obtaining a copy of - * this software and associated documentation files (the "Software"), to deal in - * the Software without restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the - * Software, and to permit persons to whom the Software is furnished to do so, - * subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in all - * copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS - * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR - * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN - * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION - * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - */ -package com.vitorpamplona.quartz.nip01Core.store.sqlite - -import com.vitorpamplona.quartz.nip01Core.core.Event -import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter -import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync -import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent -import kotlinx.coroutines.runBlocking -import kotlin.test.Test -import kotlin.test.assertEquals - -/** - * Contrast the two NIP-50 result orderings the contentless FTS index enables. - * The FTS rowid is `event_headers.row_id` (ingestion order), so: - * - default: `ORDER BY created_at DESC` — strict recency. - * - [IndexingStrategy.searchOrderByRowId]: `ORDER BY event_fts.rowid DESC` — - * ingestion order (early-terminating, corpus-independent). - * - * Events are inserted in an order that deliberately disagrees with their - * `created_at`, so the two orderings produce visibly different results. - */ -class SearchOrderByRowIdTest { - private val signer = NostrSignerSync() - - private fun note(createdAt: Long) = signer.sign(TextNoteEvent.build("uniqorder token body", createdAt = createdAt)) - - private fun store(rowIdOrder: Boolean) = - EventStore( - dbName = null, - indexStrategy = DefaultIndexingStrategy(searchOrderByRowId = rowIdOrder), - ) - - @Test - fun defaultOrdersByCreatedAt_rowIdFlagOrdersByIngestion() = - runBlocking { - // created_at: A newest, B oldest, C middle. Inserted A, B, C — so - // ingestion (row_id) order is A < B < C. - val byCreatedAt = store(rowIdOrder = false) - val byRowId = store(rowIdOrder = true) - try { - val a = note(300) - val b = note(100) - val c = note(200) - for (store in listOf(byCreatedAt, byRowId)) { - store.insert(a) - store.insert(b) - store.insert(c) - } - - val filter = Filter(search = "uniqorder", limit = 10) - - // created_at DESC: A(300), C(200), B(100) - assertEquals( - listOf(a.id, c.id, b.id), - byCreatedAt.query(filter).map { it.id }, - "default search must order by created_at DESC", - ) - - // rowid DESC = ingestion order reversed: C, B, A - assertEquals( - listOf(c.id, b.id, a.id), - byRowId.query(filter).map { it.id }, - "searchOrderByRowId must order by ingestion (row_id) DESC", - ) - } finally { - byCreatedAt.close() - byRowId.close() - } - } - - @Test - fun rowIdOrderStillHonorsLimitAndSecondaryFilters() = - runBlocking { - val store = store(rowIdOrder = true) - try { - val notes = (0 until 6).map { note(1_700_000_000L + it) } - notes.forEach { store.insert(it) } - - // limit slices the newest-ingested 3. - val top3 = store.query(Filter(search = "uniqorder", limit = 3)).map { it.id } - assertEquals(notes.takeLast(3).reversed().map { it.id }, top3) - - // kind filter still applies alongside rowid ordering. - val wrongKind = store.query(Filter(kinds = listOf(30023), search = "uniqorder", limit = 10)) - assertEquals(0, wrongKind.size) - } finally { - store.close() - } - } -} diff --git a/quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FtsSearchScalingBenchmark.kt b/quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FtsSearchScalingBenchmark.kt index e8398925a3..8a97974d04 100644 --- a/quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FtsSearchScalingBenchmark.kt +++ b/quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FtsSearchScalingBenchmark.kt @@ -20,6 +20,8 @@ */ package com.vitorpamplona.quartz.nip01Core.relay.prodbench +import androidx.sqlite.SQLiteConnection +import androidx.sqlite.driver.bundled.BundledSQLiteDriver import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter import com.vitorpamplona.quartz.nip01Core.store.sqlite.DefaultIndexingStrategy @@ -29,23 +31,24 @@ import kotlinx.coroutines.runBlocking import kotlin.test.Test /** - * Isolates the NIP-50 search-scaling curve the scale-curve report flagged - * (FTS5 falling ~18× from 25k→400k events) and the two levers against it: + * Two measurements around the contentless FTS index, motivating the v4→v5 + * schema change and being honest about what it does and does not fix. * - * 1. **segment compaction** (`INSERT INTO event_fts(event_fts) VALUES - * ('optimize')`). Incremental inserts leave the index as many small - * segments and a MATCH queries every one; measured fragmented vs - * optimized. - * 2. **rowid ordering** ([DefaultIndexingStrategy.searchOrderByRowId]). - * `ORDER BY created_at DESC` must materialize + sort *every* document - * matching the term (cost grows with the corpus); `ORDER BY - * event_fts.rowid DESC LIMIT n` early-terminates — O(limit). + * 1. **Delete scaling — the win.** The old `fts5(event_header_row_id, + * content)` schema had the `fts_foreign_key` trigger delete by a regular + * FTS column, which FTS5 cannot seek (it scans, O(n) per delete). The + * contentless schema keys deletes off the rowid (= event_headers.row_id), + * an O(log n) primary-key seek. Every event removal fires this trigger + * (replaceable rotation, kind-5, expiration, right-to-vanish). + * 2. **Search scaling — the limit.** `MATCH … ORDER BY created_at DESC + * LIMIT n` must materialize + sort *every* matching document (FTS5 only + * early-terminates on its own rowid, and NIP-01's `limit` requires newest + * *by created_at*), so search cost grows with the match set. Segment + * `optimize` compacts the index but does not change that; corpus- + * independent search needs an external engine. Shown fragmented vs + * optimized to size the (secondary) compaction effect. * - * The seed injects a common term into ~1% of events, so the match set — and - * thus the created_at sort — grows with the corpus while the limit stays 50. - * - * Size with `-DftsBenchScale=N` (default 1). Not an assertion test; run - * explicitly and read stdout. + * Size search with `-DftsBenchScale=N` (default 1). Not an assertion test. */ class FtsSearchScalingBenchmark { companion object { @@ -95,42 +98,77 @@ class FtsSearchScalingBenchmark { return events } + private inline fun timeMs(block: () -> Unit): Double { + val start = System.nanoTime() + block() + return (System.nanoTime() - start) / 1e6 + } + + private fun SQLiteConnection.exec(sql: String) = prepare(sql).use { it.step() } + + @Test + fun deleteByColumnVsByRowid() { + // Old-schema (delete by FTS column) vs contentless (delete by rowid), + // 500 deletes at two table sizes. By-column should grow with the + // table; by-rowid should stay flat. + println("─ FtsSearchScalingBenchmark.delete (500 deletes) ─") + println(" %-9s %14s %14s".format("rows", "byColumn", "byRowid")) + for (n in listOf(2_000 * SCALE, 8_000 * SCALE)) { + val db = BundledSQLiteDriver().open(":memory:") + try { + db.exec("CREATE VIRTUAL TABLE col USING fts5(event_header_row_id, content)") + db.exec("CREATE VIRTUAL TABLE row USING fts5(content, content='', contentless_delete=1)") + for (i in 1..n) { + db.prepare("INSERT INTO col(event_header_row_id, content) VALUES (?, 'alpha beta gamma')").use { + it.bindLong(1, i.toLong()) + it.step() + } + db.prepare("INSERT INTO row(rowid, content) VALUES (?, 'alpha beta gamma')").use { + it.bindLong(1, i.toLong()) + it.step() + } + } + val byColumn = + timeMs { + for (i in 1..500) { + db.prepare("DELETE FROM col WHERE event_header_row_id = ?").use { + it.bindLong(1, i.toLong()) + it.step() + } + } + } + val byRowid = + timeMs { + for (i in 1..500) { + db.prepare("DELETE FROM row WHERE rowid = ?").use { + it.bindLong(1, i.toLong()) + it.step() + } + } + } + println(" %-9s %11.2f ms %11.2f ms".format("${n / 1000}k", byColumn, byRowid)) + } finally { + db.close() + } + } + } + @Test fun searchScaling() = runBlocking { - println("─ FtsSearchScalingBenchmark (scale=$SCALE, limit=50) ─") - println(" %-9s %14s %14s %14s".format("corpus", "createdAt/frag", "createdAt/opt", "rowid/opt")) + println("─ FtsSearchScalingBenchmark.search (created_at DESC, limit=50) ─") + println(" %-9s %16s %16s".format("corpus", "fragmented", "optimized")) for (size in SIZES) { - val events = seed(size) - - val createdAt = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy()) - val rowId = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy(searchOrderByRowId = true)) + val store = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy()) try { - // Insert in 10k chunks → many FTS segments (fragmented). - events.chunked(10_000).forEach { - createdAt.batchInsert(it) - rowId.batchInsert(it) - } - + seed(size).chunked(10_000).forEach { store.batchInsert(it) } val f = Filter(search = NEEDLE, limit = 50) - val frag = time(createdAt, f) - - createdAt.store.reindexFullTextSearch() // rebuild + optimize() - rowId.store.reindexFullTextSearch() - val optCreatedAt = time(createdAt, f) - val optRowId = time(rowId, f) - - println( - " %-9s %12.2f ms %12.2f ms %12.2f ms".format( - if (size >= 1000) "${size / 1000}k" else "$size", - frag, - optCreatedAt, - optRowId, - ), - ) + val frag = time(store, f) + store.store.reindexFullTextSearch() // rebuild + optimize() + val opt = time(store, f) + println(" %-9s %13.2f ms %13.2f ms".format("${size / 1000}k", frag, opt)) } finally { - createdAt.close() - rowId.close() + store.close() } } }