mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-09 08:04:45 +00:00
fix(store): drop non-compliant rowid search ordering, keep FTS delete/size wins
searchOrderByRowId ordered NIP-50 search by the FTS rowid (ingestion order) to get O(limit) search, but NIP-01's limit requires the newest events by created_at. Once ingestion diverges from created_at (any historical sync) that returns the wrong events under a limit — a spec violation — so the flag, its QueryBuilder branch, and its test are removed. Search stays created_at-ordered; corpus-independent search is an external-engine job, not this index. The contentless + rowid=row_id schema stays for the reasons that don't touch ordering: the delete trigger now seeks by rowid (O(log n)) instead of scanning by an FTS column (O(n)) — measured ~78× faster at 8k rows and widening, on a path every deletion hits — and the index is smaller. Benchmark reframed around the delete win and the honest (unchanged) search cost; plan doc updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqGuBuSUsRudGerPqDuoKA
This commit is contained in:
Claude
@@ -7,63 +7,69 @@ ingest 16×, author-timeline 13×, follow-feed 2.4× — while point reads staye
|
||||
flat. Two of those (author-timeline, follow-feed) were already addressed
|
||||
(the pubkey-alone index and `MergeQueryExecutor`) and mostly reflect running
|
||||
the report under the *client* `DefaultIndexingStrategy` rather than
|
||||
`relayIndexingStrategy()`. This change takes the two structural read shapes
|
||||
that were still corpus-bound: **NIP-50 search** and the **large-IN tag
|
||||
watcher**, plus a statement-cache fix the merge paths needed.
|
||||
`relayIndexingStrategy()`. This change adds the **large-IN tag watcher** to the
|
||||
merge executor, fixes the **FTS delete path** (which degraded with corpus
|
||||
size) and shrinks the FTS index, and fixes a **statement-cache** miss the merge
|
||||
paths hit.
|
||||
|
||||
It does **not** fix NIP-50 *search* latency: that is bounded by the
|
||||
`created_at`-ordered sort over all matches, which FTS5 can't early-terminate
|
||||
while staying NIP-01-compliant (see §1). Corpus-independent search is an
|
||||
external-engine job.
|
||||
|
||||
Everything here is read/size work; the write path and on-disk index set are
|
||||
unchanged except the FTS table, which gets *smaller*.
|
||||
unchanged except the FTS table, which gets *smaller* and deletes faster.
|
||||
|
||||
## 1. NIP-50 search — contentless FTS5 + optional rowid ordering + segment merge
|
||||
## 1. FTS — contentless index (fast deletes + smaller), NOT a search-scaling fix
|
||||
|
||||
The old index was `fts5(event_header_row_id, content)`: it stored a second
|
||||
copy of the tokenized text, and its rowid was auto-assigned (unrelated to the
|
||||
event). Search ran `… event_fts MATCH ? ORDER BY event_headers.created_at DESC
|
||||
LIMIT n`, which must materialize **every** document matching the term and sort
|
||||
it — cost grows with the whole corpus. That is the 18× curve.
|
||||
copy of the tokenized text, its rowid was auto-assigned (unrelated to the
|
||||
event), and — the real problem — the `fts_foreign_key` delete trigger deleted
|
||||
by the `event_header_row_id` *column*, which FTS5 cannot seek, so it **scanned
|
||||
the whole index per delete**.
|
||||
|
||||
Three changes (`FullTextSearchModule`, `IndexingStrategy`, `QueryBuilder`,
|
||||
DB version 4→5):
|
||||
Changes (`FullTextSearchModule`, `QueryBuilder`, DB version 4→5):
|
||||
|
||||
- **rowid = `event_headers.row_id`.** Set explicitly on every insert; the
|
||||
delete trigger, reindex, and catch-up paths key off it. Deletes become an
|
||||
O(log n) primary-key seek instead of an O(n) column scan. This is the win:
|
||||
every event removal fires the trigger — replaceable rotation, kind-5,
|
||||
expiration, right-to-vanish — so on the old schema deletion throughput
|
||||
degraded with corpus size.
|
||||
- **Contentless** (`fts5(content, content='', contentless_delete=1)`). The
|
||||
indexed text is *derived* (`SearchableEvent.indexableContent()`, not a raw
|
||||
column), so FTS5 **external-content** — which re-reads the source column from
|
||||
the base table — cannot express it; **contentless** is the correct primitive.
|
||||
It drops the stored content copy (index shrinks) and, with
|
||||
`contentless_delete=1`, still supports the `fts_foreign_key` delete trigger.
|
||||
- **rowid = `event_headers.row_id`.** Set explicitly on every insert. The
|
||||
delete trigger and reindex/catch-up paths all key off it, so the join is
|
||||
`event_headers.row_id = event_fts.rowid` with no stored linkage column.
|
||||
- **`searchOrderByRowId`** (new `IndexingStrategy` flag, **default off**). When
|
||||
on, simple search orders by `event_fts.rowid DESC LIMIT n`, which FTS5
|
||||
early-terminates off the doclist — **O(limit)**, corpus-independent. rowid is
|
||||
*ingestion* order, so this ≈ recency only while events arrive in time order;
|
||||
a relay that bulk-syncs history (NIP-77) diverges, which is why it is off by
|
||||
default and left off for geode. Only the simple-search shape changes; tag∩
|
||||
search and the negentropy snapshot keep `created_at`.
|
||||
- **Segment compaction.** `reindexAll` finishes with a full `'optimize'`, and
|
||||
the periodic `SQLiteEventStore.optimize()` (geode's maintenance tick) folds
|
||||
in a bounded `'merge'`, so incremental / deferred-catch-up inserts don't
|
||||
leave the index as many small segments.
|
||||
It drops the stored content copy (index shrinks) and `contentless_delete=1`
|
||||
keeps the delete trigger working.
|
||||
- **Segment compaction.** `reindexAll` finishes with `'optimize'`, and the
|
||||
periodic `SQLiteEventStore.optimize()` (geode's maintenance tick) folds in a
|
||||
bounded `'merge'`, so incremental / deferred-catch-up inserts don't leave the
|
||||
index as many small segments.
|
||||
|
||||
Measured — `FtsSearchScalingBenchmark` (jvmTest prodbench), search a term in
|
||||
~1% of events, `limit=50`, in-memory:
|
||||
Delete cost — `FtsSearchScalingBenchmark.deleteByColumnVsByRowid`, 500 deletes,
|
||||
in-memory:
|
||||
|
||||
| corpus | created_at, fragmented | created_at, optimized | **rowid, optimized** |
|
||||
|---|---:|---:|---:|
|
||||
| 50k | 1.92 ms | 1.16 ms | **0.26 ms** |
|
||||
| 100k | 2.16 ms | 2.23 ms | **0.22 ms** |
|
||||
| 200k | 4.33 ms | 4.26 ms | **0.22 ms** |
|
||||
| rows | by column (old) | by rowid (new) |
|
||||
|---|---:|---:|
|
||||
| 2k | 91.8 ms | 6.6 ms |
|
||||
| 8k | 362.0 ms | 4.7 ms |
|
||||
|
||||
- **rowid ordering is the corpus-independence lever**: flat ~0.22 ms vs
|
||||
created_at's 1.16 → 4.26 ms (grows with the match set) — ~19× at 200k, and
|
||||
the gap keeps widening. This is the direct answer to the search curve, for
|
||||
deployments that accept ingestion-order results.
|
||||
- **segment `optimize`** helps most at small sizes / right after bulk
|
||||
incremental inserts (1.92 → 1.16 ms at 50k); FTS5 `automerge` already caps
|
||||
fragmentation in steady state, so at 100k/200k here it is within noise. It is
|
||||
a cheap, unconditional safety net (biggest for the deferred relay path, whose
|
||||
catch-up commits in 1k batches), not the main lever.
|
||||
By-column grows ~linearly with the table (O(n)/delete); by-rowid is flat —
|
||||
~78× at 8k rows and widening.
|
||||
|
||||
**Search is deliberately unchanged and still `created_at`-ordered.** NIP-01's
|
||||
`limit` requires the newest events *by `created_at`*, and FTS5 only
|
||||
early-terminates on its own rowid — so `MATCH … ORDER BY created_at DESC LIMIT
|
||||
n` still materializes and sorts every match, and search latency still grows
|
||||
with the match set (the report's 18× curve). An earlier draft added a
|
||||
`searchOrderByRowId` flag that ordered by the FTS rowid to get O(limit) search;
|
||||
that is *ingestion* order, which returns the wrong events under a limit once
|
||||
ingestion diverges from `created_at` (any historical sync) — a NIP-01
|
||||
violation — so it was removed. `optimize` compacts the index but does not
|
||||
change the asymptotics (measured within noise at 100k/200k). Corpus-independent
|
||||
search is genuinely an external-engine job (the Vespa side of the report), not
|
||||
this index.
|
||||
|
||||
Migration (v4→v5): the old rowids can't be remapped, so `event_fts` is dropped
|
||||
and rebuilt — synchronous stores rebuild in the upgrade transaction (client
|
||||
@@ -118,6 +124,6 @@ reuse, and cap overflow → uncached fallback.
|
||||
whose asserted EXPLAIN output updated for the new join column
|
||||
(`event_fts.rowid`) and the contentless table's virtual-index marker
|
||||
(`0:M2`→`0:M1`).
|
||||
- Defaults preserved: `searchOrderByRowId` off, so existing deployments keep
|
||||
exact `created_at DESC` search ordering; the size + optimize wins are
|
||||
unconditional.
|
||||
- Search ordering unchanged: still exact `created_at DESC` (NIP-01 `limit`
|
||||
semantics); the delete + size + optimize wins are unconditional and
|
||||
spec-neutral.
|
||||
|
||||
+19
-10
@@ -35,17 +35,26 @@ import com.vitorpamplona.quartz.utils.EventFactory
|
||||
* `contentless_delete=1`) whose `rowid` is the event's
|
||||
* `event_headers.row_id`. Two consequences:
|
||||
*
|
||||
* - **Deletes.** The `fts_foreign_key` trigger deletes by `rowid` (an FTS5
|
||||
* primary-key seek, O(log n)). The old `fts5(event_header_row_id,
|
||||
* content)` schema deleted by a *regular* column, which FTS5 cannot seek —
|
||||
* it scans the whole index per delete (O(n)), so deletion throughput
|
||||
* degraded with corpus size. Every event removal fires this trigger
|
||||
* (replaceable rotation, kind-5, expiration, right-to-vanish), so the
|
||||
* seek matters. Measured `Fts5CapabilityProbe`/`FtsSearchScalingBenchmark`:
|
||||
* ~78× at 8k rows and widening with the table.
|
||||
* - **Size.** A contentless table stores only the inverted index, not a
|
||||
* second copy of the tokenized text (the old `fts5(event_header_row_id,
|
||||
* content)` schema kept the content verbatim). The indexed text is
|
||||
* *derived* ([SearchableEvent.indexableContent], not any raw column), so
|
||||
* FTS5 external-content — which reads the source column from the base
|
||||
* table — cannot express it; contentless is the correct primitive.
|
||||
* - **Ordering.** With `rowid = event_headers.row_id`,
|
||||
* `ORDER BY event_fts.rowid DESC LIMIT n` early-terminates off the FTS
|
||||
* doclist ([IndexingStrategy.searchOrderByRowId]); the default
|
||||
* `created_at DESC` ordering still works via the join back to
|
||||
* `event_headers` but must sort all matches.
|
||||
* second copy of the tokenized text. The indexed text is *derived*
|
||||
* ([SearchableEvent.indexableContent], not any raw column), so FTS5
|
||||
* external-content — which reads the source column from the base table —
|
||||
* cannot express it; contentless is the correct primitive.
|
||||
*
|
||||
* Search still orders by `event_headers.created_at DESC` (NIP-01 `limit`
|
||||
* semantics: the newest events *by created_at*), joining back to
|
||||
* `event_headers` on `row_id = event_fts.rowid`. FTS5 cannot early-terminate
|
||||
* that — it materializes and sorts all matches — so search latency still
|
||||
* grows with the match set; corpus-independent search needs an external
|
||||
* engine, not this index.
|
||||
*
|
||||
* When [enabled] is `false` the module becomes an inert no-op: no
|
||||
* `event_fts` virtual table and no `fts_foreign_key` delete trigger are
|
||||
|
||||
-27
@@ -131,32 +131,6 @@ interface IndexingStrategy {
|
||||
*/
|
||||
val indexFullTextSearch: Boolean
|
||||
|
||||
/**
|
||||
* Order NIP-50 search results by the FTS index rowid (= event row_id,
|
||||
* i.e. **ingestion order**) instead of `created_at DESC`.
|
||||
*
|
||||
* The `event_fts` rowid is the event's `event_headers.row_id`, so
|
||||
* `ORDER BY event_fts.rowid DESC LIMIT n` walks the FTS doclist
|
||||
* newest-rowid-first and stops at the limit — **O(limit)** — where the
|
||||
* default `ORDER BY created_at DESC` must materialize *every* document
|
||||
* matching the term and sort it (cost grows with the whole corpus; the
|
||||
* NIP-50 curve that falls ~18× from 25k→400k events). The unconditional
|
||||
* contentless-index + segment-merge changes already shrink and speed the
|
||||
* default path; this flag is the one that makes search cost independent
|
||||
* of corpus size.
|
||||
*
|
||||
* The trade-off is semantic: ingestion order ≈ `created_at` order only
|
||||
* while events arrive roughly in time order. A relay that bulk-syncs
|
||||
* historical events (NIP-77) ingests old events late, so "newest rowid"
|
||||
* and "newest created_at" diverge during/after a sync. Leave it **off**
|
||||
* (the default) where strict recency ordering matters; turn it **on**
|
||||
* for stores that want corpus-independent search latency and accept
|
||||
* recently-ingested-first ordering. Only affects the simple-search query
|
||||
* shape (`search [+ kinds/authors] + limit`); tag∩search and the
|
||||
* negentropy snapshot path keep `created_at` ordering.
|
||||
*/
|
||||
val searchOrderByRowId: Boolean get() = false
|
||||
|
||||
/**
|
||||
* Maintain an always-current in-memory `(created_at, id)` set (a
|
||||
* [com.vitorpamplona.quartz.nip77Negentropy.LiveNegentropyIndex]) so
|
||||
@@ -186,7 +160,6 @@ class DefaultIndexingStrategy(
|
||||
override val useAndIndexIdOnOrderBy: Boolean = false,
|
||||
override val indexFullTextSearch: Boolean = true,
|
||||
override val deferFullTextSearchIndexing: Boolean = false,
|
||||
override val searchOrderByRowId: Boolean = false,
|
||||
override val maintainLiveNegentropyIndex: Boolean = false,
|
||||
) : IndexingStrategy {
|
||||
override fun shouldIndex(
|
||||
|
||||
+3
-12
@@ -1005,18 +1005,9 @@ class QueryBuilder(
|
||||
if (clause.conditions.isNotEmpty()) {
|
||||
append("\nWHERE ${clause.conditions}")
|
||||
}
|
||||
if (indexStrategy.searchOrderByRowId) {
|
||||
// The FTS rowid is event_headers.row_id (ingestion order).
|
||||
// Ordering by it lets FTS5 walk the doclist newest-first
|
||||
// and stop at LIMIT — O(limit) — instead of materializing
|
||||
// and sorting every match by created_at. See
|
||||
// IndexingStrategy.searchOrderByRowId for the trade-off.
|
||||
append("\nORDER BY ${fts.tableName}.rowid DESC")
|
||||
} else {
|
||||
append("\nORDER BY event_headers.created_at DESC")
|
||||
if (indexStrategy.useAndIndexIdOnOrderBy) {
|
||||
append(", event_headers.id ASC")
|
||||
}
|
||||
append("\nORDER BY event_headers.created_at DESC")
|
||||
if (indexStrategy.useAndIndexIdOnOrderBy) {
|
||||
append(", event_headers.id ASC")
|
||||
}
|
||||
if (limit != null) {
|
||||
append("\nLIMIT ")
|
||||
|
||||
-109
@@ -1,109 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.store.sqlite
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
|
||||
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
|
||||
/**
|
||||
* Contrast the two NIP-50 result orderings the contentless FTS index enables.
|
||||
* The FTS rowid is `event_headers.row_id` (ingestion order), so:
|
||||
* - default: `ORDER BY created_at DESC` — strict recency.
|
||||
* - [IndexingStrategy.searchOrderByRowId]: `ORDER BY event_fts.rowid DESC` —
|
||||
* ingestion order (early-terminating, corpus-independent).
|
||||
*
|
||||
* Events are inserted in an order that deliberately disagrees with their
|
||||
* `created_at`, so the two orderings produce visibly different results.
|
||||
*/
|
||||
class SearchOrderByRowIdTest {
|
||||
private val signer = NostrSignerSync()
|
||||
|
||||
private fun note(createdAt: Long) = signer.sign(TextNoteEvent.build("uniqorder token body", createdAt = createdAt))
|
||||
|
||||
private fun store(rowIdOrder: Boolean) =
|
||||
EventStore(
|
||||
dbName = null,
|
||||
indexStrategy = DefaultIndexingStrategy(searchOrderByRowId = rowIdOrder),
|
||||
)
|
||||
|
||||
@Test
|
||||
fun defaultOrdersByCreatedAt_rowIdFlagOrdersByIngestion() =
|
||||
runBlocking {
|
||||
// created_at: A newest, B oldest, C middle. Inserted A, B, C — so
|
||||
// ingestion (row_id) order is A < B < C.
|
||||
val byCreatedAt = store(rowIdOrder = false)
|
||||
val byRowId = store(rowIdOrder = true)
|
||||
try {
|
||||
val a = note(300)
|
||||
val b = note(100)
|
||||
val c = note(200)
|
||||
for (store in listOf(byCreatedAt, byRowId)) {
|
||||
store.insert(a)
|
||||
store.insert(b)
|
||||
store.insert(c)
|
||||
}
|
||||
|
||||
val filter = Filter(search = "uniqorder", limit = 10)
|
||||
|
||||
// created_at DESC: A(300), C(200), B(100)
|
||||
assertEquals(
|
||||
listOf(a.id, c.id, b.id),
|
||||
byCreatedAt.query<Event>(filter).map { it.id },
|
||||
"default search must order by created_at DESC",
|
||||
)
|
||||
|
||||
// rowid DESC = ingestion order reversed: C, B, A
|
||||
assertEquals(
|
||||
listOf(c.id, b.id, a.id),
|
||||
byRowId.query<Event>(filter).map { it.id },
|
||||
"searchOrderByRowId must order by ingestion (row_id) DESC",
|
||||
)
|
||||
} finally {
|
||||
byCreatedAt.close()
|
||||
byRowId.close()
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun rowIdOrderStillHonorsLimitAndSecondaryFilters() =
|
||||
runBlocking {
|
||||
val store = store(rowIdOrder = true)
|
||||
try {
|
||||
val notes = (0 until 6).map { note(1_700_000_000L + it) }
|
||||
notes.forEach { store.insert(it) }
|
||||
|
||||
// limit slices the newest-ingested 3.
|
||||
val top3 = store.query<Event>(Filter(search = "uniqorder", limit = 3)).map { it.id }
|
||||
assertEquals(notes.takeLast(3).reversed().map { it.id }, top3)
|
||||
|
||||
// kind filter still applies alongside rowid ordering.
|
||||
val wrongKind = store.query<Event>(Filter(kinds = listOf(30023), search = "uniqorder", limit = 10))
|
||||
assertEquals(0, wrongKind.size)
|
||||
} finally {
|
||||
store.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
+82
-44
@@ -20,6 +20,8 @@
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
|
||||
|
||||
import androidx.sqlite.SQLiteConnection
|
||||
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.store.sqlite.DefaultIndexingStrategy
|
||||
@@ -29,23 +31,24 @@ import kotlinx.coroutines.runBlocking
|
||||
import kotlin.test.Test
|
||||
|
||||
/**
|
||||
* Isolates the NIP-50 search-scaling curve the scale-curve report flagged
|
||||
* (FTS5 falling ~18× from 25k→400k events) and the two levers against it:
|
||||
* Two measurements around the contentless FTS index, motivating the v4→v5
|
||||
* schema change and being honest about what it does and does not fix.
|
||||
*
|
||||
* 1. **segment compaction** (`INSERT INTO event_fts(event_fts) VALUES
|
||||
* ('optimize')`). Incremental inserts leave the index as many small
|
||||
* segments and a MATCH queries every one; measured fragmented vs
|
||||
* optimized.
|
||||
* 2. **rowid ordering** ([DefaultIndexingStrategy.searchOrderByRowId]).
|
||||
* `ORDER BY created_at DESC` must materialize + sort *every* document
|
||||
* matching the term (cost grows with the corpus); `ORDER BY
|
||||
* event_fts.rowid DESC LIMIT n` early-terminates — O(limit).
|
||||
* 1. **Delete scaling — the win.** The old `fts5(event_header_row_id,
|
||||
* content)` schema had the `fts_foreign_key` trigger delete by a regular
|
||||
* FTS column, which FTS5 cannot seek (it scans, O(n) per delete). The
|
||||
* contentless schema keys deletes off the rowid (= event_headers.row_id),
|
||||
* an O(log n) primary-key seek. Every event removal fires this trigger
|
||||
* (replaceable rotation, kind-5, expiration, right-to-vanish).
|
||||
* 2. **Search scaling — the limit.** `MATCH … ORDER BY created_at DESC
|
||||
* LIMIT n` must materialize + sort *every* matching document (FTS5 only
|
||||
* early-terminates on its own rowid, and NIP-01's `limit` requires newest
|
||||
* *by created_at*), so search cost grows with the match set. Segment
|
||||
* `optimize` compacts the index but does not change that; corpus-
|
||||
* independent search needs an external engine. Shown fragmented vs
|
||||
* optimized to size the (secondary) compaction effect.
|
||||
*
|
||||
* The seed injects a common term into ~1% of events, so the match set — and
|
||||
* thus the created_at sort — grows with the corpus while the limit stays 50.
|
||||
*
|
||||
* Size with `-DftsBenchScale=N` (default 1). Not an assertion test; run
|
||||
* explicitly and read stdout.
|
||||
* Size search with `-DftsBenchScale=N` (default 1). Not an assertion test.
|
||||
*/
|
||||
class FtsSearchScalingBenchmark {
|
||||
companion object {
|
||||
@@ -95,42 +98,77 @@ class FtsSearchScalingBenchmark {
|
||||
return events
|
||||
}
|
||||
|
||||
private inline fun timeMs(block: () -> Unit): Double {
|
||||
val start = System.nanoTime()
|
||||
block()
|
||||
return (System.nanoTime() - start) / 1e6
|
||||
}
|
||||
|
||||
private fun SQLiteConnection.exec(sql: String) = prepare(sql).use { it.step() }
|
||||
|
||||
@Test
|
||||
fun deleteByColumnVsByRowid() {
|
||||
// Old-schema (delete by FTS column) vs contentless (delete by rowid),
|
||||
// 500 deletes at two table sizes. By-column should grow with the
|
||||
// table; by-rowid should stay flat.
|
||||
println("─ FtsSearchScalingBenchmark.delete (500 deletes) ─")
|
||||
println(" %-9s %14s %14s".format("rows", "byColumn", "byRowid"))
|
||||
for (n in listOf(2_000 * SCALE, 8_000 * SCALE)) {
|
||||
val db = BundledSQLiteDriver().open(":memory:")
|
||||
try {
|
||||
db.exec("CREATE VIRTUAL TABLE col USING fts5(event_header_row_id, content)")
|
||||
db.exec("CREATE VIRTUAL TABLE row USING fts5(content, content='', contentless_delete=1)")
|
||||
for (i in 1..n) {
|
||||
db.prepare("INSERT INTO col(event_header_row_id, content) VALUES (?, 'alpha beta gamma')").use {
|
||||
it.bindLong(1, i.toLong())
|
||||
it.step()
|
||||
}
|
||||
db.prepare("INSERT INTO row(rowid, content) VALUES (?, 'alpha beta gamma')").use {
|
||||
it.bindLong(1, i.toLong())
|
||||
it.step()
|
||||
}
|
||||
}
|
||||
val byColumn =
|
||||
timeMs {
|
||||
for (i in 1..500) {
|
||||
db.prepare("DELETE FROM col WHERE event_header_row_id = ?").use {
|
||||
it.bindLong(1, i.toLong())
|
||||
it.step()
|
||||
}
|
||||
}
|
||||
}
|
||||
val byRowid =
|
||||
timeMs {
|
||||
for (i in 1..500) {
|
||||
db.prepare("DELETE FROM row WHERE rowid = ?").use {
|
||||
it.bindLong(1, i.toLong())
|
||||
it.step()
|
||||
}
|
||||
}
|
||||
}
|
||||
println(" %-9s %11.2f ms %11.2f ms".format("${n / 1000}k", byColumn, byRowid))
|
||||
} finally {
|
||||
db.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun searchScaling() =
|
||||
runBlocking {
|
||||
println("─ FtsSearchScalingBenchmark (scale=$SCALE, limit=50) ─")
|
||||
println(" %-9s %14s %14s %14s".format("corpus", "createdAt/frag", "createdAt/opt", "rowid/opt"))
|
||||
println("─ FtsSearchScalingBenchmark.search (created_at DESC, limit=50) ─")
|
||||
println(" %-9s %16s %16s".format("corpus", "fragmented", "optimized"))
|
||||
for (size in SIZES) {
|
||||
val events = seed(size)
|
||||
|
||||
val createdAt = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy())
|
||||
val rowId = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy(searchOrderByRowId = true))
|
||||
val store = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy())
|
||||
try {
|
||||
// Insert in 10k chunks → many FTS segments (fragmented).
|
||||
events.chunked(10_000).forEach {
|
||||
createdAt.batchInsert(it)
|
||||
rowId.batchInsert(it)
|
||||
}
|
||||
|
||||
seed(size).chunked(10_000).forEach { store.batchInsert(it) }
|
||||
val f = Filter(search = NEEDLE, limit = 50)
|
||||
val frag = time(createdAt, f)
|
||||
|
||||
createdAt.store.reindexFullTextSearch() // rebuild + optimize()
|
||||
rowId.store.reindexFullTextSearch()
|
||||
val optCreatedAt = time(createdAt, f)
|
||||
val optRowId = time(rowId, f)
|
||||
|
||||
println(
|
||||
" %-9s %12.2f ms %12.2f ms %12.2f ms".format(
|
||||
if (size >= 1000) "${size / 1000}k" else "$size",
|
||||
frag,
|
||||
optCreatedAt,
|
||||
optRowId,
|
||||
),
|
||||
)
|
||||
val frag = time(store, f)
|
||||
store.store.reindexFullTextSearch() // rebuild + optimize()
|
||||
val opt = time(store, f)
|
||||
println(" %-9s %13.2f ms %13.2f ms".format("${size / 1000}k", frag, opt))
|
||||
} finally {
|
||||
createdAt.close()
|
||||
rowId.close()
|
||||
store.close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user