mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-09 16:14:40 +00:00
Claude
115963eb1c
FTS indexing ran inside every insert's transaction — a measurable slice of write cost (relayBench: ~18% of ingest throughput) paid at publish time for a feature only search queries read. It now runs as a watermark catch-up: - IndexingStrategy.deferFullTextSearchIndexing (default false; geode's relay strategy enables it with search). Deferred inserts skip tokenization entirely. - FullTextSearchModule keeps a fts_catchup_state watermark (everything <= last_row_id is indexed) and gains catchUpBatch(): scan past the watermark, tokenize, advance — one write transaction per batch, so publishes interleave. DATABASE_VERSION 3->4 seeds the watermark at MAX(row_id) for existing (synchronously indexed) databases. - NostrServer runs the catch-up worker, poked by IngestQueue's new onBatchCommitted hook, and *yields to publish traffic*: it only drains while the queue has no backlog (IngestQueue.hasBacklog()), so bursts ingest at no-FTS speed and tokenization fills the gaps. - LiveEventStore drains the backlog synchronously before serving any filter with a search term (query, queryRaw, count) — NIP-50 results stay exactly as fresh as the synchronous path; the deferral is invisible to correctness. Geode's existing search tests pass unchanged through this path. The first implementation reused reindexBatch and collapsed ingest 8x — its per-row 'DELETE FROM event_fts WHERE event_header_row_id = ?' matches on a plain FTS5 column, i.e. a full FTS-table scan per row (O(n²) overall), and the worker competed with the replay for the writer mutex. catchUpBatch therefore inserts without the delete (rows past the watermark are never indexed; switching a DB between deferred and synchronous strategies requires reindexAll, same rule as a searchable-kinds change), and the worker backs off whenever publishes are pending. Alternating A/B, 50k corpus, search-enabled default: 4,902/5,090/5,136 events/s synchronous vs 5,425/5,611 deferred (+8-12%), approaching the --no-search ceiling while keeping NIP-50 advertised and fresh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NeoCvXnTxsKzqurkmjdC46