From bbd8b3a596279d35de27b6cc036d54c717697804 Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Sat, 4 Jul 2026 22:45:43 +0000
Subject: [PATCH] perf(nip77): stop the live index turning bulk backfill into
 O(n^2)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.

When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
---
 .../nip77Negentropy/LiveNegentropyIndex.kt    | 27 ++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/LiveNegentropyIndex.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/LiveNegentropyIndex.kt
index dd32d50d67..0e0b134d63 100644
--- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/LiveNegentropyIndex.kt
+++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip77Negentropy/LiveNegentropyIndex.kt
@@ -156,7 +156,22 @@ class LiveNegentropyIndex {
             if (!populated) return
             val at = search(entry)
             if (at >= 0) return
-            entries.add(-(at + 1), entry)
+            val insertionPoint = -(at + 1)
+            // Far-from-tail insert = bulk / out-of-order backfill (a mirror
+            // pulling history, an import). The O(n) array shift would make the
+            // whole run O(n²) — a 1M sync crawls once the list is big. Drop to
+            // the invalidate path instead: the index rebuilds in one O(n log n)
+            // pass on the next NEG-OPEN, so backfill costs O(1) per event here.
+            // Near-tail live traffic (created_at ≈ now, the common case) stays
+            // on the cheap incremental insert.
+            if (entries.size - insertionPoint > REBUILD_THRESHOLD) {
+                entries = ArrayList()
+                populated = false
+                generation++
+                cachedSnapshot = null
+                return
+            }
+            entries.add(insertionPoint, entry)
             generation++
             cachedSnapshot = null
         }
@@ -212,4 +227,14 @@ class LiveNegentropyIndex {
         }
         return sealed
     }
+
+    private companion object {
+        /**
+         * Max element-shift an incremental [insert] will pay before it drops
+         * the whole index (rebuilt lazily on the next NEG-OPEN) instead. Keeps
+         * near-tail live inserts cheap while stopping a bulk backfill of
+         * historical events from turning ingest into O(n²).
+         */
+        const val REBUILD_THRESHOLD = 4096
+    }
 }