perf(nip77): stop the live index turning bulk backfill into O(n^2)

LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.

When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
This commit is contained in:
Claude
2026-07-04 22:45:43 +00:00
parent 91a7261555
commit bbd8b3a596
@@ -156,7 +156,22 @@ class LiveNegentropyIndex {
if (!populated) return
val at = search(entry)
if (at >= 0) return
entries.add(-(at + 1), entry)
val insertionPoint = -(at + 1)
// Far-from-tail insert = bulk / out-of-order backfill (a mirror
// pulling history, an import). The O(n) array shift would make the
// whole run O(n²) — a 1M sync crawls once the list is big. Drop to
// the invalidate path instead: the index rebuilds in one O(n log n)
// pass on the next NEG-OPEN, so backfill costs O(1) per event here.
// Near-tail live traffic (created_at ≈ now, the common case) stays
// on the cheap incremental insert.
if (entries.size - insertionPoint > REBUILD_THRESHOLD) {
entries = ArrayList()
populated = false
generation++
cachedSnapshot = null
return
}
entries.add(insertionPoint, entry)
generation++
cachedSnapshot = null
}
@@ -212,4 +227,14 @@ class LiveNegentropyIndex {
}
return sealed
}
private companion object {
/**
* Max element-shift an incremental [insert] will pay before it drops
* the whole index (rebuilt lazily on the next NEG-OPEN) instead. Keeps
* near-tail live inserts cheap while stopping a bulk backfill of
* historical events from turning ingest into O(n²).
*/
const val REBUILD_THRESHOLD = 4096
}
}