mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-08 23:54:39 +00:00
perf(nip77): stop the live index turning bulk backfill into O(n^2)
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per incremental insert. That's cheap for near-tail live traffic (created_at ≈ now), but a mirror/import backfill delivers historical, out-of-order events, so every insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained ~20k ev/s with the index off. When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the index instead of shifting: it rebuilds in one O(n log n) scan on the next NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per event. Near-tail live inserts keep the cheap incremental path. NIP-77 convergence and byte-exact tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
This commit is contained in:
+26
-1
@@ -156,7 +156,22 @@ class LiveNegentropyIndex {
|
||||
if (!populated) return
|
||||
val at = search(entry)
|
||||
if (at >= 0) return
|
||||
entries.add(-(at + 1), entry)
|
||||
val insertionPoint = -(at + 1)
|
||||
// Far-from-tail insert = bulk / out-of-order backfill (a mirror
|
||||
// pulling history, an import). The O(n) array shift would make the
|
||||
// whole run O(n²) — a 1M sync crawls once the list is big. Drop to
|
||||
// the invalidate path instead: the index rebuilds in one O(n log n)
|
||||
// pass on the next NEG-OPEN, so backfill costs O(1) per event here.
|
||||
// Near-tail live traffic (created_at ≈ now, the common case) stays
|
||||
// on the cheap incremental insert.
|
||||
if (entries.size - insertionPoint > REBUILD_THRESHOLD) {
|
||||
entries = ArrayList()
|
||||
populated = false
|
||||
generation++
|
||||
cachedSnapshot = null
|
||||
return
|
||||
}
|
||||
entries.add(insertionPoint, entry)
|
||||
generation++
|
||||
cachedSnapshot = null
|
||||
}
|
||||
@@ -212,4 +227,14 @@ class LiveNegentropyIndex {
|
||||
}
|
||||
return sealed
|
||||
}
|
||||
|
||||
private companion object {
|
||||
/**
|
||||
* Max element-shift an incremental [insert] will pay before it drops
|
||||
* the whole index (rebuilt lazily on the next NEG-OPEN) instead. Keeps
|
||||
* near-tail live inserts cheap while stopping a bulk backfill of
|
||||
* historical events from turning ingest into O(n²).
|
||||
*/
|
||||
const val REBUILD_THRESHOLD = 4096
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user