perf(quartz): batch inserts + crawl-wide dedup in GrapeRankDataCrawler

The crawl re-verified and re-inserted the same event many times: the outbox
model mirrors each event (especially kind:10002 relay lists) across relays,
indexers, and rounds, but dedup lived in a per-drain SeenIds, so only the copies
within one drain were caught. Add a crawl-wide seen-set (thread-safe ConcurrentSet
of event ids, shared across all 24 concurrent drains and every round), checked
before verify and added only after verify so a forged copy can't suppress the
genuine one. Group-commit the store writes via IEventStore.batchInsert instead of
one transaction per event.

Measured on a from-scratch --max-hops 3 crawl: events actually verified+stored
dropped ~34% (112k -> 74k) and verify time fell in lockstep. The write path now
also reports verify/insert timing + events_stored in Stats, exposed as verify_ms/
insert_ms/events_stored on the CLI, and takes an --insert-batch knob.

Finding: with the work reduced, inserts serialize on SQLite's single writer
mutex rather than transaction count, and the crawl's wall-clock ceiling is the
drain-timeout retry tail on dead outboxes, not the disk.
This commit is contained in:
Claude
2026-07-07 17:51:27 +00:00
parent 41e88695e0
commit f7c3aef6fc
2 changed files with 104 additions and 22 deletions
@@ -131,6 +131,10 @@ object GrapeRankCommand {
val offline = args.bool("offline")
val diagnose = args.bool("diagnose")
val timeoutMs = args.longFlag("timeout", 10L) * 1000
// How many verified events the crawler group-commits per store write. 1
// forces the per-event insert path (baseline); higher amortizes the SQLite
// transaction + writer-mutex cost across the batch.
val insertBatch = args.intFlag("insert-batch", 500)
val doPublish = args.bool("publish")
// Publish cutoff: only cards with rank >= this are published; existing
// cards for targets below it (or gone from the graph) are retracted. Rank
@@ -188,6 +192,7 @@ object GrapeRankCommand {
maxHops = maxHops,
timeoutMs = timeoutMs,
diagnose = diagnose,
insertBatchSize = insertBatch,
),
log = { System.err.println(it) },
)
@@ -264,6 +269,10 @@ object GrapeRankCommand {
"reports_deleted" to reportsDeleted,
"users_scored" to rankedIds.size,
"download_ms" to crawlStats?.downloadMs,
"verify_ms" to crawlStats?.verifyMs,
"insert_ms" to crawlStats?.insertMs,
"events_stored" to crawlStats?.eventsStored,
"insert_batch" to insertBatch,
"store_load_ms" to storeLoadMs,
"graph_build_ms" to buildMs,
"scoring_ms" to scoringMs,