mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-10 16:33:27 +00:00
perf(quartz): batch inserts + crawl-wide dedup in GrapeRankDataCrawler
The crawl re-verified and re-inserted the same event many times: the outbox model mirrors each event (especially kind:10002 relay lists) across relays, indexers, and rounds, but dedup lived in a per-drain SeenIds, so only the copies within one drain were caught. Add a crawl-wide seen-set (thread-safe ConcurrentSet of event ids, shared across all 24 concurrent drains and every round), checked before verify and added only after verify so a forged copy can't suppress the genuine one. Group-commit the store writes via IEventStore.batchInsert instead of one transaction per event. Measured on a from-scratch --max-hops 3 crawl: events actually verified+stored dropped ~34% (112k -> 74k) and verify time fell in lockstep. The write path now also reports verify/insert timing + events_stored in Stats, exposed as verify_ms/ insert_ms/events_stored on the CLI, and takes an --insert-batch knob. Finding: with the work reduced, inserts serialize on SQLite's single writer mutex rather than transaction count, and the crawl's wall-clock ceiling is the drain-timeout retry tail on dead outboxes, not the disk.
This commit is contained in:
@@ -131,6 +131,10 @@ object GrapeRankCommand {
|
||||
val offline = args.bool("offline")
|
||||
val diagnose = args.bool("diagnose")
|
||||
val timeoutMs = args.longFlag("timeout", 10L) * 1000
|
||||
// How many verified events the crawler group-commits per store write. 1
|
||||
// forces the per-event insert path (baseline); higher amortizes the SQLite
|
||||
// transaction + writer-mutex cost across the batch.
|
||||
val insertBatch = args.intFlag("insert-batch", 500)
|
||||
val doPublish = args.bool("publish")
|
||||
// Publish cutoff: only cards with rank >= this are published; existing
|
||||
// cards for targets below it (or gone from the graph) are retracted. Rank
|
||||
@@ -188,6 +192,7 @@ object GrapeRankCommand {
|
||||
maxHops = maxHops,
|
||||
timeoutMs = timeoutMs,
|
||||
diagnose = diagnose,
|
||||
insertBatchSize = insertBatch,
|
||||
),
|
||||
log = { System.err.println(it) },
|
||||
)
|
||||
@@ -264,6 +269,10 @@ object GrapeRankCommand {
|
||||
"reports_deleted" to reportsDeleted,
|
||||
"users_scored" to rankedIds.size,
|
||||
"download_ms" to crawlStats?.downloadMs,
|
||||
"verify_ms" to crawlStats?.verifyMs,
|
||||
"insert_ms" to crawlStats?.insertMs,
|
||||
"events_stored" to crawlStats?.eventsStored,
|
||||
"insert_batch" to insertBatch,
|
||||
"store_load_ms" to storeLoadMs,
|
||||
"graph_build_ms" to buildMs,
|
||||
"scoring_ms" to scoringMs,
|
||||
|
||||
Reference in New Issue
Block a user