mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-10 08:27:04 +00:00
perf(graperank): run the crawl on Dispatchers.IO, not the caller's loop
The CLI calls crawl() from runBlocking's single-threaded event loop, so Phase B's thousands of concurrent drain-unit coroutines (timeout timers, channels, REQ JSON encoding, signature verifies, SQLite writes) all queued on ONE thread: timers fired late, batch walls inflated ~5x, and the --diagnose ticker showed 24/24 workers pinned while completing ~1 user/s with one core pegged and three idle. It also explains why the old 64-worker A/B ran slower - more coroutines on the same thread. Hop the whole run onto Dispatchers.IO inside the crawler so every front end gets real parallelism; IO (not Default) because the store's blocking SQLite calls must not starve the cores-sized pool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
This commit is contained in:
+12
-1
@@ -44,6 +44,8 @@ import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentSet
|
||||
import kotlinx.coroutines.CompletableDeferred
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.IO
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.async
|
||||
import kotlinx.coroutines.awaitAll
|
||||
@@ -57,6 +59,7 @@ import kotlinx.coroutines.joinAll
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.selects.select
|
||||
import kotlinx.coroutines.sync.Semaphore
|
||||
import kotlinx.coroutines.withContext
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import kotlin.concurrent.atomics.AtomicLong
|
||||
import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
@@ -278,7 +281,15 @@ class GrapeRankCrawler(
|
||||
verifyNanos.store(0)
|
||||
insertNanos.store(0)
|
||||
eventsStored.store(0)
|
||||
return CrawlRun(observer, builder).run()
|
||||
// Hop off the caller's dispatcher: a CLI calls this from runBlocking's
|
||||
// SINGLE-THREADED event loop, and Phase B runs thousands of concurrent
|
||||
// drain-unit coroutines (each with a timeout timer, a channel, REQ JSON
|
||||
// encoding, signature verifies and store writes). On one thread those
|
||||
// timers fire late and batch walls inflate ~5x — measured 111 users/s
|
||||
// with the lone thread pegged at 100% while 3 cores idled. IO (not
|
||||
// Default): the store's blocking SQLite calls must not starve the
|
||||
// small cores-sized pool.
|
||||
return withContext(Dispatchers.IO) { CrawlRun(observer, builder).run() }
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user