diff --git a/cli/README.md b/cli/README.md index e11e1b838c..fa900daf0b 100644 --- a/cli/README.md +++ b/cli/README.md @@ -385,6 +385,41 @@ HTTP endpoint. Reuses quartz's `Nip86Client` and the shared `Nip86Retriever` | `amy notes feed [--author USER \| --following] [--limit N]` | Read recent kind:1 notes (yours, one user's, or your follow set). | | `amy profile show [USER]` | Print kind:0 metadata. USER accepts npub/nprofile/hex/NIP-05; defaults to self. | | `amy profile edit --name … --about … --picture URL …` | Patch and re-publish your kind:0. | +| `amy follow USER` / `amy unfollow USER` | Add/remove USER from your kind:3 contact list (fetches the freshest list first). | +| `amy graperank [OBSERVER] [--offline] [--publish] [--min-rank N] [--publish-relay URL]` | Compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. With `--publish`, reconciles NIP-85 kind:30382 cards signed by a per-observer **service key**: publishes changed/new ranks (cutoff `--min-rank`, default 2), skips unchanged, and **retracts** (kind:5) any card whose target left the graph or fell below the cutoff. | +| `amy graperank operator [status \| relay … \| providers]` | Manage the machine's operator keys (independent of any account, under `~/.amy/operator/`). `relay` sets where cards + retractions publish; `status` shows the master pubkey and relays; `providers` lists the observer → service-pubkey map. | +| `amy graperank register [PROVIDER] [--service KIND:TAG] [--relay URL]` | Declare a NIP-85 provider in your kind:10040 so clients can discover it (default: self as the `30382:rank` provider). | +| `amy graperank providers [USER]` | List a user's declared NIP-85 trusted providers (public + your own private entries). | + +#### Publishing GrapeRank scores (NIP-85) + +Ranks are published as kind:30382 cards, but **not** under your account key. A +machine holds one **operator master** seed (`~/.amy/operator/`, stored via the +same `--secret-backend` as accounts, independent of any account). From it a +distinct, deterministic **service key** is derived per observer: + +``` +serviceKey(observer) = sha256(masterPriv ‖ "graperank-provider:" ‖ observerHex) +``` + +Because kind:30382 is addressable (`pubkey + d-tag`), the stable per-observer key +means re-publishing **replaces** a target's card instead of orphaning it — and +losing everything but the master seed still re-derives every key. Set up once and +publish: + +```bash +amy graperank operator relay wss://relay.example.com # where all cards live +amy graperank --publish # sign with the observer's service key +``` + +Each publish **reconciles** against what the service key already published: new or +changed ranks (≥ `--min-rank`, default 2) are signed and sent; unchanged ranks are +skipped (no new event id); and any card whose target dropped out of the graph or +fell below the cutoff is **retracted** with a kind:5. When the observer is your +own account (we hold the key), Amy also writes their kind:10040 pointing +`30382:rank → serviceKey @ operator relay` to their outbox, so clients can find +the cards. For a third-party observer, `graperank operator providers` prints the +`observer → service-pubkey` mapping to wire their kind:10040 out-of-band. ### Direct messages (NIP-17) diff --git a/cli/ROADMAP.md b/cli/ROADMAP.md index 24fd1ff4b7..a6fdb62a10 100644 --- a/cli/ROADMAP.md +++ b/cli/ROADMAP.md @@ -60,6 +60,7 @@ Status legend: ✅ shipped · 📦 logic lives in `commons/`, needs a command · | NIP-51 lists (bookmarks, mute, follow sets) | 🆕 | `amethyst/model/nip51Lists/` | | NIP-57 zaps (send + verify) | 🆕 | Needs LN-URL plumbing; `amethyst/service/lnurl/`. | | NIP-65 outbox model queries | 🆕 | | +| NIP-85 GrapeRank web-of-trust (`amy graperank`) | ✅ | `GrapeRankCommand` — outbox-model crawl + scoring engine in `commons/wot/` (`GrapeRank`, `TrustGraph`, `TrustGraphBuilder`); publishes kind:30382 `ContactCardEvent` (diffed against prior ranks), plus `register` / `providers` for the kind:10040 `TrustProviderListEvent` discovery layer. | | NIP-72 communities | 🆕 | | | NIP-78 app-specific data (settings sync) | 🆕 | | | Long-form (NIP-23) publish / read | 🆕 | | diff --git a/cli/plans/2026-07-06-graperank-brainstorm-parity.md b/cli/plans/2026-07-06-graperank-brainstorm-parity.md new file mode 100644 index 0000000000..52f09056d9 --- /dev/null +++ b/cli/plans/2026-07-06-graperank-brainstorm-parity.md @@ -0,0 +1,134 @@ +# GrapeRank score parity with NosFabrica Brainstorm + +Goal: `amy graperank` should output scores **numerically very close** to +NosFabrica's Brainstorm service, the reference GrapeRank implementation. + +Sources analysed: +- `NosFabrica/brainstorm_graperank_algorithm` — the Java scoring worker. +- `NosFabrica/brainstorm_server` — the Python orchestration server. + +## How Brainstorm builds its service + +A four-stage pipeline: + +1. **Ingest.** `app/nostr_event_transferer/nostr_event_transferer.py` copies raw + social-graph events — **kinds 0, 3, 10000, 1984** (profiles, follows, mutes, + reports) — from a strfry relay into the server. Same four kinds we crawl. +2. **Graph.** Events land in **Neo4j** as a directed graph of follow / mute / + report edges between pubkeys. Redis + Postgres back the job queue and config. +3. **Score.** The Java worker (`grape/GrapeRankAlgorithm.java`) runs GrapeRank + from an observer, producing a **`ScoreCard`** per user + (`rank/ScoreCard.java`): `observer, observee, hops, averageScore, input, + confidence, influence, verified, trustedFollowers, trustedReporters`. + **There is no `rank` field — the trust value is `influence` ∈ [0,1].** +4. **Serve / publish.** Presets are tunable per deployment + (`DEFAULT` / `PERMISSIVE` / `RESTRICTIVE`, `graperank_preset` table, validated + by `GrapeRankPresetParams`). Java `GrapeRankParams` mirrors the Python model + field-for-field; the README states Python is the source of truth and both + repos must stay in sync. + +## The algorithm (their `grape/GrapeRankAlgorithm.java`) + +``` +rigority = -log(rigor) +confidence(sumWeights) = 1 - exp(-sumWeights * rigority) # weight -> confidence +per edge: weight = edgeConfidence * influenceOfRater * attenuationFactor + wxr = weight * edgeRating +averageScore = sumWxR / sumWeights (0 if sumWeights == 0) +influence = max(averageScore * confidence(sumWeights), 0) +``` + +- Observer seeded at `influence = 1.0` (fixed authority). +- Non-observers seeded by hop distance, then **iterated until every user's + influence delta < 0.0001** (`loopBreakDelta`). Seeding only affects the + starting guess; attenuation < 1 makes the update a contraction, so the fixed + point is unique. +- The rater weight uses the rater's **`influence`**, and + `influence = max(weightToConfidence(sumW) * sumWR/sumW, 0)`. + +## Side-by-side: Brainstorm DEFAULT vs `commons/wot` + +`Constants.java` `DEFAULT_PARAMS` (== the Pydantic `GrapeRankPresetParams` +DEFAULT) against our `GrapeRankParams` defaults: + +| Brainstorm field | value | our field | value | match | +|---|---|---|---|---| +| `attenuationFactor` | 0.85 | `attenuation` | 0.85 | ✅ | +| `rigor` | 0.5 | `rigor` | 0.5 | ✅ | +| `followRating` | 1.0 | `FOLLOW.rating` | 1.0 | ✅ | +| `muteRating` | -0.1 | `MUTE.rating` | -0.1 | ✅ | +| `reportRating` | -0.1 | `REPORT.rating` | -0.1 | ✅ | +| `followConfidenceOfObserver` | 0.5 | `directFollowConfidence` | 0.5 | ✅ | +| `followConfidence` | 0.03 | `indirectFollowConfidence` | 0.03 | ✅ | +| `muteConfidence` | 0.5 | `muteConfidence` | 0.5 | ✅ | +| `reportConfidence` | 0.5 | `reportConfidence` | 0.5 | ✅ | +| `loopBreakDelta` | 0.0001 | `convergence` | 0.0001 | ✅ | + +The three `verified*InfluenceCutoff`s (followers 0.02, reporters 0.1, +muters 0.01) only flag a derived `verified` boolean; they do **not** affect the +score. + +**Conclusion: our formula is identical and every scoring parameter matches +DEFAULT.** Our `score` *is* their `influence` +(`max(weightToConfidence(sumW) * sumWR/sumW, 0)`), propagated as the rater +weight — the exact same quantity. On the same input graph the two produce the +same influence to floating-point precision. Our published `rank = round(score * +100)` is a presentation choice on top of that influence (their `ScoreCard` +exposes `influence` as a raw float via the API). + +## Where divergence can still come from — and why it's small + +It is **data**, not math: + +1. **Graph completeness.** Brainstorm ingests the whole strfry graph into Neo4j; + we crawl outward from the observer via the outbox model. **This matters less + than it seems:** a mute/report contributes `confidence * influenceOfRater * + attenuation`, so a signal from a user with **zero influence** (someone outside + the observer's trust graph) contributes **zero**. Only follows/mutes/reports + authored by users *inside* the follow graph move a score — and those are + exactly the users our crawl discovers and whose kind 3/10000/1984 we fetch. + So the effective scoring input is the same, as long as the crawl actually + checks every discovered user's outbox — which it now does exhaustively (no + user cap, retrying an unreachable outbox a few times). +2. **Fringe users / crawl gaps.** A relay timeout that drops a contact list + removes edges and shifts nearby scores. The injector mitigates this with a + two-stage model mirroring the app's `pickRelaysToLoadUsers`, plus a + completeness loop that retries until every user's outbox has been checked: + - **Relay-list discovery** (kind:10002) queries the account's relays + + bootstrap + event-finder + **indexer relays** (purplepag.es, coracle, …). + Indexers aggregate kind:10002 (and kind:0) for the whole network, so this is + where a stranger's outbox is found — the biggest completeness lever. + - **Content** (kind:3/10000/1984/0) is fetched from each user's **own outbox** + write relays, with harvested **relay hints** (from the `p`-tag hints in + contact lists we crawl) and general-purpose relays as a best-effort fallback + when the outbox is unknown/down. **Indexers are not used for content** — they + don't serve those kinds; kind:3/mutes/reports live only on the user's outbox. + The crawl loops round by round, retrying any member whose contact list still + didn't arrive (a few times), until every discovered user's outbox + has been checked. Remaining mitigation lever: a generous `--timeout`. +3. **Convergence precision.** Both stop at delta 0.0001; residual error is + < ~0.0001 in influence ⇒ < ~0.01 rank points ⇒ identical integer `rank`. +4. **Seeding.** Their hop-distance seed vs our zero seed — same fixed point, no + effect on the result. + +## Recommendations + +- **Keep the current DEFAULT params** — they are byte-for-byte the Brainstorm + DEFAULT preset. No change needed for parity. +- **The crawl is exhaustive by default** (no user cap; every reachable user's + outbox is checked, unreachable outboxes retried a few times). An + incomplete crawl is the single biggest source of drift, so avoid capping it. +- **Optional, for fuller parity (not required for close scores):** + - Add `--preset default|permissive|restrictive`. DEFAULT is confirmed; the + PERMISSIVE / RESTRICTIVE numbers are DB-seeded in `brainstorm_server` (an + alembic seed migration) and were not extractable from the public tree — + pull them from a running instance before hard-coding. + - Optionally expose `influence` as a raw float alongside `rank` in `--json`, + and compute the `verified` flag from the cutoffs, to mirror their + `ScoreCard` shape for interop diffing. + +## Verification idea + +Point `amy graperank --offline` at a store seeded from the same +strfry snapshot Brainstorm ingested, and diff our `score` against their +`ScoreCard.influence` for the same observer. Expect agreement to ~1e-4. diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Config.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Config.kt index 8a8779b8f7..48f19276c4 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Config.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Config.kt @@ -234,6 +234,22 @@ class DataDir( val groupsDir = File(marmotDir, "groups") val keyPackageBundleFile = File(marmotDir, "keypackages.bundle") + /** + * SQLite event-store DB file, a sibling of [eventsDir] under + * `/shared/`. Used when the store backend is SQLite (the + * default — see [StoreFactory]); the FS backend uses [eventsDir] + * instead. Kept alongside the FS store so switching backends never + * clobbers the other's data. + */ + val eventsDbFile: File = File(eventsDir.parentFile ?: root, "events.db") + + /** + * Machine-level operator keys for GrapeRank trusted-assertion publishing, + * rooted at `~/.amy/operator/` (the account root's parent) so a single + * operator master is shared across accounts. See [OperatorKeys]. + */ + fun operatorKeys(): OperatorKeys = OperatorKeys(root.parentFile ?: root, secrets) + init { SecureFileIO.secureMkdirs(root) // The accountless dir only ever exposes the shared event store; don't diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt index 88238e643b..f4bb055ff1 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Context.kt @@ -38,12 +38,14 @@ import com.vitorpamplona.quartz.marmot.mip03GroupMessages.GroupEvent import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.core.HexKey import com.vitorpamplona.quartz.nip01Core.core.hexToByteArray -import com.vitorpamplona.quartz.nip01Core.crypto.verify -import com.vitorpamplona.quartz.nip01Core.jackson.JacksonMapper import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent import com.vitorpamplona.quartz.nip01Core.relay.client.NostrClient +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.AdaptiveRelayLimiter +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.DrainFailure +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.classifyDrainFailure import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPagesFromPool import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.publishAndConfirmDetailed +import com.vitorpamplona.quartz.nip01Core.relay.client.auth.RelayAuthenticator import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener import com.vitorpamplona.quartz.nip01Core.relay.client.single.newSubId import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.CachingEventDecoder @@ -55,7 +57,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.sockets.okhttp.TcpNoDelaySocketF import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerInternal import com.vitorpamplona.quartz.nip01Core.store.IEventStore -import com.vitorpamplona.quartz.nip01Core.store.fs.FsEventStore +import com.vitorpamplona.quartz.nip01Core.store.verifyAndInsert import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent import com.vitorpamplona.quartz.nip17Dm.settings.ChatMessageRelayListEvent import com.vitorpamplona.quartz.nip46RemoteSigner.signer.NostrSignerRemote @@ -78,7 +80,9 @@ import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.launch import kotlinx.coroutines.selects.select import kotlinx.coroutines.withTimeoutOrNull +import okhttp3.Dispatcher import okhttp3.OkHttpClient +import java.util.concurrent.TimeUnit /** * Per-invocation wiring. Each CLI run constructs a Context, does its work, @@ -98,9 +102,10 @@ import okhttp3.OkHttpClient * Every Nostr event Amy observes — whether received from a relay * subscription, unwrapped from a NIP-59 gift wrap, or generated locally * before publish — is verified (NIP-01 signature + id check via - * [Event.verify]) and persisted to the file-backed [IEventStore] at - * `/events-store/`. Malformed events are dropped before - * reaching command code. + * [Event.verify]) and persisted to the shared [IEventStore] under + * `/shared/` (a SQLite DB by default, or the FS tree when + * `AMY_STORE=fs` — see [StoreFactory]). Malformed events are dropped + * before reaching command code. * * This makes [store] the authoritative cache of everything Amy has ever * seen: profile metadata, relay lists, contact lists, gift wraps, @@ -124,7 +129,31 @@ class Context( */ val anonymous: Boolean = false, ) : AutoCloseable { - private val okhttp = OkHttpClient.Builder().socketFactory(TcpNoDelaySocketFactory).build() + private val okhttp = + OkHttpClient + .Builder() + .socketFactory(TcpNoDelaySocketFactory) + // The crawl opens WebSockets to thousands of relays. Each WS-upgrade + // handshake is an async call through OkHttp's shared Dispatcher, whose + // default cap (maxRequests=64) throttles the connection ramp — worse, + // a dead relay holds a slot for the whole connectTimeout, starving live + // relays queued behind it. Widen the dispatcher so handshakes fan out, + // and keep connectTimeout tight-ish so an unreachable relay frees its + // slot fast. This is orthogonal to REQ concurrency (that runs on + // already-open sockets, bounded by AdaptiveRelayLimiter), so it can't + // trip a relay's REQ rate-limit — it only speeds connection setup. The + // executor thread pool is unbounded on demand, so raising maxRequests + // just lets more of those short-lived handshakes proceed at once. 7s + // (not 5s): a 5s cap struck too many merely-busy relays as connect + // failures — the crawl treats a connect *timeout* as retryable anyway, + // but the extra headroom lets slow-but-alive relays finish the handshake. + .connectTimeout(7, TimeUnit.SECONDS) + .dispatcher( + Dispatcher().apply { + maxRequests = 256 + maxRequestsPerHost = 16 + }, + ).build() val client: NostrClient = NostrClient( @@ -154,6 +183,44 @@ class Context( ) } ?: NostrSignerInternal(identity.keyPair()) + /** + * Client-wide tally of relay feedback — NOTICE frames, CLOSED reasons + * (auth-required / rate-limited / restricted / …), and NIP-42 AUTH + * challenges — so a failed REQ can be explained instead of guessed at. + * Registered on [client] for the life of this run. + */ + val relayDiagnostics: RelayDiagnostics = RelayDiagnostics().also { client.addConnectionListener(it) } + + /** + * Adaptive per-relay concurrent-subscription cap. Starts every relay + * generous (100) and demotes only the ones that complain about concurrency + * (100 → 20 → 10), driven straight off the NOTICE/CLOSED frames it observes + * as a connection listener. [drain]'s `gatePerRelay` path holds a relay's + * permit for the life of that relay's subscription, so we never exceed the + * cap the relay itself asked for. Idle for commands that don't opt in. + */ + val relayLimiter: AdaptiveRelayLimiter = AdaptiveRelayLimiter().also { client.addConnectionListener(it) } + + /** + * NIP-42 responder: answers a relay's AUTH challenge by signing with the + * account key, so auth-gated relays serve our reads instead of CLOSing the + * subscription. Constructing it registers its own listener on [client]. + * Only a local key auto-signs — a remote bunker signer is skipped, since a + * per-relay remote round-trip during a crawl would stall it (and signing an + * auth event with any key still unlocks relays that just want *some* auth). + */ + private val relayAuth: RelayAuthenticator = + RelayAuthenticator( + client = client, + signWithAllLoggedInUsers = { _, template -> + if (signer is NostrSignerInternal) { + runCatching { listOf(signer.sign(template)) }.getOrElse { emptyList() } + } else { + emptyList() + } + }, + ) + /** * NIP-05 resolver for turning `alice@damus.io`-style identifiers into pubkeys. * Uses the same OkHttp instance as the WebSocket client so we share connection @@ -174,23 +241,13 @@ class Context( private val messageStore by lazy { FileMarmotMessageStore(dataDir.groupsDir) } /** - * Filesystem-backed Nostr event store, rooted at [DataDir.eventsDir]. - * Lazy so commands that don't touch persistent event state pay zero - * open cost (no `.lock` file, no seed allocation). Closed by - * [close] when this Context shuts down. - * - * Files are written pretty-printed (not the compact NIP-01 canonical - * form) so `cat`, `jq`, `git diff` are useful out of the box — - * humans inspect these files. Verification always re-canonicalises, - * so the stored bytes never feed back into a signature check. + * Shared Nostr event store for this run, opened via [StoreFactory] + * (SQLite by default, or the FS tree when `AMY_STORE=fs`). Lazy so + * commands that don't touch persistent event state pay zero open cost + * (no DB file / `.lock`, no seed allocation). Closed by [close] when + * this Context shuts down. */ - private val storeDelegate: Lazy = - lazy { - FsEventStore( - root = dataDir.eventsDir.toPath(), - eventToJson = JacksonMapper::toJsonPretty, - ) - } + private val storeDelegate: Lazy = lazy { StoreFactory.open(dataDir) } val store: IEventStore by storeDelegate /** Fully-wired manager. Call [prepare] once before use to load persisted state. */ @@ -424,15 +481,26 @@ class Context( * Subscribe to the given filters across the given relays, drain all events * until either every relay has sent EOSE or the timeout elapses, and * return them. Used for one-shot catch-up queries — not live subscriptions. + * + * When [deadOut] is provided, every relay that reported it could not be + * connected to (`onCannotConnect`) is added to it, so callers can prune + * proven-dead relays from future routing instead of paying the full + * [timeoutMs] on them again. Slow-but-connected relays are NOT reported — + * only hard connect failures, so a temporarily-busy relay isn't discarded. */ suspend fun drain( filters: Map>, timeoutMs: Long = 8_000, + diagnoseSlow: Boolean = false, + deadOut: MutableMap? = null, ): List> { if (filters.isEmpty()) return emptyList() val eventChannel = Channel>(UNLIMITED) - val doneChannel = Channel(UNLIMITED) + // Carries the terminal reason per relay so a timeout can distinguish a slow + // relay (never terminal) from a connect failure / CLOSED. + val doneChannel = Channel>(UNLIMITED) val remaining = filters.keys.toMutableSet() + val doneReasons = HashMap() val subId = newSubId() val listener = object : SubscriptionListener { @@ -449,7 +517,7 @@ class Context( relay: NormalizedRelayUrl, forFilters: List?, ) { - doneChannel.trySend(relay) + doneChannel.trySend(relay to "eose") } override fun onClosed( @@ -457,7 +525,7 @@ class Context( relay: NormalizedRelayUrl, forFilters: List?, ) { - doneChannel.trySend(relay) + doneChannel.trySend(relay to "closed:$message") } override fun onCannotConnect( @@ -465,37 +533,75 @@ class Context( message: String, forFilters: List?, ) { - doneChannel.trySend(relay) + doneChannel.trySend(relay to "cannot:$message") } } val collected = mutableListOf>() try { client.subscribe(subId, filters, listener) - withTimeoutOrNull(timeoutMs) { - while (remaining.isNotEmpty()) { - select { - eventChannel.onReceive { pair -> - if (verifyAndStore(pair.second)) collected.add(pair) + val completed = + withTimeoutOrNull(timeoutMs) { + while (remaining.isNotEmpty()) { + select { + eventChannel.onReceive { pair -> + if (verifyAndStore(pair.second)) collected.add(pair) + } + doneChannel.onReceive { (relay, reason) -> + remaining.remove(relay) + doneReasons[relay] = reason + } } - doneChannel.onReceive { r -> remaining.remove(r) } } + // Drain any events that landed after EOSE but before cancel + while (true) { + val r = eventChannel.tryReceive() + if (!r.isSuccess) break + val pair = r.getOrThrow() + if (verifyAndStore(pair.second)) collected.add(pair) + } + true } - // Drain any events that landed after EOSE but before cancel - while (true) { - val r = eventChannel.tryReceive() - if (!r.isSuccess) break - val pair = r.getOrThrow() - if (verifyAndStore(pair.second)) collected.add(pair) - } + if (diagnoseSlow && completed == null && remaining.isNotEmpty()) { + logSlowDrain(timeoutMs, remaining, doneReasons, collected) } } finally { client.unsubscribe(subId) eventChannel.close() doneChannel.close() } + deadOut?.let { out -> + for ((relay, reason) in doneReasons) { + classifyDrainFailure(reason)?.let { out[relay] = it } + } + } return collected } + /** + * On a [drain] timeout, report which relays stalled and why — a relay that + * never sent EOSE (slow, possibly still streaming) vs one that couldn't be + * reached (CANNOT-CONNECT, which points at our side / the network) vs one + * that CLOSED the sub. Includes how many events each slow relay did send, so + * "relay is slow" and "we never connected" are easy to tell apart. + */ + private fun logSlowDrain( + timeoutMs: Long, + stalled: Set, + doneReasons: Map, + collected: List>, + ) { + val eventsPer = collected.groupingBy { it.first }.eachCount() + val cannot = doneReasons.filterValues { it.startsWith("cannot") } + val closed = doneReasons.filterValues { it.startsWith("closed") } + val slowDetail = stalled.take(12).joinToString(", ") { "${it.url}(${eventsPer[it] ?: 0}ev)" } + val cannotDetail = cannot.entries.take(8).joinToString(", ") { "${it.key.url}=${it.value.removePrefix("cannot:").take(40)}" } + System.err.println( + "[drain] timeout ${timeoutMs}ms: ${stalled.size} slow(no EOSE), ${cannot.size} cannot-connect, ${closed.size} closed" + + (if (slowDetail.isNotEmpty()) " | slow: $slowDetail" else "") + + (if (cannotDetail.isNotEmpty()) " | cannot: $cannotDetail" else ""), + ) + } + /** * Like [drain], but paginates every relay to completion via * [fetchAllPagesFromPool] instead of stopping at the first EOSE — so a query @@ -595,26 +701,17 @@ class Context( } /** - * Verify [event]'s NIP-01 id+signature and, if valid, persist it - * to [store]. Returns `true` when the event was accepted (and - * therefore should be surfaced to callers). Persistence failures - * (I/O errors, full disk) are logged but do not propagate. + * Verify [event]'s NIP-01 id+signature and, if valid, persist it to [store]. + * Returns `true` when the event was accepted (and therefore should be surfaced + * to callers). Persistence failures (I/O errors, full disk) are logged but do + * not propagate; a UNIQUE-constraint rejection is normal and swallowed quietly. * - * Every event-arrival path in the CLI funnels through this method - * so that [store] is the authoritative cache of what Amy has seen. + * Every event-arrival path in the CLI funnels through this so that [store] is + * the authoritative cache of what Amy has seen. Delegates to the shared quartz + * [verifyAndInsert] sink so the CLI and the GrapeRank crawler apply the exact + * same verify-then-store policy. */ - suspend fun verifyAndStore(event: Event): Boolean { - if (!event.verify()) { - System.err.println("[cli] dropped event ${event.id.take(8)} kind=${event.kind} — bad signature") - return false - } - try { - store.insert(event) - } catch (t: Throwable) { - System.err.println("[cli] store insert failed for ${event.id.take(8)}: ${t.message}") - } - return true - } + suspend fun verifyAndStore(event: Event): Boolean = store.verifyAndInsert(event) // ------------------------------------------------------------------ // Cache-first reads from [store] diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Main.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Main.kt index e267c5abe3..3a3db047f1 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Main.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Main.kt @@ -38,6 +38,7 @@ import com.vitorpamplona.amethyst.cli.commands.FilterCommand import com.vitorpamplona.amethyst.cli.commands.FollowCommand import com.vitorpamplona.amethyst.cli.commands.GiftCommands import com.vitorpamplona.amethyst.cli.commands.GitCommands +import com.vitorpamplona.amethyst.cli.commands.GrapeRankCommand import com.vitorpamplona.amethyst.cli.commands.GroupCommands import com.vitorpamplona.amethyst.cli.commands.InitCommands import com.vitorpamplona.amethyst.cli.commands.KeyCommands @@ -72,6 +73,8 @@ import com.vitorpamplona.amethyst.cli.commands.cashu.CashuCommands import com.vitorpamplona.amethyst.cli.commands.cashu.CashuMintCommands import com.vitorpamplona.amethyst.cli.commands.route import com.vitorpamplona.amethyst.cli.secrets.SecretStore +import com.vitorpamplona.quartz.utils.Log +import com.vitorpamplona.quartz.utils.LogLevel import kotlinx.coroutines.runBlocking import kotlin.system.exitProcess @@ -105,6 +108,12 @@ fun main(argv: Array) { // braces guard for invocations that bypass the launcher scripts. System.setProperty("java.awt.headless", "true") + // Quiet quartz's internal DEBUG chatter (relay auth, MLS restore, URL + // rejection, throttle notices) by default so it doesn't drown a command's + // own output; --verbose / -v restores full DEBUG. Set before dispatch so + // even startup logging is gated. + Log.minLevel = if (argv.any { it == "--verbose" || it == "-v" }) LogLevel.DEBUG else LogLevel.WARN + // Set output mode before dispatch so even argument-parsing errors // honour --json. if (argv.any { it == "--json" || it == "--json=true" }) { @@ -161,6 +170,7 @@ private suspend fun dispatch(argv: Array): Int { GlobalFlag.SECRET_BACKEND -> secretBackendFlag = consumed.value GlobalFlag.PASSPHRASE_FILE -> passphraseFileFlag = consumed.value GlobalFlag.JSON -> Output.mode = Output.Mode.JSON + GlobalFlag.VERBOSE -> Unit // level already applied in main(); just strip it here null -> filteredArgs.add(a) } i += consumed.tokensConsumed @@ -254,6 +264,7 @@ private suspend fun dispatch(argv: Array): Int { "store" -> StoreCommands.dispatch(dataDir, tail) "follow" -> FollowCommand.follow(dataDir, tail) "unfollow" -> FollowCommand.unfollow(dataDir, tail) + "graperank" -> GrapeRankCommand.dispatch(dataDir, tail) "search" -> SearchCommand.dispatch(dataDir, tail) "zap" -> ZapCommand.dispatch(dataDir, tail) "offer" -> OfferCommands.dispatch(dataDir, tail) @@ -305,11 +316,13 @@ private suspend fun marmotDispatch( private enum class GlobalFlag( val long: String, val takesValue: Boolean = true, + val short: String? = null, ) { ACCOUNT("--account"), SECRET_BACKEND("--secret-backend"), PASSPHRASE_FILE("--passphrase-file"), JSON("--json", takesValue = false), + VERBOSE("--verbose", takesValue = false, short = "-v"), } private data class ConsumedFlag( @@ -328,7 +341,7 @@ private fun extractGlobalFlag( idx: Int, ): Pair { for (flag in GlobalFlag.values()) { - if (token == flag.long) { + if (token == flag.long || token == flag.short) { return if (flag.takesValue) { flag to ConsumedFlag(argv.getOrNull(idx + 1), 2) } else { @@ -353,13 +366,15 @@ private fun printUsage() { | [--secret-backend auto|keychain|ncryptsec|plaintext] | [--passphrase-file PATH] | [--json] + | [--verbose|-v] | [args...] | |Account selection: | All state lives under ~/.amy/. Per-account directories | ~/.amy// hold identity, cursors, MLS state, and | aliases; every observed Nostr event lands in the shared - | ~/.amy/shared/events-store/. ACCOUNT must match + | store under ~/.amy/shared/ (a SQLite `events.db` by default, or + | the `events-store/` tree when AMY_STORE=fs). ACCOUNT must match | [a-zA-Z0-9_-]{1,64} (no spaces, no slashes). | | Resolution order: @@ -575,6 +590,31 @@ private fun printUsage() { | unfollow USER [--timeout SECS] remove USER from your contact list | (USER: npub|nprofile|hex|name@domain) | + |Web of Trust (GrapeRank): + | graperank [OBSERVER] compute subjective trust scores (0..1) for every + | [--limit N] [--min-score X] user reachable in the follow/mute/report graph. + | [--rigor X] [--attenuation X] Exhaustively crawls each user's kind:10002 outbox + | [--max-rounds N] [--max-hops N] for their latest kind:3/10000/1984 until every + | [--offline] [--timeout SECS] discovered user has been checked (no user cap; + | [--diagnose] --max-hops bounds follow distance, e.g. 8; + | --diagnose logs slow/failed relays on timeout). + | [--publish] [--min-rank N] OBSERVER: npub|nprofile|hex|name@domain (default: + | [--publish-limit N] [--publish-relay URL] active account). --offline scores from the local + | store only. --publish reconciles NIP-85 kind:30382 + | cards signed by a per-observer service key: sends + | new/changed ranks >= --min-rank (default 2), skips + | unchanged, and retracts (kind:5) any card whose + | target left the graph or fell below the cutoff. + | graperank operator [status|relay … manage the machine's operator keys (~/.amy/operator/, + | |providers] independent of accounts): relay sets where cards + + | retractions publish; status shows master + relays; + | providers lists observer -> service-pubkey. + | graperank register [PROVIDER] declare a NIP-85 provider in your kind:10040 so + | [--service KIND:TAG] [--relay URL] clients can discover it (default: self as the + | [--private] 30382:rank provider at your first outbox relay). + | graperank providers [USER] [--refresh] list a user's declared NIP-85 trusted providers + | [--timeout SECS] (default: active account). + | |Zaps (NIP-57): | zap user USER SATS build a profile zap-request, fetch a BOLT11 | [--comment X] [--anon|--private] invoice from the recipient's LN service @@ -649,11 +689,15 @@ private fun printUsage() { | | marmot reset [--yes] wipe all local MLS/KeyPackage state (destructive) | - |Local event store (`/events-store/`): - | store stat event count, kind histogram, disk usage + |Local event store (shared, under `/shared/`): + | Backend selected by AMY_STORE: sqlite (default; `shared/events.db`) + | or fs (`AMY_STORE=fs`; the `shared/events-store/` tree). SQLite is + | far more compact at scale — the FS tree spends one file per index + | posting, so large crawls balloon on disk. + | store stat event count + disk usage (kind histogram/mtime on fs) | store sweep-expired delete events past their NIP-40 expiration - | store scrub rebuild idx/ from canonical events (after edits / crashes) - | store compact drop dangling idx entries (canonical gone) + | store scrub fs: rebuild idx/ from canonical events; sqlite: no-op + | store compact fs: drop dangling idx entries; sqlite: VACUUM | store reindex-fts rebuild the NIP-50 search index (after a searchable-kinds change) """.trimMargin(), ) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/OperatorKeys.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/OperatorKeys.kt new file mode 100644 index 0000000000..71a7dd6d68 --- /dev/null +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/OperatorKeys.kt @@ -0,0 +1,156 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.amethyst.cli + +import com.fasterxml.jackson.module.kotlin.readValue +import com.vitorpamplona.amethyst.cli.secrets.IdentitySecret +import com.vitorpamplona.amethyst.cli.secrets.SecretStore +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip01Core.core.hexToByteArray +import com.vitorpamplona.quartz.nip01Core.core.toHexKey +import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer +import com.vitorpamplona.quartz.utils.sha256.sha256 +import java.io.File + +/** + * Operator-level signing keys for GrapeRank trusted-assertion publishing. + * + * A machine holds ONE operator master seed, **independent of any amy account**, + * stored under `~/.amy/operator/` through the same [SecretStore] backend the + * accounts use (OS keychain / NIP-49 ncryptsec / plaintext). From it we + * deterministically derive ONE service key per observer: + * + * ``` + * serviceKey(observer) = sha256(masterPriv ‖ "graperank-provider:" ‖ observerHex ‖ counter) + * ``` + * + * That service key signs the observer's kind:30382 rank cards (and their kind:5 + * retractions). Deterministic derivation buys two things: + * - **Stable identity** — the same observer always maps to the same key, so + * re-signing a card *replaces* the prior one (kind:30382 is addressable) + * instead of orphaning it and spamming clients with duplicates. + * - **One-secret backup** — back up only the master seed; every service key is + * re-derivable even if the [providers] manifest is lost. + * + * The manifest (`~/.amy/operator/operator.json`) records the master pubkey, the + * configured operator relay(s), and the observer → provider-pubkey mapping. Only + * the master itself is a secret; it rides the [SecretStore] descriptor, so the + * manifest holds public data. + */ +class OperatorKeys( + amyHome: File, + private val secrets: SecretStore, +) { + private val dir = File(amyHome, DIR_NAME) + private val configFile = File(dir, CONFIG_NAME) + + data class ProviderRecord( + val providerPubKey: HexKey = "", + ) + + data class Config( + val masterPubKey: HexKey = "", + val master: IdentitySecret? = null, + val relays: List = emptyList(), + val providers: MutableMap = mutableMapOf(), + ) + + private fun load(): Config? = if (configFile.exists()) Output.mapper.readValue(configFile.readText()) else null + + private fun save(cfg: Config) { + SecureFileIO.secureMkdirs(dir) + configFile.writeText(Output.mapper.writeValueAsString(cfg)) + SecureFileIO.tighten(configFile) + } + + /** True once an operator master exists on this machine. */ + fun exists(): Boolean = load()?.master != null + + /** Load (or, on first use, create + persist) the operator master private key. */ + private fun masterPriv(): ByteArray { + load()?.master?.let { return secrets.resolve(it).hexToByteArray() } + val kp = KeyPair() + val pub = kp.pubKey.toHexKey() + val secret = secrets.store(pub, kp.privKey!!.toHexKey()) + save(Config(masterPubKey = pub, master = secret)) + System.err.println("[operator] created operator master ${pub.take(8)}… at ${configFile.path}") + return kp.privKey!! + } + + /** The operator master pubkey, creating the master on first use. */ + fun masterPubKey(): HexKey { + masterPriv() + return load()!!.masterPubKey + } + + /** + * The deterministic service key for [observerHex], recording the observer → + * provider-pubkey mapping in the manifest. The counter loop only ever runs + * once in practice — it's a guard for the ~2^-128 chance a sha256 output isn't + * a valid secp256k1 scalar. + */ + fun serviceKey(observerHex: HexKey): KeyPair { + val master = masterPriv() + var counter = 0 + while (true) { + val material = master + "$DERIVATION_LABEL$observerHex:$counter".encodeToByteArray() + val kp = runCatching { KeyPair(privKey = sha256(material)) }.getOrNull() + if (kp?.privKey != null) { + recordProvider(observerHex, kp.pubKey.toHexKey()) + return kp + } + counter++ + } + } + + private fun recordProvider( + observerHex: HexKey, + providerPubKey: HexKey, + ) { + val cfg = load() ?: return + if (cfg.providers[observerHex]?.providerPubKey == providerPubKey) return + cfg.providers[observerHex] = ProviderRecord(providerPubKey) + save(cfg) + } + + /** Relays the operator publishes all its 30382 cards + retractions to. */ + fun operatorRelays(): Set = + load() + ?.relays + .orEmpty() + .mapNotNull { RelayUrlNormalizer.normalizeOrNull(it) } + .toSet() + + fun setRelays(urls: List) { + masterPriv() // make sure the config (and master) exists first + save(load()!!.copy(relays = urls)) + } + + fun providers(): Map = load()?.providers.orEmpty() + + companion object { + private const val DIR_NAME = "operator" + private const val CONFIG_NAME = "operator.json" + private const val DERIVATION_LABEL = "graperank-provider:" + } +} diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/RelayDiagnostics.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/RelayDiagnostics.kt new file mode 100644 index 0000000000..be275a71dd --- /dev/null +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/RelayDiagnostics.kt @@ -0,0 +1,96 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.amethyst.cli + +import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener +import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.AuthMessage +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.ClosedMessage +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.Message +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.NoticeMessage +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicLong + +/** + * Client-wide tally of the relay feedback the crawl would otherwise never see: + * `NOTICE` frames, `CLOSED` reasons (`auth-required` / `rate-limited` / + * `restricted` / …), and NIP-42 `AUTH` challenges. Registered as a + * [RelayConnectionListener] on the shared client, so every incoming message + * during a run is counted and a REQ failure can be explained instead of + * guessed at. + * + * Callbacks fire on the per-relay socket threads, so all state is concurrent. + */ +class RelayDiagnostics : RelayConnectionListener { + private val closedByReason = ConcurrentHashMap() + private val noticeSamples = ConcurrentHashMap() + private val authChallenges = AtomicLong() + + override fun onIncomingMessage( + relay: IRelayClient, + msgStr: String, + msg: Message, + ) { + when (msg) { + // CLOSED reasons follow the NIP-01 machine-readable "word: text" + // convention, so the prefix categorises the failure. + is ClosedMessage -> bump(closedByReason, prefix(msg.message)) + // NOTICE is free-form; keep the (truncated) text so recurring + // relay complaints ("too many concurrent REQs", …) are visible. + is NoticeMessage -> if (noticeSamples.size < MAX_DISTINCT_NOTICES) bump(noticeSamples, msg.message.trim().take(80)) + is AuthMessage -> authChallenges.incrementAndGet() + else -> Unit + } + } + + private fun bump( + map: ConcurrentHashMap, + key: String, + ) { + map.getOrPut(key) { AtomicLong() }.incrementAndGet() + } + + /** The NIP-01 machine-readable prefix (`word` before `:`), or `other`. */ + private fun prefix(message: String): String { + val head = message.substringBefore(':').trim().lowercase() + return head.ifEmpty { "other" }.take(24) + } + + fun hadFeedback(): Boolean = authChallenges.get() > 0 || closedByReason.isNotEmpty() || noticeSamples.isNotEmpty() + + /** JSON-friendly summary for the command output. */ + fun snapshot(): Map = + mapOf( + "auth_challenges" to authChallenges.get(), + "closed_by_reason" to closedByReason.entries.associate { it.key to it.value.get() }.toSortedMap(), + "notices" to noticeSamples.values.sumOf { it.get() }, + "notice_top" to + noticeSamples.entries + .sortedByDescending { it.value.get() } + .take(TOP_NOTICES) + .map { "${it.key} (${it.value.get()})" }, + ) + + companion object { + private const val MAX_DISTINCT_NOTICES = 500 + private const val TOP_NOTICES = 8 + } +} diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/StoreFactory.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/StoreFactory.kt new file mode 100644 index 0000000000..8c75bbf25f --- /dev/null +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/StoreFactory.kt @@ -0,0 +1,89 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.amethyst.cli + +import com.vitorpamplona.quartz.nip01Core.jackson.JacksonMapper +import com.vitorpamplona.quartz.nip01Core.store.IEventStore +import com.vitorpamplona.quartz.nip01Core.store.fs.FsEventStore +import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore +import kotlin.io.path.Path + +/** On-disk backend for the shared event store. */ +enum class StoreBackend { + /** + * Single SQLite database file at [DataDir.eventsDbFile]. Postings live + * in shared B-tree pages, so an event's kind/author/tag indexes cost a + * handful of rows — not one 4 KB-block file each, the way the FS store + * lays them out. For crawl-scale corpora (hundreds of thousands of + * follow lists) this is several times smaller on disk and the default. + */ + SQLITE, + + /** + * Filesystem tree at [DataDir.eventsDir] — one pretty-printed JSON file + * per event plus one file per index posting. Human-inspectable with + * `cat`/`jq`/`git diff`, but every posting rounds up to a filesystem + * block, so a large corpus balloons. Opt in with `AMY_STORE=fs`. + */ + FS, +} + +/** + * Chooses and opens the event-store backend for `amy`. The backend is + * selected by the `AMY_STORE` environment variable and defaults to + * [StoreBackend.SQLITE]; set `AMY_STORE=fs` for the legacy filesystem + * store. Both backends implement [IEventStore], so every command works + * unchanged regardless of the choice — the only user-visible difference + * is where bytes land ([DataDir.eventsDbFile] vs [DataDir.eventsDir]) and + * how much disk they take. + */ +object StoreFactory { + const val ENV = "AMY_STORE" + + /** Resolve the configured backend. Unrecognised values fall back to the default. */ + fun backend(): StoreBackend = + when (System.getenv(ENV)?.trim()?.lowercase()) { + "fs", "file", "files", "filesystem" -> StoreBackend.FS + else -> StoreBackend.SQLITE + } + + /** + * Open the store for [dataDir] using the configured [backend]. Events + * are written pretty-printed on the FS backend so the on-disk JSON stays + * inspection-friendly; the SQLite backend stores the compact NIP-01 + * form internally. Neither is re-used for signature checks (verification + * always re-canonicalises), so the stored representation is purely an + * implementation detail. Callers own [IEventStore.close]. + */ + fun open(dataDir: DataDir): IEventStore = + when (backend()) { + StoreBackend.SQLITE -> { + // BundledSQLiteDriver won't create parent directories. + dataDir.eventsDbFile.parentFile?.mkdirs() + EventStore(dbName = dataDir.eventsDbFile.absolutePath, relay = null) + } + StoreBackend.FS -> + FsEventStore( + root = Path(dataDir.eventsDir.absolutePath), + eventToJson = JacksonMapper::toJsonPretty, + ) + } +} diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt new file mode 100644 index 0000000000..f3f5b21e68 --- /dev/null +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -0,0 +1,755 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.amethyst.cli.commands + +import com.vitorpamplona.amethyst.cli.Args +import com.vitorpamplona.amethyst.cli.Context +import com.vitorpamplona.amethyst.cli.DataDir +import com.vitorpamplona.amethyst.cli.Output +import com.vitorpamplona.amethyst.commons.defaults.Constants +import com.vitorpamplona.amethyst.commons.defaults.DefaultIndexerRelayList +import com.vitorpamplona.quartz.experimental.graperank.GrapeRank +import com.vitorpamplona.quartz.experimental.graperank.GrapeRankDataCrawler +import com.vitorpamplona.quartz.experimental.graperank.GrapeRankParams +import com.vitorpamplona.quartz.experimental.graperank.GrapeRankPublisher +import com.vitorpamplona.quartz.experimental.graperank.TrustGraphBuilder +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip01Core.core.toHexKey +import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair +import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer +import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner +import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerInternal +import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent +import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent +import com.vitorpamplona.quartz.nip09Deletions.DeletionIndex +import com.vitorpamplona.quartz.nip51Lists.muteList.MuteListEvent +import com.vitorpamplona.quartz.nip56Reports.ReportEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.list.TrustProviderListEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.list.serviceProviders +import com.vitorpamplona.quartz.nip85TrustedAssertions.list.tags.ProviderTypes +import com.vitorpamplona.quartz.nip85TrustedAssertions.list.tags.ServiceProviderTag +import com.vitorpamplona.quartz.nip85TrustedAssertions.list.tags.ServiceType +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.ContactCardEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.async +import kotlinx.coroutines.awaitAll +import kotlinx.coroutines.coroutineScope +import kotlin.math.roundToInt + +/** + * `amy graperank [OBSERVER] [flags]` — compute GrapeRank web-of-trust scores. + * + * GrapeRank assigns every user reachable in the follow/mute/report graph a + * subjective trust score in `[0, 1]` from the observer's point of view (the + * observer has full self-trust). It crawls the follow graph outward using the + * outbox model — each user's kind:10002 write relays are located first, then + * their kind:3 / kind:10000 / kind:1984 events are fetched from *their own* + * relays. The crawl is exhaustive: it keeps going, with no user cap, until every + * discovered user's outbox has been checked and their contact list pulled (an + * unreachable outbox is retried a few times), then runs the scoring engine in + * `commons/wot`. + * + * Prints a ranked list (text, or one JSON object under `--json`). With + * `--publish`, results are also published as NIP-85 kind:30382 `ContactCardEvent` + * trusted assertions (one per scored user, `rank = round(score*100)`). + * + * The crawl and the computation are separable, because the crawl persists every + * event it fetches to the store and the score is a pure function over it: + * - `amy graperank sync [OBSERVER]` — network only: crawl the reachable graph's + * kind 3/10000/1984/10002 into the local store. Idempotent and cumulative, so + * run it a few times to make sure everything is loaded. Scores nothing. + * - `amy graperank score [OBSERVER]` — local only: build the graph from the store + * and score (same as bare `--offline`). Instant and param-tunable; repeat with + * different `--rigor`/`--attenuation`/cutoffs without re-crawling. + * - bare `amy graperank [OBSERVER]` — the convenience combo: sync then score. + * + * Sub-verbs complete the NIP-85 provider experience — the discovery layer that + * lets clients find and consume those assertions: + * - `amy graperank register` — advertise a `30382:rank` provider in the + * account's kind:10040 [TrustProviderListEvent] (defaults to self, so a + * provider publishing ranks announces where to find them). + * - `amy graperank providers [USER]` — list a user's trusted providers. + */ +object GrapeRankCommand { + // Broad, big general relays that carry kind:10002 for many users, added to the + // crawler's discovery set to raise the odds of resolving a stranger's outbox. + private val EXTRA_DISCOVERY_RELAYS: Set = + listOf( + "wss://relay.damus.io", + "wss://relay.snort.social", + "wss://offchain.pub", + "wss://nostr.land", + "wss://eden.nostr.land", + ).mapNotNull { RelayUrlNormalizer.normalizeOrNull(it) }.toSet() + + suspend fun dispatch( + dataDir: DataDir, + tail: Array, + ): Int = + // Sub-verbs are explicit words; anything else (npub / hex / nprofile / + // NIP-05, or nothing) is the OBSERVER positional for a score computation. + when (tail.firstOrNull()) { + "register" -> register(dataDir, tail.drop(1).toTypedArray()) + "providers" -> providers(dataDir, tail.drop(1).toTypedArray()) + "operator" -> operator(dataDir, tail.drop(1).toTypedArray()) + "sync" -> sync(dataDir, tail.drop(1).toTypedArray()) + "score" -> run(dataDir, tail.drop(1).toTypedArray(), forceOffline = true) + else -> run(dataDir, tail) + } + + suspend fun run( + dataDir: DataDir, + rest: Array, + forceOffline: Boolean = false, + ): Int { + val args = Args(rest) + val observerArg = args.positionalOrNull(0) + // Crawl to full convergence by default (every reachable user's outbox + // checked). --max-rounds is only a safety backstop; --max-hops bounds the + // follow-graph distance from the observer that we crawl (Brainstorm uses 8). + val limit = args.intFlag("limit", 100) + val minScore = args.flag("min-score")?.toDoubleOrNull() ?: 0.0 + // `graperank score` forces the local (no-network) path; `--offline` does the + // same on the bare command. Either way we build + score from the store only. + val offline = forceOffline || args.bool("offline") + // Crawl tuning (--max-rounds/--max-hops/--timeout/--diagnose/--drain-concurrency) + // is read straight from args by [newCrawler]; only these two are surfaced in + // the result JSON, so keep local copies for that. + val parkTimeoutMs = args.longFlag("park-timeout", 40L) * 1000 + val insertBatch = args.intFlag("insert-batch", 500) + val doPublish = args.bool("publish") + // Publish cutoff: only cards with rank >= this are published; existing + // cards for targets below it (or gone from the graph) are retracted. Rank + // is round(score*100), so 2 drops the ~0.015-and-below barely-trusted tail. + val minRank = args.intFlag("min-rank", 2) + val publishLimit = args.intFlag("publish-limit", 500) + val publishRelaysArg = args.flag("publish-relay") + // Benchmark: build + sign one kind:30382 card per scored user (rank >= + // --min-rank) with a throwaway key and time it, WITHOUT publishing. + // Measures the id-hash + Schnorr-sign cost of emitting the full card set. + val benchSign = args.bool("bench-sign") + + val params = + GrapeRankParams( + attenuation = args.flag("attenuation")?.toDoubleOrNull() ?: GrapeRankParams().attenuation, + rigor = args.flag("rigor")?.toDoubleOrNull() ?: GrapeRankParams().rigor, + ) + + Context.open(dataDir).use { ctx -> + ctx.prepare() + val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex + + // Contact lists stream straight into a compact int-CSR structure as the + // crawl finds them and the Event is discarded, so the whole network fits + // in memory without holding millions of kind:3 objects. + val builder = TrustGraphBuilder() + var contactListsFed = 0 + // Wall time to read + deserialize the contact lists out of the store + // (offline path only; online streams them in during the crawl). + var storeLoadMs: Long? = null + // Crawl telemetry (online path only): rounds, relays contacted, the + // per-hop histogram, and the network-bound download time that dominates a + // from-scratch run. Null on the offline path. + var crawlStats: GrapeRankDataCrawler.Stats? = null + + if (!offline) { + val stats = newCrawler(ctx, args).crawl(observer, builder) + crawlStats = stats + contactListsFed = stats.contactListsFed + reportRelayFeedback(ctx) + } else { + // Offline: stream contact lists from the local store into the graph. + val loadStart = System.nanoTime() + for (event in ctx.store.query(Filter(kinds = listOf(ContactListEvent.KIND)))) { + if (event is ContactListEvent) { + builder.addFollows(event.pubKey, event.verifiedFollowKeySet()) + contactListsFed++ + } + } + storeLoadMs = (System.nanoTime() - loadStart) / 1_000_000 + System.err.println("[graperank] offline: $contactListsFed contact lists from local store in $storeLoadMs ms") + } + + // Mutes + reports come from the store (both paths). Far fewer than contact + // lists, so materialising them is cheap. + for (event in ctx.store.query(Filter(kinds = listOf(MuteListEvent.KIND)))) { + if (event is MuteListEvent) builder.addMutes(event.pubKey, event.linkedPubKeys()) + } + val reportsDeleted = materializeReports(ctx, builder) + + val buildStart = System.nanoTime() + val graph = builder.build() + val buildMs = (System.nanoTime() - buildStart) / 1_000_000 + System.err.println("[graperank] graph built: ${graph.nodeCount} users, ${graph.edgeCount()} edges in $buildMs ms; scoring…") + + // Live scoring progress: fires once per Gauss-Seidel sweep with the + // running node-update count and how many nodes still moved more than the + // convergence delta this sweep — that second number trends to 0, so a + // large graph shows convergence instead of hanging silently. + val scoreStart = System.nanoTime() + var sweeps = 0 + val scores = + GrapeRank(params).compute(graph, observer) { visited, stillMoving -> + sweeps++ + System.err.println("[graperank] scoring sweep $sweeps: $visited node-updates, $stillMoving still moving") + } + + fun rankOf(score: Double) = (score * 100).roundToInt() + + val observerId = graph.idOf(observer) + // Reachable users with positive trust at or above --min-score, high→low. + val rankedIds = ArrayList() + for (id in 0 until graph.nodeCount) { + if (id != observerId && scores[id] > 0.0 && scores[id] >= minScore) rankedIds.add(id) + } + rankedIds.sortByDescending { scores[it] } + val scoringMs = (System.nanoTime() - scoreStart) / 1_000_000 + System.err.println("[graperank] scored ${rankedIds.size} users in $scoringMs ms") + + val hopHistogram = crawlStats?.hopHistogram.orEmpty() + val result = + linkedMapOf( + "observer" to observer, + "crawl_rounds" to (crawlStats?.rounds ?: 0), + "relays_contacted" to (crawlStats?.relaysContacted ?: 0), + "relay_feedback" to if (ctx.relayDiagnostics.hadFeedback()) ctx.relayDiagnostics.snapshot() else null, + "relay_throttling" to if (ctx.relayLimiter.hadThrottling()) ctx.relayLimiter.snapshot() else null, + "max_hop_reached" to (hopHistogram.keys.maxOrNull() ?: 0), + "users_by_hop" to hopHistogram.mapKeys { it.key.toString() }, + "graph_users" to graph.nodeCount, + "graph_edges" to graph.edgeCount(), + "reports_deleted" to reportsDeleted, + "users_scored" to rankedIds.size, + "download_ms" to crawlStats?.downloadMs, + "verify_ms" to crawlStats?.verifyMs, + "insert_ms" to crawlStats?.insertMs, + "events_stored" to crawlStats?.eventsStored, + "insert_batch" to insertBatch, + "park_timeout_ms" to parkTimeoutMs, + "store_load_ms" to storeLoadMs, + "graph_build_ms" to buildMs, + "scoring_ms" to scoringMs, + "scoring_sweeps" to sweeps, + "scores" to + rankedIds.take(limit).map { + mapOf("pubkey" to graph.pubkeyOf(it), "score" to scores[it], "rank" to rankOf(scores[it])) + }, + ) + + if (doPublish) { + // The cards for THIS observer are signed by a dedicated, stable + // per-observer service key derived from the machine's operator + // master (see OperatorKeys) — not the account key. Same key across + // runs means re-signing a card replaces the addressable prior one. + val opKeys = ctx.dataDir.operatorKeys() + val serviceKey = opKeys.serviceKey(observer) + val serviceSigner = NostrSignerInternal(serviceKey) + val providerPubkey = serviceKey.pubKey.toHexKey() + result["provider_pubkey"] = providerPubkey + + // Cards go to the operator's own relay(s), where the whole + // trusted-assertion set lives; --publish-relay overrides. + val relays = + publishRelaysArg + ?.split(",") + ?.mapNotNull { RelayUrlNormalizer.normalizeOrNull(it.trim()) } + ?.toSet() + ?.takeIf { it.isNotEmpty() } + ?: opKeys.operatorRelays() + + if (relays.isEmpty()) { + result["published"] = 0 + result["publish_error"] = "no operator relay configured — run `amy graperank operator relay ` or pass --publish-relay" + } else { + // The scorer's desired card set: every user at or above the rank + // cutoff, as (target, rank). GrapeRankPublisher reconciles this + // against what this provider key already published and upserts / + // retracts the difference. + val publishable = + rankedIds + .filter { rankOf(scores[it]) >= minRank } + .map { graph.pubkeyOf(it) to rankOf(scores[it]) } + + val publisher = GrapeRankPublisher(ctx.store) { event, to -> ctx.publish(event, to) } + val pub = + publisher.reconcileAndPublish( + providerSigner = serviceSigner, + providerPubkey = providerPubkey, + scored = publishable, + relays = relays, + publishLimit = publishLimit, + ) + + result["skipped_unchanged"] = pub.skippedUnchanged + if (pub.truncated > 0) result["publish_truncated"] = pub.truncated + result["published"] = pub.published + result["publish_rejected"] = pub.publishRejected + result["deleted"] = pub.deleted + result["delete_rejected"] = pub.deleteRejected + result["published_kind"] = ContactCardEvent.KIND + result["published_to"] = relays.map { it.url } + + // Help the observer point clients at this provider: publish their + // kind:10040 (30382:rank -> providerPubkey @ operator relay) to + // their outbox — but only when we actually hold their key. + maybePublishObserverProviderList(ctx, observer, providerPubkey, relays.first())?.let { + result["observer_10040"] = it + } + } + } + + if (benchSign) { + // Throwaway key — these cards are for timing only and never leave + // the process, so no real identity signs them. + val tempSigner = NostrSignerInternal(KeyPair()) + val cards = + rankedIds + .filter { rankOf(scores[it]) >= minRank } + .map { graph.pubkeyOf(it) to rankOf(scores[it]) } + val signStart = System.nanoTime() + val signed = signCards(cards, tempSigner) + val signMs = (System.nanoTime() - signStart) / 1_000_000 + val perSec = if (signMs > 0) signed * 1000L / signMs else 0 + System.err.println("[graperank] signed $signed kind:30382 cards in $signMs ms ($perSec/s, temp key, not published)") + result["bench_signed"] = signed + result["bench_sign_ms"] = signMs + } + + Output.emit(result) + return 0 + } + } + + /** + * Configure the outbox-model crawler from the crawl flags on [args] plus the + * account's relay policy. Shared by the bare command and `graperank sync`. + * Relay policy — where a stranger's kind:10002 is found (index/discovery + * aggregators + general defaults) and best-effort general relays that might + * hold content when an outbox is unknown — lives in app code, so the quartz + * crawler takes it injected. + */ + private suspend fun newCrawler( + ctx: Context, + args: Args, + ): GrapeRankDataCrawler { + val discoveryRelays = + ctx.bootstrapRelays() + Constants.eventFinderRelays + DefaultIndexerRelayList + EXTRA_DISCOVERY_RELAYS + val contentFallback = ctx.bootstrapRelays() + Constants.eventFinderRelays + return GrapeRankDataCrawler( + client = ctx.client, + store = ctx.store, + limiter = ctx.relayLimiter, + config = + GrapeRankDataCrawler.Config( + relayListDiscoveryRelays = discoveryRelays, + contentFallbackRelays = contentFallback, + maxRounds = args.intFlag("max-rounds", Int.MAX_VALUE), + maxHops = args.intFlag("max-hops", Int.MAX_VALUE), + timeoutMs = args.longFlag("timeout", 10L) * 1000, + parkTimeoutMs = args.longFlag("park-timeout", 40L) * 1000, + diagnose = args.bool("diagnose"), + insertBatchSize = args.intFlag("insert-batch", 500), + drainConcurrency = args.intFlag("drain-concurrency", 24), + ), + log = { System.err.println(it) }, + ) + } + + /** Echo any relay NOTICE/CLOSED feedback + adaptive throttling the crawl saw. */ + private fun reportRelayFeedback(ctx: Context) { + if (ctx.relayDiagnostics.hadFeedback()) { + System.err.println("[graperank] relay feedback: ${ctx.relayDiagnostics.snapshot()}") + } + if (ctx.relayLimiter.hadThrottling()) { + System.err.println("[graperank] relay throttling: ${ctx.relayLimiter.snapshot()}") + } + } + + /** + * `amy graperank sync [OBSERVER]` — network-only WoT data sync. Crawls the + * reachable follow/mute/report graph into the local store (kind 3/10000/1984/ + * 10002) and reports what it loaded, WITHOUT scoring. Idempotent + cumulative: + * run it a few times to make sure everything is loaded, then `graperank score`. + */ + private suspend fun sync( + dataDir: DataDir, + rest: Array, + ): Int { + val args = Args(rest) + val observerArg = args.positionalOrNull(0) + Context.open(dataDir).use { ctx -> + ctx.prepare() + val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex + // Persist-only crawl: no in-memory graph (null builder); every event + // still lands in the store for a later `score`. + val stats = newCrawler(ctx, args).crawl(observer, null) + reportRelayFeedback(ctx) + Output.emit( + linkedMapOf( + "observer" to observer, + "crawl_rounds" to stats.rounds, + "relays_contacted" to stats.relaysContacted, + "relay_feedback" to if (ctx.relayDiagnostics.hadFeedback()) ctx.relayDiagnostics.snapshot() else null, + "relay_throttling" to if (ctx.relayLimiter.hadThrottling()) ctx.relayLimiter.snapshot() else null, + "max_hop_reached" to (stats.hopHistogram.keys.maxOrNull() ?: 0), + "users_by_hop" to stats.hopHistogram.mapKeys { it.key.toString() }, + "users_discovered" to stats.hopHistogram.values.sum(), + "contact_lists_fed" to stats.contactListsFed, + "download_ms" to stats.downloadMs, + "verify_ms" to stats.verifyMs, + "insert_ms" to stats.insertMs, + "events_stored" to stats.eventsStored, + ), + ) + } + return 0 + } + + /** + * Build + sign one kind:30382 [ContactCardEvent] per (target, rank), fanned + * out across CPU cores (id-hash + Schnorr sign is CPU-bound). The signed + * events are discarded — this only exists to time card generation. Returns + * the number signed. + */ + private suspend fun signCards( + cards: List>, + signer: NostrSigner, + ): Int { + if (cards.isEmpty()) return 0 + val cores = Runtime.getRuntime().availableProcessors().coerceAtLeast(1) + val chunkSize = ((cards.size + cores - 1) / cores).coerceAtLeast(1) + return coroutineScope { + cards + .chunked(chunkSize) + .map { chunk -> + async(Dispatchers.Default) { + for ((target, rank) in chunk) { + ContactCardEvent.create( + targetUser = target, + signer = signer, + publicInitializer = { add(RankTag.assemble(rank)) }, + ) + } + chunk.size + } + }.awaitAll() + .sum() + } + } + + /** + * `amy graperank operator [status | relay … | providers]` + * + * Manage the machine's operator keys used to sign trusted-assertion cards. + * - `status` (default): master pubkey, configured relay(s), provider count. + * - `relay …`: set the operator relay(s) the cards + retractions publish + * to; creates the operator master on first use. + * - `providers`: the observer -> provider-pubkey mapping learned so far. + */ + private fun operator( + dataDir: DataDir, + rest: Array, + ): Int { + val opKeys = dataDir.operatorKeys() + return when (rest.firstOrNull()) { + "relay" -> { + val urls = rest.drop(1).filter { it.isNotBlank() } + val normalized = urls.mapNotNull { RelayUrlNormalizer.normalizeOrNull(it) } + if (normalized.isEmpty()) return Output.error("bad_args", "usage: amy graperank operator relay [ …]") + opKeys.setRelays(urls) + Output.emit(mapOf("master_pubkey" to opKeys.masterPubKey(), "relays" to normalized.map { it.url })) + 0 + } + + "providers" -> { + Output.emit( + mapOf( + "master_pubkey" to if (opKeys.exists()) opKeys.masterPubKey() else null, + "providers" to opKeys.providers().map { (observer, rec) -> mapOf("observer" to observer, "provider_pubkey" to rec.providerPubKey) }, + ), + ) + 0 + } + + null, "status" -> { + if (!opKeys.exists()) { + Output.emit(mapOf("initialized" to false)) + } else { + Output.emit( + mapOf( + "initialized" to true, + "master_pubkey" to opKeys.masterPubKey(), + "relays" to opKeys.operatorRelays().map { it.url }, + "providers" to opKeys.providers().size, + ), + ) + } + 0 + } + + else -> Output.error("bad_args", "unknown operator subcommand '${rest.first()}' (status | relay | providers)") + } + } + + /** + * `amy graperank register [PROVIDER] [--service KIND:TAG] [--relay URL] [--private]` + * + * Add a NIP-85 provider entry to the account's kind:10040 + * [TrustProviderListEvent] — the declaration a client reads to discover which + * key publishes which assertion, and where. Defaults to declaring *self* as + * the `30382:rank` provider at the account's first outbox relay, which is the + * self-advertisement a GrapeRank provider makes so its followers can find the + * cards it publishes. Fetches the freshest list first so existing providers + * are preserved. + */ + private suspend fun register( + dataDir: DataDir, + rest: Array, + ): Int { + val args = Args(rest) + val providerArg = args.positionalOrNull(0) ?: args.flag("provider") + val serviceArg = args.flag("service") + val relayArg = args.flag("relay") + val isPrivate = args.bool("private") + val timeoutMs = args.longFlag("timeout", 8L) * 1000 + + val service = + serviceArg?.let { + ServiceType.parse(it) ?: return Output.error("bad_args", "--service must be KIND:TAG, e.g. 30382:rank") + } ?: ProviderTypes.rank + + Context.open(dataDir).use { ctx -> + ctx.prepare() + val self = ctx.identity.pubKeyHex + val provider = providerArg?.let { ctx.requireUserHex(it) } ?: self + + val outbox = ctx.outboxRelays() + val relay = + relayArg?.let { RelayUrlNormalizer.normalizeOrNull(it) } + ?: outbox.firstOrNull() + ?: return Output.error("no_relays", "no relay hint; pass --relay URL or configure outbox relays") + + val latest = fetchLatestProviderList(ctx, self, outbox, timeoutMs) + val alreadyListed = + latest?.serviceProviders()?.any { + it.service == service && it.pubkey == provider && it.relayUrl == relay + } ?: false + + if (alreadyListed) { + Output.emit( + mapOf( + "service" to service.toValue(), + "provider" to provider, + "relay" to relay.url, + "changed" to false, + "based_on" to latest?.id, + ), + ) + return 0 + } + + val tag = ServiceProviderTag(service, provider, relay) + val event = + if (latest == null) { + TrustProviderListEvent.create(tag, isPrivate = isPrivate, signer = ctx.signer) + } else { + TrustProviderListEvent.add(latest, tag, isPrivate = isPrivate, signer = ctx.signer) + } + + val ack = ctx.publish(event, outbox) + Output.emit( + mapOf( + "service" to service.toValue(), + "provider" to provider, + "relay" to relay.url, + "private" to isPrivate, + "changed" to true, + "event_id" to event.id, + "based_on" to latest?.id, + "published_to" to ack.filterValues { it }.keys.map { it.url }, + "rejected_by" to ack.filterValues { !it }.keys.map { it.url }, + ), + ) + return 0 + } + } + + /** + * `amy graperank providers [USER] [--refresh] [--timeout SECS]` + * + * List the NIP-85 trusted providers a user declares in their kind:10040 + * (default: the active account). Cache-first; falls back to a relay drain on + * a miss or with `--refresh`. For the active account, private (NIP-44) + * provider entries are decrypted and included too. + */ + private suspend fun providers( + dataDir: DataDir, + rest: Array, + ): Int { + val args = Args(rest) + val userArg = args.positionalOrNull(0) + val refresh = args.bool("refresh") + val timeoutMs = args.longFlag("timeout", 8L) * 1000 + + Context.open(dataDir).use { ctx -> + ctx.prepare() + val user = userArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex + val isSelf = user == ctx.identity.pubKeyHex + + var event = if (refresh) null else providerListOf(ctx, user) + if (event == null) { + ctx.drain( + (ctx.bootstrapRelays() + Constants.eventFinderRelays).associateWith { + listOf(Filter(kinds = listOf(TrustProviderListEvent.KIND), authors = listOf(user), limit = 1)) + }, + timeoutMs, + ) + event = providerListOf(ctx, user) + } + + if (event == null) { + Output.emit(mapOf("user" to user, "found" to false, "providers" to emptyList())) + return 0 + } + + val public = event.serviceProviders() + val private = if (isSelf) event.privateTags(ctx.signer)?.serviceProviders().orEmpty() else emptyList() + + fun render( + tag: ServiceProviderTag, + scope: String, + ) = mapOf( + "service" to tag.service.toValue(), + "provider" to tag.pubkey, + "relay" to tag.relayUrl.url, + "scope" to scope, + ) + + Output.emit( + mapOf( + "user" to user, + "found" to true, + "event_id" to event.id, + "created_at" to event.createdAt, + "providers" to public.map { render(it, "public") } + private.map { render(it, "private") }, + ), + ) + return 0 + } + } + + /** Latest known kind:10040 provider list for [pubKey] from the local store. */ + private suspend fun providerListOf( + ctx: Context, + pubKey: HexKey, + ): TrustProviderListEvent? = + ctx.store + .query(Filter(kinds = listOf(TrustProviderListEvent.KIND), authors = listOf(pubKey), limit = 1)) + .firstOrNull() as? TrustProviderListEvent + + /** + * Fetch the freshest kind:10040 for [pubKey] from [relays] so a register + * builds on top of the current provider set instead of clobbering it. + */ + private suspend fun fetchLatestProviderList( + ctx: Context, + pubKey: HexKey, + relays: Set, + timeoutMs: Long, + ): TrustProviderListEvent? { + if (relays.isEmpty()) return providerListOf(ctx, pubKey) + val filter = Filter(kinds = listOf(TrustProviderListEvent.KIND), authors = listOf(pubKey), limit = 1) + ctx.drain(relays.associateWith { listOf(filter) }, timeoutMs) + return providerListOf(ctx, pubKey) + } + + /** + * Feed reports into [builder], dropping any that a valid NIP-09 deletion has + * retracted. Uses quartz's [DeletionIndex] — the same indexer the Android + * app's LocalCache runs — which keys each deletion under the DELETER's pubkey, + * so `hasBeenDeleted(report)` is true only when the report's own author + * deleted it (NIP-09: a deletion is authoritative only from the event's + * author). It also honours created_at ordering. Returns how many were dropped. + */ + private suspend fun materializeReports( + ctx: Context, + builder: TrustGraphBuilder, + ): Int { + val reports = ctx.store.query(Filter(kinds = listOf(ReportEvent.KIND))).filterIsInstance() + if (reports.isEmpty()) return 0 + + // Everything in the store already passed verifyAndStore, so mark the + // deletions as verified and skip the redundant signature check. + val deletions = DeletionIndex() + for (ev in ctx.store.query(Filter(kinds = listOf(DeletionEvent.KIND)))) { + if (ev is DeletionEvent) deletions.add(ev, wasVerified = true) + } + + var dropped = 0 + for (r in reports) { + if (deletions.hasBeenDeleted(r)) { + dropped++ + continue + } + builder.addReports(r.pubKey, r.reportedAuthor().map { it.pubkey }) + } + if (dropped > 0) System.err.println("[graperank] dropped $dropped retracted reports (NIP-09 deletions)") + return dropped + } + + /** + * If the active account IS the observer (so we hold their key), publish/refresh + * their kind:10040 declaring `30382:rank` -> [providerPubkey] at [relay], to + * their own outbox relays — the NIP-85 pointer a client follows to find these + * cards. Returns the 10040 event id, or null when we don't hold the key (a + * third-party observer must add the provider to their 10040 out-of-band). + */ + private suspend fun maybePublishObserverProviderList( + ctx: Context, + observer: HexKey, + providerPubkey: HexKey, + relay: NormalizedRelayUrl, + ): String? { + if (observer != ctx.identity.pubKeyHex) return null + val service = ProviderTypes.rank + val outbox = ctx.outboxRelays() + val latest = fetchLatestProviderList(ctx, observer, outbox, 8_000) + val alreadyListed = + latest?.serviceProviders()?.any { + it.service == service && it.pubkey == providerPubkey && it.relayUrl == relay + } ?: false + if (alreadyListed) return latest?.id + + val tag = ServiceProviderTag(service, providerPubkey, relay) + val event = + if (latest == null) { + TrustProviderListEvent.create(tag, isPrivate = false, signer = ctx.signer) + } else { + TrustProviderListEvent.add(latest, tag, isPrivate = false, signer = ctx.signer) + } + ctx.publish(event, outbox) + return event.id + } +} diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/StoreCommands.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/StoreCommands.kt index 2ec53cf68c..16a84361d4 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/StoreCommands.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/StoreCommands.kt @@ -22,28 +22,39 @@ package com.vitorpamplona.amethyst.cli.commands import com.vitorpamplona.amethyst.cli.DataDir import com.vitorpamplona.amethyst.cli.Output -import com.vitorpamplona.amethyst.cli.StoreStats -import com.vitorpamplona.quartz.nip01Core.jackson.JacksonMapper +import com.vitorpamplona.amethyst.cli.StoreBackend +import com.vitorpamplona.amethyst.cli.StoreFactory +import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter import com.vitorpamplona.quartz.nip01Core.store.IEventStore import com.vitorpamplona.quartz.nip01Core.store.fs.FsEventStore +import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore +import java.io.File +import java.io.IOException import java.nio.file.Files import java.nio.file.Path +import java.util.concurrent.TimeUnit +import kotlin.io.path.exists /** * `amy store ` — direct introspection - * and maintenance of the file-backed event store at - * `/events-store/`. + * and maintenance of the shared event store under `/shared/`. * - * - `stat` total event count, kind histogram, disk bytes, - * mtime range — pure read, no relay traffic. + * The store backend is selected by `AMY_STORE` (SQLite by default, or the + * FS tree with `AMY_STORE=fs` — see [StoreFactory]); each verb adapts to + * whichever is active: + * + * - `stat` total event count, disk bytes, backend, plus (FS only) + * the per-kind histogram and mtime range — pure read, + * no relay traffic. * - `sweep-expired` delete events whose NIP-40 `expiration` tag has * passed (per the store's own sweep logic). Run * from cron / scheduler / `amy` periodically. - * - `scrub` rebuild every `idx/` entry from the canonical - * events. Recovers from partial-write crashes or - * external edits. - * - `compact` drop dangling `idx/` entries whose canonical is - * gone. Cheaper than scrub. + * - `scrub` FS: rebuild every `idx/` entry from the canonical + * events, recovering from partial-write crashes or + * external edits. SQLite: a no-op (indexes are updated + * transactionally and can't drift). + * - `compact` FS: drop dangling `idx/` entries whose canonical is + * gone. SQLite: `VACUUM` the database to reclaim space. * - `reindex-fts` wipe and rebuild only the NIP-50 full-text search * index from the stored events. Run after a quartz * upgrade that changes which kinds are searchable. @@ -66,16 +77,116 @@ object StoreCommands { ), ) - private fun stat(dataDir: DataDir): Int { - val stats = StoreStats.of(dataDir.eventsDir.toPath()) + private suspend fun stat(dataDir: DataDir): Int = + when (StoreFactory.backend()) { + StoreBackend.SQLITE -> sqliteStat(dataDir) + StoreBackend.FS -> fsStat(dataDir) + } + + /** + * SQLite `stat`: total count via `COUNT(*)` and on-disk bytes from the + * DB file plus its `-wal`/`-shm` sidecars. The per-kind histogram and + * mtime range are FS-store concepts (they read the `idx/kind` tree and + * file mtimes), so they're omitted here. + */ + private suspend fun sqliteStat(dataDir: DataDir): Int { + val dbFile = dataDir.eventsDbFile + if (!dbFile.exists()) { + Output.emit( + mapOf( + "backend" to "sqlite", + "events" to 0, + "disk_bytes" to 0L, + "root" to dbFile.absolutePath, + ), + ) + return 0 + } + val count = + EventStore(dbName = dbFile.absolutePath, relay = null).use { store -> + store.count(Filter()) + } + val diskBytes = + listOf("", "-wal", "-shm").sumOf { suffix -> + val f = File(dbFile.absolutePath + suffix) + if (f.isFile) f.length() else 0L + } Output.emit( mapOf( - "events" to stats.events, - "by_kind" to stats.byKind, - "disk_bytes" to stats.diskBytes, - "oldest_at" to stats.oldestAt, - "newest_at" to stats.newestAt, - "root" to stats.root.toString(), + "backend" to "sqlite", + "events" to count, + "disk_bytes" to diskBytes, + "root" to dbFile.absolutePath, + ), + ) + return 0 + } + + private fun fsStat(dataDir: DataDir): Int { + val storeRoot = dataDir.eventsDir.toPath() + if (!storeRoot.exists()) { + Output.emit( + mapOf( + "events" to 0, + "by_kind" to emptyMap(), + "disk_bytes" to 0L, + "oldest_at" to null, + "newest_at" to null, + "root" to storeRoot.toAbsolutePath().toString(), + ), + ) + return 0 + } + + val eventsRoot = storeRoot.resolve("events") + var count = 0L + var oldest: Long? = null + var newest: Long? = null + if (Files.isDirectory(eventsRoot)) { + Files.walk(eventsRoot).use { stream -> + for (p in stream) { + if (!Files.isRegularFile(p)) continue + if (!p.fileName.toString().endsWith(".json")) continue + count++ + val mt = + try { + Files.getLastModifiedTime(p).to(TimeUnit.SECONDS) + } catch (_: IOException) { + continue + } + val o = oldest + if (o == null || mt < o) oldest = mt + val n = newest + if (n == null || mt > n) newest = mt + } + } + } + + // Histogram from idx/kind// — for a healthy store this is + // exactly one entry per (kind, event), so summing matches `count`. + // Mismatch points at index drift; run `amy store scrub` to fix. + val kindRoot = storeRoot.resolve("idx/kind") + val byKind = sortedMapOf() + if (Files.isDirectory(kindRoot)) { + Files.list(kindRoot).use { stream -> + for (kindDir in stream) { + if (!Files.isDirectory(kindDir)) continue + val n = Files.list(kindDir).use { it.count() } + byKind[kindDir.fileName.toString()] = n + } + } + } + + val diskBytes = walkSize(storeRoot) + + Output.emit( + mapOf( + "events" to count, + "by_kind" to byKind, + "disk_bytes" to diskBytes, + "oldest_at" to oldest, + "newest_at" to newest, + "root" to storeRoot.toAbsolutePath().toString(), ), ) return 0 @@ -83,37 +194,65 @@ object StoreCommands { private suspend fun sweepExpired(dataDir: DataDir): Int = withStore(dataDir) { store -> - val expiresAtDir = dataDir.eventsDir.toPath().resolve("idx/expires_at") - val before = countEntries(expiresAtDir) - store.deleteExpiredEvents() - val after = countEntries(expiresAtDir) - Output.emit( - mapOf( - "swept" to (before - after).coerceAtLeast(0L), - "remaining" to after, - ), - ) + if (store is FsEventStore) { + // The FS store exposes its expiration index as a directory, + // so we can report exactly how many entries the sweep cleared. + val expiresAtDir = dataDir.eventsDir.toPath().resolve("idx/expires_at") + val before = countEntries(expiresAtDir) + store.deleteExpiredEvents() + val after = countEntries(expiresAtDir) + Output.emit( + mapOf( + "swept" to (before - after).coerceAtLeast(0L), + "remaining" to after, + ), + ) + } else { + store.deleteExpiredEvents() + Output.emit(mapOf("ok" to true)) + } 0 } - private fun scrub(dataDir: DataDir): Int = + private suspend fun scrub(dataDir: DataDir): Int = withStore(dataDir) { store -> - store.scrub() - Output.emit(mapOf("ok" to true)) + when (store) { + is FsEventStore -> { + store.scrub() + Output.emit(mapOf("ok" to true)) + } + // SQLite indexes are written in the same transaction as the + // event, so they can't drift the way the FS `idx/` tree can — + // there is nothing to rebuild. + else -> + Output.emit( + mapOf( + "ok" to true, + "note" to "scrub is a no-op for the sqlite backend (indexes update transactionally)", + ), + ) + } 0 } - private fun compact(dataDir: DataDir): Int = + private suspend fun compact(dataDir: DataDir): Int = withStore(dataDir) { store -> - store.compact() + when (store) { + // FS: drop dangling idx/ postings. SQLite: VACUUM to rebuild + // the file and hand freed pages back to the OS. + is FsEventStore -> store.compact() + is EventStore -> store.store.vacuum() + else -> Unit + } Output.emit(mapOf("ok" to true)) 0 } private suspend fun reindexFts(dataDir: DataDir): Int = withStore(dataDir) { store -> + val fsBacked = store is FsEventStore val ftsDir = dataDir.eventsDir.toPath().resolve("idx/fts") - val before = countEntries(ftsDir) + val before = if (fsBacked) countEntries(ftsDir) else 0L // Drive the resumable, batched path to completion so a huge // store is processed without holding the writer lock for the // whole pass. A real long-running caller would persist the @@ -127,35 +266,34 @@ object StoreCommands { processed += progress.processedThisBatch batches++ } while (!progress.done) - val after = countEntries(ftsDir) - Output.emit( - mapOf( + val out = + linkedMapOf( "ok" to true, "processed" to processed, "batches" to batches, - "tokens_before" to before, - "tokens_after" to after, - ), - ) + ) + if (fsBacked) { + // Token-file counts are an FS-store notion (idx/fts is a + // directory); the SQLite FTS index doesn't expose one. + out["tokens_before"] = before + out["tokens_after"] = countEntries(ftsDir) + } + Output.emit(out) 0 } /** * Maintenance verbs only need the store — not identity, not relays, - * not the signer. Skip [Context.open] (which throws if no identity - * has been bootstrapped) and construct the [FsEventStore] directly - * from [DataDir.eventsDir]. Pretty formatter matches what the rest - * of the CLI uses for inspection-friendly output. + * not the signer. Skip [Context.open] (which throws if no identity has + * been bootstrapped) and open the configured backend directly via + * [StoreFactory], so `amy store` acts on whichever store the rest of + * the CLI is using. */ - private inline fun withStore( + private suspend fun withStore( dataDir: DataDir, - body: (FsEventStore) -> Int, + body: suspend (IEventStore) -> Int, ): Int { - val store = - FsEventStore( - root = dataDir.eventsDir.toPath(), - eventToJson = JacksonMapper::toJsonPretty, - ) + val store = StoreFactory.open(dataDir) try { return body(store) } finally { @@ -163,6 +301,23 @@ object StoreCommands { } } + private fun walkSize(root: Path): Long { + if (!Files.exists(root)) return 0L + var total = 0L + Files.walk(root).use { stream -> + for (p in stream) { + if (!Files.isRegularFile(p)) continue + total += + try { + Files.size(p) + } catch (_: IOException) { + 0L + } + } + } + return total + } + private fun countEntries(dir: Path): Long { if (!Files.isDirectory(dir)) return 0L return Files.list(dir).use { it.count() } diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRank.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRank.kt new file mode 100644 index 0000000000..3744d52b41 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRank.kt @@ -0,0 +1,168 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import androidx.compose.runtime.Immutable +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import kotlin.math.abs +import kotlin.math.exp +import kotlin.math.ln + +/** + * Tunable GrapeRank parameters. Defaults mirror the reference implementation at + * and NosFabrica's Brainstorm + * `DEFAULT` preset. + */ +@Immutable +data class GrapeRankParams( + val attenuation: Double = 0.85, + val rigor: Double = 0.5, + val directFollowConfidence: Double = 0.5, + val indirectFollowConfidence: Double = 0.03, + val muteConfidence: Double = 0.5, + val reportConfidence: Double = 0.5, + val convergence: Double = 0.0001, +) + +/** + * GrapeRank — a subjective, observer-centric web-of-trust score in `[0, 1]` for + * every user reachable from an observer in a [TrustGraph]. See the algorithm + * notes in `TrustGraph`/`GrapeRankTest`; this is the single-observer + * Gauss-Seidel form, operating on the compact int-CSR graph so it scales to the + * whole network. + * + * [compute] returns a `DoubleArray` indexed by node id (`graph.idOf(pubkey)`), + * not a map — at millions of nodes a boxed map would dwarf the graph itself. The + * observer's own entry stays pinned at `1.0`; callers rank the others. + */ +class GrapeRank( + val params: GrapeRankParams = GrapeRankParams(), +) { + private val rigidity = -ln(params.rigor) + + /** Exponential saturation turning accumulated weight into a confidence in `[0, 1)`. */ + private fun weightToConfidence(weight: Double): Double = 1.0 - exp(-weight * rigidity) + + private fun confidence( + relationCode: Int, + sourceIsObserver: Boolean, + ): Double = + when (relationCode) { + TrustRelation.FOLLOW.code -> if (sourceIsObserver) params.directFollowConfidence else params.indirectFollowConfidence + TrustRelation.MUTE.code -> params.muteConfidence + else -> params.reportConfidence + } + + private fun rating(relationCode: Int): Double = + when (relationCode) { + TrustRelation.FOLLOW.code -> TrustRelation.FOLLOW.rating + TrustRelation.MUTE.code -> TrustRelation.MUTE.rating + else -> TrustRelation.REPORT.rating + } + + /** + * Score every node reachable from [observer]. Returns scores by node id, or an + * all-zero array if the observer isn't in the graph. [onProgress] fires once + * per sweep with `(totalNodeUpdates, nodesStillMoving)` — the second value is + * how many nodes moved more than [GrapeRankParams.convergence] this sweep, so + * it trends to 0 as the graph settles. + * + * Iterates synchronous **Gauss-Seidel** sweeps over every node, updating scores + * in place so a value computed earlier in a sweep is already visible to nodes + * later in the same sweep (this converges faster than a double-buffered Jacobi + * pass). A sweep that moves no node by more than the convergence delta ends the + * loop — the same per-node threshold and fixed point as NosFabrica's Brainstorm + * reference. On a dense graph this is far less total work than a + * change-propagating worklist: a worklist re-visits a node once per rater whose + * score nudges, so its cost scales with the in-degree of the churning core, + * whereas a sweep touches each node exactly once per iteration. Attenuation < 1 + * makes the update a contraction, so the fixed point is unique regardless of + * sweep order; ids run in roughly BFS order from the observer, which lets + * trust flow outward within a single sweep and keeps the iteration count low. + * + * Nodes unreachable from the observer settle to 0 for free: all of their raters + * stay at 0, so the inner loop's `sourceScore != 0.0` guard skips every edge. + */ + fun compute( + graph: TrustGraph, + observer: HexKey, + onProgress: ((visited: Long, queued: Int) -> Unit)? = null, + ): DoubleArray { + val n = graph.nodeCount + val scores = DoubleArray(n) + val observerId = graph.idOf(observer) + if (observerId < 0) return scores + + scores[observerId] = 1.0 + + val attenuation = params.attenuation + val convergence = params.convergence + val inOffsets = graph.inOffsets + val inPacked = graph.inPacked + + var visited = 0L + while (true) { + var stillMoving = 0 + var target = 0 + while (target < n) { + if (target != observerId) { + var sumOfWeights = 0.0 + var sumOfWeightedRatings = 0.0 + var i = inOffsets[target] + val end = inOffsets[target + 1] + while (i < end) { + val packed = inPacked[i] + val source = packed and TrustGraph.SOURCE_MASK + val sourceScore = scores[source] + if (sourceScore != 0.0) { + val relationCode = packed ushr TrustGraph.SOURCE_BITS + val weight = confidence(relationCode, source == observerId) * sourceScore * attenuation + sumOfWeights += weight + sumOfWeightedRatings += weight * rating(relationCode) + } + i++ + } + + val newScore = + if (abs(sumOfWeights) < 0.00001) { + 0.0 + } else { + val s = weightToConfidence(sumOfWeights) * sumOfWeightedRatings / sumOfWeights + if (s > 0.0) s else 0.0 + } + + val oldScore = scores[target] + if (newScore != oldScore) { + scores[target] = newScore + if (abs(newScore - oldScore) > convergence) stillMoving++ + } + visited++ + } + target++ + } + + onProgress?.invoke(visited, stillMoving) + if (stillMoving == 0) break + } + + return scores + } +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankDataCrawler.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankDataCrawler.kt new file mode 100644 index 0000000000..682c633066 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankDataCrawler.kt @@ -0,0 +1,1169 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip01Core.crypto.verify +import com.vitorpamplona.quartz.nip01Core.relay.client.NostrClient +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.AdaptiveRelayLimiter +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.DrainFailure +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.classifyDrainFailure +import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener +import com.vitorpamplona.quartz.nip01Core.relay.client.single.newSubId +import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.nip01Core.store.IEventStore +import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent +import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent +import com.vitorpamplona.quartz.nip51Lists.muteList.MuteListEvent +import com.vitorpamplona.quartz.nip56Reports.ReportEvent +import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent +import com.vitorpamplona.quartz.utils.Log +import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap +import com.vitorpamplona.quartz.utils.concurrent.ConcurrentSet +import kotlinx.coroutines.CompletableDeferred +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.SupervisorJob +import kotlinx.coroutines.async +import kotlinx.coroutines.awaitAll +import kotlinx.coroutines.cancel +import kotlinx.coroutines.channels.Channel +import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.delay +import kotlinx.coroutines.joinAll +import kotlinx.coroutines.launch +import kotlinx.coroutines.selects.select +import kotlinx.coroutines.withTimeoutOrNull +import kotlin.concurrent.atomics.AtomicLong +import kotlin.concurrent.atomics.ExperimentalAtomicApi +import kotlin.coroutines.coroutineContext +import kotlin.time.TimeSource + +/** + * Crawls the Nostr follow/mute/report graph outward from an observer and streams + * the contact lists it finds into a [TrustGraphBuilder], so [GrapeRank] can score + * the whole reachable network from that observer's point of view. + * + * It uses the outbox model: each user's kind:10002 write relays are located + * first, then their kind:3 / kind:10000 / kind:1984 events are fetched from + * *their own* relays. The crawl is exhaustive — no user cap; it keeps going until + * every discovered user's outbox has been checked and their contact list pulled + * (an unreachable outbox is retried a few times), bounded only by [Config.maxHops] + * (follow-graph distance) and the [Config.maxRounds] safety backstop. + * + * Every event it fetches (contact lists, mute lists, reports, relay lists, and + * the report deletions it looks up) is verified and persisted to [store], so the + * caller can materialize mutes + reports (honouring NIP-09 retractions) from the + * store afterwards. Only the contact lists are streamed into the [TrustGraphBuilder] + * during the crawl — the compact int-CSR structure keeps the whole network in + * memory without holding millions of kind:3 objects. + * + * The crawler is transport-agnostic within quartz: it takes a [NostrClient], an + * [IEventStore], and the shared [AdaptiveRelayLimiter] (which must already be + * registered as a connection listener on the client so its ladders react to + * NOTICE/CLOSED frames). Relay *policy* — which aggregators know kind:10002, which + * general relays might hold content — is injected via [Config], because those + * defaults live in application code, not the protocol library. Operator progress + * is emitted through [log]; a headless caller routes it to stderr, a UI ignores it. + */ +@OptIn(ExperimentalAtomicApi::class) +class GrapeRankDataCrawler( + private val client: NostrClient, + private val store: IEventStore, + private val limiter: AdaptiveRelayLimiter, + private val config: Config, + private val log: (String) -> Unit = {}, +) { + // Crawl-wide timing, accumulated across every drainGated consumer (24 run at + // once). Nanoseconds spent verifying signatures vs. spent in the store write, + // plus how many verified events reached the store. Surfaced in [Stats] so a + // caller can see whether a from-scratch crawl is verify-, write-, or (by + // subtraction from wall time) network-bound. Reset at the top of each [crawl]. + private val verifyNanos = AtomicLong(0) + private val insertNanos = AtomicLong(0) + private val eventsStored = AtomicLong(0) + + /** + * Relay policy + crawl bounds. The relay sets come from the caller because the + * aggregator/bootstrap defaults live outside quartz. + * + * @param relayListDiscoveryRelays where to look up a stranger's kind:10002 — + * the index/discovery aggregators (purplepag.es, coracle, …) plus general + * defaults that carry kind:10002 for most of the network. + * @param contentFallbackRelays best-effort general relays that *might* hold a + * user's kind:3/10000/1984 when their outbox is unknown or unreachable. + * @param maxRounds safety backstop on freshness passes (default: run to convergence). + * @param maxHops follow-graph distance from the observer to crawl (Brainstorm uses 8). + * @param timeoutMs the FAST per-drain timeout that gates a round's progression. + * A relay that reaches EOSE/CLOSED inside it resolves its authors this round; + * one still streaming is not cut but PARKED (see [parkTimeoutMs]) so the round + * moves on without waiting for it. Keep this short — it is the round cadence. + * @param parkTimeoutMs how long a parked (slow-but-alive) relay is allowed to + * keep delivering after it blew [timeoutMs]. Its late events are persisted and + * its late contact lists folded into the graph in a later round, so the crawl + * waits for slow relays for completeness WITHOUT paying that wait in the + * round's wall-clock. Parked sockets are bounded by the slow-relay population, + * not the whole fan-out. Set `<= timeoutMs` to disable parking. + * @param diagnose log a breakdown of slow/unreachable relays on each drain timeout. + * @param insertBatchSize how many verified events to group-commit per + * [IEventStore.batchInsert]. The outbox model streams the same events from + * many relays through a single SQLite writer, so batching amortizes the + * per-transaction + writer-mutex cost across the batch (coerced to `>= 1`). + * @param drainConcurrency how many outbox batches drain at once (the worker + * pool size). A GLOBAL bound (memory / open sockets); the per-relay + * concurrent-sub cap is enforced separately by [AdaptiveRelayLimiter]. Keep it + * moderate: a higher global fan-out re-floods busy hubs faster than demotion + * catches up (an A/B at 64 ran ~2x slower with more dead relays), so 24 is the + * validated default and raising it is a probe, not a speedup. + */ + class Config( + val relayListDiscoveryRelays: Set, + val contentFallbackRelays: Set, + val maxRounds: Int = Int.MAX_VALUE, + val maxHops: Int = Int.MAX_VALUE, + val timeoutMs: Long = 10_000, + val parkTimeoutMs: Long = 40_000, + val diagnose: Boolean = false, + val insertBatchSize: Int = 500, + val drainConcurrency: Int = 24, + ) + + /** What the crawl fetched — the counters the caller reports and the graph is built from. */ + class Stats( + val rounds: Int, + val contactListsFed: Int, + val relaysContacted: Int, + /** Users bucketed by follow-graph distance from the observer (hop -> count), ascending. */ + val hopHistogram: Map, + val downloadMs: Long, + /** Wall time verifying signatures, summed across the concurrent consumers. */ + val verifyMs: Long, + /** Wall time in the store write path, summed across the concurrent consumers. */ + val insertMs: Long, + /** Verified events handed to the store (duplicates included — the write path dedups). */ + val eventsStored: Long, + ) + + /** + * Crawl from [observer], streaming discovered contact lists into [builder] + * (follows only — mutes/reports land in the store for the caller to + * materialize). Pass `null` for a persist-only *sync*: every event still lands + * in the store, the frontier still expands off each contact list, but no graph + * is assembled in memory (the caller scores later from the store). Returns [Stats]. + */ + suspend fun crawl( + observer: HexKey, + builder: TrustGraphBuilder?, + ): Stats { + verifyNanos.store(0) + insertNanos.store(0) + eventsStored.store(0) + return CrawlRun(observer, builder).run() + } + + /** + * Holds all per-crawl mutable state. Graph state (done/hopOf/builder/ + * writeRelayFreq/liveRelays/relaysContacted) is single-writer by construction + * — Phase A and the Phase-B consumer never run concurrently, and routeByOutbox + * (the only Phase-B producer write, to writeRelayFreq) touches a disjoint field + * — so those stay plain collections. The frontier IS [hopOf]'s key set: a user + * is "discovered" iff it has a hop stamp. Only the state genuinely shared across + * the producer / consumer / drain-worker coroutines is concurrent: relayHints, + * attempts, deadRelays, relayStrikes. + */ + private inner class CrawlRun( + val observer: HexKey, + val builder: TrustGraphBuilder?, + ) { + // hop distance per discovered user; the observer seeds it at 0. Its key set + // is the discovered frontier — no separate `discovered` set to keep in sync. + val hopOf = hashMapOf(observer to 0) + val done = hashSetOf() + val relaysContacted = hashSetOf() + val writeRelayFreq = HashMap() + val liveRelays = hashSetOf() + + // Concurrent: touched by more than one of producer/consumer/drain-workers. + val relayHints = ConcurrentMap>() + val attempts = ConcurrentMap() + val deadRelays = ConcurrentSet() + val relayStrikes = ConcurrentMap() + + // Crawl-wide dedup of event ids, shared across all concurrent drains and + // every round. The outbox model mirrors the SAME event (especially kind:10002 + // relay lists) across many relays, indexers, and rounds; a per-drain set only + // catches the copies within one drain, so without this the majority of events + // would be re-verified + re-inserted (hitting the store's UNIQUE constraint) + // in a later drain. An id is added only AFTER it verifies, so a forged copy + // (valid id, bad signature) delivered first can't suppress the genuine one. + val seenIds = ConcurrentSet() + + // Per-user relays that answered (EOSE'd) without holding this user's kind:3, + // so re-querying them for this user is guaranteed-empty waste. routeByOutbox + // subtracts these from a user's candidate relays, so a straggler is retried + // only against relays that could plausibly still have it (never-asked, or + // ones that timed out — which unlike a clean EOSE might just be slow). + val askedEmpty = ConcurrentMap>() + + // Contact lists delivered LATE by parked (slow-but-alive) relays. A parked + // unit persists its events, then pushes any kind:3 it found here; the round + // loop (the single graph-writer) folds these into hopOf/done/builder between + // rounds, so a slow relay's follows still expand the frontier — just a round + // or two later than the fast ones. Unbounded: parked delivery must never + // block on the round loop draining it. + val lateHarvest = Channel>(Channel.UNLIMITED) + + // Parked units still streaming. The crawl isn't done until this hits 0 (and + // the frontier is empty), so we wait for slow relays' completeness without + // gating each round on them. Incremented when a unit parks, decremented when + // it finishes (or its park window elapses). + val parkedInFlight = AtomicLong(0) + + // Background scope owning the parked subscriptions (and Tier-2 relay-list + // sweeps). Set in [run]; cancelled once the crawl converges. + var bgScope: CoroutineScope? = null + + var rounds = 0 + var contactListsFed = 0 + + // Live-progress context the heartbeat ticker reads (plain vars set only by the + // single round-loop coroutine; the ticker's reads are benign racy int/bool + // reads — a stale value just shows in one progress line). progTarget/progBase + // frame the CURRENT round so the ticker can show a real "X of Y (Z%)" for it. + var progRound = 0 + var progTarget = 0 + var progBaseDone = 0 + var progConverging = false + + /** + * A relay that HARD-failed (bad domain, TLS misconfig, dead HTTP code) is + * dropped on the first strike: it will not fix itself. A TRANSIENT failure + * (refused/reset/unreachable, or a 429/5xx) might clear, so it takes + * MAX_DEAD_STRIKES before we give up. Pure timeouts never reach here — the + * drain treats them as busy-retry and does not report them dead at all. + */ + fun recordDead(failed: Map) { + for ((r, kind) in failed) { + when (kind) { + DrainFailure.HARD -> deadRelays.add(r) + DrainFailure.TRANSIENT -> + if (relayStrikes.merge(r, 1) { a, b -> a + b } >= MAX_DEAD_STRIKES) deadRelays.add(r) + } + } + } + + /** The busiest live relays we've learned, excluding the dead ones. */ + fun topLiveRelays(cap: Int): List = + writeRelayFreq.entries + .asSequence() + .filter { it.key in liveRelays && it.key !in deadRelays } + .sortedByDescending { it.value } + .take(cap) + .map { it.key } + .toList() + + /** + * Feed a user's contact list into the graph, harvest relay hints, stamp + * the hop distance of newly-seen follows, and add them to the frontier. + * Called once per user (guarded by `done`). Returns the count of + * newly-discovered users. + */ + fun ingest( + source: HexKey, + contacts: ContactListEvent, + ): Int { + val nextHop = (hopOf[source] ?: 0) + 1 + val follows = ArrayList() + var fresh = 0 + for (tag in contacts.follows()) { + follows.add(tag.pubKey) + tag.relayUri?.let { relayHints.getOrPut(tag.pubKey) { ConcurrentSet() }.add(it) } + if (tag.pubKey !in hopOf) { + hopOf[tag.pubKey] = nextHop + fresh++ + } + } + builder?.addFollows(source, follows) + contactListsFed++ + return fresh + } + + /** + * Feed into the graph the contact lists a drain just returned (deduped by + * author; the store's canonical latest wins), marking fed authors done. + * Only the authors we actually received are touched — no scan over the + * whole still-missing set. Returns the count newly fed. + */ + suspend fun harvest(events: List>): Int { + var got = 0 + for ((_, ev) in events) { + if (ev !is ContactListEvent) continue + val pk = ev.pubKey + if (pk in done) continue + val contacts = contactsOf(pk) ?: continue + done += pk + ingest(pk, contacts) + got++ + } + return got + } + + /** + * Sharded backbone sweep (see SHARD_RELAYS). Splits the missing authors + * across the top live relays — one shard per relay, so no relay gets the + * same list twice — drains all shards concurrently, then rotates whoever's + * still missing onto a different relay for up to SHARD_ROTATIONS passes. + * Once the remainder is small it's cheap to broadcast it to every top relay + * at once. Returns lists fed. + */ + suspend fun shardedSweep(authors: Collection): Int { + val top = topLiveRelays(SHARD_RELAYS) + if (top.isEmpty()) return 0 + val n = top.size + var missing = authors.filter { it !in done && contactsOf(it) == null } + var got = 0 + var rotation = 0 + while (missing.size > SHARD_BROADCAST_THRESHOLD && rotation < SHARD_ROTATIONS) { + val shards = Array(n) { ArrayList() } + for (pk in missing) { + val base = ((pk.hashCode() % n) + n) % n + shards[(base + rotation) % n].add(pk) + } + val results = + coroutineScope { + top + .mapIndexedNotNull { i, relay -> + val shard = shards[i] + if (shard.isEmpty()) { + null + } else { + // Each drain gets its own dead-set — the concurrent + // drains must not share a mutable HashMap. + async { + val dead = HashMap() + val filters = + mapOf(relay to shard.chunked(AUTHORS_PER_FILTER).map { Filter(kinds = FETCH_KINDS, authors = it) }) + drainGated(filters, dead) to dead + } + } + }.awaitAll() + } + for ((_, dead) in results) recordDead(dead) + relaysContacted += top + val flat = results.flatMap { it.first } + for ((relay, _) in flat) liveRelays.add(relay) + got += harvest(flat) + missing = missing.filter { it !in done } + rotation++ + } + // Once the remainder is small it's cheap to ask every top relay for it + // at once. If the rotations bailed with a still-large set, those authors + // just aren't on the popular relays — leave them to the caller's outbox + // pass rather than broadcast a huge list. + if (missing.isNotEmpty() && missing.size <= SHARD_BROADCAST_THRESHOLD) { + // Broadcast the small remainder to a wider set of busy relays than + // the rotation used — recovers users whose list is only on a relay + // ranked below the top SHARD_RELAYS. + val live = topLiveRelays(BROADCAST_RELAYS) + if (live.isNotEmpty()) { + val dead = HashMap() + val filters = + live.associateWith { missing.chunked(AUTHORS_PER_FILTER).map { Filter(kinds = FETCH_KINDS, authors = it) } } + val events = drainGated(filters, dead) + recordDead(dead) + relaysContacted += live + for ((relay, _) in events) liveRelays.add(relay) + got += harvest(events) + } + } + return got + } + + /** + * Fetch kind:10002 relay lists for any [pubkeys] we don't already know, so + * [routeByOutbox] can route their content query to their own write relays. + * + * Tier 1 queries the bounded relay-list discovery set (indexers + general + * defaults), which aggregate kind:10002 for the whole network. Blocking, + * because this round's routing needs the result. + * + * Tier 2 is a completeness net for the stragglers the indexers don't cover: + * cast the widest net — every relay we've seen deliver events. Fired + * fire-and-forget on [bgScope]: a stray 10002 might sit on any one relay, so + * we don't skip any, but we can't block the crawl on a fan-out that large. + * The results land in the store and improve routing for later rounds. + */ + suspend fun ensureRelayLists( + pubkeys: Set, + allLiveRelays: Set, + bgScope: CoroutineScope, + ) { + val missing = pubkeys.filter { relaysOf(it) == null } + if (missing.isEmpty()) return + + suspend fun query( + authors: List, + relays: Set, + ) { + if (relays.isEmpty() || authors.isEmpty()) return + val filters = + relays.associateWith { + authors.chunked(AUTHORS_PER_FILTER).map { chunk -> + Filter(kinds = listOf(AdvertisedRelayListEvent.KIND), authors = chunk) + } + } + drainGated(filters, null) + } + + val discovery = config.relayListDiscoveryRelays + query(missing, discovery) + + val stillMissing = missing.filter { relaysOf(it) == null } + val wide = allLiveRelays - discovery + if (stillMissing.isNotEmpty() && wide.isNotEmpty()) { + bgScope.launch { query(stillMissing, wide) } + } + } + + /** + * Fetch NIP-09 kind:5 deletion requests that retract any report we gathered. + * A reporter can delete their own kind:1984 report — a deletion valid only + * from the reporter's own key, published to the reporter's outbox. So we + * group report ids by their author and ask each author's write relays for + * kind:5 events that cite those ids (`#e`), pulling only the deletions that + * touch our reports. The events land in the store for the caller to apply. + */ + suspend fun fetchReportDeletions(backbone: Set) { + val idsByAuthor = HashMap>() + for (ev in store.query(Filter(kinds = listOf(ReportEvent.KIND)))) { + if (ev is ReportEvent) idsByAuthor.getOrPut(ev.pubKey) { ArrayList() }.add(ev.id) + } + if (idsByAuthor.isEmpty()) return + + // Route each reporter to their own write relays (fallback: backbone). + val perRelayAuthors = HashMap>() + for (author in idsByAuthor.keys) { + val write = relaysOf(author)?.writeRelaysNorm()?.takeIf { it.isNotEmpty() } ?: backbone + for (relay in write) if (relay !in deadRelays) perRelayAuthors.getOrPut(relay) { HashSet() }.add(author) + } + if (perRelayAuthors.isEmpty()) return + + val filters = + perRelayAuthors.mapValues { (_, authors) -> + buildList { + for (authorChunk in authors.chunked(AUTHORS_PER_FILTER)) { + // Scope #e to this author-chunk's own report ids, chunked to + // respect REQ limits. Any over-match (a filter pairing an + // author with another author's id) is harmless — the + // deleter-must-be-author check the caller runs rejects it. + val chunkIds = authorChunk.flatMap { idsByAuthor[it].orEmpty() } + for (idChunk in chunkIds.chunked(AUTHORS_PER_FILTER)) { + add(Filter(kinds = listOf(DeletionEvent.KIND), authors = authorChunk, tags = mapOf("e" to idChunk))) + } + } + } + } + drainGated(filters, null) + } + + /** + * Group [pubkeys] by the relays we should query for their events: + * - first try: the user's own kind:10002 write relays (the outbox model); + * - a retry (`attempts[pk] > 0`, its outbox already failed): outbox + + * [backbone] — the known-good relays other people write to; + * - no outbox at all: harvested hints + backbone + the general fallback. + * + * Also tallies each user's write relays into [writeRelayFreq] so the + * backbone can be learned from the crawl. Authors are chunked per relay. + */ + suspend fun routeByOutbox( + pubkeys: Set, + backbone: Set, + ): Map> { + val fallback = config.contentFallbackRelays + val perRelay = HashMap>() + + for (pk in pubkeys) { + val write = relaysOf(pk)?.writeRelaysNorm()?.takeIf { it.isNotEmpty() } + write?.forEach { writeRelayFreq[it] = (writeRelayFreq[it] ?: 0) + 1 } + val relays = + when { + write == null -> relayHints[pk]?.snapshot().orEmpty() + backbone + fallback + (attempts[pk] ?: 0) > 0 -> write + backbone + else -> write + } + // Skip relays proven dead (routing to them only burns the drain + // timeout) and relays that already EOSE'd without this user's list + // (re-querying them for this user is guaranteed-empty waste). + val emptied = askedEmpty[pk] + for (relay in relays) { + if (relay in deadRelays) continue + if (emptied != null && relay in emptied) continue + perRelay.getOrPut(relay) { HashSet() }.add(pk) + } + } + + return perRelay.mapValues { (_, authors) -> + authors.chunked(AUTHORS_PER_FILTER).map { chunk -> + Filter(kinds = FETCH_KINDS, authors = chunk) + } + } + } + + /** + * Dedup (crawl-wide [seenIds]), verify, and group-commit a unit's events, + * returning the newly-stored ones tagged by relay. Safe to call concurrently + * from many fast drain units AND parked coroutines: an id is added to + * [seenIds] only AFTER a good signature (so a forged copy delivered first + * can't suppress the genuine one), and [ConcurrentSet.add] is an atomic + * test-and-set — two relays mirroring the same event race on it and only the + * winner stores it, so a duplicate never reaches the store's UNIQUE constraint. + * The store serializes the actual writes behind its own single-writer mutex. + */ + private suspend fun persist(events: List>): List> { + if (events.isEmpty()) return emptyList() + val flushAt = config.insertBatchSize.coerceAtLeast(1) + val fresh = ArrayList>() + val buffer = ArrayList(flushAt) + + suspend fun flush() { + if (buffer.isEmpty()) return + val mark = TimeSource.Monotonic.markNow() + store.batchInsert(buffer) + insertNanos.addAndFetch(mark.elapsedNow().inWholeNanoseconds) + eventsStored.addAndFetch(buffer.size.toLong()) + buffer.clear() + } + + for ((relay, event) in events) { + if (event.id in seenIds) continue + val vMark = TimeSource.Monotonic.markNow() + val ok = event.verify() + verifyNanos.addAndFetch(vMark.elapsedNow().inWholeNanoseconds) + if (!ok) { + Log.w("GrapeRankDataCrawler") { "dropped event ${event.id.take(8)} kind=${event.kind} — bad signature" } + continue + } + if (!seenIds.add(event.id)) continue // lost the race to a mirror; it stores it + fresh.add(relay to event) + buffer.add(event) + if (buffer.size >= flushAt) flush() + } + flush() + return fresh + } + + /** + * Wait for a subscription's terminal ([done]: EOSE/CLOSED/cannot), resetting + * the [idleMs] window every time an event pings [activity]. So the wait ends + * with "timeout" only after [idleMs] of actual SILENCE — a relay that keeps + * streaming (however long its result set) is never cut mid-flight; only a + * genuinely stalled one is. Used for the patient park window. + */ + private suspend fun awaitTerminalOrIdle( + done: CompletableDeferred, + activity: Channel, + idleMs: Long, + ): String { + while (true) { + val r = + withTimeoutOrNull(idleMs) { + select { + done.onAwait { it } + activity.onReceive { ACTIVITY } + } + } + when (r) { + null -> return "timeout" // idleMs elapsed with no event and no terminal + ACTIVITY -> Unit // an event arrived — reset the idle window and keep waiting + else -> return r // terminal reason + } + } + } + + /** + * Fold one late-delivered event from a parked relay into the graph. Only the + * round loop calls this (directly or via [foldLateHarvest]), so graph state + * stays single-writer. Returns true if it fed a new contact list. + */ + private suspend fun ingestLate( + relay: NormalizedRelayUrl, + ev: Event, + ): Boolean { + liveRelays.add(relay) + if (ev !is ContactListEvent) return false + val pk = ev.pubKey + // Only authors we actually crawled (in hopOf) and haven't fed yet. A late + // list for an unknown author would get a wrong hop stamp from ingest. + if (pk in done || pk !in hopOf) return false + val contacts = contactsOf(pk) ?: return false + done += pk + ingest(pk, contacts) + return true + } + + /** Drain whatever parked relays have delivered so far. Returns lists fed. */ + private suspend fun foldLateHarvest(): Int { + var got = 0 + while (true) { + val (relay, ev) = lateHarvest.tryReceive().getOrNull() ?: break + if (ingestLate(relay, ev)) got++ + } + return got + } + + /** + * Heartbeat so a long round never goes silent: every [PROGRESS_INTERVAL_MS] + * emit a one-liner with the CURRENT round's completion (a real X/Y % — the + * round's pending set is a known target), a rolling fetch rate + rough ETA for + * it, and live counts (events stored, slow relays parked, live/dead relays). + * Runs for the whole crawl on the background scope; cancelled when it ends. + */ + private suspend fun progressTicker() { + var lastFed = 0 + var lastMark = TimeSource.Monotonic.markNow() + while (true) { + delay(PROGRESS_INTERVAL_MS) + val nowMark = TimeSource.Monotonic.markNow() + val dtMs = (nowMark - lastMark).inWholeMilliseconds.coerceAtLeast(1) + lastMark = nowMark + val fed = contactListsFed + val rate = (fed - lastFed) * 1000L / dtMs // lists/sec over this interval + lastFed = fed + val events = eventsStored.load() + val parked = parkedInFlight.load() + when { + progConverging -> + log( + "[graperank] finishing · ${human(fed.toLong())} lists · ${human(events)} events" + + (if (parked > 0) " · $parked slow relay(s) still delivering" else " · draining"), + ) + progTarget > 0 -> { + val roundDone = (done.size - progBaseDone).coerceAtLeast(0) + val pct = (100L * roundDone / progTarget).coerceIn(0, 100) + val remaining = (progTarget - roundDone).coerceAtLeast(0) + val eta = if (rate > 0) etaFmt(remaining / rate) else "…" + log( + "[graperank] round $progRound · ${human(roundDone.toLong())}/${human(progTarget.toLong())} ($pct%)" + + " · $rate/s · ~$eta · ${human(events)} ev · $parked slow · ${deadRelays.size()} dead", + ) + } + } + } + } + + /** + * Subscribe each relay to its filters behind [limiter] and drain them. A relay + * that reaches a terminal (EOSE/CLOSED/cannot-connect) within the FAST + * [Config.timeoutMs] has its events persisted and returned so this round can + * resolve the authors it was asked for. A relay still streaming when the fast + * timeout elapses is not cut but PARKED: it hands its open subscription to + * [bgScope] (releasing its limiter permit so the fast pool moves on) and keeps + * receiving for up to [Config.parkTimeoutMs] more; whatever it eventually + * delivers is persisted and its contact lists pushed to [lateHarvest] for the + * round loop to fold in — so slow relays add completeness without holding up + * the round. Each relay's filters are split into REQ-sized groups so a popular + * relay routed thousands of authors doesn't emit a frame most relays reject. + * Hard connect failures (fast into [deadOut], parked straight to [recordDead]) + * are marked dead. Returns only the FAST events, tagged by relay. + */ + private suspend fun drainGated( + filters: Map>, + deadOut: MutableMap?, + answeredOut: MutableSet? = null, + ): List> { + if (filters.isEmpty()) return emptyList() + + // Split each relay's filters into REQ-sized groups. A REQ frame carries ALL + // its filters at once, so a popular relay routed thousands of authors would + // otherwise produce a multi-MB frame that most relays reject ("message too + // large"). Grouping by total entry count keeps each REQ under the 256KB cap. + val units = ArrayList>>() + for ((relay, relayFilters) in filters) { + var group = ArrayList() + var entries = 0 + for (f in relayFilters) { + val fe = filterEntries(f) + if (group.isNotEmpty() && entries + fe > MAX_REQ_ENTRIES) { + units.add(relay to group) + group = ArrayList() + entries = 0 + } + group.add(f) + entries += fe + } + if (group.isNotEmpty()) units.add(relay to group) + } + + // Per-relay failure classification (HARD wins over TRANSIENT); which relays + // stalled past the fast window; and which did NOT cleanly EOSE (timed out, + // parked, closed, or couldn't connect) — a relay absent from that set + // answered definitively, so an author it didn't return is one it lacks. + val failures = ConcurrentMap() + val timedOut = ConcurrentSet() + val notAnswered = ConcurrentSet() + + fun classify( + reason: String, + relay: NormalizedRelayUrl, + into: ConcurrentMap, + ) { + classifyDrainFailure(reason)?.let { kind -> + into.merge(relay, kind) { a, b -> + if (a == DrainFailure.HARD || b == DrainFailure.HARD) DrainFailure.HARD else DrainFailure.TRANSIENT + } + } + } + + fun logSlow( + relay: NormalizedRelayUrl, + reason: String, + elapsedMs: Long, + groupFilters: List, + ) { + if (!config.diagnose) return + val authors = groupFilters.flatMap { it.authors.orEmpty() } + val kinds = groupFilters.flatMap { it.kinds.orEmpty() }.distinct() + log( + "[slow-relay] ${relay.url} $reason in ${elapsedMs}ms | kinds=$kinds authors=${authors.size}: " + + authors.take(30).joinToString(",") + (if (authors.size > 30) ",…" else ""), + ) + } + + val fast = + coroutineScope { + units + .map { (subRelay, groupFilters) -> + async { + limiter.withPermit(subRelay) { + val subId = newSubId() + val done = CompletableDeferred() + val unitEvents = Channel>(Channel.UNLIMITED) + // Liveness signal for the parked idle timeout: every event pings + // this (conflated, so bursts collapse to one) and resets the park + // window, so a relay actively streaming is never cut mid-flight. + val activity = Channel(Channel.CONFLATED) + val listener = + object : SubscriptionListener { + override fun onEvent( + event: Event, + isLive: Boolean, + relay: NormalizedRelayUrl, + forFilters: List?, + ) { + unitEvents.trySend(relay to event) + activity.trySend(Unit) + } + + override fun onEose( + relay: NormalizedRelayUrl, + forFilters: List?, + ) { + done.complete("eose") + } + + override fun onClosed( + message: String, + relay: NormalizedRelayUrl, + forFilters: List?, + ) { + done.complete("closed:$message") + } + + override fun onCannotConnect( + relay: NormalizedRelayUrl, + message: String, + forFilters: List?, + ) { + done.complete("cannot:$message") + } + } + client.subscribe(subId, mapOf(subRelay to groupFilters), listener) + val mark = TimeSource.Monotonic.markNow() + val reason = withTimeoutOrNull(config.timeoutMs) { done.await() } + if (reason != null) { + // Terminal within the fast window — resolve this round. + val elapsedMs = mark.elapsedNow().inWholeMilliseconds + if (reason != "eose") notAnswered.add(subRelay) + classify(reason, subRelay, failures) + if (elapsedMs > SLOW_DRAIN_LOG_MS) logSlow(subRelay, reason, elapsedMs, groupFilters) + unitEvents.close() + client.unsubscribe(subId) + persist(buildList { for (e in unitEvents) add(e) }) + } else { + // Still streaming — hand off and let the round move on. + notAnswered.add(subRelay) + timedOut.add(subRelay) + val scope = bgScope + if (scope != null && config.parkTimeoutMs > config.timeoutMs) { + parkedInFlight.addAndFetch(1) + scope.launch { + try { + // Idle timeout, not absolute: only cut after parkTimeoutMs + // of SILENCE (no event, no terminal), so a relay still + // streaming a large result set is never chopped mid-flight. + val late = awaitTerminalOrIdle(done, activity, config.parkTimeoutMs) + logSlow(subRelay, "parked→$late", mark.elapsedNow().inWholeMilliseconds, groupFilters) + // A parked relay that ends in a hard/transient failure (not a + // clean EOSE) is reported dead the same way a fast one would be. + val lateDead = ConcurrentMap() + classify(late, subRelay, lateDead) + recordDead(lateDead.snapshot()) + unitEvents.close() + for (pair in persist(buildList { for (e in unitEvents) add(e) })) lateHarvest.trySend(pair) + } finally { + client.unsubscribe(subId) + parkedInFlight.addAndFetch(-1) + } + } + } else { + logSlow(subRelay, "timeout", mark.elapsedNow().inWholeMilliseconds, groupFilters) + unitEvents.close() + client.unsubscribe(subId) + } + emptyList() + } + } + } + }.awaitAll() + .flatten() + } + + if (config.diagnose && timedOut.size() > 0) { + log("[drain] parked ${timedOut.size()} slow relay(s) past ${config.timeoutMs}ms") + } + deadOut?.putAll(failures.snapshot()) + answeredOut?.addAll(filters.keys.filter { it !in notAnswered }) + return fast + } + + suspend fun run(): Stats { + val crawlMark = TimeSource.Monotonic.markNow() + // Scope owning parked (slow-relay) subscriptions and the fire-and-forget + // Tier-2 relay-list sweeps. SupervisorJob so one failure never cancels the + // others; cancelled once the crawl converges. Published to [bgScope] so + // drainGated can hand slow subs to it. + val scope = CoroutineScope(coroutineContext + SupervisorJob()) + bgScope = scope + + // Heartbeat: keeps a long, silent round feeling alive with live % + ETA. + // Runs on [scope], so scope.cancel() at crawl end stops it. + scope.launch { progressTicker() } + + while (rounds < config.maxRounds) { + // Fold in whatever the parked (slow-but-alive) relays have delivered + // since the last round — their late contact lists expand the frontier + // a round or two behind the fast ones (single-writer: only here). + foldLateHarvest() + + // Only crawl users within the hop budget; deeper users still appear + // in the graph as follow targets, we just don't fetch their lists. + val pending = hopOf.keys.filter { it !in done && (hopOf[it] ?: 0) < config.maxHops } + if (pending.isEmpty()) { + // Frontier drained. If no slow relay is still streaming, a final + // fold catches any last-moment delivery and we're done; otherwise + // wait for a parked relay to deliver (completeness) and loop. + progConverging = true + if (parkedInFlight.load() == 0L) { + if (foldLateHarvest() == 0) break else continue + } + withTimeoutOrNull(PARK_POLL_MS) { lateHarvest.receive() }?.let { ingestLate(it.first, it.second) } + continue + } + rounds++ + // Frame this round for the heartbeat ticker: its target is the pending + // set, its baseline is how many users were already done going in. + progRound = rounds + progTarget = pending.size + progBaseDone = done.size + progConverging = false + + // Refresh the warm pool to this round's busiest relays and keep that + // subscription open — reusing the same subId just updates the + // desired-relay set, so these sockets stay up across the round. + topLiveRelays(WARM_POOL_SIZE).takeIf { it.isNotEmpty() }?.let { warm -> + client.subscribe(WARM_SUB_ID, warm.associateWith { WARM_FILTERS }, null) + } + + val discoveredBefore = hopOf.size + val fedBefore = contactListsFed + + // Phase A — bulk-fetch from the busiest relays via the sharded sweep. + // Most users' kind:3 lives on the big popular relays, so this clears + // the majority cheaply (early rounds no-op until a backbone is learned). + shardedSweep(pending) + + // Phase B — whoever the popular relays didn't have (niche outboxes): + // resolve their kind:10002, then fetch from their own write relays, + // drained a few at a time and skipping dead relays. + val stragglers = pending.filter { it !in done } + if (stragglers.isNotEmpty()) { + val backbone = topLiveRelays(BACKBONE_SIZE).toSet() + // Snapshot of every relay we've seen work, for the wide Tier-2 + // sweep (taken now, before the Phase-B workers mutate liveRelays). + val allLive = liveRelays.filterTo(HashSet()) { it !in deadRelays } + ensureRelayLists(stragglers.toSet(), allLive, scope) + + // Continuous worker pool instead of chunked awaitAll barriers, so + // no worker waits on a slow sibling and hot relays stay connected. + // Shared graph state stays single-writer: routeByOutbox runs only + // on the producer (keeps writeRelayFreq serial) and ingest runs + // only on the consumer (keeps done/builder/hopOf serial), now + // overlapped with draining instead of blocked behind each batch. + val routed = Channel, Map>>>(config.drainConcurrency * 2) + val drainedOut = Channel(Channel.UNLIMITED) + coroutineScope { + // Producer: route each batch by outbox (serial), backpressured + // by the bounded `routed` channel. + val producer = + launch { + for (batch in stragglers.chunked(USER_BATCH)) { + val filters = routeByOutbox(batch.toSet(), backbone) + routed.send(batch to filters) + } + routed.close() + } + // Drain workers: pure network, no shared graph-state writes + // except recordDead (concurrent-safe). Each captures the relays + // that cleanly EOSE'd, so the consumer can tell "answered empty" + // from "timed out" per user. + val workers = + List(config.drainConcurrency) { + launch { + for ((batch, filters) in routed) { + val dead = HashMap() + val answered = HashSet() + val events = drainGated(filters, dead, answered) + recordDead(dead) + drainedOut.send(DrainedBatch(batch, filters, answered, events)) + } + } + } + // Consumer: single-writer ingest, overlapped with draining. + val consumer = + launch { + for (d in drainedOut) { + relaysContacted += d.filters.keys + // Any relay that gave us an event is proven live + useful. + for ((relay, _) in d.events) liveRelays.add(relay) + + // Per user, record relays that answered (EOSE'd) but did + // not return their kind:3, so they aren't re-queried there. + val returnedByRelay = HashMap>() + for ((relay, ev) in d.events) { + if (ev is ContactListEvent) returnedByRelay.getOrPut(relay) { HashSet() }.add(ev.pubKey) + } + for (relay in d.answered) { + val asked = d.filters[relay]?.flatMapTo(HashSet()) { it.authors.orEmpty() } ?: continue + val returned = returnedByRelay[relay].orEmpty() + for (pk in asked) { + if (pk !in returned) askedEmpty.getOrPut(pk) { ConcurrentSet() }.add(relay) + } + } + + for (pk in d.batch) { + if (pk in done) continue + val contacts = contactsOf(pk) + if (contacts != null) { + done += pk + ingest(pk, contacts) + } else { + val tries = (attempts[pk] ?: 0) + 1 + attempts[pk] = tries + if (tries >= MAX_OUTBOX_ATTEMPTS) done += pk + } + } + } + } + producer.join() + workers.joinAll() + drainedOut.close() + consumer.join() + } + } + + log( + "[graperank] round $rounds: pending=${pending.size}, " + + "gotList=${contactListsFed - fedBefore}, newUsers=${hopOf.size - discoveredBefore}, " + + "discovered=${hopOf.size}, done=${done.size}, dead=${deadRelays.size()}", + ) + } + + // Crawl done — drop the warm pool. + client.unsubscribe(WARM_SUB_ID) + + // Reports can be retracted. Ask each reporter's outbox for NIP-09 kind:5 + // deletions that cite the reports we gathered (#e-filtered to our report + // ids). The events land in the store; the caller decides which reports + // they actually retract. Run before cancelling [scope] so it can still + // park slow relays. + fetchReportDeletions(topLiveRelays(BACKBONE_SIZE).toSet()) + + // Stop any parked subscriptions + Tier-2 relay-list sweeps still in flight + // (whatever they fetched already landed in the store). + scope.cancel() + + val hopHistogram = + hopOf.values + .groupingBy { it } + .eachCount() + .toList() + .sortedBy { it.first } + .toMap() + val downloadMs = crawlMark.elapsedNow().inWholeMilliseconds + val verifyMs = verifyNanos.load() / 1_000_000 + val insertMs = insertNanos.load() / 1_000_000 + val stored = eventsStored.load() + log( + "[graperank] crawl complete: ${hopOf.size} discovered, $contactListsFed contact lists fed, " + + "${relaysContacted.size} relays contacted, ${deadRelays.size()} dead, $rounds rounds in $downloadMs ms; " + + "by hop: " + hopHistogram.entries.joinToString(" ") { "${it.key}=${it.value}" }, + ) + log( + "[graperank] write path: $stored events stored, verify ${verifyMs}ms + insert ${insertMs}ms " + + "(summed across all drains, batch=${config.insertBatchSize})", + ) + return Stats( + rounds = rounds, + contactListsFed = contactListsFed, + relaysContacted = relaysContacted.size, + hopHistogram = hopHistogram, + downloadMs = downloadMs, + verifyMs = verifyMs, + insertMs = insertMs, + eventsStored = stored, + ) + } + } + + /** + * One Phase-B batch after draining: the users asked for, the relay->filters map + * they were routed through, the relays that cleanly EOSE'd ([answered]), and the + * fresh events. Carries enough for the consumer to attribute "answered but + * empty" per user without re-deriving the routing. + */ + private class DrainedBatch( + val batch: List, + val filters: Map>, + val answered: Set, + val events: List>, + ) + + /** Latest known kind:3 contact list for [pubKey] from the local store, or null. */ + private suspend fun contactsOf(pubKey: HexKey): ContactListEvent? = + store + .query(Filter(authors = listOf(pubKey), kinds = listOf(ContactListEvent.KIND), limit = 1)) + .firstOrNull() as? ContactListEvent + + /** Latest known kind:10002 advertised relay list for [pubKey] from the store, or null. */ + private suspend fun relaysOf(pubKey: HexKey): AdvertisedRelayListEvent? = + store + .query(Filter(authors = listOf(pubKey), kinds = listOf(AdvertisedRelayListEvent.KIND), limit = 1)) + .firstOrNull() as? AdvertisedRelayListEvent + + companion object { + // Authors per REQ filter — keeps individual subscriptions within relay limits. + private const val AUTHORS_PER_FILTER = 300 + + // Max total "entries" (authors + ids + tag values) in a single REQ frame. + // Each entry is a ~67-byte hex string, so 2500 ≈ 167KB — under the 256KB + // message cap most relays enforce. drainGated groups filters to stay within. + private const val MAX_REQ_ENTRIES = 2500 + + // --diagnose: a REQ that takes longer than this to reach a terminal (EOSE or + // timeout) is logged with its relay + filter, so slow relays can be replayed. + private const val SLOW_DRAIN_LOG_MS = 4000L + + // Once the frontier is empty but parked relays are still streaming, how long + // to block waiting for one of them to deliver before re-checking convergence. + private const val PARK_POLL_MS = 2000L + + // How often the heartbeat ticker emits a live-progress line. + private const val PROGRESS_INTERVAL_MS = 3000L + + /** Compact human count: 1234 -> "1.2k", 1_500_000 -> "1.5M". */ + private fun human(n: Long): String = + when { + n >= 1_000_000 -> "${n / 1_000_000}.${(n % 1_000_000) / 100_000}M" + n >= 1_000 -> "${n / 1_000}.${(n % 1_000) / 100}k" + else -> n.toString() + } + + /** Seconds as "45s" or "3m20s". */ + private fun etaFmt(secs: Long): String = if (secs >= 60) "${secs / 60}m${secs % 60}s" else "${secs}s" + + // Sentinel returned by the park idle-wait's select when an event arrived + // (resets the window). A control string that can't collide with a relay's + // CLOSED/cannot message, which are the only other select results. + private const val ACTIVITY = "activity" + + // Times we re-query an unreachable user's outbox before giving up, so the + // crawl still terminates on a finite graph. + private const val MAX_OUTBOX_ATTEMPTS = 3 + + // Users whose outboxes we fetch in a single drain. Draining thousands of + // distinct outbox relays at once saturates connections and times out + // (~250/drain succeeds, ~17k fails); keep the fan-out small. + private const val USER_BATCH = 256 + + // Sharded backbone sweep: split the still-missing authors into SHARD_RELAYS + // lists, one per top relay, rotating up to SHARD_ROTATIONS times; once the + // remainder drops below SHARD_BROADCAST_THRESHOLD, broadcast it at once. + private const val SHARD_RELAYS = 10 + private const val SHARD_ROTATIONS = 6 + private const val SHARD_BROADCAST_THRESHOLD = 2000 + + // The small-remainder broadcast goes to this many top live relays — a user's + // kind:3 is often mirrored on a busy relay ranked below the top 10. + private const val BROADCAST_RELAYS = 60 + + // A relay that fails to CONNECT this many times is treated as dead. Kept + // above 1 so a single transient connect blip doesn't evict a relay. + private const val MAX_DEAD_STRIKES = 3 + + // Most-used write relays kept as the known-good backbone for retrying users. + private const val BACKBONE_SIZE = 30 + + // Warm pool: hold a do-nothing subscription open to the busiest relays for + // the whole crawl, so the connections we reuse every round survive the + // between-round routing gaps. The filter matches an impossible event id, so + // the relay EOSEs immediately and streams nothing — it only keeps sockets warm. + private const val WARM_POOL_SIZE = 20 + private const val WARM_SUB_ID = "graperank-warm" + private val WARM_FILTERS = listOf(Filter(ids = listOf("0".repeat(64)))) + + // Kinds requested from relays during the crawl: the graph edges (contact + // lists, mute lists, reports) PLUS the user's own kind:10002. A user's outbox + // holds the freshest copy of their relay list, so folding 10002 into the same + // query keeps routing current. The store keeps newest-by-created_at for the + // replaceable 10002, so the freshest always wins regardless of source relay. + private val FETCH_KINDS = + listOf(ContactListEvent.KIND, MuteListEvent.KIND, ReportEvent.KIND, AdvertisedRelayListEvent.KIND) + + /** Count the size-driving entries in a filter: authors, ids, and tag values. */ + private fun filterEntries(f: Filter): Int = + (f.authors?.size ?: 0) + + (f.ids?.size ?: 0) + + (f.tags?.values?.sumOf { it.size } ?: 0) + + (f.tagsAll?.values?.sumOf { it.size } ?: 0) + } +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt new file mode 100644 index 0000000000..2735d7e08c --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt @@ -0,0 +1,207 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner +import com.vitorpamplona.quartz.nip01Core.store.IEventStore +import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.ContactCardEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag +import kotlinx.coroutines.async +import kotlinx.coroutines.awaitAll +import kotlinx.coroutines.coroutineScope + +/** + * Publishes a set of GrapeRank scores as NIP-85 kind:30382 [ContactCardEvent] + * trusted assertions (one `rank` card per scored user), reconciled against what + * this provider key has already published so a repeat run only writes what moved. + * + * Reconciliation, given the desired `(target, rank)` set the scorer produced: + * - **skip** a target whose stored card already carries the same rank string — + * re-signing an unchanged card would churn a new event id for no client benefit; + * - **upsert** a target whose rank changed (or that has no card yet), up to a + * publish limit; + * - **retract** every stored card whose target is no longer in the desired set + * (it fell below the caller's cutoff, or dropped out of the graph) with a NIP-09 + * kind:5 deletion, batched so the frame stays under the ~64KB event cap. + * + * Transport-agnostic like [GrapeRankDataCrawler]: it reads prior cards from an + * [IEventStore] and emits through an injected [publish] function (event + relays → + * per-relay ack), so the store/relay wiring stays in the application while the + * reconcile + card-construction logic is reusable (e.g. by the Android app). + */ +class GrapeRankPublisher( + private val store: IEventStore, + private val publish: suspend (Event, Set) -> Map, +) { + /** Outcome counts for one reconcile: what was written, retracted, and skipped. */ + class Result( + val published: Int, + val publishRejected: Int, + val deleted: Int, + val deleteRejected: Int, + val skippedUnchanged: Int, + /** Changed cards beyond [publishLimit] that were not upserted this run. */ + val truncated: Int, + ) + + /** + * Reconcile the desired [scored] `(target, rank)` set (the caller has already + * applied any rank cutoff) against the cards [providerPubkey] previously + * published, then upsert the changes and retract the stale cards, all signed by + * [providerSigner]. At most [publishLimit] changed cards are upserted per run. + */ + suspend fun reconcileAndPublish( + providerSigner: NostrSigner, + providerPubkey: HexKey, + scored: List>, + relays: Set, + publishLimit: Int, + publishConcurrency: Int = PUBLISH_CONCURRENCY, + ): Result { + // Newest card per target this provider already published (read back from + // the store, which every published card was persisted to). + val existing = existingCards(providerPubkey) + val publishableTargets = scored.mapTo(HashSet()) { it.first } + + // Upsert publishable targets whose rank tag STRING would change (or that + // have no card yet). RankTag.assemble writes rank.toString(), so we diff + // that exact string — an unchanged score is skipped so clients only sync + // ranks that moved. + val changed = scored.filter { (target, rank) -> existing[target]?.let(::rankTagValue) != rank.toString() } + val toUpsert = changed.take(publishLimit) + + // Retract existing cards whose target is no longer publishable — it dropped + // out of the graph, or fell below the caller's cutoff. We won't leave a + // stale assertion standing. + val toDelete = existing.filterKeys { it !in publishableTargets }.values.toList() + + val (ok, rejected) = publishCards(providerSigner, toUpsert, relays, publishConcurrency) + val (deleted, deleteRejected) = publishDeletions(providerSigner, toDelete, relays) + + return Result( + published = ok, + publishRejected = rejected, + deleted = deleted, + deleteRejected = deleteRejected, + skippedUnchanged = scored.size - changed.size, + truncated = (changed.size - toUpsert.size).coerceAtLeast(0), + ) + } + + /** + * The newest kind:30382 card [providerPubkey] published per target, read from + * the store (every card [publish] sends is persisted first, so on repeat runs + * this reflects what is already out there). + */ + private suspend fun existingCards(providerPubkey: HexKey): Map = + store + .query(Filter(kinds = listOf(ContactCardEvent.KIND), authors = listOf(providerPubkey))) + .filterIsInstance() + .groupBy { it.aboutUser() } + .mapNotNull { (target, cards) -> + val t = target ?: return@mapNotNull null + t to (cards.maxByOrNull { it.createdAt } ?: return@mapNotNull null) + }.toMap() + + /** + * The raw `rank` tag value string on a card — exactly what a client diffs, so an + * unchanged score never produces a new signature. Our cards carry only a `rank` + * tag (plus the d-tag target), so this one value decides whether a re-publish + * would differ. + */ + private fun rankTagValue(card: ContactCardEvent): String? = + card.tags.firstNotNullOfOrNull { tag -> + if (tag.size > 1 && tag[0] == RankTag.TAG_NAME) tag[1] else null + } + + /** Build + publish one kind:30382 card per (target, rank), bounded-concurrently. */ + private suspend fun publishCards( + signer: NostrSigner, + cards: List>, + relays: Set, + concurrency: Int, + ): Pair { + var published = 0 + var rejected = 0 + for (batch in cards.chunked(concurrency)) { + val acks = + coroutineScope { + batch + .map { (pubkey, rank) -> + async { + val card = + ContactCardEvent.create( + targetUser = pubkey, + signer = signer, + publicInitializer = { add(RankTag.assemble(rank)) }, + ) + publish(card, relays) + } + }.awaitAll() + } + for (ack in acks) { + if (ack.values.any { it }) published++ else rejected++ + } + } + return published to rejected + } + + /** + * Retract stale cards with NIP-09 kind:5 deletions signed by [signer] (the same + * key that signed the cards). Batches [DELETE_PER_EVENT] addressable coordinates + * per deletion so the kind:5 frame stays under the ~64KB event cap; each carries + * the card's `a` tag (30382:provider:target), so re-publishing a newer version + * later isn't blocked. Returns (deleted, rejected) card counts. + */ + private suspend fun publishDeletions( + signer: NostrSigner, + cards: List, + relays: Set, + ): Pair { + if (cards.isEmpty()) return 0 to 0 + var deleted = 0 + var rejected = 0 + for (chunk in cards.chunked(DELETE_PER_EVENT)) { + val event = signer.sign(DeletionEvent.build(chunk)) + val ack = publish(event, relays) + if (ack.values.any { it }) deleted += chunk.size else rejected += chunk.size + } + return deleted to rejected + } + + companion object { + /** Concurrent card publishes when upserting. */ + const val PUBLISH_CONCURRENCY = 16 + + /** + * Addressable coordinates cited per kind:5 retraction. Each `a` tag is + * ~130 bytes (30382:<64hex>:<64hex>), so 400 keeps the whole event ~52KB — + * under the 64KB event-size cap many relays enforce (stricter than the + * 256KB message cap). + */ + const val DELETE_PER_EVENT = 400 + } +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt new file mode 100644 index 0000000000..ad78a6e56a --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.HexKey + +/** + * A trust relationship kind and the GrapeRank rating it carries. [code] is the + * 2-bit tag packed alongside a source node id in the edge arrays (see + * [TrustGraph]); keep it in `0..3`. + */ +enum class TrustRelation( + val rating: Double, + val code: Int, +) { + FOLLOW(1.0, 0), + MUTE(-0.1, 1), + REPORT(-0.1, 2), +} + +/** + * A web-of-trust graph over Nostr pubkeys, stored compactly so it scales to the + * whole network (millions of edges) without a `String`-keyed edge object per + * relationship. + * + * Pubkeys are interned to dense `Int` node ids. Edges live in two + * compressed-sparse-row (CSR) layouts backed by flat `IntArray`s — one indexed + * by target (what [GrapeRank] reads to score a node) and one by source (what the + * propagation worklist follows). Each incoming entry packs the source id in the + * low 29 bits and the [TrustRelation.code] in the top bits, so an edge is a + * single `int`. A 100M-edge graph is then ~0.8 GB of primitive arrays instead of + * tens of GB of objects. + * + * Build one with [TrustGraphBuilder], feeding contact lists / mutes / reports in + * as they stream off the relays. + */ +class TrustGraph internal constructor( + val nodeCount: Int, + private val pubkeys: Array, + private val ids: HashMap, + // CSR by target: incoming edges of node t are inPacked[inOffsets[t] until inOffsets[t+1]], + // each packing source id (low 29 bits) + relation code (top bits). + internal val inOffsets: IntArray, + internal val inPacked: IntArray, + // CSR by source: out-neighbour targets of node s are outTargets[outOffsets[s] until outOffsets[s+1]]. + internal val outOffsets: IntArray, + internal val outTargets: IntArray, +) { + /** Node id for [pubkey], or `-1` if it never appeared in the graph. */ + fun idOf(pubkey: HexKey): Int = ids[pubkey] ?: -1 + + /** Pubkey for a node [id]. */ + fun pubkeyOf(id: Int): HexKey = pubkeys[id] + + fun edgeCount(): Int = inPacked.size + + companion object { + const val SOURCE_BITS = 29 + const val SOURCE_MASK = (1 shl SOURCE_BITS) - 1 + const val MAX_NODES = SOURCE_MASK // ids must fit in the low 29 bits + } +} + +/** A minimal growable `int[]` — avoids boxing `Int`s in an `ArrayList` at graph scale. */ +internal class IntArrayList( + initialCapacity: Int = 16, +) { + var data: IntArray = IntArray(initialCapacity.coerceAtLeast(1)) + private set + var size: Int = 0 + private set + + fun add(value: Int) { + if (size == data.size) data = data.copyOf(data.size * 2) + data[size++] = value + } + + fun get(index: Int): Int = data[index] + + fun removeLast(): Int = data[--size] + + fun isNotEmpty(): Boolean = size > 0 +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt new file mode 100644 index 0000000000..7b3f12fbe4 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt @@ -0,0 +1,127 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.HexKey + +/** + * Builds a [TrustGraph] incrementally so callers never have to hold every contact + * list in memory at once — feed each user's follows / mutes / reports as they + * stream off the relays (or out of the store), then call [build]. + * + * Interns pubkeys to dense ids on the fly and accumulates edges in flat growable + * int arrays. Follows and mutes are replaceable (one list per author, deduped by + * the caller via latest-per-author + set-valued tags); reports are regular events, + * so `(reporter → reported)` report edges are deduped here. Self-edges are dropped. + */ +class TrustGraphBuilder { + private val ids = HashMap() + private val pubkeys = ArrayList() + + // Parallel edge arrays: edge i is source edgeSource[i] --relation--> edgeTarget[i], + // with the relation packed into the top bits of edgeSource[i]. + private val edgeTargets = IntArrayList() + private val edgeSourcesPacked = IntArrayList() + + // Dedup for report edges only (reporters can file many kind:1984 for one target). + private val reportSeen = HashSet() + + private fun intern(pubkey: HexKey): Int = + ids.getOrPut(pubkey) { + val id = pubkeys.size + pubkeys.add(pubkey) + id + } + + private fun addEdge( + source: HexKey, + target: HexKey, + relation: TrustRelation, + ) { + if (source == target) return + val s = intern(source) + val t = intern(target) + if (relation == TrustRelation.REPORT) { + val key = (s.toLong() shl 32) or (t.toLong() and 0xFFFFFFFFL) + if (!reportSeen.add(key)) return + } + edgeTargets.add(t) + edgeSourcesPacked.add(s or (relation.code shl TrustGraph.SOURCE_BITS)) + } + + fun addFollows( + source: HexKey, + follows: Iterable, + ) { + for (target in follows) addEdge(source, target, TrustRelation.FOLLOW) + } + + fun addMutes( + source: HexKey, + muted: Iterable, + ) { + for (target in muted) addEdge(source, target, TrustRelation.MUTE) + } + + fun addReports( + source: HexKey, + reported: Iterable, + ) { + for (target in reported) addEdge(source, target, TrustRelation.REPORT) + } + + fun nodeCount(): Int = pubkeys.size + + fun edgeCount(): Int = edgeTargets.size + + /** Freeze the accumulated edges into the two CSR layouts. */ + fun build(): TrustGraph { + val n = pubkeys.size + val m = edgeTargets.size + + // Incoming CSR (by target). + val inOffsets = IntArray(n + 1) + for (i in 0 until m) inOffsets[edgeTargets.get(i) + 1]++ + for (i in 1..n) inOffsets[i] += inOffsets[i - 1] + val inPacked = IntArray(m) + val inCursor = inOffsets.copyOf() + for (i in 0 until m) { + val t = edgeTargets.get(i) + inPacked[inCursor[t]++] = edgeSourcesPacked.get(i) + } + + // Outgoing CSR (by source). + val outOffsets = IntArray(n + 1) + for (i in 0 until m) { + val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK + outOffsets[s + 1]++ + } + for (i in 1..n) outOffsets[i] += outOffsets[i - 1] + val outTargets = IntArray(m) + val outCursor = outOffsets.copyOf() + for (i in 0 until m) { + val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK + outTargets[outCursor[s]++] = edgeTargets.get(i) + } + + return TrustGraph(n, pubkeys.toTypedArray(), ids, inOffsets, inPacked, outOffsets, outTargets) + } +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/AdaptiveRelayLimiter.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/AdaptiveRelayLimiter.kt new file mode 100644 index 0000000000..977767fade --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/AdaptiveRelayLimiter.kt @@ -0,0 +1,275 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.nip01Core.relay.client.accessories + +import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener +import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.ClosedMessage +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.Message +import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.NoticeMessage +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.utils.Log +import com.vitorpamplona.quartz.utils.TimeUtils +import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap +import kotlinx.coroutines.CompletableDeferred +import kotlinx.coroutines.delay +import kotlinx.coroutines.sync.Mutex +import kotlinx.coroutines.sync.withLock +import kotlin.concurrent.atomics.AtomicInt +import kotlin.concurrent.atomics.AtomicLong +import kotlin.concurrent.atomics.ExperimentalAtomicApi + +/** + * Adaptive per-relay back-pressure with TWO independent controls, because relays + * push back for two different reasons that need two different responses: + * + * 1. **Subscription-count limit** — a max on how many subscriptions may be OPEN + * at once ("too many subscriptions", "maximum concurrent subscription count", + * "number of subscriptions exceeds limit"). The fix is fewer *concurrent* + * subs, so we demote the relay's concurrency cap down [subLadder] + * (100 → 20 → 10). + * 2. **Rate limit** — too many subscription *changes per second* ("rate-limited: + * too many messages", "burst exhausted", "slow down"). Fewer concurrent subs + * wouldn't help; the fix is to *space the REQs out in time*, so we impose a + * minimum interval between opens to that relay, growing it up [rateLadder] + * (250ms → 500ms → 1s → 2s). + * + * Mixing the two mishandles the relay: capping concurrency does nothing for a + * rate limit, and slowing the rate does nothing for a subscription-count cap. So + * each complaint is routed to its own actuator by matching the notice text. + * + * A well-behaved relay starts at [startCap] concurrent subs with no rate delay, + * and only the ones that push back get throttled — each only as far, and in the + * dimension, they keep pushing. + * + * Registered as a [RelayConnectionListener] on the shared client, so both signals + * are driven straight off the incoming NOTICE/CLOSED frames (which fire on the + * per-relay socket threads — all state here is concurrent). Drains gate through + * [withPermit]: the gated-drain path holds a relay's permit for the lifetime of + * that relay's subscription, and passes the rate gate before it opens, so we + * respect both limits at once. + */ +@OptIn(ExperimentalAtomicApi::class) +class AdaptiveRelayLimiter( + private val startCap: Int = 100, + private val subLadder: List = listOf(20, 10), + private val rateLadder: List = listOf(250L, 500L, 1000L, 2000L), +) : RelayConnectionListener { + private val gates = ConcurrentMap() + + // Concurrency-cap demotions per relay (== index+1 into subLadder). Capped at + // subLadder.size: past the floor we stop demoting. + private val subDemotions = ConcurrentMap() + + // Rate-limit state per relay: how far down rateLadder we've stepped, the + // current min interval between opens, and the next epoch-ms an open may fire. + private val rateSteps = ConcurrentMap() + private val rateDelayMs = ConcurrentMap() + private val nextAllowedAtMs = ConcurrentMap() + + private fun gate(relay: NormalizedRelayUrl): Gate = gates.getOrPut(relay) { Gate(startCap) } + + /** + * Run [block] against [relay] respecting both limits: first wait out any rate + * delay (spacing opens in time), then hold one of the relay's concurrency + * permits for the duration. + */ + suspend fun withPermit( + relay: NormalizedRelayUrl, + block: suspend () -> T, + ): T { + rateGate(relay) + val g = gate(relay) + g.acquire() + try { + return block() + } finally { + g.release() + } + } + + /** If [relay] is rate-limited, reserve and wait for its next allowed open slot. */ + private suspend fun rateGate(relay: NormalizedRelayUrl) { + val delayMs = rateDelayMs[relay] ?: return + if (delayMs <= 0L) return + val now = TimeUtils.nowMillis() + // Atomically claim the next slot: my turn is max(prevSlot, now); the next + // caller can't fire until delayMs after me. Serializes opens to this relay + // at one per delayMs, in arrival order. + val slot = nextAllowedAtMs.getOrPut(relay) { AtomicLong(now) } + var myTurn: Long + while (true) { + val prev = slot.load() + myTurn = maxOf(prev, now) + if (slot.compareAndSet(prev, myTurn + delayMs)) break + } + val wait = myTurn - now + if (wait > 0) delay(wait) + } + + override fun onIncomingMessage( + relay: IRelayClient, + msgStr: String, + msg: Message, + ) { + val text = + when (msg) { + is ClosedMessage -> msg.message + is NoticeMessage -> msg.message + else -> return + } + val t = text.lowercase() + // Route each complaint to the matching actuator. Not mutually exclusive: + // if a relay somehow reports both, we act on both (they don't conflict). + if (RATE_LIMIT_MARKERS.any { it in t }) throttleRate(relay.url) + if (SUB_LIMIT_MARKERS.any { it in t }) demoteConcurrency(relay.url) + } + + /** Step [relay] one rung down the concurrency-cap ladder, unless already at the floor. */ + private fun demoteConcurrency(relay: NormalizedRelayUrl) { + if ((subDemotions[relay] ?: 0) >= subLadder.size) return + val step = subDemotions.merge(relay, 1) { a, b -> a + b } + val cap = subLadder[(step - 1).coerceIn(0, subLadder.size - 1)] + gate(relay).lower(cap) + if (step <= subLadder.size) { + Log.d("AdaptiveRelayLimiter") { "${relay.url} concurrency capped at $cap subs (sub-limit #$step)" } + } + } + + /** Step [relay] one rung down the rate ladder, unless already at the slowest. */ + private fun throttleRate(relay: NormalizedRelayUrl) { + if ((rateSteps[relay] ?: 0) >= rateLadder.size) return + val step = rateSteps.merge(relay, 1) { a, b -> a + b } + val d = rateLadder[(step - 1).coerceIn(0, rateLadder.size - 1)] + rateDelayMs[relay] = d + if (step <= rateLadder.size) { + Log.d("AdaptiveRelayLimiter") { "${relay.url} rate-throttled to 1 REQ / ${d}ms (rate-limit #$step)" } + } + } + + /** JSON-friendly view of which relays we throttled, in which dimension, how far. */ + fun snapshot(): Map { + val capCounts = HashMap() + for ((_, step) in subDemotions.snapshot()) { + val cap = subLadder[(step - 1).coerceIn(0, subLadder.size - 1)] + capCounts[cap] = (capCounts[cap] ?: 0) + 1 + } + val cappedAt = capCounts.toList().sortedBy { it.first }.toMap() + val rateCounts = HashMap() + for ((_, step) in rateSteps.snapshot()) { + val d = rateLadder[(step - 1).coerceIn(0, rateLadder.size - 1)] + rateCounts[d] = (rateCounts[d] ?: 0) + 1 + } + val rateAt = rateCounts.toList().sortedBy { it.first }.toMap() + return mapOf( + "start_cap" to startCap, + "sub_ladder" to subLadder, + "rate_ladder_ms" to rateLadder, + "concurrency_capped_relays" to subDemotions.size(), + "concurrency_capped_at" to cappedAt, + "rate_limited_relays" to rateSteps.size(), + "rate_limited_at_ms" to rateAt, + ) + } + + fun hadThrottling(): Boolean = subDemotions.size() > 0 || rateSteps.size() > 0 + + /** + * A bounded-concurrency gate whose limit can only ever be *lowered* (relays + * never earn their cap back within a run). Fair FIFO hand-off: a released + * permit goes to the longest-waiting acquirer. Lowering the limit below the + * in-use count doesn't cancel live holders — it just refuses to admit new + * ones until enough release that `inUse < limit` again, so the concurrency + * converges down to the new cap as the excess subscriptions finish. + */ + private class Gate( + initialLimit: Int, + ) { + private val limit = AtomicInt(initialLimit) + private val mutex = Mutex() + private var inUse = 0 + private val waiters = ArrayDeque>() + + suspend fun acquire() { + val wait = + mutex.withLock { + if (inUse < limit.load()) { + inUse++ + null + } else { + CompletableDeferred().also { waiters.addLast(it) } + } + } + wait?.await() + } + + suspend fun release() { + mutex.withLock { + inUse-- + while (inUse < limit.load() && waiters.isNotEmpty()) { + waiters.removeFirst().complete(Unit) + inUse++ + } + } + } + + /** Monotonically shrink the cap. Safe to call from any thread. */ + fun lower(newLimit: Int) { + while (true) { + val cur = limit.load() + if (newLimit >= cur) return + if (limit.compareAndSet(cur, newLimit)) return + } + } + } + + companion object { + // A cap on how many subscriptions may be OPEN at once. Fix: fewer + // concurrent subs (demote the concurrency cap). + private val SUB_LIMIT_MARKERS = + listOf( + "too many concurrent", + "concurrent req", + "too many subscription", + "number of subscriptions", + "subscriptions exceeds", + "subscription limit", + "subscription count", + "maximum concurrent subscription", + "max subscription", + "too many req", + ) + + // Too many subscription CHANGES per second. Fix: space the REQs out in + // time (a per-relay min interval), not fewer concurrent subs. + private val RATE_LIMIT_MARKERS = + listOf( + "rate-limit", + "rate limit", + "ratelimit", + "too many messages", + "too many requests", + "burst exhausted", + "throttl", + "slow down", + ) + } +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/DrainFailure.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/DrainFailure.kt new file mode 100644 index 0000000000..b2d605d044 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/accessories/DrainFailure.kt @@ -0,0 +1,77 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.nip01Core.relay.client.accessories + +/** + * Why a relay could not be used for a one-shot drain — when the reason is worth + * acting on (dropping the relay from further routing). + * + * - [HARD]: the relay answered wrong, or cannot exist. A bad HTTP upgrade (not a + * websocket / dead status code), an unresolvable domain, or a TLS misconfig. + * This will not fix itself, so one strike is enough to drop it. + * - [TRANSIENT]: a failure that might clear — connection refused / reset, host + * unreachable, or a temporary 429/5xx on the upgrade. Struck a few times + * before we give up. + * + * A pure connect **timeout** is neither. The relay is most likely just busy, so + * we retry it and never mark it dead — [classifyDrainFailure] returns null for + * it (and for any non-failure terminal reason). + */ +enum class DrainFailure { HARD, TRANSIENT } + +/** + * Classify a drain per-relay terminal reason. Returns null when the relay should + * simply be retried (a timeout, or a non-failure like eose/closed). The reason + * shape is `cannot:` for a connect failure (see + * `BasicRelayClient.onCannotConnect`), or `eose` / `closed:…` / `timeout`. + */ +fun classifyDrainFailure(reason: String): DrainFailure? { + if (!reason.startsWith("cannot")) return null + val m = reason.removePrefix("cannot:").lowercase() + // The message now carries the exception class name (see BasicRelayClient), so + // we can key on the stable *type* rather than localized message text. + // Busy, not dead: a connect/read timeout means the handshake just didn't + // finish in time. Retry it — the relay is probably fine, only slow or loaded. + if ("timeout" in m || "timed out" in m) return null // SocketTimeoutException, etc. + // Cannot ever work: unresolvable domain (DNS) or a TLS misconfiguration. + // Dead for good — one strike is enough. + if ("unknownhost" in m || // UnknownHostException + "unable to resolve host" in m || + "no address associated" in m || + "nodename nor servname" in m || + "sslhandshake" in m || // SSLHandshakeException + "sslpeerunverified" in m || + "sslexception" in m || + "certificate" in m || // CertificateException + "trust anchor" in m || + "certpath" in m + ) { + return DrainFailure.HARD + } + // Wrong HTTP upgrade. Usually a misconfigured endpoint (not a relay), but + // 429 / 5xx mean "busy, come back later", so those stay transient. + if ("server misconfigured" in m || "not a websocket" in m || "expected http 101" in m) { + val transientCode = Regex("response: (429|500|502|503|504)").containsMatchIn(m) + return if (transientCode) DrainFailure.TRANSIENT else DrainFailure.HARD + } + // Refused / reset / unreachable / anything else: might clear — retry a few times. + return DrainFailure.TRANSIENT +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClient.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClient.kt index 5992cecdba..b495d42dbb 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClient.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClient.kt @@ -136,7 +136,9 @@ open class BasicRelayClient( socket?.connect() } catch (e: Exception) { if (e is CancellationException) throw e - listener.onCannotConnect(this, "Error when trying to connect: ${e.message ?: e::class.simpleName}") + val typeName = e::class.simpleName + val detail = e.message?.let { "$it ($typeName)" } ?: (typeName ?: "unknown error") + listener.onCannotConnect(this, "Error when trying to connect: $detail") listener.onDisconnected(this) dontTryAgainForALongTime() markConnectionAsClosed() @@ -187,9 +189,15 @@ open class BasicRelayClient( } else { socket?.disconnect() - // suppression rules below must match the raw message; displayMsg is for listener output only + // suppression rules below must match the raw message; displayMsg is for listener output only. + // Always include the exception's class name: message text is + // localized and inconsistent across platforms, but the type + // (SocketTimeoutException / UnknownHostException / SSLHandshakeException / + // ConnectException …) is stable and lets listeners classify a failure + // reliably — a busy relay (timeout) vs a dead one (bad domain / TLS). val msg = t.message - val displayMsg = msg ?: t::class.simpleName + val typeName = t::class.simpleName + val displayMsg = if (msg != null) "$msg ($typeName)" else (typeName ?: "unknown error") // checks if this is an actual failure. Closing the socket generates an onFailure as well. // ignore tor errors. diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/normalizer/RelayUrlNormalizer.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/normalizer/RelayUrlNormalizer.kt index 81b08a9666..04f8c9cc06 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/normalizer/RelayUrlNormalizer.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/normalizer/RelayUrlNormalizer.kt @@ -156,7 +156,7 @@ class RelayUrlNormalizer { if (trimmed.contains("://")) { // some other scheme we cannot connect to. - Log.w("RelayUrlNormalizer") { "Rejected $url" } + Log.d("RelayUrlNormalizer") { "Rejected $url" } return null } @@ -189,14 +189,14 @@ class RelayUrlNormalizer { normalizedUrls.put(url, NormalizationResult.Success(normalized)) normalized } else { - Log.w("NormalizedRelayUrl") { "Rejected $url" } + Log.d("NormalizedRelayUrl") { "Rejected $url" } normalizedUrls.put(url, NormalizationResult.Error) null } } catch (e: Exception) { if (e is CancellationException) throw e normalizedUrls.put(url, NormalizationResult.Error) - Log.w("NormalizedRelayUrl") { "Rejected $url" } + Log.d("NormalizedRelayUrl") { "Rejected $url" } null } } diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/VerifyAndInsert.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/VerifyAndInsert.kt new file mode 100644 index 0000000000..c2f1ff5193 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/VerifyAndInsert.kt @@ -0,0 +1,55 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.nip01Core.store + +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.crypto.verify +import com.vitorpamplona.quartz.utils.Log + +/** + * Verify [event]'s NIP-01 id + signature and, if valid, persist it to this store. + * Returns `true` when the event was accepted (verified) — even if the insert was a + * no-op — so callers can gate "surface this event" on the return. + * + * A UNIQUE-constraint rejection is normal, not a failure: the store already holds + * this id, or a newer version of a replaceable (kind 0/3/10000-19999). The outbox + * model routinely delivers the same event from several of a user's write relays, so + * a crawl produces these by the hundred-thousand — so only genuine persistence + * failures (I/O, full disk, corruption) are logged. Persistence is best-effort: an + * insert error is swallowed, not propagated, so it can't break a live subscription. + * + * This is the single verify-then-store sink every event-arrival path should funnel + * through, so the store stays the authoritative cache of what has been seen. + */ +suspend fun IEventStore.verifyAndInsert(event: Event): Boolean { + if (!event.verify()) { + Log.w("EventStore") { "dropped event ${event.id.take(8)} kind=${event.kind} — bad signature" } + return false + } + try { + insert(event) + } catch (t: Throwable) { + if (t.message?.contains("UNIQUE constraint", ignoreCase = true) != true) { + Log.w("EventStore") { "store insert failed for ${event.id.take(8)}: ${t.message}" } + } + } + return true +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.kt new file mode 100644 index 0000000000..47d3233c9b --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.kt @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +/** + * A thread-safe hash map whose compound operations — [getOrPut] and [merge] — + * apply their update **atomically**, not merely one-lock-per-primitive-op. This + * is the contract a concurrent producer/consumer pipeline needs: two coroutines + * racing `getOrPut` on the same key must agree on a single value, and racing + * `merge` must not lose an increment. + * + * commonMain has no `java.util.concurrent.ConcurrentHashMap`, so this is + * expect/actual, matching the split already used by [com.vitorpamplona.quartz.utils.cache.ConcurrentHashCache]: + * - JVM / Android → `ConcurrentHashMap` (lock-free, true atomic `computeIfAbsent` / `merge`). + * - Native (Apple + Linux) → copy-on-write over an atomic reference, with a + * CAS retry loop giving the same atomicity. Correct but O(n)-per-write; the + * native targets never run the heavy crawl this backs, they only compile it. + * + * Only the operations the crawl actually uses are exposed — no full [MutableMap] + * surface — so the native copy-on-write actual stays small and obviously correct. + */ +expect class ConcurrentMap() { + operator fun get(key: K): V? + + operator fun set( + key: K, + value: V, + ) + + /** Atomically return the value for [key], computing and inserting [defaultValue] once if absent. */ + fun getOrPut( + key: K, + defaultValue: () -> V, + ): V + + /** + * Atomically insert [value] if [key] is absent, else replace the existing + * value with `remap(existing, value)`. Returns the value now stored. + */ + fun merge( + key: K, + value: V, + remap: (old: V, new: V) -> V, + ): V + + fun size(): Int + + /** A point-in-time copy of the entries — safe to iterate without holding a lock. */ + fun snapshot(): Map +} diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.kt new file mode 100644 index 0000000000..d00545acf6 --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.kt @@ -0,0 +1,43 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +/** + * A thread-safe hash set for the crawl's cross-coroutine membership tracking + * (dead relays struck by drain workers while the router reads them, relay hints + * written by the ingest consumer while the producer reads them). + * + * commonMain has no `java.util.concurrent.ConcurrentHashMap.newKeySet()`, so this + * is expect/actual with the same JVM-vs-native split as [ConcurrentMap]: + * - JVM / Android → `ConcurrentHashMap.newKeySet()`. + * - Native → copy-on-write over an atomic reference (compile-only, never the hot path). + */ +expect class ConcurrentSet() { + /** Add [element]; returns true if it was not already present. */ + fun add(element: E): Boolean + + operator fun contains(element: E): Boolean + + fun size(): Int + + /** A point-in-time copy — safe to iterate or diff against without a lock. */ + fun snapshot(): Set +} diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankTest.kt new file mode 100644 index 0000000000..4219095809 --- /dev/null +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankTest.kt @@ -0,0 +1,246 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import kotlin.math.abs +import kotlin.math.exp +import kotlin.math.ln +import kotlin.math.max +import kotlin.random.Random +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +class GrapeRankTest { + private val obs = "observer" + + private fun graphOf(edges: List>): TrustGraph { + val b = TrustGraphBuilder() + for ((source, target, relation) in edges) { + when (relation) { + TrustRelation.FOLLOW -> b.addFollows(source, listOf(target)) + TrustRelation.MUTE -> b.addMutes(source, listOf(target)) + TrustRelation.REPORT -> b.addReports(source, listOf(target)) + } + } + return b.build() + } + + private fun graphOf(vararg edges: Triple) = graphOf(edges.toList()) + + /** Score for a pubkey (0.0 if absent from the graph). */ + private fun DoubleArray.of( + graph: TrustGraph, + pubkey: HexKey, + ): Double { + val id = graph.idOf(pubkey) + return if (id < 0) 0.0 else this[id] + } + + @Test + fun observerIsPinnedAtFullSelfTrust() { + val graph = graphOf(Triple(obs, "a", TrustRelation.FOLLOW)) + val scores = GrapeRank().compute(graph, obs) + assertEquals(1.0, scores.of(graph, obs), 1e-12) + } + + @Test + fun directFollowMatchesHandComputedValue() { + val graph = graphOf(Triple(obs, "a", TrustRelation.FOLLOW)) + val scores = GrapeRank().compute(graph, obs) + // weight = 0.5 * 1.0 * 0.85 = 0.425 ; score = conf(0.425) = 0.2551612... + assertEquals(0.25516127, scores.of(graph, "a"), 1e-6) + } + + @Test + fun trustDecaysSteeplyAcrossHops() { + val graph = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple("a", "b", TrustRelation.FOLLOW), + ) + val scores = GrapeRank().compute(graph, obs) + val a = scores.of(graph, "a") + val b = scores.of(graph, "b") + assertEquals(0.004499, b, 1e-5) + assertTrue(b < a / 10.0, "two-hop trust should be far below one-hop trust") + } + + @Test + fun aMuteFromAnEndorsedUserLowersTheScore() { + val followOnlyGraph = graphOf(Triple(obs, "b", TrustRelation.FOLLOW)) + val followOnly = GrapeRank().compute(followOnlyGraph, obs).of(followOnlyGraph, "b") + + val muteGraph = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple(obs, "b", TrustRelation.FOLLOW), + Triple("a", "b", TrustRelation.MUTE), + ) + val withMute = GrapeRank().compute(muteGraph, obs).of(muteGraph, "b") + + assertTrue(withMute < followOnly, "a mute from a trusted user should pull b below the follow-only baseline") + } + + @Test + fun purelyReportedUserFloorsAtZero() { + val graph = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple("a", "d", TrustRelation.REPORT), + ) + val scores = GrapeRank().compute(graph, obs) + assertEquals(0.0, scores.of(graph, "d"), 1e-9) + } + + @Test + fun unreachableUsersAreNotScored() { + val graph = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple("x", "y", TrustRelation.FOLLOW), + ) + val scores = GrapeRank().compute(graph, obs) + assertTrue(scores.of(graph, "a") > 0.0) + assertEquals(0.0, scores.of(graph, "y"), 1e-12, "a user with no path from the observer stays 0") + } + + @Test + fun cyclesConverge() { + val graph = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple("a", "b", TrustRelation.FOLLOW), + Triple("b", "a", TrustRelation.FOLLOW), + ) + val scores = GrapeRank().compute(graph, obs) + assertTrue(scores.of(graph, "a") > 0.0) + assertTrue(scores.of(graph, "b") > 0.0) + } + + @Test + fun deduplicatesRepeatedReportEdges() { + // Two report edges a->d collapse to one; the score matches a single report. + val once = graphOf(Triple(obs, "a", TrustRelation.FOLLOW), Triple("a", "d", TrustRelation.REPORT)) + val twice = + graphOf( + Triple(obs, "a", TrustRelation.FOLLOW), + Triple("a", "d", TrustRelation.REPORT), + Triple("a", "d", TrustRelation.REPORT), + ) + assertEquals(2, twice.edgeCount(), "duplicate report edge should be dropped") + assertEquals( + GrapeRank().compute(once, obs).of(once, "d"), + GrapeRank().compute(twice, obs).of(twice, "d"), + 1e-12, + ) + } + + /** + * Adversarial cross-check: the worklist propagation must reach the same fixed + * point as a naive full-sweep (the reference `v1FullSweep`) on random graphs. + */ + @Test + fun worklistMatchesFullSweepOnRandomGraphs() { + val params = GrapeRankParams(convergence = 1e-10) + val engine = GrapeRank(params) + repeat(50) { seed -> + val rng = Random(seed) + val n = 3 + rng.nextInt(12) + val nodes = (0 until n).map { "u$it" } + val edges = ArrayList>() + for (src in nodes) { + for (dst in nodes) { + if (src == dst) continue + if (rng.nextDouble() < 0.25) { + val relation = + when (rng.nextInt(5)) { + 0 -> TrustRelation.MUTE + 1 -> TrustRelation.REPORT + else -> TrustRelation.FOLLOW + } + edges.add(Triple(src, dst, relation)) + } + } + } + val observer = nodes.first() + val graph = graphOf(edges) + val scores = engine.compute(graph, observer) + val reference = fullSweep(edges, nodes, observer, params) + + for (node in nodes) { + if (node == observer) continue // observer self-trust is not part of a ranking + val a = scores.of(graph, node) + val b = reference[node] ?: 0.0 + assertEquals(b, a, 1e-5, "seed=$seed node=$node worklist=$a fullSweep=$b") + } + } + } + + // Reference: blind full sweep over every user until nothing changes. + private fun fullSweep( + edges: List>, + nodes: List, + observer: HexKey, + params: GrapeRankParams, + ): Map { + // Dedup identical edges (mirrors the builder: report edges dedup; follow/mute + // sets are unique per source anyway). + val incoming = HashMap>>() + for ((s, t, r) in edges) { + if (s == t) continue + incoming.getOrPut(t) { LinkedHashSet() }.add(s to r) + } + + fun confidence( + r: TrustRelation, + source: HexKey, + ) = when (r) { + TrustRelation.FOLLOW -> if (source == observer) params.directFollowConfidence else params.indirectFollowConfidence + TrustRelation.MUTE -> params.muteConfidence + TrustRelation.REPORT -> params.reportConfidence + } + + fun weightToConfidence(w: Double) = 1.0 - exp(-w * -ln(params.rigor)) + + val scores = HashMap() + scores[observer] = 1.0 + do { + var changed = false + for (target in nodes) { + if (target == observer) continue + var sumW = 0.0 + var sumWR = 0.0 + for ((source, r) in incoming[target] ?: emptySet()) { + val s = scores[source] ?: continue + val w = confidence(r, source) * s * params.attenuation + sumW += w + sumWR += w * r.rating + } + val newScore = if (abs(sumW) < 0.00001) 0.0 else max(weightToConfidence(sumW) * sumWR / sumW, 0.0) + val old = scores.put(target, newScore) ?: 0.0 + changed = changed || abs(newScore - old) > params.convergence + } + } while (changed) + return scores + } +} diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt new file mode 100644 index 0000000000..f27e064273 --- /dev/null +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt @@ -0,0 +1,107 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertTrue + +class TrustGraphBuilderTest { + private val alice = "alice" + private val bob = "bob" + private val carol = "carol" + private val dave = "dave" + + /** Decode a node's incoming edges back to (source, relation) pairs from the CSR. */ + private fun TrustGraph.incomingOf(pubkey: HexKey): Set> { + val t = idOf(pubkey) + if (t < 0) return emptySet() + val out = HashSet>() + var i = inOffsets[t] + val end = inOffsets[t + 1] + while (i < end) { + val packed = inPacked[i] + val source = pubkeyOf(packed and TrustGraph.SOURCE_MASK) + val relation = TrustRelation.entries.first { it.code == (packed ushr TrustGraph.SOURCE_BITS) } + out.add(source to relation) + i++ + } + return out + } + + @Test + fun buildsFollowMuteAndReportEdges() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob, carol)) + b.addMutes(bob, listOf(dave)) + b.addReports(carol, listOf(dave)) + val graph = b.build() + + assertEquals(setOf(alice to TrustRelation.FOLLOW), graph.incomingOf(bob)) + assertEquals(setOf(alice to TrustRelation.FOLLOW), graph.incomingOf(carol)) + assertEquals( + setOf(bob to TrustRelation.MUTE, carol to TrustRelation.REPORT), + graph.incomingOf(dave), + ) + } + + @Test + fun dropsSelfEdges() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(alice, bob)) + val graph = b.build() + assertTrue(graph.incomingOf(alice).isEmpty(), "a self-follow must not become an edge") + assertEquals(setOf(alice to TrustRelation.FOLLOW), graph.incomingOf(bob)) + } + + @Test + fun dedupesRepeatedReports() { + val b = TrustGraphBuilder() + b.addReports(alice, listOf(dave)) + b.addReports(alice, listOf(dave)) + val graph = b.build() + assertEquals(1, graph.edgeCount()) + assertEquals(setOf(alice to TrustRelation.REPORT), graph.incomingOf(dave)) + } + + @Test + fun keepsFollowAndMuteFromSameSourceAsDistinctEdges() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob)) + b.addMutes(alice, listOf(bob)) + val graph = b.build() + assertEquals( + setOf(alice to TrustRelation.FOLLOW, alice to TrustRelation.MUTE), + graph.incomingOf(bob), + ) + } + + @Test + fun internsEachPubkeyOnce() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob, carol)) + b.addFollows(bob, listOf(carol)) + val graph = b.build() + assertEquals(3, graph.nodeCount, "alice, bob, carol interned once each") + assertEquals(3, graph.edgeCount()) + } +} diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClientTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClientTest.kt index 065ef0fcaf..7a7caaec58 100644 --- a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClientTest.kt +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/client/single/basic/BasicRelayClientTest.kt @@ -72,13 +72,15 @@ class BasicRelayClientTest { } @Test - fun onFailureWithMessageKeepsExistingFormat() { + fun onFailureWithMessageAppendsExceptionClassName() { val (socket, listener) = connectAndCapture() socket.onFailure(Exception("Connection reset"), null, null) + // The exception type is appended so listeners can classify the failure by + // its stable class name rather than by localized message text. assertEquals( - listOf("WebSocket Failure: Connection reset"), + listOf("WebSocket Failure: Connection reset (Exception)"), listener.cannotConnectMessages, ) } diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentCollectionsTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentCollectionsTest.kt new file mode 100644 index 0000000000..208054f818 --- /dev/null +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentCollectionsTest.kt @@ -0,0 +1,108 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFalse +import kotlin.test.assertNull +import kotlin.test.assertTrue + +class ConcurrentCollectionsTest { + @Test + fun mapGetSet() { + val m = ConcurrentMap() + assertNull(m["a"]) + m["a"] = 1 + assertEquals(1, m["a"]) + m["a"] = 2 + assertEquals(2, m["a"]) + assertEquals(1, m.size()) + } + + @Test + fun mapGetOrPutComputesOnce() { + val m = ConcurrentMap() + var calls = 0 + assertEquals( + 7, + m.getOrPut("k") { + calls++ + 7 + }, + ) + // Present now: the default must NOT be recomputed. + assertEquals( + 7, + m.getOrPut("k") { + calls++ + 99 + }, + ) + assertEquals(1, calls) + assertEquals(7, m["k"]) + } + + @Test + fun mapMergeInsertsThenCombines() { + val m = ConcurrentMap() + // Absent -> inserts the value verbatim, remap not applied. + assertEquals(1, m.merge("k", 1) { a, b -> a + b }) + // Present -> remap(existing, value). + assertEquals(4, m.merge("k", 3) { a, b -> a + b }) + assertEquals(4, m["k"]) + } + + @Test + fun mapSnapshotIsDetached() { + val m = ConcurrentMap() + m["a"] = 1 + m["b"] = 2 + val snap = m.snapshot() + assertEquals(mapOf("a" to 1, "b" to 2), snap) + // Mutating the map after the snapshot must not change the snapshot. + m["c"] = 3 + assertEquals(2, snap.size) + assertEquals(3, m.size()) + } + + @Test + fun setAddContainsSize() { + val s = ConcurrentSet() + assertFalse("x" in s) + assertTrue(s.add("x")) + // Re-adding is a no-op and reports it. + assertFalse(s.add("x")) + assertTrue("x" in s) + assertTrue(s.add("y")) + assertEquals(2, s.size()) + } + + @Test + fun setSnapshotIsDetached() { + val s = ConcurrentSet() + s.add("a") + val snap = s.snapshot() + s.add("b") + assertEquals(setOf("a"), snap) + assertEquals(2, s.size()) + } +} diff --git a/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.jvmAndroid.kt b/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.jvmAndroid.kt new file mode 100644 index 0000000000..aaf40fdcb7 --- /dev/null +++ b/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.jvmAndroid.kt @@ -0,0 +1,55 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +import java.util.concurrent.ConcurrentHashMap + +actual class ConcurrentMap { + private val map = ConcurrentHashMap() + + actual operator fun get(key: K): V? = map[key] + + actual operator fun set( + key: K, + value: V, + ) { + map[key] = value + } + + actual fun getOrPut( + key: K, + defaultValue: () -> V, + ): V = + // Fast-path the present-key hit (the common case in the crawl's hot + // relay-hint accumulation) so it never allocates the mapping-function + // closure; only an absent key pays for the atomic computeIfAbsent. + map[key] ?: map.computeIfAbsent(key) { defaultValue() } + + actual fun merge( + key: K, + value: V, + remap: (old: V, new: V) -> V, + ): V = map.merge(key, value) { old, new -> remap(old, new) }!! + + actual fun size(): Int = map.size + + actual fun snapshot(): Map = HashMap(map) +} diff --git a/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.jvmAndroid.kt b/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.jvmAndroid.kt new file mode 100644 index 0000000000..94a0754c11 --- /dev/null +++ b/quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.jvmAndroid.kt @@ -0,0 +1,35 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +import java.util.concurrent.ConcurrentHashMap + +actual class ConcurrentSet { + private val set: MutableSet = ConcurrentHashMap.newKeySet() + + actual fun add(element: E): Boolean = set.add(element) + + actual operator fun contains(element: E): Boolean = set.contains(element) + + actual fun size(): Int = set.size + + actual fun snapshot(): Set = HashSet(set) +} diff --git a/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.native.kt b/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.native.kt new file mode 100644 index 0000000000..de4ee540d8 --- /dev/null +++ b/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentMap.native.kt @@ -0,0 +1,80 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +import kotlin.concurrent.atomics.AtomicReference +import kotlin.concurrent.atomics.ExperimentalAtomicApi + +// Copy-on-write, mirroring ConcurrentHashCache.linux: correct and simple. The +// native targets never run the crawl this backs (it is JVM/Android-only work); +// they only compile it, so the O(n)-per-write cost is irrelevant. A CAS retry +// loop gives getOrPut/merge the same atomicity the JVM actual gets for free. +@OptIn(ExperimentalAtomicApi::class) +actual class ConcurrentMap { + private val ref = AtomicReference(HashMap()) + + actual operator fun get(key: K): V? = ref.load()[key] + + actual operator fun set( + key: K, + value: V, + ) { + while (true) { + val cur = ref.load() + val copy = HashMap(cur) + copy[key] = value + if (ref.compareAndSet(cur, copy)) return + } + } + + actual fun getOrPut( + key: K, + defaultValue: () -> V, + ): V { + while (true) { + val cur = ref.load() + cur[key]?.let { return it } + val value = defaultValue() + val copy = HashMap(cur) + copy[key] = value + if (ref.compareAndSet(cur, copy)) return value + } + } + + actual fun merge( + key: K, + value: V, + remap: (old: V, new: V) -> V, + ): V { + while (true) { + val cur = ref.load() + val old = cur[key] + val merged = if (old == null) value else remap(old, value) + val copy = HashMap(cur) + copy[key] = merged + if (ref.compareAndSet(cur, copy)) return merged + } + } + + actual fun size(): Int = ref.load().size + + actual fun snapshot(): Map = HashMap(ref.load()) +} diff --git a/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.native.kt b/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.native.kt new file mode 100644 index 0000000000..70bf86f133 --- /dev/null +++ b/quartz/src/nativeMain/kotlin/com/vitorpamplona/quartz/utils/concurrent/ConcurrentSet.native.kt @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.utils.concurrent + +import kotlin.concurrent.atomics.AtomicReference +import kotlin.concurrent.atomics.ExperimentalAtomicApi + +// Copy-on-write native actual — see ConcurrentMap.native for the rationale. +@OptIn(ExperimentalAtomicApi::class) +actual class ConcurrentSet { + private val ref = AtomicReference(HashSet()) + + actual fun add(element: E): Boolean { + while (true) { + val cur = ref.load() + if (element in cur) return false + val copy = HashSet(cur) + copy.add(element) + if (ref.compareAndSet(cur, copy)) return true + } + } + + actual operator fun contains(element: E): Boolean = element in ref.load() + + actual fun size(): Int = ref.load().size + + actual fun snapshot(): Set = HashSet(ref.load()) +}