From c8ff1bb18f18f0e22f87c22ffd1212a27c6ae049 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 18:23:43 +0000 Subject: [PATCH 1/6] feat(graperank): persist follower count and hop distance on trust cards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each kind:30382 GrapeRank card now carries two more public tags alongside `rank`: - `followers` — the number of the target's followers whose own score clears a threshold (`--followers-threshold`, default 0.02), mirroring Brainstorm's trusted-follower cutoff. - `hops` — the shortest follow-graph distance from the observer (1 = a direct follow), matching the `hops` field on Brainstorm's ScoreCard. New `HopsTag` (the `followers`/`FollowerCountTag` already existed) is wired through the ContactCardEvent tag accessors/builders. TrustGraph gains `hopsFrom` (a follow-only BFS over the compact int-CSR) and `trustedFollowerCounts`; the out-CSR now packs the relation code so a forward walk can filter FOLLOW edges. The publisher's `reconcileLocal` takes a richer `ScoredCard` and diffs the full (rank, followers, hops) triple, so a card re-signs when any of them moves and older cards migrate onto the new tags once. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- cli/README.md | 16 ++-- .../amethyst/cli/commands/GrapeRankCommand.kt | 38 ++++++++- .../graperank/GrapeRankPublisher.kt | 72 +++++++++++----- .../experimental/graperank/TrustGraph.kt | 83 ++++++++++++++++++- .../graperank/TrustGraphBuilder.kt | 14 ++-- .../users/ContactCardEvent.kt | 2 + .../users/TagArrayBuilderExt.kt | 3 + .../users/TagArrayExt.kt | 3 + .../users/tags/HopsTag.kt | 45 ++++++++++ .../graperank/GrapeRankPublisherTest.kt | 49 +++++++++-- .../graperank/TrustGraphBuilderTest.kt | 72 ++++++++++++++++ 11 files changed, 354 insertions(+), 43 deletions(-) create mode 100644 quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/tags/HopsTag.kt diff --git a/cli/README.md b/cli/README.md index 468211a712..c78efa9d56 100644 --- a/cli/README.md +++ b/cli/README.md @@ -389,7 +389,7 @@ HTTP endpoint. Reuses quartz's `Nip86Client` and the shared `Nip86Retriever` | `amy profile show [USER]` | Print kind:0 metadata. USER accepts npub/nprofile/hex/NIP-05; defaults to self. | | `amy profile edit --name … --about … --picture URL …` | Patch and re-publish your kind:0. | | `amy follow USER` / `amy unfollow USER` | Add/remove USER from your kind:3 contact list (fetches the freshest list first). | -| `amy graperank [OBSERVER] [--offline] [--min-rank N]` | Crawl + score: compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, then persist the result. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. **Every score run persists its result locally**: the ranks (cutoff `--min-rank`, default 2) are reconciled into the shared store as NIP-85 kind:30382 cards signed by a per-observer **service key** — changed ranks re-signed, unchanged skipped (no event-id churn), dropped targets retracted (kind:5). `--offline` skips the crawl. | +| `amy graperank [OBSERVER] [--offline] [--min-rank N]` | Crawl + score: compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, then persist the result. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. **Every score run persists its result locally**: the cards (rank cutoff `--min-rank`, default 2) are reconciled into the shared store as NIP-85 kind:30382 cards signed by a per-observer **service key** — each carries `rank`, `followers` (trusted-follower count, cutoff `--followers-threshold`, default 0.02) and `hops` (follow distance from the observer); changed cards re-signed, unchanged skipped (no event-id churn), dropped targets retracted (kind:5). `--offline` skips the crawl. | | `amy graperank crawl [OBSERVER] [--max-hops N] [--no-preconnect]` | Pipeline stage 1 — network only: crawl the follow/mute/report graph (kind 3/10000/1984/10002) into the local store, no scoring. Idempotent and cumulative: run it a few times to load everything, then `score`. | | `amy graperank score [OBSERVER]` | Pipeline stage 2 — local only: score from the store and persist the cards (identical to bare `--offline`; same scoring flags). No network, so re-run with different `--rigor`/`--attenuation`/`--min-rank` without re-crawling. | | `amy graperank publish [OBSERVER] [--relay URL[,URL…]]` | Pipeline stage 3 — transport only: make the operator relay(s) converge to the locally persisted card set — one NIP-77 up-only reconcile per relay over the service key's kind:30382 + kind:5 (nothing is re-scored or re-signed; a relay that can't reconcile gets the full set published instead). Also refreshes the observer's kind:10040 pointer when we hold their key. | @@ -420,12 +420,16 @@ means re-signing **replaces** a target's card instead of orphaning it — and losing everything but the master seed still re-derives every key. **Every score run persists its cards.** After scoring, Amy reconciles the result -into the local store: new or changed ranks (≥ `--min-rank`, default 2) are -signed; unchanged ranks are skipped (no new event id); and any card whose target +into the local store: new or changed cards (rank ≥ `--min-rank`, default 2) are +signed; unchanged cards are skipped (no new event id); and any card whose target dropped out of the graph or fell below the cutoff is **retracted** with a kind:5 -(the store applies it; the tombstone is kept). The local store is the source of -truth — `graperank rank USER` reads it offline, and `graperank publish` mirrors -it out: +(the store applies it; the tombstone is kept). Each card carries three public +tags: `rank` (`round(score*100)`), `followers` — the number of the target's +followers whose own score clears `--followers-threshold` (default 0.02, matching +Brainstorm's trusted-follower cutoff) — and `hops`, the shortest follow-graph +distance from the observer (1 = a direct follow). A change to *any* of the three +re-signs the card. The local store is the source of truth — `graperank rank USER` +reads it offline, and `graperank publish` mirrors it out: ```bash amy graperank operator relay wss://relay.example.com # where all cards live diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt index 302fc6d02e..36a5aa6b36 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -146,6 +146,10 @@ object GrapeRankCommand { private const val PROBE_TIMEOUT_MS = 2000 + // Default score cutoff for counting a follower as "trusted" (the `followers` + // tag). Matches NosFabrica Brainstorm's verifiedFollowersInfluenceCutoff. + private const val DEFAULT_FOLLOWERS_THRESHOLD = 0.02 + // args.bool on a mistyped flag silently returns false, so the one flag read from // three different functions goes through a compile-time-checked name. private const val NO_REACHABILITY_CACHE_FLAG = "no-reachability-cache" @@ -248,6 +252,10 @@ object GrapeRankCommand { // retracted. Rank is round(score*100), so 2 drops the ~0.015-and-below // barely-trusted tail. val minRank = args.intFlag("min-rank", 2) + // A "trusted follower" (the `followers` tag) is a follower whose own score is + // at or above this. Mirrors Brainstorm's verifiedFollowersInfluenceCutoff + // (0.02), which is the same 0.02 score == rank 2 line as the default min-rank. + val followersThreshold = args.flag("followers-threshold")?.toDoubleOrNull() ?: DEFAULT_FOLLOWERS_THRESHOLD val params = GrapeRankParams( @@ -327,6 +335,14 @@ object GrapeRankCommand { val scoringMs = (System.nanoTime() - scoreStart) / 1_000_000 System.err.println("[graperank] scored ${rankedIds.size} users in $scoringMs ms") + // Two derived per-user metrics persisted alongside the rank on each card: + // - trusted-follower count: how many of a user's followers score at or + // above the cutoff (Brainstorm's trustedFollowers). + // - hops: shortest follow-graph distance from the observer (1 = direct + // follow). Both are pure functions over the same graph + scores. + val followerCounts = graph.trustedFollowerCounts(scores, followersThreshold) + val hops = graph.hopsFrom(observer) + val hopHistogram = crawlStats?.hopHistogram.orEmpty() val result = linkedMapOf( @@ -352,9 +368,16 @@ object GrapeRankCommand { "graph_build_ms" to buildMs, "scoring_ms" to scoringMs, "scoring_sweeps" to sweeps, + "followers_threshold" to followersThreshold, "scores" to rankedIds.take(limit).map { - mapOf("pubkey" to graph.pubkeyOf(it), "score" to scores[it], "rank" to rankOf(scores[it])) + mapOf( + "pubkey" to graph.pubkeyOf(it), + "score" to scores[it], + "rank" to rankOf(scores[it]), + "followers" to followerCounts[it], + "hops" to hops[it], + ) }, ) @@ -372,7 +395,16 @@ object GrapeRankCommand { val desiredCards = rankedIds .filter { rankOf(scores[it]) >= minRank } - .map { graph.pubkeyOf(it) to rankOf(scores[it]) } + .map { id -> + GrapeRankPublisher.ScoredCard( + target = graph.pubkeyOf(id), + rank = rankOf(scores[id]), + followers = followerCounts[id], + // A scored user always has a follow path from the observer, + // so hops is ≥ 1; guard the UNREACHABLE sentinel just in case. + hops = hops[id].takeIf { it >= 1 }, + ) + } val cardsStart = System.nanoTime() val local = @@ -908,6 +940,8 @@ object GrapeRankCommand { linkedMapOf( "provider" to card.pubKey, "rank" to card.rank(), + "followers" to card.followerCount(), + "hops" to card.hops(), "observer" to providerToObserver[card.pubKey], "created_at" to card.createdAt, "event_id" to card.id, diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt index 9ff48d83ef..89b90b79a4 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisher.kt @@ -31,6 +31,11 @@ import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner import com.vitorpamplona.quartz.nip01Core.store.IEventStore import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent import com.vitorpamplona.quartz.nip85TrustedAssertions.users.ContactCardEvent +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.followerCount +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.hops +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.rank +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag import com.vitorpamplona.quartz.utils.TimeUtils import kotlinx.coroutines.Dispatchers @@ -62,6 +67,21 @@ class GrapeRankPublisher( private val store: IEventStore, private val log: (String) -> Unit = {}, ) { + /** + * One desired kind:30382 card: the GrapeRank result for [target] the caller + * wants persisted. [rank] (`round(score*100)`) is always written; [followers] + * (trusted-follower count) and [hops] (follow-graph distance from the observer) + * are optional — a `null` omits that tag, so a caller that only knows the rank + * still produces a valid card, and older cards missing those tags reconcile + * cleanly against a run that now supplies them. + */ + class ScoredCard( + val target: HexKey, + val rank: Int, + val followers: Int? = null, + val hops: Int? = null, + ) + /** Outcome of one [reconcileLocal]: what was signed, retracted, and left alone. */ class LocalResult( /** New or rank-changed cards signed and inserted into the store. */ @@ -73,25 +93,27 @@ class GrapeRankPublisher( ) /** - * Reconcile the desired [scored] `(target, rank)` set (the caller has already - * applied any rank cutoff) into the local store, signed by [providerSigner]: - * upsert the changed cards, retract the stale ones, skip the unchanged rest. - * Every card and retraction is stamped [createdAt], so a replacement is - * strictly newer than what it displaces. + * Reconcile the desired [scored] card set (the caller has already applied any + * rank cutoff) into the local store, signed by [providerSigner]: upsert the + * changed cards, retract the stale ones, skip the unchanged rest. Every card and + * retraction is stamped [createdAt], so a replacement is strictly newer than what + * it displaces. */ suspend fun reconcileLocal( providerSigner: NostrSigner, providerPubkey: HexKey, - scored: List>, + scored: List, createdAt: Long = TimeUtils.now(), ): LocalResult { val existing = existingCards(providerPubkey) - val desiredTargets = scored.mapTo(HashSet()) { it.first } + val desiredTargets = scored.mapTo(HashSet()) { it.target } - // Upsert targets whose rank tag STRING would change (or that have no card - // yet). RankTag.assemble writes rank.toString(), so we diff that exact - // string — an unchanged score never produces a new signature. - val changed = scored.filter { (target, rank) -> existing[target]?.let(::rankTagValue) != rank.toString() } + // Upsert targets whose card values would change (or that have no card yet). + // We diff every tag we write — rank, follower count, and hops — so a card + // re-signs when any of them moves, but an all-unchanged run never churns an + // event id. A card predating the follower/hops tags reads null for them and + // so re-signs once to pick them up. + val changed = scored.filter { card -> existing[card.target]?.let(::cardValues) != desiredValues(card) } // Retract cards whose target is no longer desired — it dropped out of the // graph, or fell below the caller's cutoff. No stale assertion is left standing. @@ -203,15 +225,15 @@ class GrapeRankPublisher( }.toMap() /** - * The raw `rank` tag value string on a card — exactly what a client diffs, so an - * unchanged score never produces a new signature. Our cards carry only a `rank` - * tag (plus the d-tag target), so this one value decides whether a re-sign - * would differ. + * The (rank, followers, hops) triple stored on an existing card — the values a + * re-sign would overwrite. Compared against [desiredValues] so an unchanged run + * produces no new signature; a missing tag reads null and so differs from a + * desired non-null value (the one-time migration onto the new tags). */ - private fun rankTagValue(card: ContactCardEvent): String? = - card.tags.firstNotNullOfOrNull { tag -> - if (tag.size > 1 && tag[0] == RankTag.TAG_NAME) tag[1] else null - } + private fun cardValues(card: ContactCardEvent): Triple = Triple(card.rank(), card.followerCount(), card.hops()) + + /** The (rank, followers, hops) triple a [ScoredCard] would write. */ + private fun desiredValues(card: ScoredCard): Triple = Triple(card.rank, card.followers, card.hops) /** * Build + sign one kind:30382 card per (target, rank) — fanned out on @@ -221,7 +243,7 @@ class GrapeRankPublisher( */ private suspend fun signAndInsertCards( signer: NostrSigner, - cards: List>, + cards: List, createdAt: Long, ): Int { if (cards.isEmpty()) return 0 @@ -230,13 +252,17 @@ class GrapeRankPublisher( val signedBatch = coroutineScope { batch - .map { (target, rank) -> + .map { card -> async(Dispatchers.Default) { ContactCardEvent.create( - targetUser = target, + targetUser = card.target, signer = signer, createdAt = createdAt, - publicInitializer = { add(RankTag.assemble(rank)) }, + publicInitializer = { + add(RankTag.assemble(card.rank)) + card.followers?.let { add(FollowerCountTag.assemble(it)) } + card.hops?.let { add(HopsTag.assemble(it)) } + }, ) } }.awaitAll() diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt index ad78a6e56a..829ac7d3c2 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraph.kt @@ -60,9 +60,11 @@ class TrustGraph internal constructor( // each packing source id (low 29 bits) + relation code (top bits). internal val inOffsets: IntArray, internal val inPacked: IntArray, - // CSR by source: out-neighbour targets of node s are outTargets[outOffsets[s] until outOffsets[s+1]]. + // CSR by source: out-edges of node s are outPacked[outOffsets[s] until outOffsets[s+1]], + // each packing the neighbour (target) id (low 29 bits) + relation code (top bits) — + // same layout as inPacked, so a forward walk can filter by relation. internal val outOffsets: IntArray, - internal val outTargets: IntArray, + internal val outPacked: IntArray, ) { /** Node id for [pubkey], or `-1` if it never appeared in the graph. */ fun idOf(pubkey: HexKey): Int = ids[pubkey] ?: -1 @@ -72,10 +74,87 @@ class TrustGraph internal constructor( fun edgeCount(): Int = inPacked.size + /** + * Hop distance from [observer] to every node along **FOLLOW edges only** — a + * plain BFS over the follow graph (mutes/reports are not reachability). Indexed + * by node id: the observer is `0`, a user the observer follows directly is `1`, + * a user followed by a 1-hop user is `2`, and so on. A node with no follow path + * from the observer stays [UNREACHABLE]. Returns an all-[UNREACHABLE] array (bar + * a missing observer's own 0) when the observer isn't in the graph. + * + * BFS over the compact int-CSR: an `IntArray` ring queue and the distance array + * itself as the visited set, so a whole-network graph costs one int per node + * plus the queue — no boxed collections. + */ + fun hopsFrom(observer: HexKey): IntArray { + val hops = IntArray(nodeCount) { UNREACHABLE } + val observerId = idOf(observer) + if (observerId < 0) return hops + + hops[observerId] = 0 + // Ring buffer sized to the node count — BFS enqueues each node at most once. + val queue = IntArray(nodeCount) + var head = 0 + var tail = 0 + queue[tail++] = observerId + while (head < tail) { + val node = queue[head++] + val nextHop = hops[node] + 1 + var i = outOffsets[node] + val end = outOffsets[node + 1] + while (i < end) { + val packed = outPacked[i] + if ((packed ushr SOURCE_BITS) == TrustRelation.FOLLOW.code) { + val target = packed and SOURCE_MASK + if (hops[target] == UNREACHABLE) { + hops[target] = nextHop + queue[tail++] = target + } + } + i++ + } + } + return hops + } + + /** + * For every node, how many of its **followers** score at or above [minScore] in + * [scores] — the "trusted follower" count Brainstorm records on its `ScoreCard` + * (its `verifiedFollowersInfluenceCutoff` defaults to 0.02). A follower is the + * source of an incoming FOLLOW edge; mutes/reports don't count. Indexed by node + * id. [scores] must be the array [GrapeRank.compute] returned for this graph. + */ + fun trustedFollowerCounts( + scores: DoubleArray, + minScore: Double, + ): IntArray { + val counts = IntArray(nodeCount) + var target = 0 + while (target < nodeCount) { + var count = 0 + var i = inOffsets[target] + val end = inOffsets[target + 1] + while (i < end) { + val packed = inPacked[i] + if ((packed ushr SOURCE_BITS) == TrustRelation.FOLLOW.code) { + val source = packed and SOURCE_MASK + if (scores[source] >= minScore) count++ + } + i++ + } + counts[target] = count + target++ + } + return counts + } + companion object { const val SOURCE_BITS = 29 const val SOURCE_MASK = (1 shl SOURCE_BITS) - 1 const val MAX_NODES = SOURCE_MASK // ids must fit in the low 29 bits + + /** [hopsFrom] distance for a node with no follow path from the observer. */ + const val UNREACHABLE = -1 } } diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt index 7b3f12fbe4..a4bb31f632 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilder.kt @@ -108,20 +108,24 @@ class TrustGraphBuilder { inPacked[inCursor[t]++] = edgeSourcesPacked.get(i) } - // Outgoing CSR (by source). + // Outgoing CSR (by source). Each entry packs the target id + relation code + // in the same layout as the incoming CSR, so a forward walk (e.g. the + // follow-only BFS in TrustGraph.hopsFrom) can filter edges by relation. val outOffsets = IntArray(n + 1) for (i in 0 until m) { val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK outOffsets[s + 1]++ } for (i in 1..n) outOffsets[i] += outOffsets[i - 1] - val outTargets = IntArray(m) + val outPacked = IntArray(m) val outCursor = outOffsets.copyOf() for (i in 0 until m) { - val s = edgeSourcesPacked.get(i) and TrustGraph.SOURCE_MASK - outTargets[outCursor[s]++] = edgeTargets.get(i) + val packedSource = edgeSourcesPacked.get(i) + val s = packedSource and TrustGraph.SOURCE_MASK + val relationCode = packedSource ushr TrustGraph.SOURCE_BITS + outPacked[outCursor[s]++] = edgeTargets.get(i) or (relationCode shl TrustGraph.SOURCE_BITS) } - return TrustGraph(n, pubkeys.toTypedArray(), ids, inOffsets, inPacked, outOffsets, outTargets) + return TrustGraph(n, pubkeys.toTypedArray(), ids, inOffsets, inPacked, outOffsets, outPacked) } } diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/ContactCardEvent.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/ContactCardEvent.kt index 6f276a3716..d2e38ed3ff 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/ContactCardEvent.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/ContactCardEvent.kt @@ -61,6 +61,8 @@ class ContactCardEvent( fun followerCount() = tags.followerCount() + fun hops() = tags.hops() + fun firstCreatedAt() = tags.firstCreatedAt() fun postCount() = tags.postCount() diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayBuilderExt.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayBuilderExt.kt index 88d87d7359..161912e25d 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayBuilderExt.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayBuilderExt.kt @@ -25,6 +25,7 @@ import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursEnd import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursStartTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FirstCreatedAtTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PetNameTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PostCountTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag @@ -45,6 +46,8 @@ fun TagArrayBuilder.rank(rank: Int) = addUnique(RankTag.assemb fun TagArrayBuilder.followers(count: Int) = addUnique(FollowerCountTag.assemble(count)) +fun TagArrayBuilder.hops(hops: Int) = addUnique(HopsTag.assemble(hops)) + fun TagArrayBuilder.firstCreatedAt(timestamp: Long) = addUnique(FirstCreatedAtTag.assemble(timestamp)) fun TagArrayBuilder.postCount(count: Int) = addUnique(PostCountTag.assemble(count)) diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayExt.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayExt.kt index fad3a43cc8..c406e73727 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayExt.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/TagArrayExt.kt @@ -26,6 +26,7 @@ import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursEnd import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.ActiveHoursStartTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FirstCreatedAtTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.FollowerCountTag +import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.HopsTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PetNameTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.PostCountTag import com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags.RankTag @@ -46,6 +47,8 @@ fun TagArray.rank() = fastFirstNotNullOfOrNull(RankTag::parse) fun TagArray.followerCount() = fastFirstNotNullOfOrNull(FollowerCountTag::parse) +fun TagArray.hops() = fastFirstNotNullOfOrNull(HopsTag::parse) + fun TagArray.firstCreatedAt() = fastFirstNotNullOfOrNull(FirstCreatedAtTag::parse) fun TagArray.postCount() = fastFirstNotNullOfOrNull(PostCountTag::parse) diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/tags/HopsTag.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/tags/HopsTag.kt new file mode 100644 index 0000000000..cd5c2d71ce --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/users/tags/HopsTag.kt @@ -0,0 +1,45 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.nip85TrustedAssertions.users.tags + +import com.vitorpamplona.quartz.nip01Core.core.has +import com.vitorpamplona.quartz.utils.ensure + +/** + * The number of follow hops from the observer to the card's target — the length + * of the shortest follow path in the observer's web of trust. A user the observer + * follows directly is 1 hop; a user followed by someone the observer follows is 2; + * and so on. Mirrors the `hops` field on Brainstorm's GrapeRank `ScoreCard`. + */ +class HopsTag { + companion object { + const val TAG_NAME = "hops" + + fun parse(tag: Array): Int? { + ensure(tag.has(1)) { return null } + ensure(tag[0] == TAG_NAME) { return null } + ensure(tag[1].isNotEmpty()) { return null } + return tag[1].toIntOrNull() + } + + fun assemble(hops: Int) = arrayOf(TAG_NAME, hops.toString()) + } +} diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisherTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisherTest.kt index cbd3fbb6de..a5ef8d76eb 100644 --- a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisherTest.kt +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/GrapeRankPublisherTest.kt @@ -20,6 +20,7 @@ */ package com.vitorpamplona.quartz.experimental.graperank +import com.vitorpamplona.quartz.experimental.graperank.GrapeRankPublisher.ScoredCard import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter @@ -64,7 +65,7 @@ class GrapeRankPublisherTest { val c = hexKey(0xC) // First run: every card is new. - val r1 = publisher.reconcileLocal(signer, provider, listOf(a to 50, b to 10), createdAt = 1_000L) + val r1 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 50), ScoredCard(b, 10)), createdAt = 1_000L) assertEquals(2, r1.signed) assertEquals(0, r1.unchanged) assertEquals(0, r1.retracted) @@ -76,7 +77,7 @@ class GrapeRankPublisherTest { .toSet() // Same ranks again: nothing is re-signed, no event id churns. - val r2 = publisher.reconcileLocal(signer, provider, listOf(a to 50, b to 10), createdAt = 2_000L) + val r2 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 50), ScoredCard(b, 10)), createdAt = 2_000L) assertEquals(0, r2.signed) assertEquals(2, r2.unchanged) assertEquals(0, r2.retracted) @@ -88,7 +89,7 @@ class GrapeRankPublisherTest { assertEquals(firstIds, secondIds) // a's rank moved, b dropped out, c is new: a + c signed, b retracted. - val r3 = publisher.reconcileLocal(signer, provider, listOf(a to 60, c to 5), createdAt = 3_000L) + val r3 = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 60), ScoredCard(c, 5)), createdAt = 3_000L) assertEquals(2, r3.signed) assertEquals(0, r3.unchanged) assertEquals(1, r3.retracted) @@ -113,16 +114,54 @@ class GrapeRankPublisherTest { val a = hexKey(0xA) - publisher.reconcileLocal(signer, provider, listOf(a to 40), createdAt = 1_000L) + publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 40)), createdAt = 1_000L) publisher.reconcileLocal(signer, provider, emptyList(), createdAt = 2_000L) assertEquals(emptyMap(), cardsByTarget(store, provider)) // A NEWER card outranks the older kind:5 (NIP-09 deletions only cover // versions up to their created_at), so the target comes back cleanly. - val r = publisher.reconcileLocal(signer, provider, listOf(a to 45), createdAt = 3_000L) + val r = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, 45)), createdAt = 3_000L) assertEquals(1, r.signed) assertEquals(mapOf(a to 45), cardsByTarget(store, provider)) + store.close() + } + + @Test + fun writesFollowerAndHopTagsAndReSignsWhenTheyMove() = + runBlocking { + val store = EventStore(null) + val signer = NostrSignerInternal(KeyPair()) + val provider = signer.pubKey + val publisher = GrapeRankPublisher(store) + + val a = hexKey(0xA) + + suspend fun cardFor(target: String): ContactCardEvent = + store + .query(Filter(kinds = listOf(ContactCardEvent.KIND), authors = listOf(provider))) + .filterIsInstance() + .first { it.aboutUser() == target } + + // A card carrying rank + followers + hops persists all three tags. + publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 7, hops = 2)), createdAt = 1_000L) + cardFor(a).let { + assertEquals(50, it.rank()) + assertEquals(7, it.followerCount()) + assertEquals(2, it.hops()) + } + + // Rank unchanged but the follower count moved → the card re-signs. + val moved = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 9, hops = 2)), createdAt = 2_000L) + assertEquals(1, moved.signed) + assertEquals(0, moved.unchanged) + assertEquals(9, cardFor(a).followerCount()) + + // Everything identical → no re-sign. + val same = publisher.reconcileLocal(signer, provider, listOf(ScoredCard(a, rank = 50, followers = 9, hops = 2)), createdAt = 3_000L) + assertEquals(0, same.signed) + assertEquals(1, same.unchanged) + store.close() } } diff --git a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt index f27e064273..c7041264cf 100644 --- a/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt +++ b/quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/experimental/graperank/TrustGraphBuilderTest.kt @@ -104,4 +104,76 @@ class TrustGraphBuilderTest { assertEquals(3, graph.nodeCount, "alice, bob, carol interned once each") assertEquals(3, graph.edgeCount()) } + + private fun TrustGraph.hop( + observer: HexKey, + target: HexKey, + ): Int { + val id = idOf(target) + return if (id < 0) TrustGraph.UNREACHABLE else hopsFrom(observer)[id] + } + + @Test + fun hopsCountFollowDistanceFromTheObserver() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob)) // alice -> bob (1 hop) + b.addFollows(bob, listOf(carol)) // bob -> carol (2 hops) + b.addFollows(carol, listOf(dave)) // carol -> dave (3 hops) + val graph = b.build() + + assertEquals(0, graph.hop(alice, alice), "the observer is 0 hops from itself") + assertEquals(1, graph.hop(alice, bob), "a direct follow is 1 hop") + assertEquals(2, graph.hop(alice, carol)) + assertEquals(3, graph.hop(alice, dave)) + } + + @Test + fun hopsTakeTheShortestFollowPath() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob, dave)) // dave is also a direct follow… + b.addFollows(bob, listOf(carol)) + b.addFollows(carol, listOf(dave)) // …as well as reachable at 3 hops + val graph = b.build() + assertEquals(1, graph.hop(alice, dave), "BFS keeps the shortest of two follow paths") + } + + @Test + fun hopsIgnoreMuteAndReportEdges() { + val b = TrustGraphBuilder() + b.addMutes(alice, listOf(bob)) // a mute is not reachability + b.addReports(alice, listOf(carol)) // neither is a report + val graph = b.build() + assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, bob), "a muted user is not reachable by follows") + assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, carol), "a reported user is not reachable by follows") + } + + @Test + fun hopsAreUnreachableWithNoFollowPath() { + val b = TrustGraphBuilder() + b.addFollows(alice, listOf(bob)) + b.addFollows(carol, listOf(dave)) // a separate island + val graph = b.build() + assertEquals(TrustGraph.UNREACHABLE, graph.hop(alice, dave)) + } + + @Test + fun trustedFollowerCountsOnlyCountFollowersAboveTheThreshold() { + val b = TrustGraphBuilder() + // carol is followed by alice, bob and dave (three follow edges in). + b.addFollows(alice, listOf(carol)) + b.addFollows(bob, listOf(carol)) + b.addFollows(dave, listOf(carol)) + // dave also MUTES carol — a mute must never count as a follower. + b.addMutes(dave, listOf(carol)) + val graph = b.build() + + val carolId = graph.idOf(carol) + val scores = DoubleArray(graph.nodeCount) + scores[graph.idOf(alice)] = 0.9 // trusted + scores[graph.idOf(bob)] = 0.01 // below the 0.02 cutoff + scores[graph.idOf(dave)] = 0.5 // trusted + + val counts = graph.trustedFollowerCounts(scores, minScore = 0.02) + assertEquals(2, counts[carolId], "only alice and dave clear the cutoff; bob is too low and dave's mute doesn't count") + } } From 5f65405b08f5251e34586fdac8e1fe81254bb139 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 19:33:46 +0000 Subject: [PATCH 2/6] feat(graperank): add reverse follower crawl (amy graperank followers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The outbox model can't find an observer's followers — you don't know a follower exists until you've seen their kind:3, so you can't route to their outbox first. FollowerCrawler casts a wide net instead: it asks as many relays as possible for kind:3 lists that #p-tag the observer, paging each relay past its per-REQ cap via fetchAllPagesFromPool, verifies with ParallelEventVerifier, keeps only lists that genuinely tag the observer, dedups by id, and group-commits to the store. Each follower's list is a full contact list, so persisting it also enriches the graph a later `graperank score` builds — every follower becomes a FOLLOW edge into the observer. CLI: `amy graperank followers [OBSERVER]` assembles "all possible relays" from the reachability-cache live set + every kind:10002/30166 relay in the store + the index/aggregator relays, skipping proven-dead relays. Runs anonymously (no signing) when given an explicit observer. Tunable via --relay/--page-limit/--timeout/--relay-concurrency/--insert-batch. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- cli/README.md | 1 + .../amethyst/cli/commands/GrapeRankCommand.kt | 119 +++++++++++ .../experimental/graperank/FollowerCrawler.kt | 189 ++++++++++++++++++ 3 files changed, 309 insertions(+) create mode 100644 quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt diff --git a/cli/README.md b/cli/README.md index c78efa9d56..257e5af825 100644 --- a/cli/README.md +++ b/cli/README.md @@ -391,6 +391,7 @@ HTTP endpoint. Reuses quartz's `Nip86Client` and the shared `Nip86Retriever` | `amy follow USER` / `amy unfollow USER` | Add/remove USER from your kind:3 contact list (fetches the freshest list first). | | `amy graperank [OBSERVER] [--offline] [--min-rank N]` | Crawl + score: compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, then persist the result. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap), dropping reports the author retracted via NIP-09. **Every score run persists its result locally**: the cards (rank cutoff `--min-rank`, default 2) are reconciled into the shared store as NIP-85 kind:30382 cards signed by a per-observer **service key** — each carries `rank`, `followers` (trusted-follower count, cutoff `--followers-threshold`, default 0.02) and `hops` (follow distance from the observer); changed cards re-signed, unchanged skipped (no event-id churn), dropped targets retracted (kind:5). `--offline` skips the crawl. | | `amy graperank crawl [OBSERVER] [--max-hops N] [--no-preconnect]` | Pipeline stage 1 — network only: crawl the follow/mute/report graph (kind 3/10000/1984/10002) into the local store, no scoring. Idempotent and cumulative: run it a few times to load everything, then `score`. | +| `amy graperank followers [OBSERVER] [--relay URL[,URL…]]` | The **reverse** crawl: find every user who *follows* the observer by asking as many relays as possible for kind:3 lists that `#p`-tag them (paged past each relay's cap). The outbox model can't find followers — you don't know one exists until you've seen their list — so this casts a wide net over the whole relay universe (reachability-cache live set + every kind:10002/30166 relay in the store + index/aggregator relays), skipping proven-dead relays. Persists each follower's list, enriching the graph a later `score` builds. Idempotent and cumulative. | | `amy graperank score [OBSERVER]` | Pipeline stage 2 — local only: score from the store and persist the cards (identical to bare `--offline`; same scoring flags). No network, so re-run with different `--rigor`/`--attenuation`/`--min-rank` without re-crawling. | | `amy graperank publish [OBSERVER] [--relay URL[,URL…]]` | Pipeline stage 3 — transport only: make the operator relay(s) converge to the locally persisted card set — one NIP-77 up-only reconcile per relay over the service key's kind:30382 + kind:5 (nothing is re-scored or re-signed; a relay that can't reconcile gets the full set published instead). Also refreshes the observer's kind:10040 pointer when we hold their key. | | `amy graperank rank USER [--provider PUBKEY] [--refresh]` | The consumer side: read the kind:30382 cards about USER — one rank per provider, newest card each. Local store first; `--refresh` (or a miss) drains the operator relays, the relays your kind:10040 declares, and the bootstrap set. | diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt index 36a5aa6b36..e028d0335e 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -26,6 +26,7 @@ import com.vitorpamplona.amethyst.cli.DataDir import com.vitorpamplona.amethyst.cli.Output import com.vitorpamplona.amethyst.commons.defaults.Constants import com.vitorpamplona.amethyst.commons.defaults.DefaultIndexerRelayList +import com.vitorpamplona.quartz.experimental.graperank.FollowerCrawler import com.vitorpamplona.quartz.experimental.graperank.GrapeRank import com.vitorpamplona.quartz.experimental.graperank.GrapeRankCrawler import com.vitorpamplona.quartz.experimental.graperank.GrapeRankParams @@ -213,6 +214,7 @@ object GrapeRankCommand { "providers" -> providers(dataDir, tail.drop(1).toTypedArray()) "operator" -> operator(dataDir, tail.drop(1).toTypedArray()) "crawl" -> crawl(dataDir, tail.drop(1).toTypedArray()) + "followers" -> followers(dataDir, tail.drop(1).toTypedArray()) "status" -> status(dataDir) // The relay census outgrew graperank (it feeds the shared NIP-66 // reachability cache every command reads) and moved to `amy relay @@ -586,6 +588,123 @@ object GrapeRankCommand { return 0 } + /** + * `amy graperank followers [OBSERVER] [flags]` — the reverse crawl: find every + * user who FOLLOWS the observer and persist their kind:3 contact lists. + * + * The outbox model (what `crawl` uses) walks follows *outward* and can't find + * followers — you don't know a follower exists until you've seen their list, so + * you can't route to their outbox first. This casts a wide net instead: it asks + * as many relays as possible for kind:3 events that `#p`-tag the observer, paging + * each relay past its per-REQ cap. The relay universe is "all possible relays": + * the reachability-cache live set + every kind:10002 read/write relay and every + * kind:30166 monitored relay in the store + the index/aggregator relays that hold + * reverse-follow data for the network. Proven-dead relays are skipped + * (`--no-reachability-cache` to query them anyway). + * + * Each follower's list lands in the store, so it also enriches the graph a later + * `graperank score` builds — every follower becomes a FOLLOW edge into the + * observer. Idempotent and cumulative; run it a few times for completeness. + * + * Flags: `--relay URL[,URL…]` (query only these instead of the whole universe), + * `--page-limit N` (per-page REQ limit, default 500), `--timeout SECS` (per-page + * EOSE watchdog, default 15), `--relay-concurrency N` (relays paged at once, + * default 16), `--insert-batch N` (events per store commit, default 500). + */ + private suspend fun followers( + dataDir: DataDir, + rest: Array, + ): Int { + val args = Args(rest) + val observerArg = args.positionalOrNull(0) + val relayArg = args.flag("relay") + val relayConcurrency = args.intFlag("relay-concurrency", args.intFlag("concurrency", 16)) + + // Read-only + never signs, so it runs anonymously — but then an OBSERVER + // positional is required (there's no account to default to). + Context.openOrAnonymous(dataDir).use { ctx -> + ctx.prepare() + if (ctx.anonymous && observerArg == null) { + return Output.error("bad_args", "usage: amy graperank followers OBSERVER [flags] (no account — pass an observer)") + } + val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex + + val relays = + relayArg + ?.split(",") + ?.mapNotNull { RelayUrlNormalizer.normalizeOrNull(it.trim()) } + ?.toSet() + ?.takeIf { it.isNotEmpty() } + ?: allKnownRelays(ctx, args) + + if (relays.isEmpty()) { + return Output.error("no_relays", "no relays to query — run `amy relay probe` / `amy graperank crawl` first, or pass --relay") + } + + System.err.println("[followers] querying ${relays.size} relays for kind:3 #p=${observer.take(8)}…") + val crawler = + FollowerCrawler( + client = ctx.client, + store = ctx.store, + config = + FollowerCrawler.Config( + relays = relays, + pageLimit = args.intFlag("page-limit", 500), + timeoutMs = args.longFlag("timeout", 15L) * 1000, + maxConcurrentRelays = relayConcurrency, + insertBatchSize = args.intFlag("insert-batch", 500), + ), + log = { System.err.println(it) }, + ) + val stats = crawler.crawl(observer) + + Output.emit( + linkedMapOf( + "observer" to observer, + "relays_queried" to stats.relaysQueried, + "relays_answered" to stats.relaysAnswered, + "followers_found" to stats.followersFound, + "events_stored" to stats.eventsStored, + "download_ms" to stats.downloadMs, + ), + ) + } + return 0 + } + + /** + * "All possible relays" for the follower crawl: the reachability-cache live set, + * every kind:10002 read/write relay and every kind:30166 monitored relay the + * store knows, and the index/aggregator relays that hold reverse-follow data — + * minus the ones a prior probe proved dead (unless `--no-reachability-cache`). + */ + private suspend fun allKnownRelays( + ctx: Context, + args: Args, + ): Set { + val reach = ctx.reachability.snapshot() + val relays = LinkedHashSet() + relays.addAll(reach.live) + // Every advertised outbox/inbox relay in the store — the homes of the users + // whose followers we're hunting are exactly where those followers publish too. + for (event in ctx.store.query(Filter(kinds = listOf(AdvertisedRelayListEvent.KIND)))) { + if (event is AdvertisedRelayListEvent) relays.addAll(event.relaysNorm()) + } + // Relays a NIP-66 monitor has ever seen (kind:30166) — the widest census. + for (event in ctx.store.query(Filter(kinds = listOf(RelayDiscoveryEvent.KIND)))) { + if (event is RelayDiscoveryEvent) event.relay()?.let { relays.add(it) } + } + // Aggregators/indexers hold reverse-follow data for users whose own outbox + // we've never learned — the biggest completeness lever for followers. + relays.addAll(CONTENT_AGGREGATOR_RELAYS) + relays.addAll(EXTRA_DISCOVERY_RELAYS) + relays.addAll(ctx.bootstrapRelays()) + relays.addAll(Constants.eventFinderRelays) + + if (!args.bool(NO_REACHABILITY_CACHE_FLAG)) relays.removeAll(reach.dead) + return relays + } + /** * `amy graperank status` — read-only inventory of everything a GrapeRank run * depends on, straight from the local store: WoT record counts (the "do I diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt new file mode 100644 index 0000000000..4456015dae --- /dev/null +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt @@ -0,0 +1,189 @@ +/* + * Copyright (c) 2025 Vitor Pamplona + * + * Permission is hereby granted, free of charge, to any person obtaining a copy of + * this software and associated documentation files (the "Software"), to deal in + * the Software without restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the + * Software, and to permit persons to whom the Software is furnished to do so, + * subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in all + * copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR + * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN + * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + */ +package com.vitorpamplona.quartz.experimental.graperank + +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.core.HexKey +import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.ParallelEventVerifier +import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPagesFromPool +import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter +import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl +import com.vitorpamplona.quartz.nip01Core.store.IEventStore +import com.vitorpamplona.quartz.nip01Core.tags.people.isTaggedUser +import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.channels.Channel +import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.launch +import kotlinx.coroutines.withContext +import kotlin.time.TimeSource + +/** + * The reverse of [GrapeRankCrawler]: finds every user who **follows** the observer + * by asking as many relays as possible for kind:3 contact lists that `#p`-tag the + * observer, and persists each one to [store]. The outbox model can't find these — + * you don't know a follower until you've seen their list, so you can't route to + * their outbox first — so this casts a wide net across the whole relay universe + * ([Config.relays], assembled by the caller from the reachability cache, the + * kind:10002 relays in the store, and the index/aggregator relays that hold + * reverse-follow data for the network). + * + * Each discovered kind:3 is a follower's full contact list, so persisting it also + * enriches the graph a later `graperank score` builds from the store — the + * follower becomes a FOLLOW edge into the observer (and into everyone else it + * follows). A popular observer can have far more followers than a relay's per-REQ + * cap, so every relay is paged on its own `created_at` cursor via + * [fetchAllPagesFromPool]. + * + * Transport-agnostic within quartz: it takes an [INostrClient] and an + * [IEventStore]; relay *policy* (which relays make up "all possible relays") is the + * caller's, injected through [Config]. Progress is emitted through [log]. + */ +class FollowerCrawler( + private val client: INostrClient, + private val store: IEventStore, + private val config: Config, + private val log: (String) -> Unit = {}, +) { + /** + * @param relays the relay universe to query — "all possible relays" is the + * caller's policy (reachability-cache live set + every kind:10002 relay in the + * store + the index/aggregator relays). Empty means nothing to do. + * @param pageLimit per-page `limit` handed to each relay's paged REQ; the cursor + * walks past it, so this only bounds one page, not the total. + * @param timeoutMs per-page EOSE timeout for a relay before its next page fires. + * @param maxConcurrentRelays how many relays page at once (a global fan-out cap). + * @param insertBatchSize verified events group-committed per [IEventStore.batchInsert]. + */ + class Config( + val relays: Set, + val pageLimit: Int = 500, + val timeoutMs: Long = 15_000, + val maxConcurrentRelays: Int = 16, + val insertBatchSize: Int = 500, + ) + + /** What the follower crawl found. */ + class Stats( + val relaysQueried: Int, + /** Relays that returned at least one valid follower list. */ + val relaysAnswered: Int, + /** Distinct followers (kind:3 authors that `#p`-tag the observer). */ + val followersFound: Int, + /** Verified kind:3 events handed to the store (superseded versions included). */ + val eventsStored: Long, + val downloadMs: Long, + ) + + /** + * Find and persist every follower of [observer] across [Config.relays]. Verifies + * each event (a relay can lie), keeps only kind:3 that actually `#p`-tag the + * observer (a relay can over-return), dedups by event id, and group-commits to + * the store. Returns [Stats]. + */ + suspend fun crawl(observer: HexKey): Stats = + withContext(Dispatchers.IO) { + if (config.relays.isEmpty()) return@withContext Stats(0, 0, 0, 0L, 0L) + + val mark = TimeSource.Monotonic.markNow() + val filter = Filter(kinds = FOLLOW_KINDS, tags = mapOf("p" to listOf(observer)), limit = config.pageLimit) + val perRelay = config.relays.associateWith { listOf(filter) } + + // These three are read/written ONLY by the verifier's single drain + // coroutine (ParallelEventVerifier dispatches onVerified in order from one + // coroutine), so plain collections are safe — no concurrent access. + val followers = HashSet() + val seen = HashSet() + val answered = HashSet() + var eventsStored = 0L + + // Verified follower lists cross to a single persister coroutine that can + // suspend on the store; onVerified itself must not (it's a plain callback). + val toPersist = Channel(Channel.UNLIMITED) + + coroutineScope { + val persister = + launch(Dispatchers.IO) { + val flushAt = config.insertBatchSize.coerceAtLeast(1) + val buffer = ArrayList(flushAt) + + suspend fun flush() { + if (buffer.isEmpty()) return + store.batchInsert(buffer) + eventsStored += buffer.size + buffer.clear() + } + for (event in toPersist) { + buffer.add(event) + if (buffer.size >= flushAt) flush() + } + flush() + } + + val verifier = + ParallelEventVerifier( + scope = this, + onVerified = { event, relay -> + // A relay can over-return; keep only genuine followers, and + // dedup by id so a list mirrored across relays is stored once. + if (event is ContactListEvent && event.isTaggedUser(observer) && seen.add(event.id)) { + followers.add(event.pubKey) + answered.add(relay) + toPersist.trySend(event) + } + }, + ) + + client.fetchAllPagesFromPool( + filters = perRelay, + timeoutMs = config.timeoutMs, + maxConcurrentRelays = config.maxConcurrentRelays, + onRelayComplete = { relay, total -> + if (total > 0) log("[followers] ${relay.url}: $total kind:3 pages drained") + }, + ) { event, relay -> + // onEvent runs on the relay reader thread and must not suspend; + // submit() is a non-suspending channel send that backpressures. + if (event.kind == ContactListEvent.KIND) verifier.submit(event, relay) + } + + // No more events will arrive: drain the verifier, then the persister. + verifier.close() + verifier.join() + toPersist.close() + persister.join() + } + + Stats( + relaysQueried = config.relays.size, + relaysAnswered = answered.size, + followersFound = followers.size, + eventsStored = eventsStored, + downloadMs = mark.elapsedNow().inWholeMilliseconds, + ) + } + + companion object { + /** Reverse-follow lookup is kind:3 only — a follower IS the author of a kind:3 that `#p`-tags the observer. */ + private val FOLLOW_KINDS = listOf(ContactListEvent.KIND) + } +} From dc157e32d23f8330b9431d4ec6407f251f591ea2 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 21:32:16 +0000 Subject: [PATCH 3/6] fix(graperank): page through ALL followers, not just the first limit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FollowerCrawler set the reverse-lookup filter's `limit` to the page size, but fetchAllPages treats a filter `limit` as the TOTAL cap across all pages and stops paging once it's reached — so the crawl silently capped at ~500 followers per relay (verified live: relay.damus.io returned exactly 500 for a many-thousand-follower observer). Leave the filter limit null by default so pagination walks the whole result set (the same observer now returns 12,823 followers from damus alone); Config gains `maxPerRelay` and the CLI a `--max N` flag to opt back into a bounded spot check. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- .../amethyst/cli/commands/GrapeRankCommand.kt | 11 +++++++---- .../experimental/graperank/FollowerCrawler.kt | 14 ++++++++++---- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt index e028d0335e..3b1a8fa95a 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -607,9 +607,10 @@ object GrapeRankCommand { * observer. Idempotent and cumulative; run it a few times for completeness. * * Flags: `--relay URL[,URL…]` (query only these instead of the whole universe), - * `--page-limit N` (per-page REQ limit, default 500), `--timeout SECS` (per-page - * EOSE watchdog, default 15), `--relay-concurrency N` (relays paged at once, - * default 16), `--insert-batch N` (events per store commit, default 500). + * `--max N` (cap followers pulled per relay; default: pull every follower each + * relay holds), `--timeout SECS` (per-page EOSE watchdog, default 15), + * `--relay-concurrency N` (relays paged at once, default 16), `--insert-batch N` + * (events per store commit, default 500). */ private suspend fun followers( dataDir: DataDir, @@ -649,7 +650,9 @@ object GrapeRankCommand { config = FollowerCrawler.Config( relays = relays, - pageLimit = args.intFlag("page-limit", 500), + // Default null → pull EVERY follower each relay holds; --max + // N caps the total per relay for a quick spot check. + maxPerRelay = args.flag("max")?.toIntOrNull(), timeoutMs = args.longFlag("timeout", 15L) * 1000, maxConcurrentRelays = relayConcurrency, insertBatchSize = args.intFlag("insert-batch", 500), diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt index 4456015dae..c1f97e2989 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt @@ -68,15 +68,19 @@ class FollowerCrawler( * @param relays the relay universe to query — "all possible relays" is the * caller's policy (reachability-cache live set + every kind:10002 relay in the * store + the index/aggregator relays). Empty means nothing to do. - * @param pageLimit per-page `limit` handed to each relay's paged REQ; the cursor - * walks past it, so this only bounds one page, not the total. + * @param maxPerRelay TOTAL cap on followers pulled from each relay, or `null` + * (default) to pull **every** follower a relay holds. This is NOT a page size: + * [fetchAllPages] treats a filter's `limit` as the total across all pages and + * stops paging once it's reached, so a non-null value cuts the crawl short at + * that many per relay. Leave it null for completeness; set it only to bound a + * spot check. The per-page size is the relay's own default either way. * @param timeoutMs per-page EOSE timeout for a relay before its next page fires. * @param maxConcurrentRelays how many relays page at once (a global fan-out cap). * @param insertBatchSize verified events group-committed per [IEventStore.batchInsert]. */ class Config( val relays: Set, - val pageLimit: Int = 500, + val maxPerRelay: Int? = null, val timeoutMs: Long = 15_000, val maxConcurrentRelays: Int = 16, val insertBatchSize: Int = 500, @@ -105,7 +109,9 @@ class FollowerCrawler( if (config.relays.isEmpty()) return@withContext Stats(0, 0, 0, 0L, 0L) val mark = TimeSource.Monotonic.markNow() - val filter = Filter(kinds = FOLLOW_KINDS, tags = mapOf("p" to listOf(observer)), limit = config.pageLimit) + // No limit by default → fetchAllPages walks the whole result set page by + // page; a non-null maxPerRelay caps the total per relay (see Config). + val filter = Filter(kinds = FOLLOW_KINDS, tags = mapOf("p" to listOf(observer)), limit = config.maxPerRelay) val perRelay = config.relays.associateWith { listOf(filter) } // These three are read/written ONLY by the verifier's single drain From 331dca79cd46ff031769fd7f7dff0229438c5731 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 22:41:26 +0000 Subject: [PATCH 4/6] feat(graperank): run crawl/score without a personal account MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit crawl and score used Context.open, which requires an account solely to default the observer to the logged-in user — but neither uses account keys: crawl never signs, and score signs cards with the machine-level operator key (~/.amy/operator/, independent of any account). Switch both to Context.openOrAnonymous and require an explicit OBSERVER when running anonymously, matching `graperank followers`. A machine acting as a GrapeRank provider for arbitrary observers no longer needs a personal account (the operator key still needs its SecretStore passphrase, as before — that's the reachability monitor + card signer, a real secret). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- .../amethyst/cli/commands/GrapeRankCommand.kt | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt index 3b1a8fa95a..c6c1e5c1a0 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -265,8 +265,15 @@ object GrapeRankCommand { rigor = args.flag("rigor")?.toDoubleOrNull() ?: GrapeRankParams().rigor, ) - Context.open(dataDir).use { ctx -> + // Crawl never signs and score signs cards with the machine-level operator + // key (`~/.amy/operator/`, independent of any account), so neither needs a + // personal account — run anonymously when there is none, requiring an + // explicit observer since there's no logged-in user to default to. + Context.openOrAnonymous(dataDir).use { ctx -> ctx.prepare() + if (ctx.anonymous && observerArg == null) { + return Output.error("bad_args", "no account — pass an OBSERVER (npub / hex / nprofile / NIP-05)") + } val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex // Contact lists stream straight into a compact int-CSR structure as the @@ -558,8 +565,13 @@ object GrapeRankCommand { ): Int { val args = Args(rest) val observerArg = args.positionalOrNull(0) - Context.open(dataDir).use { ctx -> + // Crawl never signs, so no account is needed — run anonymously when there is + // none, requiring an explicit observer (no logged-in user to default to). + Context.openOrAnonymous(dataDir).use { ctx -> ctx.prepare() + if (ctx.anonymous && observerArg == null) { + return Output.error("bad_args", "no account — pass an OBSERVER (npub / hex / nprofile / NIP-05)") + } val observer = observerArg?.let { ctx.requireUserHex(it) } ?: ctx.identity.pubKeyHex // Persist-only crawl: no in-memory graph (null builder); every event // still lands in the store for a later `score`. From 33f4c648e9ae9f44fda27cf12aa5c2cb8b4850de Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 22:57:25 +0000 Subject: [PATCH 5/6] fix(graperank): don't create the operator master for a read-only follower crawl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `amy graperank followers ` (no --relay) assembles the relay universe via allKnownRelays, which read the reachability cache through ctx.reachability — and that lazily derives the monitor key from the operator master, forcing a passphrase prompt (or failing on a fresh machine) purely to READ kind:30166 records. The follower crawl signs nothing, so this defeats its anonymous path. Read the reachability snapshot with a throwaway signer instead (snapshot() only reads; the signer is used solely for writes) — the same trick `graperank status` already uses to stay side-effect-free. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- .../amethyst/cli/commands/GrapeRankCommand.kt | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt index c6c1e5c1a0..9da16dce86 100644 --- a/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt +++ b/cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt @@ -697,7 +697,12 @@ object GrapeRankCommand { ctx: Context, args: Args, ): Set { - val reach = ctx.reachability.snapshot() + // Read the reachability records (kind:30166) with a throwaway signer, NOT + // ctx.reachability — the latter derives the monitor key and would CREATE the + // operator master (a passphrase prompt) purely to read a cache. The follower + // crawl never signs, so keep it truly account-and-key-free. snapshot() only + // reads; the signer is used solely for writes. Same trick as `status`. + val reach = RelayReachabilityStore(store = ctx.store, signer = NostrSignerInternal(KeyPair())).snapshot() val relays = LinkedHashSet() relays.addAll(reach.live) // Every advertised outbox/inbox relay in the store — the homes of the users From 699c2dc7e5e82b560b49e2ad2c3e20832b493864 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 23:26:46 +0000 Subject: [PATCH 6/6] fix(graperank): import kotlinx.coroutines.IO for Kotlin/Native compile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FollowerCrawler is in commonMain but used `Dispatchers.IO` without importing the multiplatform `kotlinx.coroutines.IO` extension. On JVM `Dispatchers.IO` resolves to the JVM member (compiled fine locally), but on Kotlin/Native that member is internal, so `:quartz:compileKotlinIosSimulatorArm64` failed with "Cannot access 'val IO': it is internal". Add the explicit import — the same idiom the sibling GrapeRankCrawler already uses across all targets. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Xc3Wm4qVCrAGvSAotTUVt4 --- .../quartz/experimental/graperank/FollowerCrawler.kt | 1 + 1 file changed, 1 insertion(+) diff --git a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt index c1f97e2989..14cbd15adf 100644 --- a/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt +++ b/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/experimental/graperank/FollowerCrawler.kt @@ -31,6 +31,7 @@ import com.vitorpamplona.quartz.nip01Core.store.IEventStore import com.vitorpamplona.quartz.nip01Core.tags.people.isTaggedUser import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.IO import kotlinx.coroutines.channels.Channel import kotlinx.coroutines.coroutineScope import kotlinx.coroutines.launch