laantungir/amethyst
1
0
Fork 0
mirror of https://github.com/vitorpamplona/amethyst.git synced 2026-08-08 23:54:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(cli): exhaustive graperank crawl — no user cap, check every outbox

Browse Source 
Replace the depth-limited, user-capped BFS with a completeness loop that runs
until every discovered user's kind:10002 outbox has been checked and their
latest kind:3/10000/1984 pulled from it:

- Delete the --max-users cap entirely.
- Crawl round by round until the pending set (discovered minus done) is empty.
  A user is "done" once we download its contact list, or after --max-attempts
  (default 3) failed tries of its outbox — so an unreachable outbox can't stall
  the crawl, and it still terminates on a finite graph.
- --max-rounds replaces --max-depth as an (unbounded by default) safety backstop.
- Track and report the pool of relays actually contacted (relays_contacted),
  the "running relays" we connect to as more outboxes are discovered.

JSON: `depth_reached` -> `crawl_rounds`, add `relays_contacted`. Per-round and
final crawl-summary progress on stderr. Local regression: scores unchanged
(rank 26); the crawl retries contact-list-less users then terminates cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
This commit is contained in:
Claude
2026-07-06 18:23:10 +00:00
parent dd86b617c1
commit 4ec3da1e86
4 changed files with 90 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/README.md
+1 -1
View File
@@ -384,7 +384,7 @@ HTTP endpoint. Reuses quartz's `Nip86Client` and the shared `Nip86Retriever`
| `amy profile show [USER]` | Print kind:0 metadata. USER accepts npub/nprofile/hex/NIP-05; defaults to self. |
| `amy profile edit --name … --about … --picture URL …` | Patch and re-publish your kind:0. |
| `amy follow USER` / `amy unfollow USER` | Add/remove USER from your kind:3 contact list (fetches the freshest list first). |
| `amy graperank [OBSERVER] [--max-depth N] [--offline] [--publish]` | Compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph, crawled via the outbox model; optionally publish results as NIP-85 kind:30382 cards (unchanged ranks are skipped). |
| `amy graperank [OBSERVER] [--offline] [--publish]` | Compute GrapeRank web-of-trust scores (0..1) over the follow/mute/report graph. Exhaustively crawls each user's kind:10002 outbox for their latest kind:3/10000/1984 until every discovered user is checked (no user cap); optionally publishes results as NIP-85 kind:30382 cards (unchanged ranks are skipped). |
| `amy graperank register [PROVIDER] [--service KIND:TAG] [--relay URL]` | Declare a NIP-85 provider in your kind:10040 so clients can discover it (default: self as the `30382:rank` provider). |
| `amy graperank providers [USER]` | List a user's declared NIP-85 trusted providers (public + your own private entries). |
cli/plans/2026-07-06-graperank-brainstorm-parity.md
+13 -10
View File
@@ -87,11 +87,13 @@ It is **data**, not math:
the observer's trust graph) contributes **zero**. Only follows/mutes/reports
authored by users *inside* the follow graph move a score — and those are
exactly the users our crawl discovers and whose kind 3/10000/1984 we fetch.
So the effective scoring input is the same, provided the crawl runs to
convergence (our default) rather than a shallow `--max-depth`.
2. **Fringe users / crawl gaps.** Relay timeouts that drop a contact list, or a
`--max-users` cap, remove edges and shift nearby scores. The injector mitigates
this with a two-stage model mirroring the app's `pickRelaysToLoadUsers`:
So the effective scoring input is the same, as long as the crawl actually
checks every discovered user's outbox — which it now does exhaustively (no
user cap, retrying an unreachable outbox up to `--max-attempts` times).
2. **Fringe users / crawl gaps.** A relay timeout that drops a contact list
removes edges and shifts nearby scores. The injector mitigates this with a
two-stage model mirroring the app's `pickRelaysToLoadUsers`, plus a
completeness loop that retries until every user's outbox has been checked:
- **Relay-list discovery** (kind:10002) queries the account's relays +
bootstrap + event-finder + **indexer relays** (purplepag.es, coracle, …).
Indexers aggregate kind:10002 (and kind:0) for the whole network, so this is
@@ -101,9 +103,9 @@ It is **data**, not math:
contact lists we crawl) and general-purpose relays as a best-effort fallback
when the outbox is unknown/down. **Indexers are not used for content** — they
don't serve those kinds; kind:3/mutes/reports live only on the user's outbox.
A per-hop **retry pass** re-queries any member whose contact list still didn't
arrive against that hint + general-relay set. Remaining mitigation levers: a
full crawl (default) and a generous `--timeout`.
The crawl loops round by round, retrying any member whose contact list still
didn't arrive (up to `--max-attempts`), until every discovered user's outbox
has been checked. Remaining mitigation lever: a generous `--timeout`.
3. **Convergence precision.** Both stop at delta 0.0001; residual error is
< ~0.0001 in influence ⇒ < ~0.01 rank points ⇒ identical integer `rank`.
4. **Seeding.** Their hop-distance seed vs our zero seed — same fixed point, no
@@ -113,8 +115,9 @@ It is **data**, not math:
- **Keep the current DEFAULT params** — they are byte-for-byte the Brainstorm
DEFAULT preset. No change needed for parity.
- **Crawl to convergence** (the default) rather than a small `--max-depth`; a
shallow crawl is the single biggest source of drift.
- **The crawl is exhaustive by default** (no user cap; every reachable user's
outbox is checked, unreachable outboxes retried up to `--max-attempts`). An
incomplete crawl is the single biggest source of drift, so avoid capping it.
- **Optional, for fuller parity (not required for close scores):**
- Add `--preset default|permissive|restrictive`. DEFAULT is confirmed; the
PERMISSIVE / RESTRICTIVE numbers are DB-seeded in `brainstorm_server` (an
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/Main.kt
+10 -8
View File
@@ -528,14 +528,16 @@ private fun printUsage() {
|
|Web of Trust (GrapeRank):
| graperank [OBSERVER] compute subjective trust scores (0..1) for every
| [--max-depth N] [--max-users N] user reachable in the follow/mute/report graph,
| [--limit N] [--min-score X] crawled via the outbox model until no new users
| [--rigor X] [--attenuation X] appear (OBSERVER: npub|nprofile|hex|name@domain,
| [--offline] [--timeout SECS] default: active account). --offline scores from
| [--publish] [--min-rank N] the local store only. --publish writes NIP-85
| [--publish-limit N] [--publish-relay URL] kind:30382 trusted-assertion cards
| (rank = round(score*100)) for each user at or
| above --min-rank (unchanged ranks are skipped).
| [--limit N] [--min-score X] user reachable in the follow/mute/report graph.
| [--rigor X] [--attenuation X] Crawls each user's kind:10002 outbox for their
| [--max-attempts N] [--max-rounds N] latest kind:3/10000/1984 until every discovered
| [--offline] [--timeout SECS] user has been checked (no user cap; --max-attempts
| [--publish] [--min-rank N] bounds retries of an unreachable outbox, default 3).
| [--publish-limit N] [--publish-relay URL] OBSERVER: npub|nprofile|hex|name@domain (default:
| active account). --offline scores from the local
| store only. --publish writes NIP-85 kind:30382
| cards (rank = round(score*100)) for each user at
| or above --min-rank (unchanged ranks skipped).
| graperank register [PROVIDER] declare a NIP-85 provider in your kind:10040 so
| [--service KIND:TAG] [--relay URL] clients can discover it (default: self as the
| [--private] 30382:rank provider at your first outbox relay).
cli/src/main/kotlin/com/vitorpamplona/amethyst/cli/commands/GrapeRankCommand.kt
+66 -48
View File
@@ -58,8 +58,10 @@ import kotlin.math.roundToInt
* observer has full self-trust). It crawls the follow graph outward using the
* outbox model — each user's kind:10002 write relays are located first, then
* their kind:3 / kind:10000 / kind:1984 events are fetched from *their own*
* relays — until no new users appear (typically ~8 hops), then runs the scoring
* engine in `commons/wot`.
* relays. The crawl is exhaustive: it keeps going, with no user cap, until every
* discovered user's outbox has been checked and their contact list pulled (an
* unreachable outbox is retried up to `--max-attempts` times), then runs the
* scoring engine in `commons/wot`.
*
* Prints a ranked list (text, or one JSON object under `--json`). With
* `--publish`, results are also published as NIP-85 kind:30382 `ContactCardEvent`
@@ -100,8 +102,11 @@ object GrapeRankCommand {
): Int {
val args = Args(rest)
val observerArg = args.positionalOrNull(0)
val maxDepth = args.intFlag("max-depth", 8)
val maxUsers = args.intFlag("max-users", 50_000)
// Crawl to full convergence by default (every reachable user's outbox
// checked). --max-rounds is only a safety backstop; --max-attempts bounds
// how many times we re-try an unreachable user's outbox before giving up.
val maxRounds = args.intFlag("max-rounds", Int.MAX_VALUE)
val maxAttempts = args.intFlag("max-attempts", 3)
val limit = args.intFlag("limit", 100)
val minScore = args.flag("min-score")?.toDoubleOrNull() ?: 0.0
val offline = args.bool("offline")
@@ -123,7 +128,8 @@ object GrapeRankCommand {
val graphKinds = listOf(ContactListEvent.KIND, MuteListEvent.KIND, ReportEvent.KIND)
var depthReached = 0
var rounds = 0
var relaysContactedCount = 0
val events: List<Event>
if (offline) {
@@ -132,60 +138,71 @@ object GrapeRankCommand {
} else {
val collected = mutableListOf<Event>()
val discovered = hashSetOf(observer)
// Per-user relay hints harvested from the `p`-tag relay hints in
// the contact lists we crawl (A's follow of B says where B writes).
// A second discovery tier below each user's kind:10002 outbox.
// Per-user relay hints harvested from the `p`-tag relay hints in the
// contact lists we crawl (A's follow of B says where B writes) — a
// discovery tier below each user's kind:10002 outbox.
val relayHints = HashMap<HexKey, MutableSet<NormalizedRelayUrl>>()
var frontier: Set<HexKey> = setOf(observer)
// Users we're finished with: their outbox was queried and we either
// downloaded their kind:3 or ran out of retry attempts. Growing this
// set toward `discovered` is what drives the crawl to completion.
val done = hashSetOf<HexKey>()
val attempts = HashMap<HexKey, Int>()
// The pool of relays we actually route outbox queries to, grown as
// more users' kind:10002 outboxes are discovered.
val relaysContacted = hashSetOf<NormalizedRelayUrl>()
for (hop in 0 until maxDepth) {
if (frontier.isEmpty()) break
depthReached = hop + 1
// Loop until every discovered user has had their outbox checked and
// their kind:3/10000/1984 pulled from it — no user cap. A user whose
// outbox stays unreachable is dropped after --max-attempts tries so
// the crawl still terminates.
while (rounds < maxRounds) {
val pending = discovered.filterNot { it in done }
if (pending.isEmpty()) break
rounds++
// 1. Locate each frontier member's kind:10002 write relays.
ensureRelayLists(ctx, frontier, relayHints, timeoutMs)
// 1. Resolve kind:10002 outboxes for pending users missing them.
ensureRelayLists(ctx, pending.toSet(), relayHints, timeoutMs)
// 2. Fetch their follows/mutes/reports from those relays.
val filters = routeByOutbox(ctx, frontier, relayHints, graphKinds)
// 2. Pull kind:3/10000/1984 from each pending user's own outbox
// (hints + general relays only when the outbox is unknown).
val before = collected.size
val filters = routeByOutbox(ctx, pending.toSet(), relayHints, graphKinds)
relaysContacted += filters.keys
collected += ctx.drain(filters, timeoutMs).map { it.second }
// 3. Completeness retry: any member whose contact list still
// didn't arrive (no kind:10002, or its outbox was down) gets
// re-queried against its relay hints plus the general-purpose
// fallback relays. kind:3/10000/1984 live on the user's own
// outbox — NOT on indexers (those only aggregate kind:10002) —
// so this is best-effort recovery from general relays that may
// hold a copy, not a guaranteed find.
val stillMissing = frontier.filter { ctx.contactsOf(it) == null }
if (stillMissing.isNotEmpty()) {
val retryRelays = contentFallbackRelays(ctx) + stillMissing.flatMap { relayHints[it].orEmpty() }
val retry =
retryRelays.associateWith {
stillMissing.chunked(AUTHORS_PER_FILTER).map { chunk -> Filter(kinds = graphKinds, authors = chunk) }
// 3. Mark done / retry, harvest hints, expand the follow graph.
var downloaded = 0
var newUsers = 0
for (pk in pending) {
val contacts = ctx.contactsOf(pk)
if (contacts != null) {
done += pk
downloaded++
for (tag in contacts.follows()) {
tag.relayUri?.let { relayHints.getOrPut(tag.pubKey) { HashSet() }.add(it) }
if (discovered.add(tag.pubKey)) newUsers++
}
collected += ctx.drain(retry, timeoutMs).map { it.second }
}
// 4. Harvest relay hints + expand the follow frontier.
val next = hashSetOf<HexKey>()
for (pk in frontier) {
val contacts = ctx.contactsOf(pk) ?: continue
for (tag in contacts.follows()) {
tag.relayUri?.let { relayHints.getOrPut(tag.pubKey) { HashSet() }.add(it) }
if (discovered.size < maxUsers && discovered.add(tag.pubKey)) next += tag.pubKey
} else {
val tries = (attempts[pk] ?: 0) + 1
attempts[pk] = tries
// Give up once we've exhausted retries: either the user has
// no contact list, or their outbox is unreachable.
if (tries >= maxAttempts) done += pk
}
}
val recovered = stillMissing.count { ctx.contactsOf(it) != null }
System.err.println(
"[graperank] hop ${hop + 1}: frontier=${frontier.size}, recovered=$recovered/${stillMissing.size}, new=${next.size}, total=${discovered.size}",
"[graperank] round $rounds: queried=${pending.size}, +events=${collected.size - before}, " +
"downloaded=$downloaded, newUsers=$newUsers, discovered=${discovered.size}, done=${done.size}",
)
if (discovered.size >= maxUsers) {
System.err.println("[graperank] reached --max-users=$maxUsers cap; stopping crawl")
break
}
frontier = next
}
relaysContactedCount = relaysContacted.size
val unreached = discovered.count { it !in done || ctx.contactsOf(it) == null }
System.err.println(
"[graperank] crawl complete: ${discovered.size} users discovered, " +
"${discovered.size - unreached} contact lists downloaded, $unreached without one, " +
"$relaysContactedCount relays contacted, $rounds rounds",
)
events = collected
}
@@ -215,7 +232,8 @@ object GrapeRankCommand {
val result =
linkedMapOf<String, Any?>(
"observer" to observer,
"depth_reached" to depthReached,
"crawl_rounds" to rounds,
"relays_contacted" to relaysContactedCount,
"graph_users" to graph.users.size,
"graph_edges" to graph.edgeCount(),
"users_scored" to scores.size,