Answers "do we try relays we know work from other people's lists when a user
isn't in the first set?" — now yes.
- Drop dead relays I'd added unchecked (relay.nostr.band, relayable.org,
relay.nostr.bg); keep only ones that reply to a limit:1 query. NIP-11 presence
is not used for liveness (it's optional).
- Learn a backbone dynamically from the crawl: tally how often each relay appears
as someone's kind:10002 write relay, and mark relays that actually delivered
events as live. The most-used live relays form the backbone.
- Route retried users (whose own outbox already failed) and outbox-less users to
outbox + backbone, since popular relays usually hold a copy of their kind:3.
Effect for Vitor (--max-hops 3): hop-3 coverage rose from ~112,600 to 146,413
(95% of Brainstorm's 153,409), as round-3 contact-list fetches more than doubled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Fixes the hop-2 (and general) completeness undercount, and makes the cause
observable.
- Add a broad set of aggregator + big general relays (relay.nostr.band,
relay.damus.io, snort, offchain.pub, relayable.org, …) to the kind:10002
discovery set. Effect: for Vitor, hop-2 discovery went from ~16,980 to 19,886
(~83% -> ~98% of Brainstorm's 20,332).
- `Context.drain` gains a `diagnoseSlow` flag: on a timeout it logs which relays
stalled and why — slow (no EOSE, with the event count they did send) vs
cannot-connect (with the failure reason) vs closed. `amy graperank --diagnose`
turns it on. This showed the remaining misses are relay-side: users advertise
dead/misconfigured write relays (HTTP 404/530/503, "Unexpected response",
read/connect timeouts), not anything blocking on our end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Stamp each discovered user with its follow-graph hop distance from the observer
(observer=0, a user's fresh follows = its hop+1). Report the per-hop histogram
on the crawl-complete line and as `max_hop_reached` / `users_by_hop` in --json,
and add `--max-hops N` to bound how deep the crawl fetches (deeper users still
appear as follow targets). Brainstorm's graph for an observer saturates within
~8 hops, so `--max-hops 8` matches its scope.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Point #1 (freshness): each run now fetches every discovered user's LATEST
kind:3/10000/1984 once from their outbox (grouped by write relay in
routeByOutbox; empty-tagged relays already count as write), instead of skipping
users whose list is already cached. `done` still guarantees once-per-run and
that a fresh download isn't repeated.
Point #2 (memory): replace the HexKey-keyed, TrustEdge-object graph with a
compact representation that scales to the whole network:
- commons/wot: pubkeys interned to dense Int ids; edges stored in two CSR
IntArray layouts (by target for scoring, by source for the worklist), each
incoming entry packing source id + relation into one int. GrapeRank.compute
returns a DoubleArray by node id (no boxed map at millions of nodes).
TrustGraphBuilder is now stateful/streaming (addFollows/addMutes/addReports).
Tests rewritten; the full-sweep cross-check still passes.
- cli: contact lists stream straight into the builder as they arrive and the
Event is discarded — the crawl never holds millions of kind:3 objects. Mutes
and reports (far fewer) are fed from the store. Output de-interns the top-N.
Validated: a fresh online run built a 108,961-user / 1.67M-edge graph and
scored 83,613 users in a 4 GB heap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Two related fixes so a warm/shared store isn't re-downloaded every run:
- Only DOWNLOAD contact lists we don't already have. Each round now splits
pending users into "already in the store" (expanded from disk with zero
network) vs "need to fetch" (routed to their outbox). Verified on a warm
store: round 2 pending=250 -> cached=236, downloaded=7; round 3 pending=19491
-> cached=15827, downloaded=87.
- Build the trust graph from the store (kind:3/10000/1984 query) instead of the
in-run `collected` list, so cached-and-skipped lists still contribute their
edges. Online and offline paths now share the same graph source.
No behavioural change on a cold store (nothing cached -> download everything
once), and the crawl still runs to full graph depth with no user cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
A live run against Vitor's ~110k-user WoT exposed the crawl's real bottleneck:
draining every pending user's outbox in one subscription saturates connections
and times out. Round-by-round evidence — 250 users queried downloaded 205
contact lists (82%), but 17,055 queried downloaded only 137 (0.8%). Net: 93k
outboxes found but only ~14k contact lists pulled, so most users had no
outgoing edges and scores came out far below Brainstorm's.
Fix: fetch content in bounded batches (USER_BATCH=256) drained a few at a time
(DRAIN_CONCURRENCY=8). Routing (store reads) runs serially; only the drains run
concurrently — inserts serialize on the store write lock, so that is safe.
kind:10002 discovery stays a bulk indexer query (they aggregate 10002 and
handle bulk author filters fine); only the per-user outbox fan-out is batched.
Effect on the same graph: round 3 went from 137 → 4,157 contact lists
downloaded (+36k events), and users scored after 3 rounds jumped 34,976 →
109,760. Full-run parity verification against the live Brainstorm set is in
progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
The per-user outbox retry bound doesn't need to be tunable — replace the
--max-attempts flag with a MAX_OUTBOX_ATTEMPTS = 3 constant. Same behaviour,
one fewer knob. Updates usage text, README, and the parity doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Replace the depth-limited, user-capped BFS with a completeness loop that runs
until every discovered user's kind:10002 outbox has been checked and their
latest kind:3/10000/1984 pulled from it:
- Delete the --max-users cap entirely.
- Crawl round by round until the pending set (discovered minus done) is empty.
A user is "done" once we download its contact list, or after --max-attempts
(default 3) failed tries of its outbox — so an unreachable outbox can't stall
the crawl, and it still terminates on a finite graph.
- --max-rounds replaces --max-depth as an (unbounded by default) safety backstop.
- Track and report the pool of relays actually contacted (relays_contacted),
the "running relays" we connect to as more outboxes are discovered.
JSON: `depth_reached` -> `crawl_rounds`, add `relays_contacted`. Per-round and
final crawl-summary progress on stderr. Local regression: scores unchanged
(rank 26); the crawl retries contact-list-less users then terminates cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Correct the injector's relay model: indexer relays (purplepag.es, coracle, …)
aggregate kind:10002 (and kind:0) for the whole network but do NOT serve
kind:3/10000/1984. Those live only on each user's own outbox.
- Split the relay sets: `relayListDiscoveryRelays` (bootstrap + event-finder +
indexers) is used only to locate kind:10002; `contentFallbackRelays`
(bootstrap + event-finder, no indexers) is the best-effort fallback for
content when a user's outbox is unknown/down.
- Content is fetched from each user's outbox write relays, with harvested
relay hints and general relays as fallback — never indexers.
Also add progress status (all on stderr, stdout stays the JSON contract):
- loading already logs per-hop frontier/recovered/new/total counts;
- "graph built: N users, E edges; scoring…" and "scored N users" bracket the
calculation;
- GrapeRank.compute gains an optional (visited, scored, queued) progress
callback, wired to emit a scoring line every 5000 worklist visits so a large
graph shows movement instead of hanging silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
To match Brainstorm's full-graph ingest, the crawl now discovers data through
three tiers (mirroring the app's pickRelaysToLoadUsers) instead of just the
outbox + a small fallback:
- Indexer relays (purplepag.es, coracle, …) join the discovery set. They serve
kind:0/3/10002 for the whole network and are where a stranger's relay list
and contact list are actually found — the biggest completeness lever.
- Per-follow relay hints are harvested from the `p`-tag hints in every contact
list we crawl and used as a discovery tier below each user's kind:10002.
- A per-hop retry pass re-queries any frontier member whose contact list still
didn't arrive (no kind:10002, or its outbox was unreachable) against the
indexer + hint set, recovering users the outbox model alone would miss.
This tightens the only real source of score divergence from Brainstorm — data
completeness — since a signal's weight scales by the rater's influence, so the
users that matter are exactly the in-graph ones this crawl now reaches more
reliably. Local regression check: scores unchanged (rank 26 for direct follows).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Analysis of NosFabrica/brainstorm_graperank_algorithm (Java scoring worker)
and NosFabrica/brainstorm_server (Python orchestration) to confirm amy's
scores match the reference GrapeRank service.
Finding: our commons/wot formula and every scoring parameter are already
identical to Brainstorm's DEFAULT preset (attenuation 0.85, rigor 0.5,
follow 1.0/0.03, from-observer 0.5, mute/report -0.1/0.5, delta 0.0001). Our
`score` is exactly their ScoreCard `influence`. Remaining divergence is data
completeness, not math — and because a signal's weight scales by the rater's
influence, only in-graph raters move a score, which our outbox crawl already
captures. Documents the pipeline, side-by-side params, divergence sources,
and follow-ups (presets, influence/verified fields).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Publishing kind:30382 rank cards is only half of NIP-85 — clients also need
the kind:10040 TrustProviderListEvent to discover which key provides which
assertion, and where. Add the discovery layer as two sub-verbs:
- `amy graperank register [PROVIDER]` — append a ServiceProviderTag
(default `30382:rank`, self, first outbox relay) to the account's kind:10040,
fetching the freshest list first so existing providers are preserved.
Idempotent, supports `--service KIND:TAG`, `--relay`, and `--private`.
- `amy graperank providers [USER]` — list a user's declared providers
(cache-first; own private entries are decrypted and included).
Bare `amy graperank [OBSERVER]` still computes scores; the dispatcher only
peels off the `register` / `providers` words. All built on quartz's existing
`TrustProviderListEvent` / `ServiceProviderTag` / `ProviderTypes`.
Verified against a local geode relay: register creates the 10040 and is
idempotent on re-run; providers lists both a public 30382:rank entry and a
private 30382:followers entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Previously `graperank --publish` rebuilt and rebroadcast a NIP-85 kind:30382
ContactCard for every scored user on every run, minting a new event id and
created_at even when the rank was identical — pure churn for a parameterized-
replaceable event.
Read back the ranks we last published from the account's own kind:30382 cards
in the local store (ctx.publish already persists them) and publish only the
targets whose rank is new or changed. Report the count left alone as
`skipped_unchanged`.
Verified against a local geode relay: first run publishes N cards
(skipped_unchanged=0); an immediate re-run with identical ranks publishes 0
(skipped_unchanged=N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
- Remove `--target USER`: the command already emits the full ranking, and a
single-user lookup is a trivial slice of it.
- Remove `--no-mutes` / `--no-reports` and the include* parameters on
TrustGraphBuilder.build. GrapeRank is defined over follows, mutes and
reports together; scoring with a signal disabled isn't a meaningful WoT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Bring the GrapeRank algorithm into Amethyst as a WoT service calculator
on the CLI.
commons/wot (protocol-agnostic, CLI-safe, reusable by the apps):
- TrustGraph / TrustEdge / TrustRelation — pubkey-keyed graph model.
- GrapeRank — single-observer scoring engine, a faithful port of the
reference v3 TargetedBFS variant using a worklist that reaches the same
fixed point a full sweep would while only touching reachable users.
- TrustGraphBuilder — pure kind:3 / kind:10000 / kind:1984 events -> graph
(latest-replaceable-per-author, dedup, self-edge drop).
- Unit tests: hand-computed values plus an adversarial full-sweep
cross-check over 50 random graphs.
cli: `amy graperank [OBSERVER]` crawls the follow/mute/report graph via the
outbox model (locate each user's kind:10002 write relays, then fetch their
lists from their own relays, with a broad event-finder fallback) until no
new users appear, scores it, and prints a ranked list (text / --json).
--target queries one user, --offline scores from the local store, and
--publish writes NIP-85 kind:30382 ContactCard assertions
(rank = round(score*100)) per user at or above --min-rank.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Restructure `amy relay` from verb-first `relay add URL --type T` to noun-first
`relay <noun> <verb>`, matching amy's `marmot group …` / `cashu mint …`
convention. The relay-list type is now a required path segment (no implicit
default), and a bare noun lists that bucket.
NIP-65 (kind:10002) is fronted by two facet-nouns, `outbox` (write) and `inbox`
(read), replacing the `--marker` flag. They edit the single 10002 event and
apply the spec's merge rules:
- outbox add R on a read-only R → both
- inbox add R on a write-only R → both
- outbox remove R on a both-R → read (stays in inbox)
- inbox remove R on a both-R → write (stays in outbox)
- dropping the last facet removes R entirely
`relay nip65` shows the combined view; `nip65 remove`/`clear` edit the whole
event.
Other buckets are noun+verb: `relay dm|key-package|search|private|blocked|
trusted|proxy|indexer|broadcast|feeds <add|remove|set|clear|list>`. `set` needs
≥1 URL; `clear` empties. `relay add|remove URL` (no noun) stays as the
transport fan-out (nip65 both + dm + key-package).
BREAKING (cli --json/args): removes `relay add/remove/set --type T` and
`--marker`; `relay list` overview now keys nip65 as `outbox`/`inbox`/`nip65`
and the DM bucket as `dm` (was `inbox`). In-repo harnesses updated
(cache/dm/marmot setup drop `--type all`; cache T5 asserts `.dm`).
Verified end-to-end: merge semantics, encrypted NIP-51 round-trips, facet
set/clear, fan-out, aliases, and error paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
`relay set --type T` with no URLs is now rejected (bad_args, exit 2) instead
of silently wiping the list — a bare empty `set` is almost always a shell
variable that expanded to nothing. Emptying a bucket is explicit: pass
`--clear` (mutually exclusive with URLs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
Expand `amy relay` from the 3 transport lists (nip65/inbox/key_package) to
every relay-list bucket Amethyst's relay-settings screen manages, and add
remove/set verbs alongside add/list.
Buckets (kind): nip65 (10002, read/write markers), inbox/dm (10050),
key_package (10051), search (10007), private (10013), blocked (10006),
trusted (10089), proxy (10087), indexer (10086), broadcast (10088),
feeds/favorites (10012). The private NIP-51 lists are signed NIP-44-encrypted
via the quartz event factories, exactly like the app. Local relays (device
pref, no event) and named relay sets (30002) are intentionally out of scope.
New/changed commands:
- `relay add URL --type T [--marker read|write|both]` — `--marker` sets the
nip65 role; `all` still means nip65+inbox+key_package.
- `relay remove URL --type T` — new.
- `relay set --type T [URL…] [--marker …]` — new; replace a whole bucket
(no URLs clears it).
- `relay list [--type T]` — lists every bucket, or one.
- `relay publish-lists` — now broadcasts every configured list.
Thin-assembly only: buckets are a small registry over the existing quartz
`create`/`relays` factories; adds one generic `Context.latestReplaceable`
helper. `--json` is additive — legacy keys (`nip65`/`inbox`/`key_package`,
`nip65_event_id`/…) are unchanged, so the existing test harnesses keep passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
`putPending` stores each relay's pending map as
`Collections.synchronizedMap(LinkedHashMap(...))` and wraps every writer
path (`putPending`, `recordSent`, `recordIncoming`, `recordDisconnect`)
in `synchronized(perRelay)`. `sweep` iterated `pending.entries.iterator()`
without taking the same lock, violating the wrapper's Javadoc contract.
Any concurrent websocket-thread write during `RelayHealthStore.reclassify`'s
sweep threw `ConcurrentModificationException` on the underlying
`LinkedHashMap$LinkedHashIterator`. Because `reclassify` schedules sweep
on the AWT dispatcher, the CME killed Amethyst Desktop's Compose render
thread and froze the UI.
Wrap both inner iterator loops in `synchronized(pending) { ... }` — the
exact synchronization the wrapper's Javadoc prescribes for manual
iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Desktop, tapping a sidebar nav item while a detail screen (profile,
thread, article, editor) was open only mutated the sidebar destination.
The opaque `AnimatedContent` overlay driven by `ColumnNavigationState`
kept covering the (already-swapped) root content until the user hit
Back, creating the impression that the click did nothing.
Fix: emit a `clearOverlaySignal` from `SinglePaneState.navigate` and
`DeckState.focusExistingColumn`. Each layout collects the signal in a
`LaunchedEffect` and calls `navState.clear()`, draining any pending
detail stack so the tapped destination is what the user actually sees.
- SINGLE_PANE: one signal (Unit), one layout-local `navState`.
- DECK: signal payload is the column id; each `DeckColumnContainer`
filters on `column.id`, so only the focused column's detail clears —
other columns' navigation stacks are preserved.
- Same-item taps also clear (signal fires unconditionally, unlike a
StateFlow value comparison).
- `onOpenSettings` uses the same navigate / focusExistingColumn paths
and inherits the fix automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use the app's cached stringRes helper instead of raw stringResource,
matching the dominant convention in ui/note/types
- Drive the icon animation from the Compose frame clock (withFrameMillis)
instead of a delay loop, so the ticker suspends whenever the composition
stops drawing rather than waking the main dispatcher 4x/sec from the
back stack
- Drop the unconsumed memoryCardId/blockState/blockHash accessors; the
tag schema stays documented in the class KDoc
- Document the frames arrays as frozen: mutating them in place would
silently break the @Immutable skip contract; build a new instance
to change pixels
- Replace the API-29-deprecated Bitmap.createBitmap(IntArray, ...)
overload with createBitmap(w, h, config) + setPixels
Extend the showOnchainWallet preference to the zap flows, which also surface an
on-chain rail:
- the on-chain rail on each zap-amount chip in ZapAmountChoicePopup (folded into
the shared railCapability so every caller — reaction row, ReusableZapButton —
is covered)
- the "Send on-chain instead" hand-off button in the custom zap dialog
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
Some users don't want the on-chain wallet surfaced. Add a `showOnchainWallet`
UI preference (default true, so behavior is unchanged) that hides it across the
app when turned off:
- the "Bitcoin" card on the Wallet screen (OnchainSection)
- the on-chain chip on profiles (DisplayPaymentRailChips)
- the on-chain rail in the Send Payment screen
The flag lives in the existing UiSettings/UiSettingsFlow display-preferences
system (persisted to the shared-settings DataStore, alongside the other
show/hide profile toggles) and is exposed as a switch on the Profile UI
settings screen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
A relay that refuses negentropy with a NEG-ERR whose reason merely starts
with "blocked" (e.g. "blocked: Negentropy sync is disabled" from a relay
that has NIP-77 turned off, or an auth/ban refusal) was misclassified as a
strfry `max_sync_events` overflow by `isOverflow`. Overflow triggers
created_at window-splitting, so every split re-opened, was refused again,
and the splitter fanned out breadth-first across the whole created_at range
(~2^31 windows). The call therefore never threw NegentropySyncException (so
`negentropySyncOrFetch` never took its paging fallback) and never tripped
the idle watchdog (the relay answered every NEG-OPEN promptly), so it hung
indefinitely. A second relay whose refusal string did not start with
"blocked" fell through to `Failed` -> paging and completed, which is why the
two behaved differently despite advertising the same NIPs.
Narrow `isOverflow` to genuine "result set too large" signals only; a bare
`blocked:` refusal now maps to a hard failure and fails over to paging.
Adds a regression test driving an in-process relay that refuses every
NEG-OPEN with "blocked: Negentropy sync is disabled" while still serving
plain REQ: negentropySyncOrFetch now pages and delivers every event instead
of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015UF3eh76rRiwAuPg32rwiz
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):
geode import [--db …] [--no-verify] [FILE…] # files, or stdin when none
geode export [--db …] # NDJSON to stdout
Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.
This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.
Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.
Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two benchmark-integrity bugs that could invalidate sync numbers while a run
reports success:
- CorpusServerMain keyed its serve-existing decision only on port + row
count, so re-running a port with a different corpus/maxCount, or after a
load crashed mid-way, silently served a stale/partial DB. Gate reuse on a
completion sentinel keyed on corpus identity (path + byte length) and
maxCount, written only after a full load; on any mismatch the prior DB is
dropped and reloaded.
- CorpusDownloader treated CLOSED identically to EOSE, so a relay ending a
sub early (rate-limit/policy) after sending a partial page advanced the
cursor past the unsent tail — silent corpus loss. Treat CLOSED as a soft
failure (null → reconnect and retry the same cursor; the id dedup set
absorbs the re-fetch), distinct from EOSE which means the page is
complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
runCatchUpUp used negentropyReconcileIds, which builds the FULL need+have
id lists in memory even though the up direction only needs haveIds — on a
large window (e.g. 1M local events against an empty upstream) that is a
~100 MB+ heap spike per convergence round, plus a needIds list built and
immediately discarded.
Switch to the streaming negentropyReconcile: publish each haveIds batch as
it arrives (bounded to one batch of ids) and drop the need direction via a
no-op onNeedIds. The publish still suspends the reconcile round, so the
back-pressure and the reconcile-as-delivery-check convergence loop are
unchanged — only the peak memory drops from O(window) to O(batch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:
- id tie-break: thread the IndexingStrategy through run()/prepareStreams
and append ", id ASC" to the per-stream ORDER BY when
useAndIndexIdOnOrderBy is set — matching every sibling query in
QueryBuilder. The id-indexed order comes straight off the index (no
extra sort, lazy cursor preserved), so the merge now matches the
single-SQL path byte-for-byte on same-second same-author events. Without
the id index the tie stays in rowid order (a valid NIP-01 newest-N);
documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
kinds, so a filter with a repeated pubkey can no longer open two
identical cursors and emit each matching event twice (the single-SQL
IN(…) path already dedups).
Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode paged-REQ sink (verify on, FTS off):
geode→geode 4,963 ev/s (completes) strfry→geode 5,801 ev/s (completes)
Findings: strfry has no REQ bulk sync (stream is live-only limit:0), so the
strfry-sink REQ pairs are n/a. strfry→geode over REQ now completes (earlier
FTS-on stall was strfry killing the slow client at its 32MB pending cap; FTS-off
keeps pace). REQ is ~15-30% slower than negentropy for the geode sink, and
geode-as-a-REQ-source is slower than strfry (SQLite range scans vs LMDB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
All four source→sink pairings via NIP-77, geode sink configured like strfry
(verify on, FTS off), run sequentially:
geode→geode 6,971 ev/s strfry→geode 6,690 ev/s
strfry→strfry 2,682 ev/s geode→strfry 2,127 ev/s
Key findings: the sink sets the rate (geode ~6.7-7.0k, strfry ~2.1-2.7k from
either source — geode ingests ~2.6-3.3x faster); full geode↔strfry negentropy
interop both directions; strfry-source count is lower because its negentropy
snapshot drops NIP-40-expired events (geode, no cutoff, offers its full set).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Reuse the file-backed source DB when it already holds events instead of
deleting + reloading the corpus every boot, so re-running a single sync pair
skips the multi-minute load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
- MirrorSyncThroughputTest: add -DsyncVerify (default true — `strfry sync` always
verifies received events, so the negentropy sink verifies too for an
apples-to-apples comparison) and -DsyncFts (default true; pass false to match
strfry, which has no NIP-50). Both forwarded through the geode test task.
- CorpusServerMain: a benchmark-only tool that boots a real geode relay
(geode's default indexing) preloaded with an NDJSON corpus over a file-backed
store and serves forever, so `strfry sync` / another geode / the negentropy
sink can reconcile against a geode source holding the same 1M corpus a strfry
source does.
Used to run the 4-pair 1M negentropy sync comparison (geode↔geode, strfry→geode,
strfry↔strfry, geode→strfry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
strfry's `sync --dir` is bidirectional (its source: doUp = both||up,
doDown = both||down), but geode's catch-up was down-only. Add the up half so
`dir=up`/`dir=both` reconcile-and-push matches `strfry sync --dir both`.
runCatchUpUp reconciles the local set against the upstream (negentropyReconcileIds)
and publishes the events we hold that the upstream lacks (the reconcile's `have`
ids). Symmetric to the down catch-up: same one `dir`, live up-session starts at
`now` when the up catch-up covers history.
Reliability: client.publish's outbox is best-effort under a bulk burst (each
publish also churns a reconnect — measured ~1-2% dropped per pass), so the push
runs as a reconcile→push convergence loop. Each round re-reconciles — the
reconcile IS the delivery check against the upstream — and re-pushes only the
stragglers until the have-diff is empty. Test observed 3000 → 69 → 2 → 0 across
3 rounds, lossless.
Test: MirrorNegentropyCatchUpTest.negentropyCatchUpPushesUp pushes 3000 local
events to an empty (no-verify) sink and asserts all 3000 land. The four existing
mirror tests still pass (up catch-up needs a store + negentropyBackfill, both off
by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).
MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:
- Catch-up reconciles the local set against the upstream over the
[now - backfill_seconds, now] window and downloads only the diff via the
existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
overrun us) and it completes the pull. Reconcile-against-local means a warm
restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
the sync's job). The windows overlap at `now`; the store's unique-id
constraint dedups the seam.
Changes:
- quartz: add a backward-compatible `localEntries` param to the public
negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
trusted-scope re-check as the live path; failure is non-fatal). New `store` +
`negentropyBackfill` ctor params; default off so existing live-REQ tests are
unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
preloading historical events a live-only sub cannot deliver, then proves the
post-boot event still arrives (3000 catch-up + 1 live = 3001).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a NIP-77 negentropy client path to MirrorSyncThroughputTest (default for
external sources; `-DsyncMode=req` keeps the paged-REQ drain). geode reconciles
its empty set against strfry, then client-paced fetch-by-ids + ingest.
Result on the 1M damus.io corpus (strfry source):
- reconcile (empty local → 995,024 need-ids): 11.4 s, 64 rounds.
- fetch + ingest: ~7,000 ev/s steady-state.
- overall: 994,936 / 997,980 in 171.0 s => 5,818 ev/s — COMPLETES.
This is the apples-to-apples counterpart to strfry→strfry's `strfry sync`
(both negentropy, same corpus, empty→full): strfry→geode 5,818 ev/s vs
strfry→strfry ~2,550 ev/s — geode ingests real content ~2.3x faster and
finishes the pull. The paged-REQ path, by contrast, stalls at ~310k every time
(strfry kills a slow REQ client at its 32 MB maxPendingOutboundBytes cap),
confirming that cross-relay bulk sync from strfry requires negentropy, not REQ.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a strfry→geode drain mode to MirrorSyncThroughputTest and documents the
three-way 1M-corpus sync throughput comparison.
Findings (relayBench/plans/2026-07-04-sync-throughput-1m.md):
- strfry→strfry (strfry sync / negentropy): ~2,550 ev/s, completes.
- geode→geode (MirrorWorker REQ): 13,161 ev/s, completes — but inflated by
synthetic 4-byte content; real-content ingest is ~7,000 ev/s.
- strfry→geode (paged REQ drain, real corpus): sustains ~7,000 ev/s but does
NOT finish — strfry hard-kills a slower REQ client at its 32 MB
maxPendingOutboundBytes cap (~310k events in; confirmed by the source log's
"Pending: 32.01M" disconnects). Independent of heap (2G/10G) and the live
negentropy index (on/off).
Conclusion: strfry's REQ serving is structurally hostile to any client slower
than its scan (buffers outbound, then kills or OOMs). Only NIP-77 negentropy —
pull/reconcile-based and client-paced — completes a cross-relay bulk pull, which
is why strfry's own sync uses it. A production geode backfill from a foreign
relay should use negentropy, not the live-tail REQ path.
Also documents MirrorWorker's unbounded intake channel (a deliberate live-tail
trade) OOMing under a 1M bulk backfill, and forwards the sync* system
properties through the geode test task.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.
When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Claude
Vitor PamplonaandGitHub
nrobi144andClaude Opus 4.7
David KasparandGitHub
vitorpamplonaandgithub-actions[bot]