Replace the depth-limited, user-capped BFS with a completeness loop that runs
until every discovered user's kind:10002 outbox has been checked and their
latest kind:3/10000/1984 pulled from it:
- Delete the --max-users cap entirely.
- Crawl round by round until the pending set (discovered minus done) is empty.
A user is "done" once we download its contact list, or after --max-attempts
(default 3) failed tries of its outbox — so an unreachable outbox can't stall
the crawl, and it still terminates on a finite graph.
- --max-rounds replaces --max-depth as an (unbounded by default) safety backstop.
- Track and report the pool of relays actually contacted (relays_contacted),
the "running relays" we connect to as more outboxes are discovered.
JSON: `depth_reached` -> `crawl_rounds`, add `relays_contacted`. Per-round and
final crawl-summary progress on stderr. Local regression: scores unchanged
(rank 26); the crawl retries contact-list-less users then terminates cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Correct the injector's relay model: indexer relays (purplepag.es, coracle, …)
aggregate kind:10002 (and kind:0) for the whole network but do NOT serve
kind:3/10000/1984. Those live only on each user's own outbox.
- Split the relay sets: `relayListDiscoveryRelays` (bootstrap + event-finder +
indexers) is used only to locate kind:10002; `contentFallbackRelays`
(bootstrap + event-finder, no indexers) is the best-effort fallback for
content when a user's outbox is unknown/down.
- Content is fetched from each user's outbox write relays, with harvested
relay hints and general relays as fallback — never indexers.
Also add progress status (all on stderr, stdout stays the JSON contract):
- loading already logs per-hop frontier/recovered/new/total counts;
- "graph built: N users, E edges; scoring…" and "scored N users" bracket the
calculation;
- GrapeRank.compute gains an optional (visited, scored, queued) progress
callback, wired to emit a scoring line every 5000 worklist visits so a large
graph shows movement instead of hanging silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
To match Brainstorm's full-graph ingest, the crawl now discovers data through
three tiers (mirroring the app's pickRelaysToLoadUsers) instead of just the
outbox + a small fallback:
- Indexer relays (purplepag.es, coracle, …) join the discovery set. They serve
kind:0/3/10002 for the whole network and are where a stranger's relay list
and contact list are actually found — the biggest completeness lever.
- Per-follow relay hints are harvested from the `p`-tag hints in every contact
list we crawl and used as a discovery tier below each user's kind:10002.
- A per-hop retry pass re-queries any frontier member whose contact list still
didn't arrive (no kind:10002, or its outbox was unreachable) against the
indexer + hint set, recovering users the outbox model alone would miss.
This tightens the only real source of score divergence from Brainstorm — data
completeness — since a signal's weight scales by the rater's influence, so the
users that matter are exactly the in-graph ones this crawl now reaches more
reliably. Local regression check: scores unchanged (rank 26 for direct follows).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Analysis of NosFabrica/brainstorm_graperank_algorithm (Java scoring worker)
and NosFabrica/brainstorm_server (Python orchestration) to confirm amy's
scores match the reference GrapeRank service.
Finding: our commons/wot formula and every scoring parameter are already
identical to Brainstorm's DEFAULT preset (attenuation 0.85, rigor 0.5,
follow 1.0/0.03, from-observer 0.5, mute/report -0.1/0.5, delta 0.0001). Our
`score` is exactly their ScoreCard `influence`. Remaining divergence is data
completeness, not math — and because a signal's weight scales by the rater's
influence, only in-graph raters move a score, which our outbox crawl already
captures. Documents the pipeline, side-by-side params, divergence sources,
and follow-ups (presets, influence/verified fields).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Publishing kind:30382 rank cards is only half of NIP-85 — clients also need
the kind:10040 TrustProviderListEvent to discover which key provides which
assertion, and where. Add the discovery layer as two sub-verbs:
- `amy graperank register [PROVIDER]` — append a ServiceProviderTag
(default `30382:rank`, self, first outbox relay) to the account's kind:10040,
fetching the freshest list first so existing providers are preserved.
Idempotent, supports `--service KIND:TAG`, `--relay`, and `--private`.
- `amy graperank providers [USER]` — list a user's declared providers
(cache-first; own private entries are decrypted and included).
Bare `amy graperank [OBSERVER]` still computes scores; the dispatcher only
peels off the `register` / `providers` words. All built on quartz's existing
`TrustProviderListEvent` / `ServiceProviderTag` / `ProviderTypes`.
Verified against a local geode relay: register creates the 10040 and is
idempotent on re-run; providers lists both a public 30382:rank entry and a
private 30382:followers entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Previously `graperank --publish` rebuilt and rebroadcast a NIP-85 kind:30382
ContactCard for every scored user on every run, minting a new event id and
created_at even when the rank was identical — pure churn for a parameterized-
replaceable event.
Read back the ranks we last published from the account's own kind:30382 cards
in the local store (ctx.publish already persists them) and publish only the
targets whose rank is new or changed. Report the count left alone as
`skipped_unchanged`.
Verified against a local geode relay: first run publishes N cards
(skipped_unchanged=0); an immediate re-run with identical ranks publishes 0
(skipped_unchanged=N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
- Remove `--target USER`: the command already emits the full ranking, and a
single-user lookup is a trivial slice of it.
- Remove `--no-mutes` / `--no-reports` and the include* parameters on
TrustGraphBuilder.build. GrapeRank is defined over follows, mutes and
reports together; scoring with a signal disabled isn't a meaningful WoT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Bring the GrapeRank algorithm into Amethyst as a WoT service calculator
on the CLI.
commons/wot (protocol-agnostic, CLI-safe, reusable by the apps):
- TrustGraph / TrustEdge / TrustRelation — pubkey-keyed graph model.
- GrapeRank — single-observer scoring engine, a faithful port of the
reference v3 TargetedBFS variant using a worklist that reaches the same
fixed point a full sweep would while only touching reachable users.
- TrustGraphBuilder — pure kind:3 / kind:10000 / kind:1984 events -> graph
(latest-replaceable-per-author, dedup, self-edge drop).
- Unit tests: hand-computed values plus an adversarial full-sweep
cross-check over 50 random graphs.
cli: `amy graperank [OBSERVER]` crawls the follow/mute/report graph via the
outbox model (locate each user's kind:10002 write relays, then fetch their
lists from their own relays, with a broad event-finder fallback) until no
new users appear, scores it, and prints a ranked list (text / --json).
--target queries one user, --offline scores from the local store, and
--publish writes NIP-85 kind:30382 ContactCard assertions
(rank = round(score*100)) per user at or above --min-rank.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWk2ZMrGBSr4WenKgwqmbB
Restructure `amy relay` from verb-first `relay add URL --type T` to noun-first
`relay <noun> <verb>`, matching amy's `marmot group …` / `cashu mint …`
convention. The relay-list type is now a required path segment (no implicit
default), and a bare noun lists that bucket.
NIP-65 (kind:10002) is fronted by two facet-nouns, `outbox` (write) and `inbox`
(read), replacing the `--marker` flag. They edit the single 10002 event and
apply the spec's merge rules:
- outbox add R on a read-only R → both
- inbox add R on a write-only R → both
- outbox remove R on a both-R → read (stays in inbox)
- inbox remove R on a both-R → write (stays in outbox)
- dropping the last facet removes R entirely
`relay nip65` shows the combined view; `nip65 remove`/`clear` edit the whole
event.
Other buckets are noun+verb: `relay dm|key-package|search|private|blocked|
trusted|proxy|indexer|broadcast|feeds <add|remove|set|clear|list>`. `set` needs
≥1 URL; `clear` empties. `relay add|remove URL` (no noun) stays as the
transport fan-out (nip65 both + dm + key-package).
BREAKING (cli --json/args): removes `relay add/remove/set --type T` and
`--marker`; `relay list` overview now keys nip65 as `outbox`/`inbox`/`nip65`
and the DM bucket as `dm` (was `inbox`). In-repo harnesses updated
(cache/dm/marmot setup drop `--type all`; cache T5 asserts `.dm`).
Verified end-to-end: merge semantics, encrypted NIP-51 round-trips, facet
set/clear, fan-out, aliases, and error paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
`relay set --type T` with no URLs is now rejected (bad_args, exit 2) instead
of silently wiping the list — a bare empty `set` is almost always a shell
variable that expanded to nothing. Emptying a bucket is explicit: pass
`--clear` (mutually exclusive with URLs).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
Expand `amy relay` from the 3 transport lists (nip65/inbox/key_package) to
every relay-list bucket Amethyst's relay-settings screen manages, and add
remove/set verbs alongside add/list.
Buckets (kind): nip65 (10002, read/write markers), inbox/dm (10050),
key_package (10051), search (10007), private (10013), blocked (10006),
trusted (10089), proxy (10087), indexer (10086), broadcast (10088),
feeds/favorites (10012). The private NIP-51 lists are signed NIP-44-encrypted
via the quartz event factories, exactly like the app. Local relays (device
pref, no event) and named relay sets (30002) are intentionally out of scope.
New/changed commands:
- `relay add URL --type T [--marker read|write|both]` — `--marker` sets the
nip65 role; `all` still means nip65+inbox+key_package.
- `relay remove URL --type T` — new.
- `relay set --type T [URL…] [--marker …]` — new; replace a whole bucket
(no URLs clears it).
- `relay list [--type T]` — lists every bucket, or one.
- `relay publish-lists` — now broadcasts every configured list.
Thin-assembly only: buckets are a small registry over the existing quartz
`create`/`relays` factories; adds one generic `Context.latestReplaceable`
helper. `--json` is additive — legacy keys (`nip65`/`inbox`/`key_package`,
`nip65_event_id`/…) are unchanged, so the existing test harnesses keep passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EjHzNewJ2sfBGCcSwe35Mc
`putPending` stores each relay's pending map as
`Collections.synchronizedMap(LinkedHashMap(...))` and wraps every writer
path (`putPending`, `recordSent`, `recordIncoming`, `recordDisconnect`)
in `synchronized(perRelay)`. `sweep` iterated `pending.entries.iterator()`
without taking the same lock, violating the wrapper's Javadoc contract.
Any concurrent websocket-thread write during `RelayHealthStore.reclassify`'s
sweep threw `ConcurrentModificationException` on the underlying
`LinkedHashMap$LinkedHashIterator`. Because `reclassify` schedules sweep
on the AWT dispatcher, the CME killed Amethyst Desktop's Compose render
thread and froze the UI.
Wrap both inner iterator loops in `synchronized(pending) { ... }` — the
exact synchronization the wrapper's Javadoc prescribes for manual
iteration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Desktop, tapping a sidebar nav item while a detail screen (profile,
thread, article, editor) was open only mutated the sidebar destination.
The opaque `AnimatedContent` overlay driven by `ColumnNavigationState`
kept covering the (already-swapped) root content until the user hit
Back, creating the impression that the click did nothing.
Fix: emit a `clearOverlaySignal` from `SinglePaneState.navigate` and
`DeckState.focusExistingColumn`. Each layout collects the signal in a
`LaunchedEffect` and calls `navState.clear()`, draining any pending
detail stack so the tapped destination is what the user actually sees.
- SINGLE_PANE: one signal (Unit), one layout-local `navState`.
- DECK: signal payload is the column id; each `DeckColumnContainer`
filters on `column.id`, so only the focused column's detail clears —
other columns' navigation stacks are preserved.
- Same-item taps also clear (signal fires unconditionally, unlike a
StateFlow value comparison).
- `onOpenSettings` uses the same navigate / focusExistingColumn paths
and inherits the fix automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use the app's cached stringRes helper instead of raw stringResource,
matching the dominant convention in ui/note/types
- Drive the icon animation from the Compose frame clock (withFrameMillis)
instead of a delay loop, so the ticker suspends whenever the composition
stops drawing rather than waking the main dispatcher 4x/sec from the
back stack
- Drop the unconsumed memoryCardId/blockState/blockHash accessors; the
tag schema stays documented in the class KDoc
- Document the frames arrays as frozen: mutating them in place would
silently break the @Immutable skip contract; build a new instance
to change pixels
- Replace the API-29-deprecated Bitmap.createBitmap(IntArray, ...)
overload with createBitmap(w, h, config) + setPixels
Extend the showOnchainWallet preference to the zap flows, which also surface an
on-chain rail:
- the on-chain rail on each zap-amount chip in ZapAmountChoicePopup (folded into
the shared railCapability so every caller — reaction row, ReusableZapButton —
is covered)
- the "Send on-chain instead" hand-off button in the custom zap dialog
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
Some users don't want the on-chain wallet surfaced. Add a `showOnchainWallet`
UI preference (default true, so behavior is unchanged) that hides it across the
app when turned off:
- the "Bitcoin" card on the Wallet screen (OnchainSection)
- the on-chain chip on profiles (DisplayPaymentRailChips)
- the on-chain rail in the Send Payment screen
The flag lives in the existing UiSettings/UiSettingsFlow display-preferences
system (persisted to the shared-settings DataStore, alongside the other
show/hide profile toggles) and is exposed as a switch on the Profile UI
settings screen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UM57Rq5iUPL1SGhzhzTusa
A relay that refuses negentropy with a NEG-ERR whose reason merely starts
with "blocked" (e.g. "blocked: Negentropy sync is disabled" from a relay
that has NIP-77 turned off, or an auth/ban refusal) was misclassified as a
strfry `max_sync_events` overflow by `isOverflow`. Overflow triggers
created_at window-splitting, so every split re-opened, was refused again,
and the splitter fanned out breadth-first across the whole created_at range
(~2^31 windows). The call therefore never threw NegentropySyncException (so
`negentropySyncOrFetch` never took its paging fallback) and never tripped
the idle watchdog (the relay answered every NEG-OPEN promptly), so it hung
indefinitely. A second relay whose refusal string did not start with
"blocked" fell through to `Failed` -> paging and completed, which is why the
two behaved differently despite advertising the same NIPs.
Narrow `isOverflow` to genuine "result set too large" signals only; a bare
`blocked:` refusal now maps to a hard failure and fails over to paging.
Adds a regression test driving an in-process relay that refuses every
NEG-OPEN with "blocked: Negentropy sync is disabled" while still serving
plain REQ: negentropySyncOrFetch now pages and delivers every event instead
of hanging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015UF3eh76rRiwAuPg32rwiz
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):
geode import [--db …] [--no-verify] [FILE…] # files, or stdin when none
geode export [--db …] # NDJSON to stdout
Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.
This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.
Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.
Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two benchmark-integrity bugs that could invalidate sync numbers while a run
reports success:
- CorpusServerMain keyed its serve-existing decision only on port + row
count, so re-running a port with a different corpus/maxCount, or after a
load crashed mid-way, silently served a stale/partial DB. Gate reuse on a
completion sentinel keyed on corpus identity (path + byte length) and
maxCount, written only after a full load; on any mismatch the prior DB is
dropped and reloaded.
- CorpusDownloader treated CLOSED identically to EOSE, so a relay ending a
sub early (rate-limit/policy) after sending a partial page advanced the
cursor past the unsent tail — silent corpus loss. Treat CLOSED as a soft
failure (null → reconnect and retry the same cursor; the id dedup set
absorbs the re-fetch), distinct from EOSE which means the page is
complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
runCatchUpUp used negentropyReconcileIds, which builds the FULL need+have
id lists in memory even though the up direction only needs haveIds — on a
large window (e.g. 1M local events against an empty upstream) that is a
~100 MB+ heap spike per convergence round, plus a needIds list built and
immediately discarded.
Switch to the streaming negentropyReconcile: publish each haveIds batch as
it arrives (bounded to one batch of ids) and drop the need direction via a
no-op onNeedIds. The publish still suspends the reconcile round, so the
back-pressure and the reconcile-as-delivery-check convergence loop are
unchanged — only the peak memory drops from O(window) to O(batch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:
- id tie-break: thread the IndexingStrategy through run()/prepareStreams
and append ", id ASC" to the per-stream ORDER BY when
useAndIndexIdOnOrderBy is set — matching every sibling query in
QueryBuilder. The id-indexed order comes straight off the index (no
extra sort, lazy cursor preserved), so the merge now matches the
single-SQL path byte-for-byte on same-second same-author events. Without
the id index the tie stays in rowid order (a valid NIP-01 newest-N);
documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
kinds, so a filter with a repeated pubkey can no longer open two
identical cursors and emit each matching event twice (the single-SQL
IN(…) path already dedups).
Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode paged-REQ sink (verify on, FTS off):
geode→geode 4,963 ev/s (completes) strfry→geode 5,801 ev/s (completes)
Findings: strfry has no REQ bulk sync (stream is live-only limit:0), so the
strfry-sink REQ pairs are n/a. strfry→geode over REQ now completes (earlier
FTS-on stall was strfry killing the slow client at its 32MB pending cap; FTS-off
keeps pace). REQ is ~15-30% slower than negentropy for the geode sink, and
geode-as-a-REQ-source is slower than strfry (SQLite range scans vs LMDB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
All four source→sink pairings via NIP-77, geode sink configured like strfry
(verify on, FTS off), run sequentially:
geode→geode 6,971 ev/s strfry→geode 6,690 ev/s
strfry→strfry 2,682 ev/s geode→strfry 2,127 ev/s
Key findings: the sink sets the rate (geode ~6.7-7.0k, strfry ~2.1-2.7k from
either source — geode ingests ~2.6-3.3x faster); full geode↔strfry negentropy
interop both directions; strfry-source count is lower because its negentropy
snapshot drops NIP-40-expired events (geode, no cutoff, offers its full set).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Reuse the file-backed source DB when it already holds events instead of
deleting + reloading the corpus every boot, so re-running a single sync pair
skips the multi-minute load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
- MirrorSyncThroughputTest: add -DsyncVerify (default true — `strfry sync` always
verifies received events, so the negentropy sink verifies too for an
apples-to-apples comparison) and -DsyncFts (default true; pass false to match
strfry, which has no NIP-50). Both forwarded through the geode test task.
- CorpusServerMain: a benchmark-only tool that boots a real geode relay
(geode's default indexing) preloaded with an NDJSON corpus over a file-backed
store and serves forever, so `strfry sync` / another geode / the negentropy
sink can reconcile against a geode source holding the same 1M corpus a strfry
source does.
Used to run the 4-pair 1M negentropy sync comparison (geode↔geode, strfry→geode,
strfry↔strfry, geode→strfry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
strfry's `sync --dir` is bidirectional (its source: doUp = both||up,
doDown = both||down), but geode's catch-up was down-only. Add the up half so
`dir=up`/`dir=both` reconcile-and-push matches `strfry sync --dir both`.
runCatchUpUp reconciles the local set against the upstream (negentropyReconcileIds)
and publishes the events we hold that the upstream lacks (the reconcile's `have`
ids). Symmetric to the down catch-up: same one `dir`, live up-session starts at
`now` when the up catch-up covers history.
Reliability: client.publish's outbox is best-effort under a bulk burst (each
publish also churns a reconnect — measured ~1-2% dropped per pass), so the push
runs as a reconcile→push convergence loop. Each round re-reconciles — the
reconcile IS the delivery check against the upstream — and re-pushes only the
stragglers until the have-diff is empty. Test observed 3000 → 69 → 2 → 0 across
3 rounds, lossless.
Test: MirrorNegentropyCatchUpTest.negentropyCatchUpPushesUp pushes 3000 local
events to an empty (no-verify) sink and asserts all 3000 land. The four existing
mirror tests still pass (up catch-up needs a store + negentropyBackfill, both off
by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).
MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:
- Catch-up reconciles the local set against the upstream over the
[now - backfill_seconds, now] window and downloads only the diff via the
existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
overrun us) and it completes the pull. Reconcile-against-local means a warm
restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
the sync's job). The windows overlap at `now`; the store's unique-id
constraint dedups the seam.
Changes:
- quartz: add a backward-compatible `localEntries` param to the public
negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
trusted-scope re-check as the live path; failure is non-fatal). New `store` +
`negentropyBackfill` ctor params; default off so existing live-REQ tests are
unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
preloading historical events a live-only sub cannot deliver, then proves the
post-boot event still arrives (3000 catch-up + 1 live = 3001).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a NIP-77 negentropy client path to MirrorSyncThroughputTest (default for
external sources; `-DsyncMode=req` keeps the paged-REQ drain). geode reconciles
its empty set against strfry, then client-paced fetch-by-ids + ingest.
Result on the 1M damus.io corpus (strfry source):
- reconcile (empty local → 995,024 need-ids): 11.4 s, 64 rounds.
- fetch + ingest: ~7,000 ev/s steady-state.
- overall: 994,936 / 997,980 in 171.0 s => 5,818 ev/s — COMPLETES.
This is the apples-to-apples counterpart to strfry→strfry's `strfry sync`
(both negentropy, same corpus, empty→full): strfry→geode 5,818 ev/s vs
strfry→strfry ~2,550 ev/s — geode ingests real content ~2.3x faster and
finishes the pull. The paged-REQ path, by contrast, stalls at ~310k every time
(strfry kills a slow REQ client at its 32 MB maxPendingOutboundBytes cap),
confirming that cross-relay bulk sync from strfry requires negentropy, not REQ.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a strfry→geode drain mode to MirrorSyncThroughputTest and documents the
three-way 1M-corpus sync throughput comparison.
Findings (relayBench/plans/2026-07-04-sync-throughput-1m.md):
- strfry→strfry (strfry sync / negentropy): ~2,550 ev/s, completes.
- geode→geode (MirrorWorker REQ): 13,161 ev/s, completes — but inflated by
synthetic 4-byte content; real-content ingest is ~7,000 ev/s.
- strfry→geode (paged REQ drain, real corpus): sustains ~7,000 ev/s but does
NOT finish — strfry hard-kills a slower REQ client at its 32 MB
maxPendingOutboundBytes cap (~310k events in; confirmed by the source log's
"Pending: 32.01M" disconnects). Independent of heap (2G/10G) and the live
negentropy index (on/off).
Conclusion: strfry's REQ serving is structurally hostile to any client slower
than its scan (buffers outbound, then kills or OOMs). Only NIP-77 negentropy —
pull/reconcile-based and client-paced — completes a cross-relay bulk pull, which
is why strfry's own sync uses it. A production geode backfill from a foreign
relay should use negentropy, not the live-tail REQ path.
Also documents MirrorWorker's unbounded intake channel (a deliberate live-tail
trade) OOMing under a 1M bulk backfill, and forwards the sync* system
properties through the geode test task.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.
When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds -DsyncLiveIndex and a fast (no-live-index) source preload. Rate curves show
geode's mirror sustains ~20k ev/s with the live index off, but collapses to
O(n^2) with it on: the LiveNegentropyIndex is a sorted ArrayList whose per-event
insert is O(n), and a backfill delivers historical (non-near-tail) events, so
every insert memmoves ~n/2 entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Default EventStore(null) tokenizes FTS synchronously on every insert, which
dominates ingest and misrepresents mirror sync throughput. Use geode's actual
RelayIndexingStrategy (deferred FTS, live negentropy index) for both source and
sink, and log instantaneous events/s every 3s so the rate is visible during the
run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two pieces measuring how fast each sink pulls a large corpus from a source using
its native sync client:
- relayBench/sync-throughput-strfry.sh: `strfry import` N events into a source
strfry, boot it, `strfry sync --dir down` an empty sink, report events/s.
- MirrorSyncThroughputTest: geode downstream pulls via the real MirrorWorker
(WebSocket). Default in-process geode source (geode→geode); with
-DsyncSourceUrl it mirrors an external relay (e.g. strfry) for strfry→geode.
Sized by -DsyncN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The ~19/40k event shortfall in the geode↔geode sync is not a geode event-loss
bug: geode is proven lossless across the store, the concurrent IngestQueue
pipeline, RelaySession (OK-true only post-commit), and the real MirrorWorker
WebSocket path (50k/50k). The shortfall is in the benchmark's hand-rolled
fetchByIds+publish delta transfer. Points the fix at the harness (or at driving
convergence through geode's real mirror) rather than geode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Streams 50k events from an upstream KtorRelay to a downstream via the production
MirrorWorker (real OkHttp WebSocket + trusted skipVerify ingest) and asserts the
downstream receives every one — 50000/50000, 0 missing.
This closes the last untested layer: the in-process guards (BatchInsertLossTest,
ConcurrentIngestLossTest) call IngestQueue.submit directly, bypassing the wire.
With this, geode is proven lossless end-to-end — store, concurrent pipeline,
background pool contention, RelaySession, and the real WS mirror path. The
~0.05% shortfall seen in the geode↔geode relayBench sync is therefore an
artifact of the harness's hand-rolled fetchByIds+publish delta transfer, not a
geode event-loss bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Drives the full IngestQueue pipeline — parallel verify, greedy-drain group
commit, a concurrent deferred-FTS catch-up worker taking the pool writer, and
windowed concurrent submits via the trusted (skipVerify) mirror path — over the
clean 200k corpus, asserting every Accepted regular event is queryable after.
Passes (199,612 in/accepted/stored, 0 lost), together with BatchInsertLossTest
proving geode's ingest is lossless at every in-process layer. The geode↔geode
sync event-loss therefore lives above the store+queue — in the real Ktor
WebSocket path or the benchmark harness's hand-rolled delta transfer, which the
in-process paths bypass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Claude
Vitor PamplonaandGitHub
nrobi144andClaude Opus 4.7
David KasparandGitHub
vitorpamplonaandgithub-actions[bot]