Follow-up to the linux-arm64 CI leg (feat/release-linux-arm64). Extends the
release matrix to Windows in three places, all on free public-repo hosted
GitHub runners:
* build-desktop: adds windows-11-arm (arm64) alongside the existing
windows-latest (x64). jpackage/jlink on Windows arm64 produce arm64 MSIs
natively; the same packageReleaseMsi + createReleaseDistributable task
list is used unchanged and the portable-archive step already parameterises
on ${{ matrix.arch }}.
* build-cli: adds windows-latest (x64) and windows-11-arm (arm64) legs
running :cli:amyImage. Windows has no jpackageDeb/Rpm and MSI-for-CLI is
deferred (portable zip is the documented Windows install path); the
headless-lib assertion runs unchanged under git-bash. amyImage now emits
both a POSIX `bin/amy` shell launcher AND a Windows `bin/amy.bat`
launcher into the flat image so the tree layout is uniform regardless of
build host. The .bat pins UTF-8 (chcp 65001) for sun.jnu.encoding, same
reason the installDist .bat was already patched.
* build-geode: adds windows-latest + windows-11-arm legs running
:geode:geodeImage. The existing --port smoke test is generalised to pick
bin/geode.bat on Windows; NIP-11 fetch via curl works unchanged under
git-bash on GH windows runners. Same dual-launcher pattern as amy.
scripts/asset-name.sh: collect_cli_assets and collect_geode_assets now
package the flat image as .zip on Windows (7z when available, falling
back to `zip`, then a portable python3 zipfile.ZipFile invocation). Every
other OS continues to use tar.gz. Adds the expected Windows examples to
the header block.
BUILDING.md: mentions the windows-11-arm runner and updates the asset
count in the Release runbook. No asset-naming contract changes — the
existing amethyst-desktop-<v>-windows-<arch>.<ext>, amy-<v>-windows-<arch>.zip,
and geode-<v>-windows-<arch>.zip shapes were already in scope, they just
weren't produced by any CI leg before.
Local validation on macOS arm64 (build host: JDK 21, gradle 9.5.0):
./gradlew :cli:amyImage -> bin/amy + bin/amy.bat both present
./gradlew :geode:geodeImage -> bin/geode + bin/geode.bat both present
./bin/amy --help -> parses (unix launcher unbroken)
./bin/geode --port 17447 -> NIP-11 served, "supported_nips" present
collect_cli_assets windows arm64 ... -> valid .zip with bin/amy.bat
collect_geode_assets windows x64 ... -> valid .zip with bin/geode.bat
actionlint .github/workflows/create-release.yml -> no new findings
Cross-compile is impossible for jlink/jpackage, so end-to-end
Windows-runtime validation still happens on GH CI on the first PR
build; nothing in this change can be verified any harder locally.
Deep-audit pass over the branch. Nothing here changes what a band claims;
these are the defects that pass tests and bite later.
- record() read the clock twice PER KIND. A 40-kind map took 80 readings,
and worse, a span's floor and ceiling were judged against two different
instants — so a span could be accepted at one end and rejected at the
other on a clock tick. One read, one instant, for the whole call. The
aggregate path had the same double read and now shares it.
- legs() handed the SAME MutableList instance to every Filter in a group,
publishing its accumulator through a public return value. Filters are
treated as immutable everywhere else; this keeps that true by
construction rather than by nobody having tried yet.
- The state file's round trip was asserted only for the fields, never for
the behaviour. Three tests now pin it: per-kind spans survive a restart
AND still narrow per kind afterwards; the ALL_KINDS sentinel survives
its negative key through toString/toInt; and a pre-split file (min/max,
no spans) loads as the claim it always was. Plus the rollback contract
— `min`/`max` must remain the OUTER edges, since a binary from before
per-kind spans reads those and would otherwise skip ground it has not
covered.
Checked and found sound, recorded so the next reader need not re-derive
it: ConcurrentMap.snapshot() copies, so export() cannot be mutated under
a writer; Band is immutable (widen() copies its map), so a shared Band
across threads is safe; merge() keeps old.fullAt, preserving the
re-walk clock across widening; and coveringWindow does NOT regress —
a paged band gave >1 leg before this change too, and a reconciled band
still collapses to one leg and narrows the shared snapshot.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A consumer that cannot suspend has to block, and blocking here deadlocks
the whole client.
Measured on a mirror built against this library, twice, ~13 minutes after
each start: all 64 shared coroutine workers parked in `runBlocking` beneath
`trySendBlocking`, called from the websocket message callback. The consumer
draining that channel needed threads from the same pool to reach its store,
so it could never make room, so the producers never woke. Every stream, the
health reporter, all of it stopped, at 2% CPU with a healthy, idle backend.
A full queue was the symptom; producers eating the threads the drain needed
was the cause.
The coroutine context was already there — BasicOkHttpWebSocket has always
processed messages inside `scope.launch { for (message in incomingMessages) }`
— so the only thing forcing a blocking hand-off was that the hops in between
were declared non-suspend. Now they are not:
WebSocketListener.onMessage
RelayConnectionListener.onIncomingMessage
PoolRequests/PoolCounts/PoolEventOutbox.onIncomingMessage
SubscriptionListener.onEvent
fetchAllPages / negentropy accessories' onEvent parameter
A consumer that fills its buffer now suspends and releases its thread rather
than holding it, which is the same reasoning BasicOkHttpWebSocket already
documents for keeping its own channel UNLIMITED so a slow consumer cannot
block OkHttp reader threads. This extends it one layer down.
BLE is the one transport whose callback genuinely cannot suspend — the
platform hands notifications to a plain callback — so BleNostrClient gets
the same treatment the websocket transport already had: an UNLIMITED
hand-off channel so the BLE stack is never blocked, drained by ONE coroutine
so message order survives the boundary.
Tests that drove these entry points directly now do so from `runTest`, or
from `runBlocking` where the call sits inside a raw thread or Runnable that
models a platform callback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A band held ONE created_at interval per (relay, filter). For a filter
naming several kinds that is a claim no walk can support: ask for
`kinds: [0, 30382]`, find profiles going back years and score cards only
from last month, and the band records 2020..now for the pair. The next
run then skips that whole interior for BOTH — so score cards written
inside it are never asked for again, and nothing anywhere says so. A
long-lived kind vouched for a short-lived one.
Band.spans is now per kind. Each carries only the evidence actually
collected for it, so the profile kind keeps its wide interval and the
score kind keeps its narrow one, and legs() re-opens the interior for
the second while still skipping it for the first.
Three things keep the cost of that where it was:
- legs() REGROUPS kinds by the windows they want. Identical coverage —
the common case, and the only case until they diverge — collapses back
into one ask, so a filter that produced two legs still produces two
rather than two per kind. Only a kind whose evidence genuinely differs
earns its own.
- A finished reconcile needs no per-kind evidence and is given none:
negentropy compares the filter's whole id set in one pass, so it
covers every kind in the filter or none. Only the PAGED path changed.
- Filters naming no kinds keep a single span under ALL_KINDS, which is
the same claim as before, correctly scoped to the case where it is the
only claim available.
record() takes observedByKind, and SyncCoverage.observe() accumulates it
as events arrive — replacing the pair of hand-rolled vars each caller
kept, and moving the per-event isPlausible guard in with it. A paged
walk over a MULTI-kind filter that supplies none earns no band at all,
loudly, once: attributing one interval to every kind is exactly the
over-claim this removes, and a band that over-claims skips events
silently, which is worse than re-reading them. Single-kind filters are
untouched — there the aggregate always was the per-kind answer.
The state file gains a per-kind `spans` object and keeps `min`/`max` as
the outer edges, so a rollback to a binary from before this reads the
file and behaves as it always did. A file written BEFORE this loads its
one interval under ALL_KINDS — the old, wider claim, kept rather than
discarded because discarding it would re-download every upstream's
corpus once on upgrade. The first per-kind walk replaces it.
All 26 existing SyncCoverage tests pass unchanged, which is the evidence
that single-kind behaviour did not move. The five new ones were checked
against the pre-fix rule reinstated in place: the two behavioural ones
fail there and pass here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
quartz:
- SQLiteEventStore: classify per-row savepoint errors — policy refusals
(blocked:/constraint/not allowed) stay Rejected, everything else is now
Failed, so disk-full no longer masquerades as 2M duplicate rejections
- IEventStore.batchInsert default: rethrow CancellationException and map
unknown throws to Failed (re-offering a duplicate is idempotent;
dropping a good event on a transient store error is not)
- IngestQueue: rethrow CancellationException instead of stamping a
cancelled batch Failed and continuing
- HostStrikes: make the eviction verdict exactly-once under concurrency
(deadHosts.add is the atomic gate) and re-check produced before
publishing
- SyncCoverage: bound the identity fingerprint cache (a caller minting
fresh Filter instances per cycle could grow it forever); legs() gains a
floor parameter so a complete band re-opens its older span when the
caller's window deepens; coveringWindow no longer treats a fully
covered relay as needing the whole filter
- PagingWindowProgress: accept single-second windows (a band's re-read
edge leg is exactly that shape)
geode:
- MirrorWorker: cap reconciledThrough at the leg's own ceiling — the
older leg of a resumed catch-up no longer stamps the band complete
through 'now' before the newer leg has run (silent event loss for up
to fullResyncSeconds if that leg failed)
- MirrorWorker: run negentropy and the paged fallback by hand instead of
negentropySyncOrFetch: drops the O(delivered-ids) dedup set from the
mirror path, and a fallback resets the observed span so a band never
claims interior ranges only a half-finished reconcile scattered over
- MirrorWorker: clamp a paged band's ceiling to the snapshot instant so
one future-dated event cannot suppress the next boot's newer leg
- MirrorWorker.close(): join the workers (bounded) so the final coverage
flush carries the last records
- Main: gate the coverage file on the store actually being persistent —
database.file with in_memory=true (the default) persisted bands over a
volatile store, and the next boot skipped the backfill over an empty
database; honor --db overrides
- SyncCoverageFile: request ATOMIC_MOVE explicitly; fix the restore/dirty
comment
- Import summary now prints the failed count; document
mirror_sync_state_file in config.example.toml
Renames from review: SyncBands -> SyncCoverage ("sync" reads negentropy-ish
in quartz, and coverage is the role — the bands are the records), and
PagingProgress moves into relay.client.paging as PagingWindowProgress,
beside RelayLoadingCursors and RelayPagingProgress, with its docs swept from
"walk" dialect to quartz's pagination vocabulary and cross-references
delineating the three: cursors are in-memory positions for demand-driven UI
paging, the window progress is fraction/ETA for a bulk pagination over a
known window, coverage is persistent intervals that license skipping work.
geode adopts both halves of the new contract. MirrorWorker counts
InsertOutcome.Failed in its own `failed` counter instead of folding it into
`rejected`, and the down catch-up gains resume memory: SyncCoverageFile
persists SyncCoverage next to the event database (admin state-file
convention, temp-file + atomic move, daemon flush), and runCatchUpDown asks
only for the legs outside the covered band. Bands are keyed on the stable
scoped filter — never the boot window, whose since/until change every start
— and clamped to the window, which only slides forward, so an old band can
never license skipping a range an earlier boot could not ask about. A clean
reconcile records completeness through its snapshot instant; a paged
fallback earns only the span it saw. For an upstream without NIP-77 this
turns the every-boot full re-download of the backfill window into a
resumed walk.
Off unless wired: MirrorWorker's coverage parameter defaults to null and
in-memory stores keep no state file, so existing tests and setups are
unchanged. Full :quartz:jvmTest and :geode:test pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Pi9YYMhdzTFxRiV2jF9R
Bisecting existed to reconstruct per-event attribution after a store threw a
batch-wide exception. Better to never lose the attribution: batchInsert's
contract now requires per-row isolation, with a third outcome telling whose
fault a miss was. Rejected is the EVENT's fault (duplicate, expired, invalid,
blocked) and is final; Failed is the STORE's fault (schema drift, a failed
feed, a resource error) — the event was good, it is lost unless re-offered,
and a rising Failed count means the store is broken rather than that
upstreams send junk. Throwing is reserved for failures with no per-event
answer (engine unreachable, transaction never started), readable as "nothing
in this batch was written".
Consumers updated: RelaySession maps Failed to OK false with NIP-01's
"error:" prefix; IngestQueue converts a thrown batch and a missing outcome to
Failed instead of Rejected; NdjsonImportExport counts failed apart from
rejected; geode's MirrorWorker logs store failures at warn instead of
folding them into debug-level rejections. BisectingInsert and its test are
removed — with attribution guaranteed by the contract, retry-by-splitting
has nothing left to do.
Full :quartz:jvmTest passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Pi9YYMhdzTFxRiV2jF9R
multiRelayPoolReturnsContentFromEachRelay flaked with
"expected:<from-b> but was:<null>": the SubscriptionListener wrote the
per-relay results into a plain HashMap/HashSet, but each relay delivers
its EVENT/EOSE on its own InProcessWebSocket scope (Dispatchers.Default)
and PoolRequests dispatches the listener callbacks outside any lock. Two
relays therefore call `received[relay] = ...` concurrently, and a
HashMap.put racing a rehash can drop an entry, leaving a relay's value
null and failing the assertion.
Use ConcurrentHashMap and ConcurrentHashMap.newKeySet() for the shared
collections. Reproduced within 7 runs before the fix; 80 stress runs
clean after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HeAjLDNBvGPjb5bfViU3ad
The first cut appended :geode:test to the build-desktop matrix command.
That was wrong twice over: geode is JVM-only, so it ran 3× across the
ubuntu/macos/windows matrix, and — because org.gradle.parallel=true —
its default suite's CPU-heavy throughput benchmarks (a 1M-event mirror
sync, WireReqFloor, NegentropyServerReconcile) ran concurrently with the
timing-sensitive quartz relay-client tests, flaking
NostrClientReqBypassingRelayLimitsTest.denseSecondBeyondCapIsSteppedPastWithoutStalling.
Move :geode:test into its own test-geode job (needs: lint, ubuntu, JVM
21) so it runs once and its benchmark load can't starve another module's
timing assertions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KCdJwdhGtmLZ12ViS56S3k
geode was runnable only via ./gradlew :geode:run and was absent from CI.
Give it the same release process as the amy CLI (it's the same kind of
application-plugin JVM module), plus the pieces a long-running server
daemon needs that a one-shot CLI does not.
- Main.kt: add terminal --version/-V and --help/-h flags so a packaged
binary has a fast, exit-0 command (Homebrew test block, package smoke
checks, Docker healthcheck).
- build.gradle.kts: jlinkRuntime + geodeImage (portable flat app-image
with a bundled JRE, plus config.example.toml + geode.service under
share/) + jpackageDeb/jpackageRpm, mirroring cli/. No Compose to
exclude — geode depends only on :quartz.
- Dockerfile + .dockerignore: multi-stage image (gradle installDist ->
temurin JRE), the primary channel for relay operators.
- packaging/: systemd unit, macOS hardened-runtime entitlements, and a
reference Homebrew formula.
- scripts/asset-name.sh: geode_asset_name/collect_geode_assets under the
canonical geode-<version>-<family>-<arch>.<ext> scheme.
- create-release.yml: build-geode matrix (tarball + deb/rpm + no-JRE jvm
bundle, with a serve+NIP-11 smoke test of the jlink image) and a
docker-geode job pushing ghcr.io/<owner>/geode:<version> (+ :latest).
- bump-homebrew-geode-formula.yml: auto-sync the reference formula on
stable releases.
- build.yml: run :geode:test in CI (it ran in no workflow before).
- README.md + plans/2026-07-24-geode-release.md: operator docs + design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KCdJwdhGtmLZ12ViS56S3k
Geode hard-wired the SQLite EventStore. Add a `[database].backend`
selector (and `--store` CLI flag) so an operator can choose the store
implementation:
- "sqlite" (default): the SQLite EventStore, unchanged.
- "fs": quartz's filesystem FsEventStore, rooted at [database].file.
- any other value: a fully-qualified class name of a custom
IEventStore on the classpath, instantiated reflectively via one of
`(NormalizedRelayUrl?, IndexingStrategy)`, `(NormalizedRelayUrl?)`,
or `()` — the "plug in anything" escape hatch.
Store construction moves into a new StoreFactory (mirrors cli's
StoreFactory) shared by the serve path and the import/export verbs, so
both open the same store from the same config. The SQLite-only
`PRAGMA optimize` maintenance loop now runs only when the resolved
store is the SQLite one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GG3TBvLUv5uB5js1naG5sc
TagAuthorIndexBenchmark at 1M events settles the flag: the DM-room
shape (kinds + authors + #p, 65 client assembler call sites) drops
14.2 ms -> 0.66 ms (~21x, growing with corpus size) while batch-insert
cost stays inside run noise (49.0 vs 47.4 us/event). Existing relay
DBs build the index on next open via ensureOptionalIndexes.
Also refreshes the docs the numbers made stale: IndexingStrategy KDoc
now records the 200k and 1M measurements instead of a TODO,
MergeQueryExecutor's tag-merge note points at the new relayBench
reactions-watch scenario, FsQueryPlanner/FsDriverSelectionBenchmark
reflect the landed cost-based pick (149 ms -> 4.0 ms at 30k events),
and RELAY.md documents that strategy flag flips materialize indexes on
the next open.
Verified: quartz jvmTest store suites, geode test (126), desktopApp
LocalRelayStore tests (5, incl. reopening a default-strategy DB with
the new pubkey-alone flag), relayBench compiles.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RzkoN3SJHCZWRAiadXXG4w
RelayAuthenticator.signWithAllLoggedInUsers gained an `interactive`
Boolean parameter, but these two geode auth tests still passed a
two-arg lambda and no longer compiled. Accept and ignore the flag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0129yvP2hmVeDFfuKKy94tqX
Drive a real NostrClient against an in-process geode relay running
FullAuthPolicy, publishing a NIP-17 gift wrap through the PoolEventOutbox
retry queue. The first EVENT races ahead of AUTH and is rejected
`auth-required`; a RelayAuthenticator answers the challenge and the
still-pending wrap is resent on the post-AUTH resync and stored. This is
the integration counterpart to PoolEventOutboxAuthTest and exercises the
"auth-required must not burn the retry budget" fix end-to-end. A control
test (no authenticator) proves the relay genuinely gates, so the delivery
assertion isn't vacuous.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Drop the flaky publish-into-large-relay warmup from DeletionSettleBenchmark
(it timed out the measured reconcile at N=100k — the container noise the
docstring already warns against) and remove the throwaway ScratchSettleTiming
investigation tool. Record in the docstring what the phase breakdown proved:
the settle's extra time over a bare reconcile is O(K) relay-ingest of the K
residual deletions, dominated by one-time JVM/JIT warmup of the publish path
(consecutive K-note batches fell ~3100->570ms), not the deletion algorithm and
not O(N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
relayBench measures relay-to-relay reconcile, not the amy/quartz client feature,
so the deletion-settle perf claim belongs in an in-process benchmark of
negentropySettleDeletions itself.
Models the post-content-settle state: a relay with N notes, a local store with
the same N except K it deleted (keeping the K kind-5s). The reconcile residual is
exactly those K, so a sendUp settle fetches K — not N. Asserts residual==K,
sentUp==K, and relay convergence (correctness guard at the small default N),
and prints one-reconcile vs full-settle so the deletion overhead reads as
"a few reconciles + K", never "+ a content re-download". Measured:
N=2000 K=20: settle ~2x one reconcile, fetched K=20 not N
N=100000 K=20: settle ~5x one reconcile, fetched K=20 not N=100000
The growth is the relay rebuilding its negentropy index after the deletions
(O(N) once) — inherent to applying deletions, and still far cheaper than
re-fetching the need set, which the old per-need-fetch approach did.
Scale with -DdelBenchN / -DdelBenchK (forwarded by the geode test task).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The two-pass deletion convergence is protocol logic, not CLI assembly, and the
geode mirror is a near-term second consumer — so move it out of SyncCommand into
a reusable accessory alongside the rest of the negentropy family.
quartz: negentropySettleDeletions(relay, filter, store, sendUp, applyDown, …) —
re-reconciles after a content settle and resolves only the residual: publishes
our covering deletions up (sendUp) and/or ingests the relay's kind-5 down
(applyDown, vanish never auto-applied), looping until a round resolves nothing.
Returns DeletionSettleResult(sentUp, appliedDown, rounds). Everything it needs is
already quartz (negentropyReconcileIds, fetchAll, deletionsCovering,
publishAndConfirm, Event.verify, IEventStore), so it carries no CLI dependency.
SyncCommand's pass 2 collapses to a single call; pass 1 (content) is unchanged.
Catalogued in the accessories README.
Tests: DeletionSyncTest drives the accessory end-to-end both ways (sendUp → relay
converges to gone; applyDown → local converges to gone), on top of the existing
deletionsCovering unit + manual-wiring cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Replace the per-need-event fetch (which pulled the whole need set just to read
metadata — an O(db) regression on large syncs) with a second reconcile pass over
the residual, per the "settle, then diff, then explain what didn't converge" idea.
Pass 1 is the plain content sync again (drain needs, publish haves) — zero
deletion overhead. Pass 2+ re-reconciles; the leftover diff is exactly the
deletion mismatches, and only that (tiny) set is fetched:
- residual need (relay has it, we still lack it after --down) = we deleted it →
publish our covering deletion up so the relay drops it;
- residual have (we have it, relay still lacks it after --up) = the relay deleted
it → pull the relay's covering kind-5 down and apply locally (vanish is NOT
auto-applied on pull — account-wide blast radius).
Loops until a round resolves nothing (converges + self-verifies).
So `amy sync` makes the relay honor our deletions; `--up` makes us honor the
relay's; `--up --down` converges both ways. Cost is one cheap reconcile + the
residual regardless of database size — the large-DB bottleneck is gone by
construction, not by heuristics.
quartz: deletionsCovering is now source-agnostic (takes a query lambda) so the
same coverage rule runs against the local store (up) or the relay (down); the
IEventStore overload is the local convenience.
Tests: DeletionSyncTest gains the down-direction end-to-end (relay deleted →
local removes) alongside the up-direction and the per-form unit cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Refine the sync deletion rule to what was asked: for the events the relay HAS
that we LACK (the reconcile need set), publish only the local deletions that
would actually make the relay remove them — and nothing else, not other
deletions by the same author.
Determining coverage needs the need event's author/address/created_at, which we
don't have for an id we lack, so we fetch the need events (raw — no verify, no
store) purely for metadata. quartz gains IEventStore.deletionsCovering(events,
relay), which maps server-held events to the covering local deletions across all
three forms:
- NIP-09 id-based: a kind-5 with an `e` tag naming the event id;
- NIP-09 address-based: a kind-5 with an `a` tag naming the event's
addressable/replaceable coordinate, at/after it (created_at <= deletion);
- NIP-62 vanish: a kind-62 by the event's author, targeting this relay, issued
after it (created_at < vanish).
SyncCommand's need workers now fetch each need batch once (Context.fetchRaw),
publish its covering deletions (deduped across workers), and — when --down —
store the rest; anything we deleted is rejected by the store's own tombstone.
Nothing is pulled down or applied locally, so it cannot over-delete the store.
DeletionSyncTest covers each form (with cutoff and wrong-relay negatives) plus an
end-to-end reconcile → cover → publish that removes the note on the relay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Per the actual requirement, deletion propagation is exactly: for the ids the
relay HAS that we LACK (the negentropy need set), if we hold a kind-5 deletion
targeting one of them, publish that deletion up — so a note we deleted is
deleted on the relay too instead of being re-downloaded. Only the need ids,
only kind-5, up only.
This removes all the machinery the earlier approach accreted and that the audit
flagged as over-broad / data-loss-prone:
- deleted NostrClientDeletionSyncExt (the bidirectional side-channel, author
scoping, vanish gating, kind selection);
- reverted geode MirrorWorker to base (no deletion side-channel, live-sub
changes, catch-up ordering, or convergence changes);
- dropped the 3-phase SyncCommand flow (deletions-first pull, author-scope
derivation, reject-reaction backstop, --sync-vanish, deletions_* output).
The new path pulls nothing down and applies nothing locally, so it cannot
over-delete the store, and it needs no author scoping — the need set already
bounds it. Kind-62 is intentionally excluded: a vanish is not "of an id".
Emits deletions_sent. DeletionSyncTest now exercises the exact wiring
(reconcile → look up local kind-5 by its e tag for the need ids → publish),
including the negative case (a need id we never had sends nothing).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
An audit (adversarial-verified) found the deletion side-channel over-deletes
and over-propagates. Root cause: deletionSideChannelFilter fell open to
authors=null for any non-author-scoped content sync, so `amy sync --kind 1`
reconciled the RELAY'S ENTIRE kind-5/62 history and applied it to the personal
FsEventStore — every kind-5 deleting its targets + installing an id-tombstone
for every target id, every ALL_RELAYS kind-62 wiping all of a pubkey's events
(all kinds), and pushing our whole local deletion history up. Data loss plus a
full-history reconcile on every scoped sync.
Fixes:
- Bound the side-channel to the authors we actually hold content for (filter
authors ∪ local matched-set authors), never the relay's population. Skip when
that scope is empty; Phase 3's reject-reaction covers the author-less case.
- Kind-5 (precise, owner-scoped) propagates by default; kind-62 vanish is opt-in
via --sync-vanish (its blast radius always exceeds a content sync's scope).
- excludesDeletionKinds() now checks each deletion kind independently
(`--kind 1,5` no longer silently drops kind-62); the side-channel reconciles
only the missing kinds.
- amy Phase 1 is best-effort: a deletion-reconcile failure records deletions_error
and falls through to content, never aborting the primary sync (matches geode).
- Mirror up-catch-up converges on whether a PUBLISHABLE event was pushed, not raw
haveCount — a vanish targeting another relay no longer burns all 8 rounds every
startup. Mirror keeps its (correct) global scope for relay-to-relay replication.
Helper API: negentropyPropagateDeletions gains scopeAuthors + deletionKinds;
deletionSideChannelFilter takes authors + deletionKinds and returns only the
missing kinds. Tests updated for the new semantics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
Reorder amy sync so both sides fully reflect each other, including deletions:
1. Deletion side-channel now runs FIRST, before content, in both directions.
The content snapshot is taken AFTER it, closing a resurrection bug: a
deletion pulled down mid-sync removes a local event, but the old top-of-run
snapshot still listed it and would re-offer it up — resurrecting it on a
relay that also lacked the deletion.
2. Content reconcile, over the post-deletion snapshot.
3. Reject-reaction backstop: when the relay blocks a content push (usually it
holds a deletion we lack), pull that author's kind-5/62 and ingest locally
so we stop re-offering the dead event. Verify-by-fetch — only a real
deletion the store accepts has any effect; fires only on an actual reject.
The up-push of deletions is already verified per-event: ctx.publish awaits the
relay's OK, and ingesting the kind-5 runs the delete synchronously, so OK=true
confirms the remote applied it.
Mirror catch-up gets the same deletions-first ordering (down and up), so a
deletion lands, or the reject-trigger is armed, before its target — no
add-then-delete churn.
Tests: MirrorDeletionSyncTest gains scopedUpMirrorPushesDeletion (authoritative
push — local holds the deletion, remote drops the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
NIP-77 reconciles by event id over the content filter, so a scoped sync
(`--kind 1`) never carries the kind-5/62 that deletes one of those notes:
the deletion stays stuck on whichever side issued it while the target
lives on forever on the other. Add a deletion side-channel that reconciles
kinds 5 & 62 on their own, independent of the content filter.
quartz: NostrClientDeletionSyncExt — DELETION_PROPAGATION_KINDS,
Filter.excludesDeletionKinds()/deletionSideChannelFilter() (kinds 5/62 scoped
to the same authors, no time window since a deletion's created_at is not its
target's), shouldPropagateDeletionUp() (kind-5 always; kind-62 only to a relay
it targets, honoring the vanish's declared relays), and
negentropyPropagateDeletions() — one bidirectional reconcile that streams
have→upload and need→download.
amy sync: run the side-channel bidirectionally regardless of --up/--down
whenever the filter excludes 5/62; emits deletions_{need,have,downloaded,
uploaded}; --no-sync-deletions opts out.
geode MirrorWorker: thread a per-upstream deletionScope through both catch-up
phases and both live subs (down + up), in the mirror's configured direction;
relax down containment to accept in-scope deletions, gate kind-62 pushes by
target relay, and carry the deletion filter on re-subscribe so a reconnect
never drops it.
Tests: DeletionSyncTest (up/down propagation + filter/vanish-gate units) and
MirrorDeletionSyncTest (a kind-scoped down mirror still removes the note).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JgL1WTV4Hkp2uuXcUHCHGt
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.
- move `com.vitorpamplona.geode.ImportExport` →
`com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
NostrSignerSync (real Schnorr signing) instead of geode fixtures.
No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):
geode import [--db …] [--no-verify] [FILE…] # files, or stdin when none
geode export [--db …] # NDJSON to stdout
Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.
This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.
Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.
Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Follow-up to the audit fixes so nothing describes the pre-fix behavior:
- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
the reload" comment above `val dbFile` — the sentinel-gated reuse it
described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
(publishing each onHaveIds batch) instead of materializing the full diff
via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
id-ASC tie-break is byte-exact vs the single-SQL path only when the store
indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two benchmark-integrity bugs that could invalidate sync numbers while a run
reports success:
- CorpusServerMain keyed its serve-existing decision only on port + row
count, so re-running a port with a different corpus/maxCount, or after a
load crashed mid-way, silently served a stale/partial DB. Gate reuse on a
completion sentinel keyed on corpus identity (path + byte length) and
maxCount, written only after a full load; on any mismatch the prior DB is
dropped and reloaded.
- CorpusDownloader treated CLOSED identically to EOSE, so a relay ending a
sub early (rate-limit/policy) after sending a partial page advanced the
cursor past the unsent tail — silent corpus loss. Treat CLOSED as a soft
failure (null → reconnect and retry the same cursor; the id dedup set
absorbs the re-fetch), distinct from EOSE which means the page is
complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
runCatchUpUp used negentropyReconcileIds, which builds the FULL need+have
id lists in memory even though the up direction only needs haveIds — on a
large window (e.g. 1M local events against an empty upstream) that is a
~100 MB+ heap spike per convergence round, plus a needIds list built and
immediately discarded.
Switch to the streaming negentropyReconcile: publish each haveIds batch as
it arrives (bounded to one batch of ids) and drop the need direction via a
no-op onNeedIds. The publish still suspends the reconcile round, so the
back-pressure and the reconcile-as-delivery-check convergence loop are
unchanged — only the peak memory drops from O(window) to O(batch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Reuse the file-backed source DB when it already holds events instead of
deleting + reloading the corpus every boot, so re-running a single sync pair
skips the multi-minute load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
- MirrorSyncThroughputTest: add -DsyncVerify (default true — `strfry sync` always
verifies received events, so the negentropy sink verifies too for an
apples-to-apples comparison) and -DsyncFts (default true; pass false to match
strfry, which has no NIP-50). Both forwarded through the geode test task.
- CorpusServerMain: a benchmark-only tool that boots a real geode relay
(geode's default indexing) preloaded with an NDJSON corpus over a file-backed
store and serves forever, so `strfry sync` / another geode / the negentropy
sink can reconcile against a geode source holding the same 1M corpus a strfry
source does.
Used to run the 4-pair 1M negentropy sync comparison (geode↔geode, strfry→geode,
strfry↔strfry, geode→strfry).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
strfry's `sync --dir` is bidirectional (its source: doUp = both||up,
doDown = both||down), but geode's catch-up was down-only. Add the up half so
`dir=up`/`dir=both` reconcile-and-push matches `strfry sync --dir both`.
runCatchUpUp reconciles the local set against the upstream (negentropyReconcileIds)
and publishes the events we hold that the upstream lacks (the reconcile's `have`
ids). Symmetric to the down catch-up: same one `dir`, live up-session starts at
`now` when the up catch-up covers history.
Reliability: client.publish's outbox is best-effort under a bulk burst (each
publish also churns a reconnect — measured ~1-2% dropped per pass), so the push
runs as a reconcile→push convergence loop. Each round re-reconciles — the
reconcile IS the delivery check against the upstream — and re-pushes only the
stragglers until the have-diff is empty. Test observed 3000 → 69 → 2 → 0 across
3 rounds, lossless.
Test: MirrorNegentropyCatchUpTest.negentropyCatchUpPushesUp pushes 3000 local
events to an empty (no-verify) sink and asserts all 3000 land. The four existing
mirror tests still pass (up catch-up needs a store + negentropyBackfill, both off
by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).
MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:
- Catch-up reconciles the local set against the upstream over the
[now - backfill_seconds, now] window and downloads only the diff via the
existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
overrun us) and it completes the pull. Reconcile-against-local means a warm
restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
the sync's job). The windows overlap at `now`; the store's unique-id
constraint dedups the seam.
Changes:
- quartz: add a backward-compatible `localEntries` param to the public
negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
trusted-scope re-check as the live path; failure is non-fatal). New `store` +
`negentropyBackfill` ctor params; default off so existing live-REQ tests are
unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
preloading historical events a live-only sub cannot deliver, then proves the
post-boot event still arrives (3000 catch-up + 1 live = 3001).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a NIP-77 negentropy client path to MirrorSyncThroughputTest (default for
external sources; `-DsyncMode=req` keeps the paged-REQ drain). geode reconciles
its empty set against strfry, then client-paced fetch-by-ids + ingest.
Result on the 1M damus.io corpus (strfry source):
- reconcile (empty local → 995,024 need-ids): 11.4 s, 64 rounds.
- fetch + ingest: ~7,000 ev/s steady-state.
- overall: 994,936 / 997,980 in 171.0 s => 5,818 ev/s — COMPLETES.
This is the apples-to-apples counterpart to strfry→strfry's `strfry sync`
(both negentropy, same corpus, empty→full): strfry→geode 5,818 ev/s vs
strfry→strfry ~2,550 ev/s — geode ingests real content ~2.3x faster and
finishes the pull. The paged-REQ path, by contrast, stalls at ~310k every time
(strfry kills a slow REQ client at its 32 MB maxPendingOutboundBytes cap),
confirming that cross-relay bulk sync from strfry requires negentropy, not REQ.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds a strfry→geode drain mode to MirrorSyncThroughputTest and documents the
three-way 1M-corpus sync throughput comparison.
Findings (relayBench/plans/2026-07-04-sync-throughput-1m.md):
- strfry→strfry (strfry sync / negentropy): ~2,550 ev/s, completes.
- geode→geode (MirrorWorker REQ): 13,161 ev/s, completes — but inflated by
synthetic 4-byte content; real-content ingest is ~7,000 ev/s.
- strfry→geode (paged REQ drain, real corpus): sustains ~7,000 ev/s but does
NOT finish — strfry hard-kills a slower REQ client at its 32 MB
maxPendingOutboundBytes cap (~310k events in; confirmed by the source log's
"Pending: 32.01M" disconnects). Independent of heap (2G/10G) and the live
negentropy index (on/off).
Conclusion: strfry's REQ serving is structurally hostile to any client slower
than its scan (buffers outbound, then kills or OOMs). Only NIP-77 negentropy —
pull/reconcile-based and client-paced — completes a cross-relay bulk pull, which
is why strfry's own sync uses it. A production geode backfill from a foreign
relay should use negentropy, not the live-tail REQ path.
Also documents MirrorWorker's unbounded intake channel (a deliberate live-tail
trade) OOMing under a 1M bulk backfill, and forwards the sync* system
properties through the geode test task.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Adds -DsyncLiveIndex and a fast (no-live-index) source preload. Rate curves show
geode's mirror sustains ~20k ev/s with the live index off, but collapses to
O(n^2) with it on: the LiveNegentropyIndex is a sorted ArrayList whose per-event
insert is O(n), and a backfill delivers historical (non-near-tail) events, so
every insert memmoves ~n/2 entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Default EventStore(null) tokenizes FTS synchronously on every insert, which
dominates ingest and misrepresents mirror sync throughput. Use geode's actual
RelayIndexingStrategy (deferred FTS, live negentropy index) for both source and
sink, and log instantaneous events/s every 3s so the rate is visible during the
run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Two pieces measuring how fast each sink pulls a large corpus from a source using
its native sync client:
- relayBench/sync-throughput-strfry.sh: `strfry import` N events into a source
strfry, boot it, `strfry sync --dir down` an empty sink, report events/s.
- MirrorSyncThroughputTest: geode downstream pulls via the real MirrorWorker
(WebSocket). Default in-process geode source (geode→geode); with
-DsyncSourceUrl it mirrors an external relay (e.g. strfry) for strfry→geode.
Sized by -DsyncN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Streams 50k events from an upstream KtorRelay to a downstream via the production
MirrorWorker (real OkHttp WebSocket + trusted skipVerify ingest) and asserts the
downstream receives every one — 50000/50000, 0 missing.
This closes the last untested layer: the in-process guards (BatchInsertLossTest,
ConcurrentIngestLossTest) call IngestQueue.submit directly, bypassing the wire.
With this, geode is proven lossless end-to-end — store, concurrent pipeline,
background pool contention, RelaySession, and the real WS mirror path. The
~0.05% shortfall seen in the geode↔geode relayBench sync is therefore an
artifact of the harness's hand-rolled fetchByIds+publish delta transfer, not a
geode event-loss bug.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
The 1M relayBench run had geode losing the negentropy phase to strfry
(initial reconcile 6066ms/27r vs 1270ms/14r; identical-set 1947ms vs
557ms). Three layered benchmarks pin where the time actually goes:
- NegentropyReconcileBenchmark (quartz): the kmp-negentropy server loop
in isolation is ~200ms for the full 14-round exchange — the reconcile
ALGORITHM is not the bottleneck. (An early version showed 22s/139r;
that was a benchmark bug — index slices over randomly-sorted ids
scatter the diff. Real relayBench slices are contiguous time ranges;
monotonic created_at fixes it and matches strfry's round count.)
- NegentropyServerReconcileBenchmark (geode): the real in-process geode
server over loopback is 3214ms — 15x the library loop. JFR of the
server call-trees: ~40% hex/UTF-8/JSON serialization of the payloads,
~26% actual reconcile, rest allocation. The gap is the JVM
constant-factor tax on hex-in-JSON, which strfry pays in C++, not a
single hotspot.
- NegentropyPrefixFingerprintTest (quartz): the one algorithmic lever.
Negentropy's fingerprint is an additive sum mod 2^256, so a prefix-sum
table answers any range in O(1). Proven bit-for-bit identical to the
library over 2000 random ranges, and 460x faster per call — the fix
for the ~26% reconcile slice (dominant in the identical-set case).
Not yet wired: the library instantiates FingerprintCalculator
internally, so shipping prefix-sum needs a kmp-negentropy change (or a
quartz-side fast server). Full write-up + artifacts in
quartz/plans/2026-07-04-negentropy-reconcile-profiling.md.
Benchmarks are CI-safe (small defaults / opt-in gates); JFR via
-PnegProfile, scale via -DnegBenchN, geode server bench via
-DnegServerBench=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
Fail-loud on config that would silently degrade the running relay:
- readers = 0 makes every query hang forever on an empty reader pool;
readers < 0 crashes with an unrelated message. optimize_interval_seconds
<= 0 busy-loops PRAGMA optimize under the writer mutex. StaticConfig
.validate() (called at boot) rejects both.
- A typo in [[mirror]].filter (e.g. `kindss`) parsed to an empty
match-everything filter through the tolerant deserializer — silently
widening a trusted upstream's skip-verify scope to the whole firehose.
MirrorFilterValidator strict-checks the filter JSON at boot: unknown
keys and non-array list fields fail startup.
- The self-mirror guard now compares scheme-insensitively (ws:// vs
wss:// for the same host is still us) via displayUrl().
- The maintenance loop rethrows CancellationException and no longer
swallows Errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Two mirror hardening fixes from the audit:
- Trust leak: the skip-verify decision was keyed on the subscription id,
but the client pool dispatches EVENTs by subscription id alone and
every [[mirror]] upstream shares one client. A hostile untrusted
upstream could answer with the trusted upstream's subscription id and
ride its skip-verify into the store. startDown now drops any event
whose delivering relay isn't the one that subscription dialed
(relay != up.url). New MirrorWorkerTrustOriginTest injects a foreign
subId frame from a hostile relay and asserts it never lands.
- Reconnect replay: `since` was frozen at boot, so a long-lived daemon
re-streamed the whole backfill window on every upstream flap. The down
path now tracks the newest ingested created_at and, on the
connected->disconnected edge, advances the REQ's since to
watermark - overlap before the reconnect re-sends it. Advancing only on
disconnect means a stable link never re-queries. The reconnect test now
asserts the watermark advanced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Audit follow-ups on the live NIP-77 index, all with the store/scan
equivalence test extended to cover them:
- Same-batch replaceable displacement left a dead id in the index.
applyAfterCommit applied all removes before all adds, so when a later
row in one transaction displaced an earlier row of the same batch (two
versions of one replaceable — the mirror-backfill hot path), the
displaced row's remove no-op'd against an index that hadn't taken its
add yet, then the add re-inserted it: the index advertised an id the
trigger had already deleted. recordAccepted now cancels the pending add
instead of queueing a remove (added is a LinkedHashSet for O(1)
cancel).
- A kind-5 that deleted nothing (a delete broadcast for events this relay
never stored — the common case) still invalidated the whole index,
forcing a full-scan rebuild under the writer mutex on the next
NEG-OPEN. DeletionRequestModule.insert now returns the rows it deleted;
recordAccepted only invalidates when that count is > 0, else records the
kind-5 as a plain row.
- The first NEG-OPEN over a corpus larger than the serve cap scanned the
whole table uncapped, built a full index that could never produce a
snapshot, and then maintained it forever for zero benefit.
liveNegentropySnapshot now caps the rebuild scan at maxEntries + 1 and
leaves the index unpopulated when the corpus is over-cap (the scan path
answers NEG-ERR, as before).
- delete/deleteExpired/clearDB invalidate() moved inside the writer mutex
so no NEG-OPEN can seal a snapshot of just-deleted rows, and no
concurrent rebuild can be discarded by a late invalidate.
Also corrects the ~40 B/event heap figure to ~140 B (IdAndTime keeps the
id as a 64-char hex string, not 32 bytes) in the strategy/plan docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Each [[mirror]] entry now takes strfry-router's dir:
- down (default): pull — subscribe to the upstream and ingest, exactly
as before.
- up: push — an in-process session on the LOCAL relay subscribes with
the same scoped filter (so backfill_seconds and filter behave
identically in both directions, and the relay's own policy chain
gates what leaves), and every matching event is handed to the
client's outbox for the upstream, which owns delivery and re-sends
across reconnects.
- both: pull and push, with echo suppression: a per-upstream LRU of
recently exchanged ids keeps an event pulled down from being pushed
straight back (and vice versa when the upstream fans our own publish
back). Eviction only costs a duplicate round trip — the stores'
unique-id constraints stay the correctness backstop.
trusted (verify skip) remains a down-only concept; the up direction
never verifies since the upstream does its own gatekeeping.
Tests: up pushes both the backfill window and the live tail; both
converges two stores with disjoint content and holds exact counts after
the echo settles (no ping-pong). Live-published test events are
genuinely signed — the local publish path verifies, which is also what
the debugging showed: the mechanism was fine, the first version of the
tests was pushing forged events into a verifying relay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
Backlog items 4-5 close. Both-variants-in-one-run A/B at 50k, repeated
with relay order reversed: every delta flipped with the order (the
second-running relay won queries and ingest latency in BOTH runs), so
readers=8 / mmap_size=256MiB / temp_store=MEMORY / periodic PRAGMA
optimize are all noise-level on container-class hardware. The config
plumbing stays (hardware-dependent, operators should measure their own
box); the example config now says so explicitly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtDNpayEYvJH7QuPswND3A
mstrofnoneandVitor Pamplona
Vitor PamplonaandGitHub
vitorpamplonaandgithub-actions[bot]
davotoula