Merge pull request #3498 from vitorpamplona/claude/negentropy-sync-deletions-t2a7sf

NIP-77 deletion sync: two-pass settle over the reconcile residual
This commit is contained in:
Vitor Pamplona
2026-07-08 12:05:07 -04:00
committed by GitHub
11 changed files with 988 additions and 16 deletions
+11
View File
@@ -160,6 +160,17 @@ Summarize the survey in your plan: for each component, note whether it's
reused as-is, extracted from `amethyst/` to `commons/`, genuinely new
(platform-specific only), or a duplicate of an existing pattern to avoid.
**Relay client ops already exist — don't hand-roll subscribe/REQ/publish loops.**
One-shot and high-level relay operations (fetch a set, fetch one, page past the
relay cap, publish-and-confirm, NIP-45 count, NIP-77 sync/reconcile) are
`INostrClient` **extension functions** in
`quartz/…/nip01Core/relay/client/accessories/` (+ `…/reqs/` for the flow/subscribe
helpers). Because they're extensions, they don't surface under "usages of
`NostrClient`" or in completion — grep that package (or read its `README.md`, which
catalogs them) before writing a new subscription/collect loop. Reuse `fetchAll`,
`fetchFirst`, `fetchAllPages`, `publishAndConfirm`, `count`, `negentropyReconcile`,
etc. instead of re-implementing them.
**Share vs keep platform-native:**
- **Share** → `quartz/commonMain/` (business logic, data models, protocol) and
+6
View File
@@ -123,6 +123,12 @@ Each subscription tracks "End of Stored Events" per relay. The eose manager in `
## Related
- **Headless / one-shot client ops** (CLI, geode, tests, non-compose code): don't go
through `Subscribable` — use the `INostrClient` extension functions in
`quartz/…/nip01Core/relay/client/accessories/` (`fetchAll`, `fetchFirst`,
`fetchAllPages`, `publishAndConfirm`, `count`, `negentropyReconcile`/`negentropySync`,
…). They're extensions, so they don't show up under "usages of `NostrClient`" — see
that package's `README.md` for the catalog before writing a raw subscribe/collect loop.
- `nostr-expert/references/tag-patterns.md` — how tags inform what a filter needs to look for.
- `kotlin-coroutines/references/relay-patterns.md` — relay pool internals (sibling layer beneath assemblers).
- `feed-patterns` skill — feeds compose several Subscribables (content + metadata + reactions).
@@ -26,8 +26,10 @@ import com.vitorpamplona.amethyst.cli.DataDir
import com.vitorpamplona.amethyst.cli.Output
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.DeletionSettleResult
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.NegentropySyncException
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropyReconcile
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySettleDeletions
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
@@ -54,15 +56,29 @@ import java.util.concurrent.atomic.AtomicInteger
* Pass both for a full bidirectional sync. The filter flags are the same as
* `fetch`/`subscribe`; an empty filter reconciles the whole store.
*
* Both directions are pipelined with the reconcile: need-id batches feed
* [DOWNLOAD_WORKERS] concurrent by-id REQ drains and have-ids feed a single
* uploader, so downloads and uploads overlap the remaining reconcile rounds
* instead of waiting for the full diff. Every downloaded event funnels
* through `Context.drain`'s verify-and-store path, unchanged.
* Deletion propagation (on by default; disable with `--no-sync-deletions`) is a
* **second pass over the residual**, not per-event work in the content pass — so it
* costs the same whether the database is tiny or huge. After the content settle, a
* re-reconcile's leftover diff is (barring races) exactly the events a deletion kept
* from converging:
*
* Thin assembly only: the windowing, streaming, and back-pressure live in
* quartz (`negentropyReconcile`); this file only routes ids to
* `Context.drain` / `Context.publish`.
* - a residual **need** (relay has it, we still lack it after `--down` tried to
* download) = we deleted it → publish OUR covering deletion up so the relay drops it;
* - a residual **have** (we have it, relay still lacks it after `--up` tried to upload)
* = the relay deleted it → pull the relay's covering kind-5 down and apply it locally.
*
* Coverage is any way a deletion reaches an event ([deletionsCovering]): a NIP-09 kind-5
* by id (`e`) or address (`a`, cutoff-checked), or a NIP-62 vanish targeting this relay
* (up direction only — a pulled vanish is not auto-applied, its blast radius being the
* whole account). The residual is small (only real deletion mismatches), so only it is
* fetched — never the whole need set. The loop repeats until a round resolves nothing.
* So `amy sync` (default `--down`) makes the relay honor your deletions; `--up` makes
* your store honor the relay's; `--up --down` converges both ways.
*
* Content is pipelined with the reconcile: need-id batches feed [DOWNLOAD_WORKERS]
* concurrent by-id REQ drains and have-ids feed a single uploader. Thin assembly only:
* the windowing, streaming, and back-pressure live in quartz (`negentropyReconcile`);
* this file only routes ids to `Context.drain` / `Context.publish`.
*/
object SyncCommand {
private const val ID_CHUNK = 500
@@ -78,6 +94,15 @@ object SyncCommand {
/** Overlapped `created_at`-window reconciles after an over-cap split. */
private const val RECONCILE_CONCURRENCY = 2
/**
* Cap on deletion-settle rounds. Each round resolves the residual it can and
* re-reconciles; a healthy sync converges in 12 (round N sends/applies, round
* N+1 confirms empty). The cap only bounds pathological non-convergence (e.g. a
* relay that refuses a deletion), which the "resolved nothing → stop" check
* normally catches first.
*/
private const val MAX_DELETION_ROUNDS = 4
suspend fun run(
dataDir: DataDir,
rest: Array<String>,
@@ -93,6 +118,7 @@ object SyncCommand {
// Default direction is download; --up adds upload.
val up = args.bool("up")
val down = args.bool("down") || !up
val syncDeletions = !args.bool("no-sync-deletions")
val filter = RawEventSupport.buildFilter(args)
Context.openOrAnonymous(dataDir).use { ctx ->
@@ -104,23 +130,22 @@ object SyncCommand {
val downloaded = AtomicInteger(0)
val uploaded = AtomicInteger(0)
// ── Pass 1: content settle — download needs, upload haves. No deletion
// logic, so a plain sync costs exactly what it always did.
val result =
try {
coroutineScope {
// needIds = relay has, we lack; haveIds = we have, relay lacks.
// Bounded so a slow download back-pressures the reconcile
// rounds instead of piling ids up in memory.
val needBatches = Channel<List<HexKey>>(DOWNLOAD_WORKERS * 2)
// Unbounded is fine here: have-ids reference events we already
// hold locally, so memory is bounded by the local set.
val haveBatches = Channel<List<HexKey>>(Channel.UNLIMITED)
val downloaders =
List(DOWNLOAD_WORKERS) {
launch {
for (batch in needBatches) {
val got = ctx.drain(mapOf(relay to listOf(Filter(ids = batch))), timeoutMs)
downloaded.addAndGet(got.size)
// drain verifies + stores; anything we deleted is
// rejected by our own tombstone and stays a "need".
downloaded.addAndGet(ctx.drain(mapOf(relay to listOf(Filter(ids = batch))), timeoutMs).size)
}
}
}
@@ -129,8 +154,7 @@ object SyncCommand {
for (batch in haveBatches) {
for (id in batch) {
val ev = localById[id] ?: continue
val ack = ctx.publish(ev, setOf(relay))
if (ack.values.any { it }) uploaded.incrementAndGet()
if (ctx.publish(ev, setOf(relay)).values.any { it }) uploaded.incrementAndGet()
}
}
}
@@ -160,6 +184,28 @@ object SyncCommand {
return Output.error("sync_error", e.message ?: "negentropy sync failed")
}
// ── Pass 2+: deletion settle. The reusable quartz accessory re-reconciles
// and resolves only the residual — send our deletions up for what we deleted
// (bounded by --down), apply the relay's kind-5 down for what it deleted
// (bounded by --up) — looping until stable. Cheap regardless of database size
// (see negentropySettleDeletions), and best-effort so it can't fail the sync.
val deletions =
if (syncDeletions) {
ctx.client.negentropySettleDeletions(
relay = relay,
filter = filter,
store = ctx.store,
sendUp = down,
applyDown = up,
batchSize = ID_CHUNK,
idleTimeoutMs = timeoutMs,
maxRounds = MAX_DELETION_ROUNDS,
reconcileConcurrency = RECONCILE_CONCURRENCY,
)
} else {
DeletionSettleResult(0, 0, 0)
}
Output.emit(
mapOf(
"relay" to relay.url,
@@ -169,6 +215,9 @@ object SyncCommand {
"have" to result.haveCount,
"downloaded" to downloaded.get(),
"uploaded" to uploaded.get(),
"deletions_sent_up" to deletions.sentUp,
"deletions_applied_down" to deletions.appliedDown,
"deletion_rounds" to deletions.rounds,
),
)
return 0
+1
View File
@@ -3,3 +3,4 @@ marmot/state-headless/
dm/state-dm-headless/
nests/state/
clink/state-clink-headless/
sync/state-sync-deletions/
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env bash
#
# sync-deletions-headless.sh — drives the real `amy` binary against a real
# `amy serve` relay to prove NIP-77 deletion propagation end-to-end.
#
# `amy sync` converges deletions in a second pass over the reconcile residual
# (see quartz `negentropySettleDeletions`). This exercises both directions plus
# the opt-out:
#
# T1 (up) — we deleted a note the relay still has → `amy sync` sends our
# kind-5 up and the relay drops the note. Verified by an ISOLATED
# third account whose store reads the relay only (no tombstone).
# T2 (off) — same setup with `--no-sync-deletions` → the relay keeps the note
# and nothing is sent.
# T3 (down) — the relay deleted a note we still hold → `amy sync --up` pulls the
# relay's kind-5 down and applies it locally (converges on re-sync).
#
# Each amy account gets its OWN $HOME so their file stores don't share (accounts
# under one $HOME share ~/.amy/shared/events-store). The relay (amy serve) keeps
# a separate store from any client store.
#
# Usage: ./sync-deletions-headless.sh [--port N] [--no-build]
set -uo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd -- "$SCRIPT_DIR/../../.." && pwd)"
TESTS_DIR="$(cd -- "$SCRIPT_DIR/.." && pwd)"
STATE_DIR="$SCRIPT_DIR/state-sync-deletions"
LOG_DIR="$STATE_DIR/logs"
RUN_TS="$(date +%Y%m%d-%H%M%S)"
LOG_FILE="$LOG_DIR/run-$RUN_TS.log"
RESULTS_FILE="$STATE_DIR/results-$RUN_TS.tsv"
AMY_BIN="$REPO_ROOT/cli/build/install/amy/bin/amy"
RELAY_HOST="127.0.0.1"
RELAY_PORT="${RELAY_PORT:-7790}"
RELAY_URL="ws://$RELAY_HOST:$RELAY_PORT"
NO_BUILD=0
while [[ $# -gt 0 ]]; do
case "$1" in
--port) RELAY_PORT="$2"; RELAY_URL="ws://$RELAY_HOST:$RELAY_PORT"; shift ;;
--no-build) NO_BUILD=1 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
shift
done
# Fresh state every run — stale per-account $HOME dirs from a prior run must not
# leak into this one.
rm -rf "$STATE_DIR"
mkdir -p "$LOG_DIR"
: >"$RESULTS_FILE"
# shellcheck source=../lib.sh
source "$TESTS_DIR/lib.sh"
# Leniently-trimmed equality assertion (assert helpers live in the DM-specific
# helpers.sh, which hardcodes its own amy wrappers — so define our own here).
assert_eq() {
local actual="$1" expected="$2" test_id="$3" note="${4:-}"
if [[ "${actual// /}" == "${expected// /}" ]]; then
info "assert: $test_id \"$actual\" == \"$expected\""
return 0
fi
fail_msg "$test_id: expected \"$expected\", got \"$actual\" (${note:-})"
record_result "$test_id" fail "${note:-mismatch}"
return 1
}
SERVE_PID=""
RELAY_HOME=""
cleanup() {
[[ -n "$SERVE_PID" ]] && kill "$SERVE_PID" 2>/dev/null
trap - EXIT INT TERM HUP
print_summary
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
banner "amy sync — NIP-77 deletion propagation headless ($RUN_TS)"
# ---- build ------------------------------------------------------------------
if [[ "$NO_BUILD" -eq 0 ]]; then
step "Building amy (installDist)…"
(cd "$REPO_ROOT" && ./gradlew -q :cli:installDist) >>"$LOG_FILE" 2>&1 \
|| { fail_msg "build failed (see $LOG_FILE)"; exit 1; }
fi
[[ -x "$AMY_BIN" ]] || { fail_msg "amy binary not found at $AMY_BIN"; exit 1; }
# ---- amy wrappers (one isolated $HOME per account) --------------------------
strip() { grep -vE "Picked up JAVA_TOOL|DEBUG:|INFO:|MarmotManager|MlsGroup"; }
mk_home() { mktemp -d "$STATE_DIR/home.XXXXXX"; }
# amy_run <home> <account> args...
amy_run() {
local home="$1" acct="$2"; shift 2
HOME="$home" "$AMY_BIN" --account "$acct" --secret-backend plaintext --json "$@" 2>>"$LOG_FILE" | strip
}
RELAY_HOME="$(mk_home)"
amy_run "$RELAY_HOME" a init >/dev/null
step "Starting amy serve on $RELAY_URL"
HOME="$RELAY_HOME" "$AMY_BIN" --account a --secret-backend plaintext \
serve --host "$RELAY_HOST" --port "$RELAY_PORT" >>"$LOG_FILE" 2>&1 &
SERVE_PID=$!
# Wait for the relay to accept connections (poll the serve log).
for _ in $(seq 1 60); do
grep -q "relay up at" "$LOG_FILE" && break
sleep 0.5
done
grep -q "relay up at" "$LOG_FILE" || { fail_msg "relay did not come up"; exit 1; }
# Isolated verifier: its own empty store, reads the relay only (no tombstone).
VERIFY_HOME="$(mk_home)"
amy_run "$VERIFY_HOME" v init >/dev/null
relay_count() { amy_run "$VERIFY_HOME" v fetch --id "$1" --relay "$RELAY_URL" | jq -r '.count // 0'; }
# =============================================================================
# T1 — up direction: we deleted it, the relay still has it → sync sends it up.
# =============================================================================
banner "T1 — amy sync sends our deletion up (relay drops the note)"
NOTE="$(amy_run "$RELAY_HOME" a event --kind 1 --content "delete-me-t1" | jq -c '.event')"
NID="$(echo "$NOTE" | jq -r '.id')"
echo "$NOTE" | amy_run "$RELAY_HOME" a publish --relay "$RELAY_URL" >/dev/null
before="$(relay_count "$NID")"
assert_eq "$before" "1" T1.setup "relay should hold the note before sync" \
&& record_result T1.setup pass "relay has the note"
# Delete locally only (no --relay → stored, applied, not sent to the relay).
amy_run "$RELAY_HOME" a event --kind 5 --tags "[[\"e\",\"$NID\"]]" --content "" --publish >/dev/null
SYNC="$(amy_run "$RELAY_HOME" a sync --relay "$RELAY_URL")"
info "sync: $SYNC"
sent="$(echo "$SYNC" | jq -r '.deletions_sent_up // 0')"
assert_eq "$sent" "1" T1.sent_up "sync should report one deletion sent up" \
&& record_result T1.sent_up pass "deletions_sent_up=1"
sleep 1
after="$(relay_count "$NID")"
assert_eq "$after" "0" T1.relay_dropped "relay must have removed the note after sync" \
&& record_result T1.relay_dropped pass "relay note count 1 → 0"
# =============================================================================
# T2 — opt-out: --no-sync-deletions leaves the relay untouched.
# =============================================================================
banner "T2 — --no-sync-deletions propagates nothing"
NOTE2="$(amy_run "$RELAY_HOME" a event --kind 1 --content "keep-me-t2" | jq -c '.event')"
NID2="$(echo "$NOTE2" | jq -r '.id')"
echo "$NOTE2" | amy_run "$RELAY_HOME" a publish --relay "$RELAY_URL" >/dev/null
amy_run "$RELAY_HOME" a event --kind 5 --tags "[[\"e\",\"$NID2\"]]" --content "" --publish >/dev/null
SYNC2="$(amy_run "$RELAY_HOME" a sync --relay "$RELAY_URL" --no-sync-deletions)"
info "sync: $SYNC2"
sent2="$(echo "$SYNC2" | jq -r '.deletions_sent_up // 0')"
assert_eq "$sent2" "0" T2.no_send "--no-sync-deletions must send nothing" \
&& record_result T2.no_send pass "deletions_sent_up=0"
sleep 1
kept="$(relay_count "$NID2")"
assert_eq "$kept" "1" T2.relay_kept "relay must still hold the note" \
&& record_result T2.relay_kept pass "relay note untouched"
# =============================================================================
# T3 — down direction: the relay deleted it, we still hold it → sync --up pulls
# the relay's deletion down and applies it locally.
# =============================================================================
banner "T3 — amy sync --up applies the relay's deletion locally"
BOB_HOME="$(mk_home)"
amy_run "$BOB_HOME" b init >/dev/null
NOTE3="$(amy_run "$RELAY_HOME" a event --kind 1 --content "delete-me-t3" | jq -c '.event')"
NID3="$(echo "$NOTE3" | jq -r '.id')"
echo "$NOTE3" | amy_run "$RELAY_HOME" a publish --relay "$RELAY_URL" >/dev/null
# bob's isolated store learns the note from the relay…
amy_run "$BOB_HOME" b fetch --id "$NID3" --relay "$RELAY_URL" >/dev/null
# …then the relay deletes it (author pushes a kind-5 straight to the relay).
amy_run "$RELAY_HOME" a event --kind 5 --tags "[[\"e\",\"$NID3\"]]" --content "" | jq -c '.event' \
| amy_run "$RELAY_HOME" a publish --relay "$RELAY_URL" >/dev/null
SYNC3="$(amy_run "$BOB_HOME" b sync --up --relay "$RELAY_URL")"
info "sync: $SYNC3"
applied="$(echo "$SYNC3" | jq -r '.deletions_applied_down // 0')"
assert_eq "$applied" "1" T3.applied_down "sync --up should apply one relay deletion locally" \
&& record_result T3.applied_down pass "deletions_applied_down=1"
# Converged: a second --up sync finds nothing left to apply.
SYNC3B="$(amy_run "$BOB_HOME" b sync --up --relay "$RELAY_URL")"
applied2="$(echo "$SYNC3B" | jq -r '.deletions_applied_down // 0')"
assert_eq "$applied2" "0" T3.converged "re-sync applies nothing (converged)" \
&& record_result T3.converged pass "second sync stable"
# print_summary runs from the cleanup trap; exit non-zero if any test failed.
grep -q $'\tfail\t' "$RESULTS_FILE" && exit 1
exit 0
+3
View File
@@ -72,6 +72,9 @@ tasks.withType<Test>().configureEach {
// NegentropyServerReconcileBenchmark opt-in + sizing.
System.getProperty("negServerBench")?.let { systemProperty("negServerBench", it) }
System.getProperty("negBenchN")?.let { systemProperty("negBenchN", it) }
// DeletionSettleBenchmark sizing.
System.getProperty("delBenchN")?.let { systemProperty("delBenchN", it) }
System.getProperty("delBenchK")?.let { systemProperty("delBenchK", it) }
// MirrorSyncThroughputTest sizing + external-source opt-in.
System.getProperty("syncN")?.let { systemProperty("syncN", it) }
System.getProperty("syncExpect")?.let { systemProperty("syncExpect", it) }
@@ -0,0 +1,128 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.geode
import com.vitorpamplona.geode.testing.RelayClientTest
import com.vitorpamplona.geode.testing.preload
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropyReconcileIds
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySettleDeletions
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import kotlin.test.AfterTest
import kotlin.test.Test
import kotlin.test.assertEquals
/**
* Cost of the deletion side-channel ([negentropySettleDeletions]) at database scale.
*
* The whole point of the two-pass design is that turning deletions on does NOT re-fetch
* content — the content sync already downloaded the need set, and the settle only touches
* the reconcile *residual* (the events a deletion stopped from converging). So its cost is
* one reconcile per round plus the residual, independent of how big the database is.
*
* This models the post-content-settle state: a relay holding N notes, and a local store
* holding the same N notes EXCEPT K it deleted (it keeps the K kind-5s). The residual is
* exactly those K — so a `sendUp` settle fetches K, not N. It prints the reconcile cost
* (the O(N) part it shares with any sync) next to the settle cost, so the deletion
* overhead is visible as "≈ a couple of reconciles + K", not "+ a content re-download".
*
* Why the printed settle can read as several× a bare reconcile at large N: the extra time
* is NOT the deletion algorithm (a phase breakdown showed reconciles stay ~sub-second at
* N=100k, and the settle re-fetches K=20, not N). It is entirely the K `publishAndConfirm`
* ingests into a large geode relay — publishing K *plain* notes costs the same — and that
* ingest path is JVM-cold on first use: consecutive K-note batches dropped monotonically
* (~3100 → ~570 ms) purely from JIT warmup. So the cost is O(K) relay-ingest dominated by
* one-time warmup, independent of N.
*
* Default N is small so it doubles as a fast correctness guard; scale it with
* `-DdelBenchN=200000` to see the shape at size. Not a speed assertion (container noise).
*/
class DeletionSettleBenchmark : RelayClientTest() {
private val signer = NostrSignerSync(KeyPair())
private val local = EventStore(null)
@AfterTest fun closeLocal() = local.close()
private val n = System.getProperty("delBenchN")?.toInt() ?: 2_000
private val k = System.getProperty("delBenchK")?.toInt() ?: 20
@Test
fun settleCostIsResidualNotDatabase() =
runBlocking {
val base = TimeUtils.now() - n
// N notes with monotonic created_at (sorted order == index order).
val notes = (0 until n).map { signer.sign(TextNoteEvent.build("n$it", createdAt = base + it.toLong())) }
// The last K are the ones we deleted locally.
val deleted = notes.takeLast(k)
val kept = notes.dropLast(k)
val deletions = deleted.map { signer.sign(DeletionEvent.build(listOf(it), createdAt = it.createdAt + 1)) }
// Relay holds all N notes; we hold the N-K we didn't delete, plus the K kind-5s.
defaultRelay.preload(notes)
kept.forEach { local.insert(it) }
deletions.forEach { local.insert(it) }
assertEquals(n - k, local.query<Event>(Filter(kinds = listOf(1))).size, "local kept N-K notes")
// Cost of one reconcile — the O(N) work every sync round already does.
val r0 = System.nanoTime()
val diff =
withTimeout(120_000) {
client.negentropyReconcileIds(defaultRelayUrl, Filter(kinds = listOf(1)), local.snapshotIdsForNegentropy(listOf(Filter(kinds = listOf(1)))))
}
val reconcileMs = (System.nanoTime() - r0) / 1e6
assertEquals(k, diff.needIds.size, "the residual is exactly the K deleted notes, not N")
// Cost of the whole settle: reconcile(s) + resolve the K-event residual.
val s0 = System.nanoTime()
val res =
withTimeout(120_000) {
client.negentropySettleDeletions(
relay = defaultRelayUrl,
filter = Filter(kinds = listOf(1)),
store = local,
sendUp = true,
applyDown = false,
idleTimeoutMs = 60_000,
)
}
val settleMs = (System.nanoTime() - s0) / 1e6
assertEquals(k, res.sentUp, "sent exactly K deletions up")
assertEquals(
n - k,
defaultRelay.store.query<Event>(Filter(kinds = listOf(1))).size,
"relay converged: the K deleted notes are gone",
)
println("─ DeletionSettleBenchmark @ N=$n K=$k")
println(" one reconcile: ${"%.1f".format(reconcileMs)} ms (O(N), shared with any sync)")
println(" full settle: ${"%.1f".format(settleMs)} ms (${res.rounds} rounds, sentUp=${res.sentUp})")
println(" deletion cost: settle is ~${"%.1f".format(settleMs / reconcileMs)}× one reconcile — fetched K=$k, not N=$n")
}
}
@@ -0,0 +1,267 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.geode
import com.vitorpamplona.geode.testing.RelayClientTest
import com.vitorpamplona.geode.testing.preload
import com.vitorpamplona.geode.testing.publish
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropyReconcileIds
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySettleDeletions
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
import com.vitorpamplona.quartz.nip01Core.store.deletionsCovering
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.nip02FollowList.ContactListEvent
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.nip62RequestToVanish.RequestToVanishEvent
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import kotlin.test.AfterTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* The `amy sync` deletion rule: for the events the relay HAS that we LACK (the
* negentropy need set), publish the local deletions that would make the relay remove
* them — and only those. [deletionsCovering] is the core: it maps a set of server-held
* events to the local deletions that cover them, across id-based (NIP-09 `e`),
* address-based (NIP-09 `a`, cutoff-checked) and NIP-62 vanish (relay-targeted, cutoff).
*/
class DeletionSyncTest : RelayClientTest() {
private val signer = NostrSignerSync(KeyPair())
private val here: NormalizedRelayUrl get() = defaultRelayUrl
private val elsewhere = RelayUrlNormalizer.normalize("wss://elsewhere.example/")
private val store = EventStore(null)
@AfterTest fun closeStore() = store.close()
private fun note(text: String): Event = signer.sign(TextNoteEvent.build(text))
// ---- deletionsCovering: the three coverage forms --------------------------
@Test
fun idBasedDeletionCoversByETag() =
runBlocking {
val target = note("delete me")
val deletion = signer.sign(DeletionEvent.build(listOf(target), createdAt = target.createdAt + 1))
store.insert(deletion)
assertEquals(listOf(deletion.id), store.deletionsCovering(listOf(target), here).map { it.id })
// A different note the deletion doesn't name is not covered.
assertTrue(store.deletionsCovering(listOf(note("unrelated")), here).isEmpty())
}
@Test
fun addressBasedDeletionCoversByATagWithCutoff() =
runBlocking {
val contacts = ContactListEvent.createFromScratch(emptyList(), null, signer)
// Address-only deletion (no `e` tag) → only the `a`-tag path can match it.
val delAddr = signer.sign(DeletionEvent.buildAddressOnly(listOf(contacts), createdAt = contacts.createdAt + 1))
store.insert(delAddr)
assertEquals(
listOf(delAddr.id),
store.deletionsCovering(listOf(contacts), here).map { it.id },
"a replaceable event is covered by an address deletion at/after it",
)
// NIP-09 cutoff: a deletion OLDER than the event does not delete it.
val stale = EventStore(null)
stale.insert(signer.sign(DeletionEvent.buildAddressOnly(listOf(contacts), createdAt = contacts.createdAt - 1)))
assertTrue(stale.deletionsCovering(listOf(contacts), here).isEmpty(), "an older address deletion does not cover")
stale.close()
}
@Test
fun vanishCoversAuthorsEventsWhenTargetedAndNewer() =
runBlocking {
val old = note("before the vanish")
val vanishHere = signer.sign(RequestToVanishEvent.build(here, createdAt = old.createdAt + 1))
store.insert(vanishHere)
assertEquals(
listOf(vanishHere.id),
store.deletionsCovering(listOf(old), here).map { it.id },
"a relay-targeted vanish issued after the event covers it",
)
// Not targeting this relay → not sent here.
val otherStore = EventStore(null)
otherStore.insert(signer.sign(RequestToVanishEvent.build(elsewhere, createdAt = old.createdAt + 1)))
assertTrue(otherStore.deletionsCovering(listOf(old), here).isEmpty(), "a vanish for another relay is not sent")
// A newer event (created after the vanish) is NOT deleted by it.
val newer = signer.sign(TextNoteEvent.build("after", createdAt = vanishHere.createdAt + 10))
assertTrue(store.deletionsCovering(listOf(newer), here).isEmpty(), "the vanish does not cover a later event")
otherStore.close()
}
// ---- end-to-end through the relay ----------------------------------------
// UP direction: we deleted it, the relay still has it → send our deletion up.
@Test
fun sendsCoveringDeletionSoRelayRemovesTheNote() =
runBlocking {
val target = note("delete me e2e")
val deletion = signer.sign(DeletionEvent.build(listOf(target), createdAt = target.createdAt + 1))
// Relay holds the note; we already deleted it locally (hold only the kind-5).
defaultRelay.preload(listOf(target))
val local = hub.getOrCreate(RelayUrlNormalizer.normalize("ws://local/"))
local.preload(listOf(target, deletion))
assertTrue(local.store.query<Event>(Filter(ids = listOf(target.id))).isEmpty(), "local deleted the note")
// Reconcile → the note is a need id. (No local kind-1 remains.)
val diff =
withTimeout(20_000) {
client.negentropyReconcileIds(relay = defaultRelayUrl, filter = Filter(kinds = listOf(1)), localEntries = emptyList<IdAndTime>())
}
assertEquals(setOf(target.id), diff.needIds.toSet())
// What SyncCommand does: fetch the need events, ask the local store which of
// our deletions cover them, publish those.
val serverEvents = defaultRelay.store.query<Event>(Filter(ids = diff.needIds))
val covering = local.store.deletionsCovering(serverEvents, defaultRelayUrl)
assertEquals(listOf(deletion.id), covering.map { it.id })
covering.forEach { defaultRelay.publish(it) }
assertTrue(
defaultRelay.store.query<Event>(Filter(ids = listOf(target.id))).isEmpty(),
"relay applied the pushed deletion and removed the note",
)
}
// DOWN direction: the relay deleted it, we still have it → pull the relay's deletion
// down and apply it locally (the residual-have resolution).
@Test
fun appliesRelaysDeletionSoLocalRemovesTheNote() =
runBlocking {
val target = note("delete me down")
val deletion = signer.sign(DeletionEvent.build(listOf(target), createdAt = target.createdAt + 1))
// Relay already applied the deletion → holds only the kind-5.
defaultRelay.preload(listOf(target, deletion))
assertTrue(defaultRelay.store.query<Event>(Filter(ids = listOf(target.id))).isEmpty(), "relay deleted the note")
// Local still holds the note (never saw the deletion).
val local = hub.getOrCreate(RelayUrlNormalizer.normalize("ws://local-down/"))
local.preload(listOf(target))
assertEquals(1, local.store.query<Event>(Filter(ids = listOf(target.id))).size)
// Reconcile → the note is a HAVE (we have it, the relay lacks it).
val diff =
withTimeout(20_000) {
client.negentropyReconcileIds(
relay = defaultRelayUrl,
filter = Filter(kinds = listOf(1)),
localEntries = listOf(IdAndTime(target.createdAt, target.id)),
)
}
assertEquals(setOf(target.id), diff.haveIds.toSet())
// What SyncCommand does for the down direction: take our have events, ask the
// RELAY which of ITS deletions cover them, and apply those locally.
val ourEvents = local.store.query<Event>(Filter(ids = diff.haveIds))
val relayDeletions = deletionsCovering(ourEvents, defaultRelayUrl) { f -> defaultRelay.store.query<Event>(f) }
assertEquals(listOf(deletion.id), relayDeletions.map { it.id })
relayDeletions.filterIsInstance<DeletionEvent>().forEach { local.store.insert(it) }
assertTrue(
local.store.query<Event>(Filter(ids = listOf(target.id))).isEmpty(),
"local applied the pulled deletion and removed the note",
)
}
// ---- the full accessory loop (negentropySettleDeletions) -----------------
// sendUp: local holds the deletion, relay still has the note → the loop pushes it
// up and the relay converges to gone.
@Test
fun settleSendsOurDeletionUp() =
runBlocking {
val target = note("settle up")
val deletion = signer.sign(DeletionEvent.build(listOf(target), createdAt = target.createdAt + 1))
val localStore = EventStore(null)
localStore.insert(target)
localStore.insert(deletion) // deletes target locally, keeps the kind-5
defaultRelay.preload(listOf(target))
val res =
withTimeout(30_000) {
client.negentropySettleDeletions(
relay = defaultRelayUrl,
filter = Filter(kinds = listOf(1)),
store = localStore,
sendUp = true,
applyDown = false,
idleTimeoutMs = 20_000,
)
}
assertEquals(1, res.sentUp)
assertEquals(0, res.appliedDown)
assertTrue(
defaultRelay.store.query<Event>(Filter(ids = listOf(target.id))).isEmpty(),
"relay converged: the deleted note is gone",
)
localStore.close()
}
// applyDown: relay deleted the note (holds only the kind-5), local still has it →
// the loop pulls the relay's deletion down and local converges to gone.
@Test
fun settleAppliesRelayDeletionDown() =
runBlocking {
val target = note("settle down")
val deletion = signer.sign(DeletionEvent.build(listOf(target), createdAt = target.createdAt + 1))
defaultRelay.preload(listOf(target, deletion)) // relay deletes target, keeps the kind-5
val localStore = EventStore(null)
localStore.insert(target)
val res =
withTimeout(30_000) {
client.negentropySettleDeletions(
relay = defaultRelayUrl,
filter = Filter(kinds = listOf(1)),
store = localStore,
sendUp = false,
applyDown = true,
idleTimeoutMs = 20_000,
)
}
assertEquals(0, res.sentUp)
assertEquals(1, res.appliedDown)
assertTrue(
localStore.query<Event>(Filter(ids = listOf(target.id))).isEmpty(),
"local converged: the relay-deleted note is gone",
)
localStore.close()
}
}
@@ -0,0 +1,145 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.accessories
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.crypto.verify
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
import com.vitorpamplona.quartz.nip01Core.store.deletionsCovering
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
/**
* Outcome of a [negentropySettleDeletions] run.
*
* @property sentUp distinct local deletions published to the relay (up direction).
* @property appliedDown distinct relay deletions ingested into [store] (down direction).
* @property rounds reconcile rounds run before convergence (or the cap).
*/
class DeletionSettleResult(
val sentUp: Int,
val appliedDown: Int,
val rounds: Int,
)
/**
* Converge deletions between [store] and [relay] AFTER a content sync has settled the
* two sides — the second half of a two-pass sync. NIP-77 reconciles by id, so a plain
* content sync converges everything except events a deletion physically stops from
* moving; those survive as the reconcile's residual, which this resolves:
*
* - **[sendUp]** — a residual **need** (relay has it, [store] still lacks it after the
* content pass tried to download it) means we deleted it. Publish OUR covering
* deletion up ([IEventStore.deletionsCovering]) so the relay drops it.
* - **[applyDown]** — a residual **have** ([store] has it, relay still lacks it after
* the content pass tried to upload it) means the relay deleted it. Pull the RELAY'S
* covering **kind-5** down and ingest it, so [store] drops it too. A NIP-62 vanish is
* deliberately NOT applied on pull — its blast radius is the author's whole account.
*
* Because it works off the residual — not every id — the cost is one cheap reconcile
* per round plus the (small) residual, independent of database size. It loops until a
* round resolves nothing (converged, and thereby self-verified) or [maxRounds] is hit.
*
* **Direction requires the matching content pass.** A residual need is a clean signal
* only after the content sync attempted the download ([sendUp] pairs with a `--down`
* content pass); a residual have only after it attempted the upload ([applyDown] pairs
* with `--up`). Passing a direction whose content pass didn't run makes its residual the
* full unsettled set, not a deletion signal — so drive this with the same directions the
* content pass used.
*
* Best-effort: a reconcile failure ([NegentropySyncException]) stops the loop and returns
* what already settled rather than throwing — the content sync is the primary work.
*
* @param batchSize ids per reconcile chunk and per by-id fetch.
* @param idleTimeoutMs idle watchdog for the reconciles and fetches.
* @param maxRounds hard cap on rounds; the "resolved nothing" check usually stops first.
* @param reconcileConcurrency overlapped `created_at`-window reconciles after an over-cap split.
*/
suspend fun INostrClient.negentropySettleDeletions(
relay: NormalizedRelayUrl,
filter: Filter,
store: IEventStore,
sendUp: Boolean,
applyDown: Boolean,
batchSize: Int = 500,
idleTimeoutMs: Long = 120_000L,
maxRounds: Int = 4,
reconcileConcurrency: Int = 1,
): DeletionSettleResult {
if ((!sendUp && !applyDown) || maxRounds <= 0) return DeletionSettleResult(0, 0, 0)
val publishTimeoutSecs = (idleTimeoutMs / 1000).coerceAtLeast(1)
val sentUp = HashSet<HexKey>()
val appliedDown = HashSet<HexKey>()
var rounds = 0
while (rounds < maxRounds) {
rounds++
val diff =
try {
negentropyReconcileIds(
relay = relay,
filter = filter,
localEntries = store.snapshotIdsForNegentropy(listOf(filter)),
batchSize = batchSize,
idleTimeoutMs = idleTimeoutMs,
reconcileConcurrency = reconcileConcurrency,
)
} catch (e: NegentropySyncException) {
break
}
var resolved = 0
// residual needs → publish our covering deletions up.
if (sendUp) {
for (chunk in diff.needIds.chunked(batchSize)) {
val events = fetchAll(relay, Filter(ids = chunk), idleTimeoutMs)
for (del in store.deletionsCovering(events, relay)) {
if (sentUp.add(del.id)) {
if (publishAndConfirm(del, setOf(relay), publishTimeoutSecs)) resolved++
}
}
}
}
// residual haves → ingest the relay's covering kind-5 (never a vanish).
if (applyDown) {
for (chunk in diff.haveIds.chunked(batchSize)) {
val ours = store.query<Event>(Filter(ids = chunk))
val relayDeletions = deletionsCovering(ours, relay) { f -> fetchAll(relay, f, idleTimeoutMs) }
for (del in relayDeletions.filterIsInstance<DeletionEvent>()) {
if (del.verify() && appliedDown.add(del.id)) {
store.insert(del)
resolved++
}
}
}
}
if (resolved == 0) break
}
return DeletionSettleResult(sentUp.size, appliedDown.size, rounds)
}
@@ -0,0 +1,62 @@
# `INostrClient` accessories
One-shot / high-level relay operations, written as **extension functions** on
`INostrClient`. They live here (and in `../reqs/`) rather than on the client class,
so they don't show up under "usages of `NostrClient`" or in method completion — you
only find them by knowing this package exists.
**Before writing a new subscribe / REQ / publish loop, look here first.** Most of what
a caller needs (fetch a set, fetch one, page past the relay cap, publish-and-confirm,
count, negentropy sync/reconcile) already exists.
Import as `com.vitorpamplona.quartz.nip01Core.relay.client.accessories.<name>` (or
`...client.reqs.<name>` for the flow/subscribe helpers).
## One-shot reads (subscribe → collect → return)
| Function | File | Use when |
| --- | --- | --- |
| `fetchAll(relay, filter, timeoutMs)` | `NostrClientFetchAllExt` | Get every event matching a filter in one REQ, deduped by id, until EOSE or timeout. **No verify, no store** — just the events. |
| `fetchFirst(relay, filter, timeoutMs)` | `NostrClientFetchFirstExt` | Get the first matching event and stop (returns `null` on none/timeout). |
| `fetchAllPages(relay, filters, timeoutMs)` | `NostrClientFetchAllPagesExt` | Fully retrieve a result set larger than the relay's per-REQ cap (strfry `limit`, ~500) by walking a `created_at` cursor. Bound it with the filter's `limit`. |
| `fetchAllPagesFromPool(filters, ...)` | `NostrClientFetchAllPagesPoolExt` | Same paging, across several relays at once, deduped across them. |
## Streaming (`Flow`)
| Function | File | Use when |
| --- | --- | --- |
| `fetchAsFlow(relay, filter)` | `../reqs/NostrClientFetchAsFlowExt` | Emit the accumulating list on each arrival; completes on EOSE. One-shot query as a flow. |
| `subscribeAsFlow(relay, filter)` | `../reqs/NostrClientSubscribeAsFlowExt` | Live subscription as a flow (stays open past EOSE; re-sends the REQ on reconnect). |
| `subscribe(subId, filters, listener)` | `../reqs/StaticSubscription`, `DynamicSubscription` | Raw live subscription with a `SubscriptionListener`. The lowest-level primitive the above build on. |
## Publish
| Function | File | Use when |
| --- | --- | --- |
| `publishAndConfirm(event, relays, timeout)` | `NostrClientPublishExt` | Send an EVENT and wait for `OK`; returns whether any relay accepted it. |
| `publishAndConfirmDetailed(event, relays, timeout)` | `NostrClientPublishExt` | Same, but returns the per-relay accepted/rejected map. |
## Count (NIP-45)
| Function | File | Use when |
| --- | --- | --- |
| `count(relay, filter, timeoutMs)` | `NostrClientCountExt` | NIP-45 `COUNT` against one relay (`null` on timeout / no support). |
| `countMerged(relays, filter, ...)` | `NostrClientCountExt` | Merged count across relays. |
## Negentropy (NIP-77)
| Function | File | Use when |
| --- | --- | --- |
| `negentropySync(relay, filter, ...)` | `NostrClientNegentropySyncExt` | Download everything a relay holds for a filter, diffing against `localEntries` and by-id downloading only the diff. Throws `NegentropySyncException` if the relay can't reconcile (no fallback). |
| `negentropySyncOrFetch(relay, filter, ...)` | `NostrClientNegentropySyncExt` | Same, but transparently falls back to `fetchAllPages` when the relay can't reconcile. The "just get the events" combinator. |
| `negentropySyncEvents` / `negentropySyncOrFetchEvents` | `NostrClientNegentropySyncEventsExt` | The two above as an O(1)-memory `Flow<Event>`. |
| `negentropyReconcile(relay, filter, localEntries, onNeedIds, onHaveIds)` | `NostrClientNegentropySyncExt` | **Pure diff, no I/O** — streams the two directions (`need` = relay has & we lack; `have` = we have & relay lacks) to callbacks. Compose your own download/upload on top. |
| `negentropyReconcileIds(relay, filter, localEntries)` | `NostrClientNegentropySyncExt` | Same diff, materialized into `needIds` / `haveIds` lists (small sets only). |
| `negentropySettleDeletions(relay, filter, store, sendUp, applyDown)` | `NostrClientNegentropyDeletionSettleExt` | Second pass of a two-pass sync: after a content sync settles, re-reconcile and resolve only the residual — send our covering deletions up (`sendUp`) and/or apply the relay's kind-5 down (`applyDown`), looping until stable. Cost is O(residual), not O(db). Pairs with `IEventStore.deletionsCovering`. |
`fetchByIds`, `reconcileStreaming`, `syncPipeline` in `NostrClientNegentropySyncExt`
are `internal` implementation details — not part of the public surface.
---
_Keep this table in sync when you add a public `INostrClient` extension here._
@@ -0,0 +1,104 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store
import com.vitorpamplona.quartz.nip01Core.core.Address
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.core.isAddressable
import com.vitorpamplona.quartz.nip01Core.core.isReplaceable
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
import com.vitorpamplona.quartz.nip62RequestToVanish.RequestToVanishEvent
/** The addressable/replaceable coordinate of [event] as a NIP-01 `a`-tag value. */
private fun addressValue(event: Event): String {
val dTag = if (event.kind.isAddressable()) event.tags.firstOrNull { it.size > 1 && it[0] == "d" }?.get(1) ?: "" else ""
return Address.assemble(event.kind, event.pubKey, dTag)
}
/**
* The local deletion events that would make [relay] remove one of [serverEvents] — the
* events the relay HAS that we LACK. Used by sync to push *only* the deletions that
* actually apply to what the relay holds, and nothing else (not other deletions by the
* same author). Covers every way a stored deletion can reach an event:
*
* - **NIP-09, id-based** — a kind-5 with an `e` tag naming a server event's id.
* - **NIP-09, address-based** — a kind-5 with an `a` tag naming a server event's
* addressable/replaceable coordinate, at or after that event's `created_at`
* (NIP-09 only deletes `created_at <= deletion.created_at`).
* - **NIP-62 vanish** — a kind-62 by a server event's author, targeting [relay] (its
* `relay` tags name the URL or `ALL_RELAYS`), issued after that event (a vanish
* deletes `created_at < vanish.created_at`).
*
* Deduped by event id; a single deletion covering several events is returned once.
*
* [query] is where the deletions are looked up — it is source-agnostic on purpose, so
* the same coverage rule runs in both sync directions:
* - **up** (send our deletions): `events` are the relay's, `query` is the local store —
* which of OUR deletions would delete what the relay still holds.
* - **down** (apply the relay's deletions): `events` are ours, `query` fetches from the
* relay — which of the RELAY'S deletions would delete what we still hold.
*/
suspend fun deletionsCovering(
events: List<Event>,
relay: NormalizedRelayUrl,
query: suspend (Filter) -> List<Event>,
): List<Event> {
if (events.isEmpty()) return emptyList()
val covering = LinkedHashMap<HexKey, Event>()
// 1. id-based NIP-09: a kind-5 `e`-tagging an event's id.
query(Filter(kinds = listOf(DeletionEvent.KIND), tags = mapOf("e" to events.map { it.id })))
.forEach { covering[it.id] = it }
// 2. address-based NIP-09: a kind-5 `a`-tagging an event's coordinate, cutoff-checked.
val byAddress = events.filter { it.kind.isAddressable() || it.kind.isReplaceable() }.groupBy(::addressValue)
if (byAddress.isNotEmpty()) {
query(Filter(kinds = listOf(DeletionEvent.KIND), tags = mapOf("a" to byAddress.keys.toList())))
.forEach { del ->
if (del !is DeletionEvent) return@forEach
for (addr in del.deleteAddresses()) {
val hit = byAddress[addr.toValue()] ?: continue
if (hit.any { it.createdAt <= del.createdAt }) {
covering[del.id] = del
break
}
}
}
}
// 3. NIP-62 vanish: a kind-62 by an event's author, targeting this relay, issued after it.
query(Filter(kinds = listOf(RequestToVanishEvent.KIND), authors = events.mapTo(HashSet()) { it.pubKey }.toList()))
.forEach { vanish ->
if (vanish !is RequestToVanishEvent || !vanish.shouldVanishFrom(relay)) return@forEach
if (events.any { it.pubKey == vanish.pubKey && it.createdAt < vanish.createdAt }) covering[vanish.id] = vanish
}
return covering.values.toList()
}
/** [deletionsCovering] with the local store as the deletion source (the "up" direction). */
suspend fun IEventStore.deletionsCovering(
serverEvents: List<Event>,
relay: NormalizedRelayUrl,
): List<Event> = deletionsCovering(serverEvents, relay) { query<Event>(it) }