From f3637dd7c1d2e44d2fc9fa02dd55fd48f7fbb933 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 18:14:30 +0000 Subject: [PATCH 1/3] =?UTF-8?q?docs(quartz):=20revise=20local-headers-expl?= =?UTF-8?q?orer=20plan=20=E2=80=94=20SQLite,=20no=20bundle,=20OTS-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lock the design choices from the 2026-05-19 review against the intervening 2026-05-14 onchain-zaps work: - Move the module from quartz/.../nip03Timestamp/bitcoin/ to a sibling quartz/.../bitcoin/ package so the headers explorer, header validator, peer pool and store can be reused by the future onchain-zap merkle-proof work without an inverted import path. - Use androidx.sqlite + BundledSQLiteDriver for HeaderStore, matching the existing SQLiteEventStore. Schema lives in commonMain via IModule. Drops the hand-rolled flat-file + sidecar height index. - Drop the bundled headers blob. Ship a single hardcoded PinnedCheckpoint constant; first-run sync starts from the checkpoint and pulls forward over P2P. APK growth: 0 bytes. - Pre-checkpoint OTS heights fall through to OkHttpBitcoinExplorer via BitcoinExplorerEndpoint (shared with the onchain-zap EsploraBackend). Strict-mode users get an explicit error instead of a network call. - Mark trustless NIP-BC onchain-zap verification as out of scope and capture it as a follow-up plan (BIP-37 merkleblock or full-block fetch on top of this stack). Resolves open questions Q1, Q2 and Q4 from the original plan; Q3 (Quartz public API vs internal) left open for Phase 0. --- .../2026-05-08-local-headers-explorer.md | 411 +++++++++++++----- 1 file changed, 292 insertions(+), 119 deletions(-) diff --git a/quartz/plans/2026-05-08-local-headers-explorer.md b/quartz/plans/2026-05-08-local-headers-explorer.md index 5f621c1839..4244760285 100644 --- a/quartz/plans/2026-05-08-local-headers-explorer.md +++ b/quartz/plans/2026-05-08-local-headers-explorer.md @@ -1,10 +1,35 @@ # Local Bitcoin headers explorer for NIP-03 OTS -**Date:** 2026-05-08 -**Branch:** `claude/review-ots-blockchain-deps-bKns7` +**Date:** 2026-05-08 (revised 2026-05-19) +**Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`) **Module:** `quartz/` (with thin Android wiring in `amethyst/`) **Status:** Plan — not yet implemented +## Revision log + +**2026-05-19 — decisions locked, scope tightened.** After review against the +intervening 2026-05-14 onchain-zaps work: + +- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps + (kind 8333) needs more than headers — at minimum BIP-37 merkleblock or + full-block fetch on top of this stack. That becomes a follow-up plan + (see §20). +- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/` + and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block + phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`. +- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`, + matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`). + No flat append-only file; no hand-rolled height index. See §9. +- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height, + blockHash, chainWork, time)` constant, bumped at release. First-run sync + starts at the checkpoint and pulls forward to the tip over P2P. See §7. +- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`). + No backward-from-checkpoint sync. Strict-mode users get a clear error + instead of opening an HTTP connection. +- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is + now the shared Esplora-URL resolver for OTS and onchain zaps. The + composite resolver builder must read from it for the HTTP fallback path. + ## 1. Motivation NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting @@ -42,8 +67,9 @@ proof-of-work chain instead of a trusted explorer. verification logic. - Work over Tor when the user has Tor enabled. - Keep existing HTTP explorers as a fallback for proofs older than the - bundled checkpoint, or when local sync hasn't completed yet. -- Live in `quartz/` so `cli/` and `desktopApp/` benefit too. + pinned checkpoint, or when local sync hasn't completed yet. +- Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and + any future merkle-block / onchain-zap work inherit the same primitives. ### Non-goals @@ -70,12 +96,13 @@ proof-of-work chain instead of a trusted explorer. ┌──────────────┼──────────────┐ │ │ │ OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW - (existing, fallback) │ │ - │ ▼ - │ ┌─────────────────────────┐ - │ │ HeaderStore │ - │ │ (append-only file + │ - │ │ height index) │ + (existing, fallback, │ │ + URL from │ ▼ + BitcoinExplorer- │ ┌─────────────────────────┐ + Endpoint) │ │ HeaderStore │ + │ │ (SQLite, androidx- │ + │ │ sqlite + Bundled- │ + │ │ SQLiteDriver) │ │ └─────────────────────────┘ │ ▲ │ │ @@ -105,17 +132,20 @@ proof-of-work chain instead of a trusted explorer. │ CompositeBitcoinExplorer (try local first; if height - not synced yet, fall back to HTTP) + below pinned checkpoint OR not yet + reached by sync, fall back to HTTP + via BitcoinExplorerEndpoint) ``` ## 4. Module layout -All new code in `quartz/` so non-Android targets (CLI, Desktop) inherit it. -Most of the protocol code is platform-independent; only socket I/O is in -`jvmAndroid`. +All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of +`nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is +platform-independent; socket I/O is the only `expect`/`actual` boundary. +Storage uses `androidx.sqlite` which is already KMP — no platform split. ``` -quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ +quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/ ├── proto/ │ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet │ ├── MessageHeader.kt # 24-byte envelope codec @@ -128,47 +158,50 @@ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ │ │ ├── PingMessage.kt │ │ ├── PongMessage.kt │ │ ├── GetHeadersMessage.kt # locator + stop hash -│ │ └── HeadersMessage.kt # up to 2,000 × (80B header + varint=0) +│ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0) +│ │ └── SendHeadersMessage.kt # BIP-130 │ └── BitcoinMessage.kt # sealed interface ├── header/ │ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce) -│ ├── BlockHash.kt # value class wrapping ByteArray(32) +│ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32) │ ├── HeaderHasher.kt # dsha256 over 80B → blockHash │ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form -│ ├── ChainWork.kt # cumulative work accumulator +│ ├── ChainWork.kt # cumulative work accumulator (uint256 BE) │ └── PowValidator.kt # checks hash ≤ target ├── consensus/ │ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules │ ├── MedianTimePast.kt # last-11 median, for time sanity │ └── HeaderValidator.kt # composes the above + linkage check ├── store/ -│ ├── HeaderStore.kt # append-only file + height→offset index +│ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip +│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic) +│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9) │ ├── Checkpoint.kt # (height, blockHash, chainWork, time) -│ └── HeaderRecord.kt # 80B header + cached blockHash + cumulativeChainWork +│ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant ├── sync/ │ ├── LocatorBuilder.kt # exponential locator │ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists │ └── ReorgHandler.kt # rare; switches to higher-chainwork tip ├── peer/ +│ ├── PeerSocket.kt # expect (open / send / recv / close) │ ├── PeerPool.kt # holds N BitcoinPeer connections │ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg -│ └── PeerScoring.kt # ban misbehaving peers -└── LocalHeadersBitcoinExplorer.kt # implements quartz.../ots/BitcoinExplorer +│ ├── PeerScoring.kt # ban misbehaving peers +│ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds +└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer + # (with small LRU on top, like OtsBlockHeightCache) -quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ -├── peer/ -│ ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket) -│ └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames -└── store/ - └── FileHeaderStore.jvmAndroid.kt # RandomAccessFile-backed HeaderStore - -quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/ -└── PeerSocket.kt # expect (open / send / recv / close) -└── HeaderStorage.kt # expect (append, getByHeight, tip) +quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/ +└── peer/ + ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS) + └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames ``` -The `expect`/`actual` split keeps protocol logic testable cross-platform; the -socket layer and disk layer are the only platform-dependent pieces. +Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor) +and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs, +store, sync engine, peer pool — all compile in `commonMain`, all testable +cross-platform without a single platform shim. Storage compiles in +`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP. ### Wiring on Android @@ -182,7 +215,17 @@ amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/ `AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a composite that prefers the local explorer and falls back to HTTP only when -local is missing the height. +local is missing the height. The HTTP path resolves its base URL via the +existing `BitcoinExplorerEndpoint` (shared with the onchain-zap +`EsploraBackend`), so a user-configured custom Esplora is honoured on both +the OTS and the zap side. + +`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }` +into Quartz; the headers SQLite file goes under +`$filesDir/bitcoin/headers.db` via a new entry in that wiring. + +On desktop and CLI, the equivalent data-dir paths are already plumbed for +`SQLiteEventStore` — same hook reused here. ## 5. Bitcoin P2P protocol — minimum we implement @@ -274,23 +317,35 @@ mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned checkpoint** updated at app build time near the current tip. Sync must agree with all checkpoints — disagreement aborts and bans the peer. -## 7. Bundled data +## 7. Bundled data — none. Just a pinned checkpoint constant. -To avoid making every fresh install download 75 MB: +The point of this work is "verify against the Bitcoin network, not a trusted +party," so we ship no headers in the APK. Instead: -- App ships an **assets file** `bitcoin_headers_.bin`: raw 80-byte - headers from genesis up to a release-pinned height (≤ 1 month before build). - ~72 MB — too large for Play Store delta-update friendliness. -- Ship instead a **compact bundled checkpoint**: just one `Checkpoint(height, - blockHash, chainWork, time)` per app release, plus the most recent ~50 K - headers (~4 MB) so users can verify any attestation from the past ~year - immediately. -- For older proofs, lazy-sync from the checkpoint *backwards* down to genesis - on demand (rare path). -- Forward sync from checkpoint to current tip is what runs at first launch - (~10 KB to a few MB depending on release age). +- **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`: + `(height, blockHash, chainWork, time)`. ~80 bytes of source. +- The checkpoint is pinned ~1 month before each release and bumped via a + one-line PR per release. This is the **only** thing trusted at build time + — and it's a single block hash any user can independently verify against + any Bitcoin node before installing. +- **First-run sync** starts at the checkpoint and pulls forward to the + current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers + (~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at + worst), seconds over WiFi. +- **Pre-checkpoint heights** are not synced locally at all. OTS proofs + whose height precedes the checkpoint fall through to + `OkHttpBitcoinExplorer` via the composite. Users in strict-mode + (`localHeadersTrustOnly = true`) get a clear + "this OTS proof predates the bundled checkpoint" error and no network call. +- **Catch-up after long absence.** A device that hasn't run the app for + months will, on next sync, fast-forward from `tip_local` (its stored + best_height) to `tip_network`. Each `headers` message brings 2,000 + headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round + trips. +- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9). -This keeps APK growth at ~4 MB and first-run sync time at seconds. +APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one +block hash, code-reviewable in git. ## 8. Multi-peer strategy & eclipse mitigation @@ -315,42 +370,85 @@ A single peer can lie. We protect against that with: (`bitcoin.example.com:8333`) — same path as the existing custom explorer URL setting in `OtsSettings.kt`. -## 9. Storage format +## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver` -Append-only flat file plus a sparse index. +We reuse the same storage primitive Quartz already uses for `SQLiteEventStore` +(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled +driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm` +(already a Quartz dependency), and is the project's standard storage choice. -### `headers.bin` +The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`, +`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations, +`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and +`HeaderStoreSchema`. -Records, fixed 112 bytes each, indexed by height (record N at offset `N*112`): +### Schema -``` -+0 80B raw block header -+80 32B cumulative chainwork (uint256, big-endian) +```sql +PRAGMA journal_mode = WAL; +PRAGMA synchronous = NORMAL; +PRAGMA foreign_keys = OFF; + +CREATE TABLE block_header ( + height INTEGER PRIMARY KEY, -- 0..tip, dense + hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order + header BLOB NOT NULL, -- 80 B raw header + chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian +); + +CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash); + +CREATE TABLE chain_tip ( + id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row + best_height INTEGER NOT NULL, + best_hash BLOB NOT NULL, + best_work BLOB NOT NULL +); + +CREATE TABLE checkpoint_seen ( + height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed + hash BLOB NOT NULL, + source TEXT NOT NULL -- 'pinned' | 'historical' +); ``` -The block hash and target are recomputed from bytes 0..80 lazily; we don't -store them. Total at height 1M: 112 MB. We can drop this to 80 B/record -(no chainwork) if we accept recomputing chainwork on startup — TBD; default -is 112 B for fast tip selection. +Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as +the event store's modules). v1 is the schema above; future additions +(prev-hash cache column, merkle-block annotations) ALTER from there. -### `tips.json` +### Sizing -A small JSON file holding: +| Item | Per record | At height 1,000,000 | +|---|---|---| +| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB | +| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB | +| `chain_tip` | 1 row | negligible | +| **Total** | | **~220–260 MB at height 1 M** | -```json -{ - "best_height": 945123, - "best_hash": "00000000000000000001abc...", - "best_work": "0x000...", - "checkpoints_passed": [600000, 700000, 800000, 940000] -} -``` +In practice users will start from a recent checkpoint and grow ~4 MB raw +(~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M +height" line is the long-tail worst case if someone with a multi-year-old +device's local store retroactively backfills (which we explicitly don't do +in v1 — see §15 Q1). -### Atomicity +### Atomicity & crash safety -Sync writes to `headers.bin.tmp`, fsyncs, then atomically renames or -appends-and-fsyncs in batches of 2000. Crash recovery: on startup, scan from -last fsynced offset, validate linkage, and discard partial trailing records. +- `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a + single `transaction { }`. WAL gives us atomic-or-nothing per batch. +- `chain_tip` is updated in the same transaction as the headers it points to, + so the tip can never reference rows that didn't commit. +- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on + next open. No custom recovery code. +- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and + confirms the referenced header exists. If not (unexpected DB corruption, + ransomware-like sandbox damage), we delete the db file and fall back to + re-sync from the pinned checkpoint. + +### Concurrency + +Same model as `SQLiteEventStore`: one connection pool, all writes go through +the sync engine on a dedicated dispatcher; reads use the pool's read slots. +OTS verifier reads are short and indexed — no contention with sync writes. ## 10. Resolving a height query (the actual `BitcoinExplorer` impl) @@ -410,20 +508,25 @@ Extend `OtsSettings.kt`: ```kotlin data class OtsSettings( - val customExplorerUrl: String? = null, + val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint // NEW: val localHeadersEnabled: Boolean = true, - val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback + val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error val localHeadersWifiOnly: Boolean = true, - val customP2pNode: String? = null, // e.g. "myriad.example.com:8333" + val customP2pNode: String? = null, // e.g. "myriad.example.com:8333" ) ``` A short "Bitcoin verification" settings screen lets the user pick: -- Local headers (default) -- HTTP explorer only (current behavior, lighter) -- Local + HTTP fallback -- Local only (strict, may delay verification of brand-new attestations) +- **Local headers + HTTP fallback** (default) — best UX, trustless for + post-checkpoint heights, HTTP for older proofs. +- **Local headers only (strict)** — refuses to verify proofs older than the + pinned checkpoint. Most paranoid setting. +- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU. + +The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`, +so a custom HTTP Esplora is honoured both by the fallback in this feature +and by the onchain-zaps `EsploraBackend`. No new Esplora setting. ## 12. Tor integration @@ -461,10 +564,15 @@ Each phase produces a shippable artifact with measurable behavior. ### Phase 2 — Storage layer (2–3 days) -- `HeaderStore` interface + `FileHeaderStore` actual on jvmAndroid. -- Append, get-by-height, get-by-hash (in-memory hash→height map built at - startup, ~60 MB at height 1 M — acceptable, or use a sparse on-disk index). -- Crash-recovery test. +- `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain` + (`androidx.sqlite` is KMP, no platform shim needed). +- `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore` + migration pattern. +- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`. +- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }` + from `nip01Core/store/sqlite/` — no duplication. +- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback; + confirm `verifyTipIntegrity()` recovers cleanly. ### Phase 3 — P2P codec, no socket (3–5 days) @@ -488,13 +596,16 @@ Each phase produces a shippable artifact with measurable behavior. - Header-batch cross-validation across peers. - Hardcoded checkpoints in `Checkpoints.kt`. -### Phase 6 — Bundled checkpoint & lazy-genesis-sync (2–3 days) +### Phase 6 — Pinned checkpoint constant (½ day) -- Build-time generation of the release-pinned checkpoint and trailing-50K - headers blob, packaged as `quartz/src/jvmAndroid/resources/bitcoin/...`. -- Forward sync from checkpoint to tip on first launch. -- Backward (genesis-direction) sync triggered only when an OTS proof requests - a height before the checkpoint. +- `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded + `Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source. +- A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes + `(height, hash)` arguments and writes the constant. Run by the release + process; otherwise no automation. +- No backward-sync, no bundled headers blob, no asset resources. + Pre-checkpoint heights are routed to the HTTP fallback by the composite + resolver (or refused in strict mode). ### Phase 7 — Android wiring (3–4 days) @@ -512,9 +623,14 @@ Each phase produces a shippable artifact with measurable behavior. ### Phase 9 — Desktop & CLI integration (1–2 days) -- `desktopApp/` already inherits the explorer because it's in `quartz/`. - Confirm it works end-to-end on JVM. -- `cli/` (`amy verify-ots`) — confirm CLI sync also works headlessly. +- `desktopApp/` inherits the explorer for free (everything is in + `quartz/commonMain/`). Confirm headers.db path is plumbed via the same + data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync + runs end-to-end on the desktop app. +- `cli/` — add an `amy verify-ots ` command if not present, + wired through the same composite resolver. The CLI is a thin assembly + layer over Quartz (per `amy-expert`), so the actual sync logic is reused + as-is. ### Phase 10 — Rollout & telemetry (ongoing) @@ -523,8 +639,9 @@ Each phase produces a shippable artifact with measurable behavior. fallback rate. Surface in a debug screen. - Default-on in the next release if metrics look healthy. -Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk -(protocol + validation); phases 5–10 are integration. +Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk +(protocol + validation); phases 5–10 are integration. Phase 6 dropped from +2–3 d to ½ d because we ship no bundled data — just one constant. ## 14. Testing strategy @@ -544,8 +661,12 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk (committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`). - Local-network test against a Bitcoin Core regtest node spun up in CI (Docker). Validates full handshake + `getheaders` loop end to end. -- Crash-and-resume test: kill mid-sync, restart, confirm we pick up at the - correct height. +- `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a + `tmpdir`): append 5,000 headers in batches, getByHeight, getByHash, + reopen, confirm tip survives. +- Crash-and-resume test: kill the writer in the middle of a `transaction { }`, + reopen, confirm SQLite WAL rolls the partial batch back and sync resumes + from the last committed tip. ### Property tests @@ -564,32 +685,41 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk | # | Risk | Mitigation | |---|------|------------| -| R1 | First-run UX cost (1–10 min sync) puts users off | Bundled checkpoint + 50 K trailing headers means most installs need only seconds. Show progress prominently. | -| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design. | +| R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. | +| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). | | R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. | -| R4 | Disk corruption | Atomic appends, checksum on close, fall back to re-sync from last checkpoint. | -| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer when sync stalls. | -| R6 | APK size growth from bundled headers | Cap bundle at most-recent ~50 K headers (~4 MB). Older headers fetched on demand. | +| R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. | +| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. | +| R6 | APK size growth | Zero — no bundled data ships in the APK. | | R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. | | R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. | | R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. | | R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. | +| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. | ### Open questions -- **Q1.** Do we actually want backward-from-checkpoint sync, or just refuse - to verify proofs older than the bundled checkpoint and tell users to use - the HTTP fallback? Backward sync is operational complexity for a rare path. -- **Q2.** Storage: 80 B/record (recompute chainwork on startup) vs - 112 B/record (cache it)? At height 1M, 32 MB difference. Probably worth - caching. -- **Q3.** Should we expose this as a Quartz public API (so 3rd-party apps - using Quartz also benefit) or keep it internal to NIP-03? Recommendation: - public — it's generally useful. -- **Q4.** Do we ship a "headers blob" download URL (HTTPS) as a faster - alternative to P2P for first-run? Trade-off: faster but adds a trust point. - Could be made cross-checked (download blob, then verify every header's PoW - locally — same trust as P2P then). Worth considering as Phase 6b. +#### Resolved 2026-05-19 + +- **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are + routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode + users (`localHeadersTrustOnly = true`) get a clear error. No backward + sync code path. +- **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one + extra BLOB column). Reading current tip work without a full scan matters + for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk. +- **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is + trustless verification, and the pinned-checkpoint approach already gives + fast first-run. Adding a second ingestion path (signed blob over HTTPS) + doubles the test surface for marginal benefit. + +#### Still open + +- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz + public API for 3rd-party Quartz consumers, or internal to Amethyst? + Recommendation: public — it's a clean reusable primitive. Decide before + Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration` + skill for the public-API contract Quartz exposes today.)* ## 16. Migration & rollout @@ -605,22 +735,55 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk - A NIP-03 attestation that verifies via Blockstream also verifies via the local explorer (bit-for-bit identical timestamp). - 95th-percentile cold-launch verification of a recent attestation: ≤2 s. -- 95th-percentile first-run sync (post-checkpoint): ≤30 s on WiFi. +- 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on + WiFi for a checkpoint ≤2 months old. - No new crash-rate regression in the verification path. -- App size growth: ≤5 MB. +- App size growth: **0 MB** (no bundled data). - A user with HTTP explorer disabled can still verify any post-checkpoint attestation. ## 18. Out of scope (explicit) - BIP-157/158 compact filters +- BIP-37 merkleblock or full-block fetch +- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or + full-block fetch on top of this stack. See §20 for the follow-up plan. +- **Trustless wallet operations** for NIP-BC onchain zap *sending* — + needs UTXO-by-address and broadcast paths that headers can't provide. + Stays on user-configured Esplora. - Lightning, BIP-32 wallet, mempool monitoring - Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in the spec) - Replacing the calendar-server upgrade path (`OtsState.upgrade()`) — unrelated, those are simple HTTPS calls and stay as-is -## 19. References +## 19. Follow-up: trustless NIP-BC onchain zap verification + +*Not in this plan. Tracked here so the design choices above stay aligned.* + +The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`) +trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both +tx-existence-in-a-block and the output values used to compute +`verifiedSats`. A malicious Esplora can fabricate or suppress zaps. + +Closing that gap is a **separate plan** that builds on this one: + +- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`, + `HeaderValidator`, `LocalHeadersBitcoinExplorer`. +- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` / + `merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream + parse; partial-merkle-tree verification; new `OnchainBackend` variant + that asks the local stack for the inclusion proof and the matched tx + while keeping Esplora for UTXO listing / broadcast / fee estimates on + the *send* side. +- **Net win:** zap-receive verification stops trusting Esplora. Zap-send + still needs an address-indexed server (no headers-only equivalent + exists), so that part of the trust story is unchanged. +- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`, + written once S1 has shipped and we have real numbers on first-run sync, + store size, and battery cost. + +## 20. References - NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md - OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps @@ -632,6 +795,16 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt` - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt` - `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt` - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt` + - `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt` +- Storage precedent (reused as-is): + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt` +- Follow-up area: + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt` + - `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt` From d19bcf07b07cc78d0a388af90ed1967a825e7c4a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 18:35:30 +0000 Subject: [PATCH 2/3] docs(quartz): add interop-test-vectors section to local-headers-explorer plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every consensus-relevant layer of the planned headers explorer (BlockHeader80 parser, DifficultyTarget compact↔target, CalculateNextWorkRequired retarget, MedianTimePast, header validator end-to-end, P2P wire codecs, reorg/chain selection, OTS proofs) is pinned to upstream test vectors committed under quartz/src/commonTest/resources/bitcoin/, matching the existing nip44.vectors.json / bip39.vectors.json / mls/*.json pattern. The single highest-value test is a nightly differential check asserting LocalHeadersBitcoinExplorer.blockHash(h) == OkHttpBitcoinExplorer.blockHash(h) for every height in [checkpoint, tip] — any consensus drift surfaces as a disagreeing height. Maps cleanly onto the existing Phase 1/3/4/5/7/9 work, adding ~5–7 engineer-days total to the plan budget. No new top-level phase needed. --- .../2026-05-08-local-headers-explorer.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/quartz/plans/2026-05-08-local-headers-explorer.md b/quartz/plans/2026-05-08-local-headers-explorer.md index 4244760285..5103663dd8 100644 --- a/quartz/plans/2026-05-08-local-headers-explorer.md +++ b/quartz/plans/2026-05-08-local-headers-explorer.md @@ -29,6 +29,13 @@ intervening 2026-05-14 onchain-zaps work: - **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is now the shared Esplora-URL resolver for OTS and onchain zaps. The composite resolver builder must read from it for the HTTP fallback path. +- **Interop test vectors** (§14a) added — every consensus-relevant layer + (header parser, compact-target, retargeter, P2P codec, OTS proofs) is + pinned to upstream vectors committed under + `quartz/src/commonTest/resources/bitcoin/`. A nightly differential test + asserts `LocalHeadersBitcoinExplorer.blockHash(h)` == + `OkHttpBitcoinExplorer.blockHash(h)` for every height in + `[checkpoint, tip]`, catching any consensus-rule drift end-to-end. ## 1. Motivation @@ -681,6 +688,93 @@ Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk - Tor on — sync completes (slowly) over Orbot. - Disk-full simulation — fail gracefully, don't corrupt store. +## 14a. Interop test vectors + +Consensus code without external test vectors is "trust me." Every layer +that could disagree with the rest of the Bitcoin network gets pinned to +authoritative upstream vectors, mirroring how `nip44.vectors.json`, +`bip39.vectors.json` and the `mls/*.json` set are already organised in +`quartz/src/commonTest/resources/`. The onchain-zaps work uses the same +discipline (BIP-341 wallet test vectors at every layer of +`nipBCOnchainZaps/taproot/` + `psbt/`). + +### Sources, by component + +| Component | Source(s) | Format | Storage | Effort | Priority | +|---|---|---|---|---|---| +| **`BlockHeader80` parse/serialize** | Genesis (0), block 1, block 209999 (last pre-halving), 481824 (first post-segwit), 709631 (last pre-taproot), one recent | JSON: `[{height, hash, headerHex, version, prevHash, merkleRoot, time, bits, nonce}, …]` | commit `commonTest/resources/bitcoin/known_headers.json` | 1 d | **High** | +| **`DifficultyTarget` compact↔target** | bitcoinj `UtilsTest.testCompactBitsToBigInteger`, btcd `chaincfg/chainhash/compact_test.go`. Vectors: `0x1d00ffff`, `0x1b0404cb`, `0x170398bb`, overflow + negative-bit edges | JSON pairs `(compact, targetHex)` | commit `bitcoin/compact_target.json` | ½ d | **High** | +| **Retarget (`CalculateNextWorkRequired`)** | Mainnet retarget heights 2016, 4032, 32256, 60480, ~600000 — extract `(prev_retarget_time, current_time, prev_bits, expected_new_bits)` from any full node | JSON list of retarget cases | commit `bitcoin/retargets.json` (~50 entries, ~5 KB) | 1 d | **High** | +| **`MedianTimePast`** | Any 11 consecutive mainnet headers synthesised from `known_headers.json` | inline test | inline | ½ d | Medium | +| **HeaderValidator end-to-end on real chain** | First 2016 mainnet headers (~160 KB; covers genesis + first retarget). Larger 50 K-header set lazy-fetched in nightly CI from a pinned-hash mirror, not committed | binary blob | commit `bitcoin/first_2016_headers.bin` | 1–2 d | **High** | +| **P2P wire codec — `version`/`verack`/`ping`/`pong`** | btcd `wire/msg*_test.go` has canonical byte strings; Bitcoin Wiki "Protocol documentation" has annotated hex | hex strings in JSON | commit `bitcoin/p2p_messages.json` | 1 d | **High** | +| **P2P `getheaders` / `headers` round-trip** | Capture against a real bitcoind once, ~50 KB | raw bytes | commit `bitcoin/p2p_capture.bin` | 1 d | **High** | +| **DNS seed list** | Bitcoin Core `chainparams.cpp` (current 9 seeds) | inline constant | n/a | <¼ d | Low | +| **Hardcoded fallback IPs** | Bitcoin Core `chainparamsseeds.h` (auto-generated by `contrib/seeds/`) | binary blob | commit `bitcoin/chainparamsseeds.bin` | ½ d | Medium | +| **Reorg / chain selection** | Synthetic 2-branch fork — verify higher-cumulative-chainwork wins | inline builder | inline | 1 d | **High** | +| **End-to-end OTS proofs** | `python-opentimestamps/tests/test_data/*.ots` + the 3 wild events already inline in `OtsTest.kt`. Import 5–10 more `.ots` for variety (calendar operators, eras) | `.ots` binaries + JSON manifest of expected timestamps | commit `commonTest/resources/ots/*.ots` | 1 d | **High** | +| **NIP-03 1040 events from real relays** | Scrape any relay for `kind:1040`; pick a diverse set across calendar operators and years | inline JSON | extend `OtsTest.kt` | ½ d | Medium | +| **Differential test: local vs HTTP** | For every `h ∈ [checkpoint, tip]`: assert `LocalHeadersBitcoinExplorer.blockHash(h)` == `OkHttpBitcoinExplorer.blockHash(h)`. Catches consensus drift end-to-end | runtime check, nightly CI | no fixture | 1 d | **High** | +| **Regtest end-to-end** | Bitcoin Core in Docker — mine N blocks, sync, assert tip matches | infrastructure | CI script | 2 d | Medium | +| **Adversarial peer** | Synthetic peer that lies about retarget, replays old headers, drops mid-batch, sends 2,001-headers messages. Validates ban + reconnect | inline test harness | inline | 2 d | **High** | + +### Committed layout + +``` +quartz/src/commonTest/resources/bitcoin/ +├── README.md # provenance + bump procedure for each file +├── known_headers.json # ~10 hand-picked mainnet headers across history +├── compact_target.json # ~20 compact-bits ↔ target pairs +├── retargets.json # ~50 retarget cases from historical mainnet +├── first_2016_headers.bin # ~160 KB; genesis + first retarget cycle +├── p2p_messages.json # hex of canonical version/verack/ping/pong +├── p2p_capture.bin # captured handshake + getheaders/headers (~50 KB) +└── chainparamsseeds.bin # current Bitcoin Core fallback IP list + +quartz/src/commonTest/resources/ots/ +├── manifest.json # {filename → expected unix timestamp | "pending"} +└── *.ots # 5–10 fixtures from python-opentimestamps +``` + +### Provenance discipline + +The `README.md` next to each fixture records, in this order: + +1. **Where it came from** — upstream URL + commit SHA. +2. **How to regenerate** — a single command, no manual steps. +3. **What it tests** — one sentence. +4. **When to bump** — e.g. *"retargets.json is stable; never bumps."* / + *"chainparamsseeds.bin bumped each Bitcoin Core major release."* + +Same pattern as the header comment of `nip44.vectors.json`. New committers +should never have to ask "where did this 32-byte blob come from." + +### Phase mapping + +These test imports map cleanly onto the existing phases — total extra +effort **~5–7 engineer-days** spread across the work, no new top-level +phase needed: + +| Phase | Vectors consumed | +|---|---| +| **Phase 1** (header parsing & validation, no I/O) | `known_headers.json`, `compact_target.json`, `retargets.json`, `first_2016_headers.bin` | +| **Phase 3** (P2P codec, no socket) | `p2p_messages.json`, `p2p_capture.bin` | +| **Phase 4** (single-peer sync) | adversarial-peer harness + regtest-in-Docker | +| **Phase 5** (multi-peer & eclipse) | `chainparamsseeds.bin` | +| **Phase 7** (Android wiring) | differential test against `OkHttpBitcoinExplorer` | +| **Phase 9** (CLI / Desktop) | OTS `.ots` fixtures via `amy verify-ots --explorer=local` | + +### Why the differential test matters most + +The single highest-value test in the list is the differential one: for +every height between the pinned checkpoint and the current tip, the local +explorer's answer must equal the HTTP explorer's answer. Run it nightly in +CI. Any consensus-rule drift — a wrong retarget formula, a missed +overflow case in compact-target decoding, a partial-merkle bug — will +surface as a height where the two disagree, and the height is the bug +report. This is the test that catches the bug nobody thought to write +a unit test for. + ## 15. Risks & open questions | # | Risk | Mitigation | From 24cf3fdc5fe36f64383d07e638c6bc43a328b92d Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 19:17:36 +0000 Subject: [PATCH 3/3] docs: update onchain-zap + headers-explorer plans for NIP-BC inline SPV tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nostr-protocol/nips#2332 adds optional ["block", …] and ["proof", …] tags on kind:8333 that ship the SPV proof inline. The data-model side is already shipped in Quartz (BlockTag, ProofTag, OnchainZapEvent.block()/.proof() and the TagArray helpers), but no production code path produces or consumes either tag yet. amethyst/plans/2026-05-14-onchain-zaps.md — onchain zaps plan: - Update the "Chain backend" decision bullet to flag the spec change and the two production gaps (send-side emit, receive-side consume). - Add a dedicated "Inline SPV proofs" section covering: spec status and the merkle-proof encoding ambiguity flagged back to the PR; per-layer status matrix (shipped vs gap, with file locations); send-side two-publish design (Design B — keep instant receipt, add post-confirm republish with block+proof; dedupe by (txid, target)); receive-side fast-path with fall-through on proof failure (never hard-reject); phased delivery G.1–G.7 with effort estimates (~3–4 d after S1 ships and spec encoding lands). - Add the two new pending items to the existing "What's still pending" list to keep that section authoritative. quartz/plans/2026-05-08-local-headers-explorer.md — headers-explorer plan: - Rewrite §19 (follow-up onchain-zap verification) to reflect the spec change. Original section assumed we'd need BIP-37 merkleblock or full-block fetch over P2P; with the proof inline none of that infrastructure is required. Estimate collapses from 10–15 d to ~3–4 d. - Point §19 at the full implementation plan in the onchain-zaps file, keeping S1 focused on OTS while the follow-up details live with the rest of the NIP-BC work. --- amethyst/plans/2026-05-14-onchain-zaps.md | 124 +++++++++++++++++- .../2026-05-08-local-headers-explorer.md | 59 ++++++--- 2 files changed, 159 insertions(+), 24 deletions(-) diff --git a/amethyst/plans/2026-05-14-onchain-zaps.md b/amethyst/plans/2026-05-14-onchain-zaps.md index f61494e015..d2725b58e5 100644 --- a/amethyst/plans/2026-05-14-onchain-zaps.md +++ b/amethyst/plans/2026-05-14-onchain-zaps.md @@ -19,10 +19,15 @@ rest of the system. there. `NostrSignerRemote` (NIP-46) is deferred. - **Chain backend.** User-configured Esplora-compatible API (mempool.space, blockstream.info, self-hosted). The configured server sees - the user's UTXO queries — accepted tradeoff for v1. Header-only SPV mode is - a future-phase add. The explorer endpoint is shared with OpenTimestamps via - `BitcoinExplorerEndpoint`: same user-configured server, same Tor-aware - default selection. + the user's UTXO queries — accepted tradeoff for v1. **Spec PR + nostr-protocol/nips#2332 adds optional `["block", …]` and + `["proof", …]` tags on `kind:8333` that let a verifier with only the + Bitcoin header chain check a zap end-to-end without trusting the + explorer. Amethyst already parses both tags (see `BlockTag.kt` / + `ProofTag.kt`) but no production code path produces or consumes them + yet — see the "Inline SPV proofs" section below.** The explorer + endpoint is shared with OpenTimestamps via `BitcoinExplorerEndpoint`: + same user-configured server, same Tor-aware default selection. - **Scope.** Full send + receive + display loop on Android. Desktop is out of scope for v1. @@ -232,6 +237,117 @@ Lightning fast path. Two minimal hooks: 2. **NIP-46 `sign_psbt`.** `NostrSignerRemote.signPsbt` still throws `UnsupportedMethodException` — the bunker-side command is not standardized yet. +3. **Send-side `block` + `proof` emit.** See "Inline SPV proofs" below. +4. **Receive-side SPV verification.** See "Inline SPV proofs" below. + +## Inline SPV proofs (`block` + `proof` tags) — pending + +### Spec status (2026-05-19) + +`nostr-protocol/nips#2332` adds two new optional tags on `kind:8333`: + +``` +["block", "", ""] +["proof", "", ""] +``` + +Together they let a verifier with only the Bitcoin header chain check a +zap end-to-end without trusting the chain backend. The recipient hashes +`raw-tx-hex` to recover the txid, walks the merkle proof from that leaf +up to the header's `merkleRoot`, parses outputs from `raw-tx-hex`, and +sums those that pay the recipient's derived Taproot script. + +**One spec ambiguity outstanding** (raised back to the PR author): +direction-at-each-step encoding for `` — Bitcoin's +merkle tree is tx-order-sensitive, so concatenated siblings alone don't +specify left vs right. Suggested resolution: `1-byte direction || 32-byte +sibling` per step (33 B stride). This blocks Phase G.2 (walker +implementation), nothing else. + +### Current state in Amethyst + +| Layer | Status | Location | +|---|---|---| +| `BlockTag(blockHashHex, height: Long)` | ✅ shipped | `quartz/.../nipBCOnchainZaps/zap/tags/BlockTag.kt` | +| `ProofTag(rawTxHex, merkleProofHex)` | ✅ shipped | `…/zap/tags/ProofTag.kt` | +| `OnchainZapEvent.block()` / `.proof()` accessors | ✅ shipped | `…/zap/OnchainZapEvent.kt:89-93` | +| TagArray + TagArrayBuilder helpers | ✅ shipped | `…/zap/TagArrayExt.kt`, `…/zap/TagArrayBuilderExt.kt` | +| Tx parser + witness-stripped txid (reusable for the walker) | ✅ shipped | `…/psbt/BitcoinTransaction.kt` | +| Recipient scriptPubKey derivation | ✅ shipped | `taproot/TaprootAddress.scriptPubKeyHexForRecipient` | +| `OnchainZapSender` populates `block` / `proof` after broadcast | ❌ gap | `commons/.../onchain/OnchainZapSender.kt:209-214` (calls `build()` / `buildProfileZap()` with no SPV tags) | +| `OnchainBackend.getMerkleProof(txid)` to source the proof on send | ❌ gap | `chain/OnchainBackend.kt` | +| Merkle-proof parser + walker | ❌ gap | new `verify/MerkleProofVerifier.kt` | +| `OnchainZapVerifier` prefers inline proof when present | ❌ gap | `verify/OnchainZapVerifier.kt` | +| Validated `header.merkleRoot` lookup by block hash | ❌ gap | depends on the headers-explorer plan (S1) at `quartz/plans/2026-05-08-local-headers-explorer.md` | + +### Send-side: Design B (two-publish) + +`OnchainZapSender` publishes the `kind:8333` receipt **immediately** after +`backend.broadcast(rawTxHex)` returns, when confirmations = 0. The chosen +design keeps that instant UX and adds a second publish after the tx +confirms: + +1. Today's path stays. Publish `kind:8333` right after broadcast, with + `i`/`p`/`amount`/`alt` only — no `block`, no `proof`. +2. A new post-confirmation worker watches the tx via the backend. When it + sees ≥1 confirmation: + - Call `backend.getMerkleProof(txid)` → `(block_hash, height, siblings, pos)`. + - Encode `siblings + pos` into the spec's `` wire format + (depends on the resolved spec encoding). + - Build a second `kind:8333` referencing the same `i` tag, with `block` + and `proof` filled in. Publish. +3. Verifiers MUST dedupe by `(txid, target)` and prefer the variant + carrying a valid SPV proof. (This dedupe rule is also in the suggested + spec amendment.) + +Two relay broadcasts per zap, ~2 KB extra each. The post-confirmation +worker is per-account, persists across app restarts, and exits cleanly +once each pending tx is either confirmed (publish second receipt) or +abandoned (after some timeout — TBD). + +### Receive-side + +`OnchainZapVerifier.verify(event)` gets a new fast path: + +1. If `event.block() != null && event.proof() != null`: + - Look up header by hash via `LocalHeadersBitcoinExplorer.byHash()` (S1). + If unavailable yet, fall through to the existing `backend.getTx()` path. + - `BitcoinTransaction.parse(rawTxHex)` → tx; recompute + `dsha256(stripWitness(tx))` → expected_txid; check it matches the + `i` tag's txid. + - `MerkleProofVerifier.verify(expected_txid, proofBytes, header.merkleRoot)` → boolean. + - Parse outputs from the same parsed tx; sum outputs paying the + recipient's `scriptPubKeyHexForRecipient`, skipping outputs paying + the sender's own derived script (change). + - Confirmations = `tip.height − block.height + 1`. + - Return `Confirmed`. +2. If the inline path fails verification (bad proof, missing header, + tx-mismatch), the event is treated as if `block` / `proof` were + absent — fall back to the existing Esplora path. **Never** treat a + failed SPV proof as a hard-reject of the whole event; that would let + a buggy producer poison legitimate zaps. +3. If neither tag is present (old-shape events), the existing + `backend.getTx()` path runs unchanged. + +The end state: a strict-mode user with HTTP fallback disabled can verify +any zap whose sender included `block` + `proof`, and only those. + +### Phased delivery for the SPV work + +| Phase | Deliverable | Depends on | Effort | +|---|---|---|---| +| **G.1** | `MerkleProofVerifier` in `nipBCOnchainZaps/verify/`; test vectors per the resolved spec encoding | spec encoding locked | ½ d | +| **G.2** | `OnchainZapVerifier` consumes inline proof, falls back on failure; needs `LocalHeadersBitcoinExplorer.byHash()` from S1 | S1 Phase 2 (storage) | ½ d | +| **G.3** | `OnchainBackend.getMerkleProof(txid)` interface + `EsploraBackend` impl (`/tx/{txid}/merkle-proof`) | — | ½ d | +| **G.4** | Post-confirmation worker + second-publish flow in `OnchainZapSender` | G.3 | 1 d | +| **G.5** | Dedupe by `(txid, target)` preferring SPV-attached variant; `Note.zapsAmount` only counts the winner | — | ½ d | +| **G.6** | End-to-end tests: real mainnet block + proof + zap; negative cases (wrong root, truncated, wrong direction) | G.1 | 1 d | +| **G.7** | (Optional) Settings toggle "Verify zaps trustlessly when possible" — off by default until S1 ships | S1 Phase 7 | ½ d | +| **Total** | | | ~3–4 d (after S1 ships and spec encoding lands) | + +S1 (the headers explorer at `quartz/plans/2026-05-08-local-headers-explorer.md`) +is the long pole. The SPV work itself is small — every primitive needed +already exists in the codebase, the only new thing is the merkle walker. ## Risks / open questions diff --git a/quartz/plans/2026-05-08-local-headers-explorer.md b/quartz/plans/2026-05-08-local-headers-explorer.md index 5103663dd8..fe499e3fcc 100644 --- a/quartz/plans/2026-05-08-local-headers-explorer.md +++ b/quartz/plans/2026-05-08-local-headers-explorer.md @@ -853,29 +853,48 @@ a unit test for. ## 19. Follow-up: trustless NIP-BC onchain zap verification -*Not in this plan. Tracked here so the design choices above stay aligned.* +*Not in this plan. Tracked here so the design choices above stay aligned. +Full implementation plan lives in `amethyst/plans/2026-05-14-onchain-zaps.md` +under "Inline SPV proofs (`block` + `proof` tags) — pending".* -The shipped `OnchainZapVerifier` (`quartz/.../nipBCOnchainZaps/verify/`) -trusts `OnchainBackend.getTx(txid)` — i.e. Esplora — for both -tx-existence-in-a-block and the output values used to compute -`verifiedSats`. A malicious Esplora can fabricate or suppress zaps. +### What the 2026-05-19 spec change buys us -Closing that gap is a **separate plan** that builds on this one: +`nostr-protocol/nips#2332` adds optional `["block", "", ""]` +and `["proof", "", ""]` tags on `kind:8333`. +The proof rides inline in the Nostr event, so the trustless-verify path +needs **no new P2P messages on top of S1** — no `getdata MSG_BLOCK`, no +BIP-37 `filterload`, no full-block parser. Just header lookup + merkle +walk against bytes already in the event. -- **Foundation reused:** `BitcoinPeer`, `PeerPool`, `HeaderStore`, - `HeaderValidator`, `LocalHeadersBitcoinExplorer`. -- **New work (rough ~10–15 days on top of S1):** BIP-37 `filterload` / - `merkleblock` / matched-`tx` handling, or `getdata MSG_BLOCK` + stream - parse; partial-merkle-tree verification; new `OnchainBackend` variant - that asks the local stack for the inclusion proof and the matched tx - while keeping Esplora for UTXO listing / broadcast / fee estimates on - the *send* side. -- **Net win:** zap-receive verification stops trusting Esplora. Zap-send - still needs an address-indexed server (no headers-only equivalent - exists), so that part of the trust story is unchanged. -- **Filename:** `quartz/plans/YYYY-MM-DD-onchain-zap-merkle-proofs.md`, - written once S1 has shipped and we have real numbers on first-run sync, - store size, and battery cost. +This collapses the original "follow-up plan" estimate from 10–15 days to +**~3–4 days** of work after S1 ships and the spec's merkle-proof +encoding is locked. + +### Foundation reused from S1 + +- `LocalHeadersBitcoinExplorer.byHash(blockHash)` → validated header (for + the `merkleRoot`). +- `HeaderStore` for the by-hash index. +- `LocalHeadersBitcoinExplorer.tipHeight()` for confirmation count. + +### New work (in `nipBCOnchainZaps/`, not in `bitcoin/`) + +- `MerkleProofVerifier.verify(txid, proofBytes, expectedMerkleRoot)` — small + pure-function module. Test-vector-driven. +- `OnchainZapVerifier` rewire to prefer the inline proof path when both + tags are present; fall back to `backend.getTx` on absence or proof + failure (never hard-reject). +- `OnchainBackend.getMerkleProof(txid)` on the send side, plus a + post-confirmation worker in `OnchainZapSender` that publishes a second + `kind:8333` once the tx confirms, this time carrying `block` + `proof`. + Receivers dedupe by `(txid, target)` preferring the variant with a + valid SPV proof. + +Detailed phase breakdown (G.1–G.7), gap matrix, and design rationale +live in the onchain-zaps plan referenced above. The previous text of +this section (BIP-37 / full-block / partial-merkle-tree options) is +obsolete now that the proof is carried inline — kept in git history if +anyone needs to revisit the assumption. ## 20. References