mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-09 08:04:45 +00:00
Merge pull request #2988 from vitorpamplona/claude/review-bitcoin-blockchain-plan-6NGZ5
Plan: local Bitcoin headers explorer for NIP-03 OTS (revised 2026-05-19)
This commit is contained in:
@@ -19,10 +19,15 @@ rest of the system.
|
||||
there. `NostrSignerRemote` (NIP-46) is deferred.
|
||||
- **Chain backend.** User-configured Esplora-compatible API
|
||||
(mempool.space, blockstream.info, self-hosted). The configured server sees
|
||||
the user's UTXO queries — accepted tradeoff for v1. Header-only SPV mode is
|
||||
a future-phase add. The explorer endpoint is shared with OpenTimestamps via
|
||||
`BitcoinExplorerEndpoint`: same user-configured server, same Tor-aware
|
||||
default selection.
|
||||
the user's UTXO queries — accepted tradeoff for v1. **Spec PR
|
||||
nostr-protocol/nips#2332 adds optional `["block", …]` and
|
||||
`["proof", …]` tags on `kind:8333` that let a verifier with only the
|
||||
Bitcoin header chain check a zap end-to-end without trusting the
|
||||
explorer. Amethyst already parses both tags (see `BlockTag.kt` /
|
||||
`ProofTag.kt`) but no production code path produces or consumes them
|
||||
yet — see the "Inline SPV proofs" section below.** The explorer
|
||||
endpoint is shared with OpenTimestamps via `BitcoinExplorerEndpoint`:
|
||||
same user-configured server, same Tor-aware default selection.
|
||||
- **Scope.** Full send + receive + display loop on Android. Desktop is out of
|
||||
scope for v1.
|
||||
|
||||
@@ -232,6 +237,117 @@ Lightning fast path. Two minimal hooks:
|
||||
2. **NIP-46 `sign_psbt`.** `NostrSignerRemote.signPsbt` still throws
|
||||
`UnsupportedMethodException` — the bunker-side command is not standardized
|
||||
yet.
|
||||
3. **Send-side `block` + `proof` emit.** See "Inline SPV proofs" below.
|
||||
4. **Receive-side SPV verification.** See "Inline SPV proofs" below.
|
||||
|
||||
## Inline SPV proofs (`block` + `proof` tags) — pending
|
||||
|
||||
### Spec status (2026-05-19)
|
||||
|
||||
`nostr-protocol/nips#2332` adds two new optional tags on `kind:8333`:
|
||||
|
||||
```
|
||||
["block", "<block-hash-hex>", "<height>"]
|
||||
["proof", "<raw-tx-hex>", "<merkle-proof-hex>"]
|
||||
```
|
||||
|
||||
Together they let a verifier with only the Bitcoin header chain check a
|
||||
zap end-to-end without trusting the chain backend. The recipient hashes
|
||||
`raw-tx-hex` to recover the txid, walks the merkle proof from that leaf
|
||||
up to the header's `merkleRoot`, parses outputs from `raw-tx-hex`, and
|
||||
sums those that pay the recipient's derived Taproot script.
|
||||
|
||||
**One spec ambiguity outstanding** (raised back to the PR author):
|
||||
direction-at-each-step encoding for `<merkle-proof-hex>` — Bitcoin's
|
||||
merkle tree is tx-order-sensitive, so concatenated siblings alone don't
|
||||
specify left vs right. Suggested resolution: `1-byte direction || 32-byte
|
||||
sibling` per step (33 B stride). This blocks Phase G.2 (walker
|
||||
implementation), nothing else.
|
||||
|
||||
### Current state in Amethyst
|
||||
|
||||
| Layer | Status | Location |
|
||||
|---|---|---|
|
||||
| `BlockTag(blockHashHex, height: Long)` | ✅ shipped | `quartz/.../nipBCOnchainZaps/zap/tags/BlockTag.kt` |
|
||||
| `ProofTag(rawTxHex, merkleProofHex)` | ✅ shipped | `…/zap/tags/ProofTag.kt` |
|
||||
| `OnchainZapEvent.block()` / `.proof()` accessors | ✅ shipped | `…/zap/OnchainZapEvent.kt:89-93` |
|
||||
| TagArray + TagArrayBuilder helpers | ✅ shipped | `…/zap/TagArrayExt.kt`, `…/zap/TagArrayBuilderExt.kt` |
|
||||
| Tx parser + witness-stripped txid (reusable for the walker) | ✅ shipped | `…/psbt/BitcoinTransaction.kt` |
|
||||
| Recipient scriptPubKey derivation | ✅ shipped | `taproot/TaprootAddress.scriptPubKeyHexForRecipient` |
|
||||
| `OnchainZapSender` populates `block` / `proof` after broadcast | ❌ gap | `commons/.../onchain/OnchainZapSender.kt:209-214` (calls `build()` / `buildProfileZap()` with no SPV tags) |
|
||||
| `OnchainBackend.getMerkleProof(txid)` to source the proof on send | ❌ gap | `chain/OnchainBackend.kt` |
|
||||
| Merkle-proof parser + walker | ❌ gap | new `verify/MerkleProofVerifier.kt` |
|
||||
| `OnchainZapVerifier` prefers inline proof when present | ❌ gap | `verify/OnchainZapVerifier.kt` |
|
||||
| Validated `header.merkleRoot` lookup by block hash | ❌ gap | depends on the headers-explorer plan (S1) at `quartz/plans/2026-05-08-local-headers-explorer.md` |
|
||||
|
||||
### Send-side: Design B (two-publish)
|
||||
|
||||
`OnchainZapSender` publishes the `kind:8333` receipt **immediately** after
|
||||
`backend.broadcast(rawTxHex)` returns, when confirmations = 0. The chosen
|
||||
design keeps that instant UX and adds a second publish after the tx
|
||||
confirms:
|
||||
|
||||
1. Today's path stays. Publish `kind:8333` right after broadcast, with
|
||||
`i`/`p`/`amount`/`alt` only — no `block`, no `proof`.
|
||||
2. A new post-confirmation worker watches the tx via the backend. When it
|
||||
sees ≥1 confirmation:
|
||||
- Call `backend.getMerkleProof(txid)` → `(block_hash, height, siblings, pos)`.
|
||||
- Encode `siblings + pos` into the spec's `<merkle-proof-hex>` wire format
|
||||
(depends on the resolved spec encoding).
|
||||
- Build a second `kind:8333` referencing the same `i` tag, with `block`
|
||||
and `proof` filled in. Publish.
|
||||
3. Verifiers MUST dedupe by `(txid, target)` and prefer the variant
|
||||
carrying a valid SPV proof. (This dedupe rule is also in the suggested
|
||||
spec amendment.)
|
||||
|
||||
Two relay broadcasts per zap, ~2 KB extra each. The post-confirmation
|
||||
worker is per-account, persists across app restarts, and exits cleanly
|
||||
once each pending tx is either confirmed (publish second receipt) or
|
||||
abandoned (after some timeout — TBD).
|
||||
|
||||
### Receive-side
|
||||
|
||||
`OnchainZapVerifier.verify(event)` gets a new fast path:
|
||||
|
||||
1. If `event.block() != null && event.proof() != null`:
|
||||
- Look up header by hash via `LocalHeadersBitcoinExplorer.byHash()` (S1).
|
||||
If unavailable yet, fall through to the existing `backend.getTx()` path.
|
||||
- `BitcoinTransaction.parse(rawTxHex)` → tx; recompute
|
||||
`dsha256(stripWitness(tx))` → expected_txid; check it matches the
|
||||
`i` tag's txid.
|
||||
- `MerkleProofVerifier.verify(expected_txid, proofBytes, header.merkleRoot)` → boolean.
|
||||
- Parse outputs from the same parsed tx; sum outputs paying the
|
||||
recipient's `scriptPubKeyHexForRecipient`, skipping outputs paying
|
||||
the sender's own derived script (change).
|
||||
- Confirmations = `tip.height − block.height + 1`.
|
||||
- Return `Confirmed`.
|
||||
2. If the inline path fails verification (bad proof, missing header,
|
||||
tx-mismatch), the event is treated as if `block` / `proof` were
|
||||
absent — fall back to the existing Esplora path. **Never** treat a
|
||||
failed SPV proof as a hard-reject of the whole event; that would let
|
||||
a buggy producer poison legitimate zaps.
|
||||
3. If neither tag is present (old-shape events), the existing
|
||||
`backend.getTx()` path runs unchanged.
|
||||
|
||||
The end state: a strict-mode user with HTTP fallback disabled can verify
|
||||
any zap whose sender included `block` + `proof`, and only those.
|
||||
|
||||
### Phased delivery for the SPV work
|
||||
|
||||
| Phase | Deliverable | Depends on | Effort |
|
||||
|---|---|---|---|
|
||||
| **G.1** | `MerkleProofVerifier` in `nipBCOnchainZaps/verify/`; test vectors per the resolved spec encoding | spec encoding locked | ½ d |
|
||||
| **G.2** | `OnchainZapVerifier` consumes inline proof, falls back on failure; needs `LocalHeadersBitcoinExplorer.byHash()` from S1 | S1 Phase 2 (storage) | ½ d |
|
||||
| **G.3** | `OnchainBackend.getMerkleProof(txid)` interface + `EsploraBackend` impl (`/tx/{txid}/merkle-proof`) | — | ½ d |
|
||||
| **G.4** | Post-confirmation worker + second-publish flow in `OnchainZapSender` | G.3 | 1 d |
|
||||
| **G.5** | Dedupe by `(txid, target)` preferring SPV-attached variant; `Note.zapsAmount` only counts the winner | — | ½ d |
|
||||
| **G.6** | End-to-end tests: real mainnet block + proof + zap; negative cases (wrong root, truncated, wrong direction) | G.1 | 1 d |
|
||||
| **G.7** | (Optional) Settings toggle "Verify zaps trustlessly when possible" — off by default until S1 ships | S1 Phase 7 | ½ d |
|
||||
| **Total** | | | ~3–4 d (after S1 ships and spec encoding lands) |
|
||||
|
||||
S1 (the headers explorer at `quartz/plans/2026-05-08-local-headers-explorer.md`)
|
||||
is the long pole. The SPV work itself is small — every primitive needed
|
||||
already exists in the codebase, the only new thing is the merkle walker.
|
||||
|
||||
## Risks / open questions
|
||||
|
||||
|
||||
@@ -1,10 +1,42 @@
|
||||
# Local Bitcoin headers explorer for NIP-03 OTS
|
||||
|
||||
**Date:** 2026-05-08
|
||||
**Branch:** `claude/review-ots-blockchain-deps-bKns7`
|
||||
**Date:** 2026-05-08 (revised 2026-05-19)
|
||||
**Branch:** `claude/review-bitcoin-blockchain-plan-6NGZ5` (originally `claude/review-ots-blockchain-deps-bKns7`)
|
||||
**Module:** `quartz/` (with thin Android wiring in `amethyst/`)
|
||||
**Status:** Plan — not yet implemented
|
||||
|
||||
## Revision log
|
||||
|
||||
**2026-05-19 — decisions locked, scope tightened.** After review against the
|
||||
intervening 2026-05-14 onchain-zaps work:
|
||||
|
||||
- **Scope stays OTS-only.** Trustless verification of NIP-BC onchain zaps
|
||||
(kind 8333) needs more than headers — at minimum BIP-37 merkleblock or
|
||||
full-block fetch on top of this stack. That becomes a follow-up plan
|
||||
(see §20).
|
||||
- **Module moves to `quartz/.../bitcoin/`** — sibling of `nip03Timestamp/`
|
||||
and `nipBCOnchainZaps/`, not nested under NIP-03. The future merkle-block
|
||||
phase will reuse `BitcoinPeer` / `HeaderStore` / `HeaderValidator`.
|
||||
- **Storage is SQLite** via `androidx.sqlite` + `BundledSQLiteDriver`,
|
||||
matching the existing `SQLiteEventStore` (`nip01Core/store/sqlite/`).
|
||||
No flat append-only file; no hand-rolled height index. See §9.
|
||||
- **No bundled headers file.** Ship a single hardcoded `Checkpoint(height,
|
||||
blockHash, chainWork, time)` constant, bumped at release. First-run sync
|
||||
starts at the checkpoint and pulls forward to the tip over P2P. See §7.
|
||||
- **Pre-checkpoint OTS proofs fall through to HTTP** (`OkHttpBitcoinExplorer`).
|
||||
No backward-from-checkpoint sync. Strict-mode users get a clear error
|
||||
instead of opening an HTTP connection.
|
||||
- **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is
|
||||
now the shared Esplora-URL resolver for OTS and onchain zaps. The
|
||||
composite resolver builder must read from it for the HTTP fallback path.
|
||||
- **Interop test vectors** (§14a) added — every consensus-relevant layer
|
||||
(header parser, compact-target, retargeter, P2P codec, OTS proofs) is
|
||||
pinned to upstream vectors committed under
|
||||
`quartz/src/commonTest/resources/bitcoin/`. A nightly differential test
|
||||
asserts `LocalHeadersBitcoinExplorer.blockHash(h)` ==
|
||||
`OkHttpBitcoinExplorer.blockHash(h)` for every height in
|
||||
`[checkpoint, tip]`, catching any consensus-rule drift end-to-end.
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
NIP-03 (OpenTimestamps attestations, kind 1040) is currently verified by hitting
|
||||
@@ -42,8 +74,9 @@ proof-of-work chain instead of a trusted explorer.
|
||||
verification logic.
|
||||
- Work over Tor when the user has Tor enabled.
|
||||
- Keep existing HTTP explorers as a fallback for proofs older than the
|
||||
bundled checkpoint, or when local sync hasn't completed yet.
|
||||
- Live in `quartz/` so `cli/` and `desktopApp/` benefit too.
|
||||
pinned checkpoint, or when local sync hasn't completed yet.
|
||||
- Live in `quartz/src/commonMain/.../bitcoin/` so `cli/`, `desktopApp/` and
|
||||
any future merkle-block / onchain-zap work inherit the same primitives.
|
||||
|
||||
### Non-goals
|
||||
|
||||
@@ -70,12 +103,13 @@ proof-of-work chain instead of a trusted explorer.
|
||||
┌──────────────┼──────────────┐
|
||||
│ │ │
|
||||
OkHttpBitcoinExplorer │ LocalHeadersBitcoinExplorer ◄── NEW
|
||||
(existing, fallback) │ │
|
||||
│ ▼
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ HeaderStore │
|
||||
│ │ (append-only file + │
|
||||
│ │ height index) │
|
||||
(existing, fallback, │ │
|
||||
URL from │ ▼
|
||||
BitcoinExplorer- │ ┌─────────────────────────┐
|
||||
Endpoint) │ │ HeaderStore │
|
||||
│ │ (SQLite, androidx- │
|
||||
│ │ sqlite + Bundled- │
|
||||
│ │ SQLiteDriver) │
|
||||
│ └─────────────────────────┘
|
||||
│ ▲
|
||||
│ │
|
||||
@@ -105,17 +139,20 @@ proof-of-work chain instead of a trusted explorer.
|
||||
│
|
||||
CompositeBitcoinExplorer
|
||||
(try local first; if height
|
||||
not synced yet, fall back to HTTP)
|
||||
below pinned checkpoint OR not yet
|
||||
reached by sync, fall back to HTTP
|
||||
via BitcoinExplorerEndpoint)
|
||||
```
|
||||
|
||||
## 4. Module layout
|
||||
|
||||
All new code in `quartz/` so non-Android targets (CLI, Desktop) inherit it.
|
||||
Most of the protocol code is platform-independent; only socket I/O is in
|
||||
`jvmAndroid`.
|
||||
All new code in `quartz/src/commonMain/.../bitcoin/`, sibling of
|
||||
`nip03Timestamp/` and `nipBCOnchainZaps/`. Protocol code is
|
||||
platform-independent; socket I/O is the only `expect`/`actual` boundary.
|
||||
Storage uses `androidx.sqlite` which is already KMP — no platform split.
|
||||
|
||||
```
|
||||
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
|
||||
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/bitcoin/
|
||||
├── proto/
|
||||
│ ├── BitcoinNetMagic.kt # 0xf9beb4d9 mainnet
|
||||
│ ├── MessageHeader.kt # 24-byte envelope codec
|
||||
@@ -128,47 +165,50 @@ quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
|
||||
│ │ ├── PingMessage.kt
|
||||
│ │ ├── PongMessage.kt
|
||||
│ │ ├── GetHeadersMessage.kt # locator + stop hash
|
||||
│ │ └── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
|
||||
│ │ ├── HeadersMessage.kt # up to 2,000 × (80B header + varint=0)
|
||||
│ │ └── SendHeadersMessage.kt # BIP-130
|
||||
│ └── BitcoinMessage.kt # sealed interface
|
||||
├── header/
|
||||
│ ├── BlockHeader80.kt # 80-byte parsed header (ver, prev, merkle, time, bits, nonce)
|
||||
│ ├── BlockHash.kt # value class wrapping ByteArray(32)
|
||||
│ ├── BlockHash.kt # @JvmInline value class wrapping ByteArray(32)
|
||||
│ ├── HeaderHasher.kt # dsha256 over 80B → blockHash
|
||||
│ ├── DifficultyTarget.kt # bits ↔ target (256-bit) compact form
|
||||
│ ├── ChainWork.kt # cumulative work accumulator
|
||||
│ ├── ChainWork.kt # cumulative work accumulator (uint256 BE)
|
||||
│ └── PowValidator.kt # checks hash ≤ target
|
||||
├── consensus/
|
||||
│ ├── RetargetCalculator.kt # every 2016 blocks, BIP-? rules
|
||||
│ ├── MedianTimePast.kt # last-11 median, for time sanity
|
||||
│ └── HeaderValidator.kt # composes the above + linkage check
|
||||
├── store/
|
||||
│ ├── HeaderStore.kt # append-only file + height→offset index
|
||||
│ ├── HeaderStore.kt # public API: append/getByHeight/getByHash/tip
|
||||
│ ├── SqliteHeaderStore.kt # androidx.sqlite impl (commonMain — driver-agnostic)
|
||||
│ ├── HeaderStoreSchema.kt # IModule for schema + migrations (see §9)
|
||||
│ ├── Checkpoint.kt # (height, blockHash, chainWork, time)
|
||||
│ └── HeaderRecord.kt # 80B header + cached blockHash + cumulativeChainWork
|
||||
│ └── PinnedCheckpoint.kt # hardcoded release-time checkpoint constant
|
||||
├── sync/
|
||||
│ ├── LocatorBuilder.kt # exponential locator
|
||||
│ ├── HeadersSyncEngine.kt # drives getheaders loop, applies validator, persists
|
||||
│ └── ReorgHandler.kt # rare; switches to higher-chainwork tip
|
||||
├── peer/
|
||||
│ ├── PeerSocket.kt # expect (open / send / recv / close)
|
||||
│ ├── PeerPool.kt # holds N BitcoinPeer connections
|
||||
│ ├── BitcoinPeer.kt # high-level: handshake, send msg, receive msg
|
||||
│ └── PeerScoring.kt # ban misbehaving peers
|
||||
└── LocalHeadersBitcoinExplorer.kt # implements quartz.../ots/BitcoinExplorer
|
||||
│ ├── PeerScoring.kt # ban misbehaving peers
|
||||
│ └── PeerDiscovery.kt # expect — DNS seeds + fixed seeds
|
||||
└── LocalHeadersBitcoinExplorer.kt # implements nip03Timestamp.ots.BitcoinExplorer
|
||||
# (with small LRU on top, like OtsBlockHeightCache)
|
||||
|
||||
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
|
||||
├── peer/
|
||||
│ ├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket)
|
||||
│ └── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
|
||||
└── store/
|
||||
└── FileHeaderStore.jvmAndroid.kt # RandomAccessFile-backed HeaderStore
|
||||
|
||||
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/bitcoin/
|
||||
└── PeerSocket.kt # expect (open / send / recv / close)
|
||||
└── HeaderStorage.kt # expect (append, getByHeight, tip)
|
||||
quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/bitcoin/
|
||||
└── peer/
|
||||
├── TcpPeerSocket.jvmAndroid.kt # actual TCP socket (java.net.Socket / SOCKS)
|
||||
└── DnsSeedDiscovery.jvmAndroid.kt # InetAddress.getAllByName on seed hostnames
|
||||
```
|
||||
|
||||
The `expect`/`actual` split keeps protocol logic testable cross-platform; the
|
||||
socket layer and disk layer are the only platform-dependent pieces.
|
||||
Two expect/actual seams only: `PeerSocket` (TCP plus optional SOCKS5 for Tor)
|
||||
and `PeerDiscovery` (DNS-seed resolution). Header validation, message codecs,
|
||||
store, sync engine, peer pool — all compile in `commonMain`, all testable
|
||||
cross-platform without a single platform shim. Storage compiles in
|
||||
`commonMain` too because `androidx.sqlite` + `BundledSQLiteDriver` are KMP.
|
||||
|
||||
### Wiring on Android
|
||||
|
||||
@@ -182,7 +222,17 @@ amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/
|
||||
|
||||
`AppModules.kt` switches from `TorAwareOkHttpOtsResolverBuilder` to a
|
||||
composite that prefers the local explorer and falls back to HTTP only when
|
||||
local is missing the height.
|
||||
local is missing the height. The HTTP path resolves its base URL via the
|
||||
existing `BitcoinExplorerEndpoint` (shared with the onchain-zap
|
||||
`EsploraBackend`), so a user-configured custom Esplora is honoured on both
|
||||
the OTS and the zap side.
|
||||
|
||||
`AppModules.kt:433` already passes `rootFilesDir = { appContext.filesDir }`
|
||||
into Quartz; the headers SQLite file goes under
|
||||
`$filesDir/bitcoin/headers.db` via a new entry in that wiring.
|
||||
|
||||
On desktop and CLI, the equivalent data-dir paths are already plumbed for
|
||||
`SQLiteEventStore` — same hook reused here.
|
||||
|
||||
## 5. Bitcoin P2P protocol — minimum we implement
|
||||
|
||||
@@ -274,23 +324,35 @@ mirroring Bitcoin Core's own historical checkpoints, plus one **release-pinned
|
||||
checkpoint** updated at app build time near the current tip. Sync must agree
|
||||
with all checkpoints — disagreement aborts and bans the peer.
|
||||
|
||||
## 7. Bundled data
|
||||
## 7. Bundled data — none. Just a pinned checkpoint constant.
|
||||
|
||||
To avoid making every fresh install download 75 MB:
|
||||
The point of this work is "verify against the Bitcoin network, not a trusted
|
||||
party," so we ship no headers in the APK. Instead:
|
||||
|
||||
- App ships an **assets file** `bitcoin_headers_<height>.bin`: raw 80-byte
|
||||
headers from genesis up to a release-pinned height (≤ 1 month before build).
|
||||
~72 MB — too large for Play Store delta-update friendliness.
|
||||
- Ship instead a **compact bundled checkpoint**: just one `Checkpoint(height,
|
||||
blockHash, chainWork, time)` per app release, plus the most recent ~50 K
|
||||
headers (~4 MB) so users can verify any attestation from the past ~year
|
||||
immediately.
|
||||
- For older proofs, lazy-sync from the checkpoint *backwards* down to genesis
|
||||
on demand (rare path).
|
||||
- Forward sync from checkpoint to current tip is what runs at first launch
|
||||
(~10 KB to a few MB depending on release age).
|
||||
- **One hardcoded `PinnedCheckpoint` constant** in `bitcoin/store/PinnedCheckpoint.kt`:
|
||||
`(height, blockHash, chainWork, time)`. ~80 bytes of source.
|
||||
- The checkpoint is pinned ~1 month before each release and bumped via a
|
||||
one-line PR per release. This is the **only** thing trusted at build time
|
||||
— and it's a single block hash any user can independently verify against
|
||||
any Bitcoin node before installing.
|
||||
- **First-run sync** starts at the checkpoint and pulls forward to the
|
||||
current tip over P2P. At a ~1-month checkpoint that's ~4,300 headers
|
||||
(~340 KB on the wire, ~15 minutes' worth of `getheaders` round-trips at
|
||||
worst), seconds over WiFi.
|
||||
- **Pre-checkpoint heights** are not synced locally at all. OTS proofs
|
||||
whose height precedes the checkpoint fall through to
|
||||
`OkHttpBitcoinExplorer` via the composite. Users in strict-mode
|
||||
(`localHeadersTrustOnly = true`) get a clear
|
||||
"this OTS proof predates the bundled checkpoint" error and no network call.
|
||||
- **Catch-up after long absence.** A device that hasn't run the app for
|
||||
months will, on next sync, fast-forward from `tip_local` (its stored
|
||||
best_height) to `tip_network`. Each `headers` message brings 2,000
|
||||
headers; even 6 months of absence is ~26,000 headers (~2 MB) in 13 round
|
||||
trips.
|
||||
- **Header growth on disk** is ~4 MB raw per year (+ SQLite overhead, see §9).
|
||||
|
||||
This keeps APK growth at ~4 MB and first-run sync time at seconds.
|
||||
APK growth: **0 bytes**. Trust delta vs the old HTTP-explorer model: one
|
||||
block hash, code-reviewable in git.
|
||||
|
||||
## 8. Multi-peer strategy & eclipse mitigation
|
||||
|
||||
@@ -315,42 +377,85 @@ A single peer can lie. We protect against that with:
|
||||
(`bitcoin.example.com:8333`) — same path as the existing custom explorer
|
||||
URL setting in `OtsSettings.kt`.
|
||||
|
||||
## 9. Storage format
|
||||
## 9. Storage — SQLite via `androidx.sqlite` + `BundledSQLiteDriver`
|
||||
|
||||
Append-only flat file plus a sparse index.
|
||||
We reuse the same storage primitive Quartz already uses for `SQLiteEventStore`
|
||||
(`quartz/.../nip01Core/store/sqlite/`): `androidx.sqlite` with the bundled
|
||||
driver. This is KMP, ships in `androidx-sqlite-bundled` / `…-bundled-jvm`
|
||||
(already a Quartz dependency), and is the project's standard storage choice.
|
||||
|
||||
### `headers.bin`
|
||||
The complete `SQLiteEventStore` infrastructure — `BundledSQLiteDriver`,
|
||||
`SQLiteConnectionPool`, `transaction { }`, `IModule` for migrations,
|
||||
`QueryBuilder` helpers — is reused by `SqliteHeaderStore` and
|
||||
`HeaderStoreSchema`.
|
||||
|
||||
Records, fixed 112 bytes each, indexed by height (record N at offset `N*112`):
|
||||
### Schema
|
||||
|
||||
```
|
||||
+0 80B raw block header
|
||||
+80 32B cumulative chainwork (uint256, big-endian)
|
||||
```sql
|
||||
PRAGMA journal_mode = WAL;
|
||||
PRAGMA synchronous = NORMAL;
|
||||
PRAGMA foreign_keys = OFF;
|
||||
|
||||
CREATE TABLE block_header (
|
||||
height INTEGER PRIMARY KEY, -- 0..tip, dense
|
||||
hash BLOB NOT NULL, -- 32 B, dsha256(header), big-endian display order
|
||||
header BLOB NOT NULL, -- 80 B raw header
|
||||
chainwork BLOB NOT NULL -- 32 B cumulative work, big-endian
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX idx_block_header_hash ON block_header(hash);
|
||||
|
||||
CREATE TABLE chain_tip (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1), -- singleton row
|
||||
best_height INTEGER NOT NULL,
|
||||
best_hash BLOB NOT NULL,
|
||||
best_work BLOB NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE checkpoint_seen (
|
||||
height INTEGER PRIMARY KEY, -- which checkpoints we've passed and confirmed
|
||||
hash BLOB NOT NULL,
|
||||
source TEXT NOT NULL -- 'pinned' | 'historical'
|
||||
);
|
||||
```
|
||||
|
||||
The block hash and target are recomputed from bytes 0..80 lazily; we don't
|
||||
store them. Total at height 1M: 112 MB. We can drop this to 80 B/record
|
||||
(no chainwork) if we accept recomputing chainwork on startup — TBD; default
|
||||
is 112 B for fast tip selection.
|
||||
Migrations live in `HeaderStoreSchema.kt` as an `IModule` (same pattern as
|
||||
the event store's modules). v1 is the schema above; future additions
|
||||
(prev-hash cache column, merkle-block annotations) ALTER from there.
|
||||
|
||||
### `tips.json`
|
||||
### Sizing
|
||||
|
||||
A small JSON file holding:
|
||||
| Item | Per record | At height 1,000,000 |
|
||||
|---|---|---|
|
||||
| `block_header` row (80+32+32 B payload + SQLite overhead) | ~180–220 B | ~180–220 MB |
|
||||
| `idx_block_header_hash` B-tree (32 B key + rowid) | ~40 B | ~40 MB |
|
||||
| `chain_tip` | 1 row | negligible |
|
||||
| **Total** | | **~220–260 MB at height 1 M** |
|
||||
|
||||
```json
|
||||
{
|
||||
"best_height": 945123,
|
||||
"best_hash": "00000000000000000001abc...",
|
||||
"best_work": "0x000...",
|
||||
"checkpoints_passed": [600000, 700000, 800000, 940000]
|
||||
}
|
||||
```
|
||||
In practice users will start from a recent checkpoint and grow ~4 MB raw
|
||||
(~10 MB stored) per year, so most installs sit at 10–40 MB. The "1 M
|
||||
height" line is the long-tail worst case if someone with a multi-year-old
|
||||
device's local store retroactively backfills (which we explicitly don't do
|
||||
in v1 — see §15 Q1).
|
||||
|
||||
### Atomicity
|
||||
### Atomicity & crash safety
|
||||
|
||||
Sync writes to `headers.bin.tmp`, fsyncs, then atomically renames or
|
||||
appends-and-fsyncs in batches of 2000. Crash recovery: on startup, scan from
|
||||
last fsynced offset, validate linkage, and discard partial trailing records.
|
||||
- `HeadersSyncEngine` writes each `headers` batch (up to 2,000) inside a
|
||||
single `transaction { }`. WAL gives us atomic-or-nothing per batch.
|
||||
- `chain_tip` is updated in the same transaction as the headers it points to,
|
||||
so the tip can never reference rows that didn't commit.
|
||||
- A crash mid-batch leaves the WAL inconsistent; SQLite rolls it back on
|
||||
next open. No custom recovery code.
|
||||
- On open, `SqliteHeaderStore.verifyTipIntegrity()` re-reads the tip row and
|
||||
confirms the referenced header exists. If not (unexpected DB corruption,
|
||||
ransomware-like sandbox damage), we delete the db file and fall back to
|
||||
re-sync from the pinned checkpoint.
|
||||
|
||||
### Concurrency
|
||||
|
||||
Same model as `SQLiteEventStore`: one connection pool, all writes go through
|
||||
the sync engine on a dedicated dispatcher; reads use the pool's read slots.
|
||||
OTS verifier reads are short and indexed — no contention with sync writes.
|
||||
|
||||
## 10. Resolving a height query (the actual `BitcoinExplorer` impl)
|
||||
|
||||
@@ -410,20 +515,25 @@ Extend `OtsSettings.kt`:
|
||||
|
||||
```kotlin
|
||||
data class OtsSettings(
|
||||
val customExplorerUrl: String? = null,
|
||||
val customExplorerUrl: String? = null, // existing — shared via BitcoinExplorerEndpoint
|
||||
// NEW:
|
||||
val localHeadersEnabled: Boolean = true,
|
||||
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback
|
||||
val localHeadersTrustOnly: Boolean = false, // disables HTTP fallback; pre-checkpoint proofs error
|
||||
val localHeadersWifiOnly: Boolean = true,
|
||||
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
|
||||
val customP2pNode: String? = null, // e.g. "myriad.example.com:8333"
|
||||
)
|
||||
```
|
||||
|
||||
A short "Bitcoin verification" settings screen lets the user pick:
|
||||
- Local headers (default)
|
||||
- HTTP explorer only (current behavior, lighter)
|
||||
- Local + HTTP fallback
|
||||
- Local only (strict, may delay verification of brand-new attestations)
|
||||
- **Local headers + HTTP fallback** (default) — best UX, trustless for
|
||||
post-checkpoint heights, HTTP for older proofs.
|
||||
- **Local headers only (strict)** — refuses to verify proofs older than the
|
||||
pinned checkpoint. Most paranoid setting.
|
||||
- **HTTP explorer only** — pre-local-headers behaviour. Lightest disk/CPU.
|
||||
|
||||
The `customExplorerUrl` field continues to feed `BitcoinExplorerEndpoint`,
|
||||
so a custom HTTP Esplora is honoured both by the fallback in this feature
|
||||
and by the onchain-zaps `EsploraBackend`. No new Esplora setting.
|
||||
|
||||
## 12. Tor integration
|
||||
|
||||
@@ -461,10 +571,15 @@ Each phase produces a shippable artifact with measurable behavior.
|
||||
|
||||
### Phase 2 — Storage layer (2–3 days)
|
||||
|
||||
- `HeaderStore` interface + `FileHeaderStore` actual on jvmAndroid.
|
||||
- Append, get-by-height, get-by-hash (in-memory hash→height map built at
|
||||
startup, ~60 MB at height 1 M — acceptable, or use a sparse on-disk index).
|
||||
- Crash-recovery test.
|
||||
- `HeaderStore` interface + `SqliteHeaderStore` implementation in `commonMain`
|
||||
(`androidx.sqlite` is KMP, no platform shim needed).
|
||||
- `HeaderStoreSchema` as an `IModule` matching the `SQLiteEventStore`
|
||||
migration pattern.
|
||||
- `append(batch)`, `getByHeight(h)`, `getByHash(hash)`, `tip()`.
|
||||
- Reuse `BundledSQLiteDriver`, `SQLiteConnectionPool`, `transaction { }`
|
||||
from `nip01Core/store/sqlite/` — no duplication.
|
||||
- Crash-safety test: kill mid-transaction, reopen, confirm WAL rollback;
|
||||
confirm `verifyTipIntegrity()` recovers cleanly.
|
||||
|
||||
### Phase 3 — P2P codec, no socket (3–5 days)
|
||||
|
||||
@@ -488,13 +603,16 @@ Each phase produces a shippable artifact with measurable behavior.
|
||||
- Header-batch cross-validation across peers.
|
||||
- Hardcoded checkpoints in `Checkpoints.kt`.
|
||||
|
||||
### Phase 6 — Bundled checkpoint & lazy-genesis-sync (2–3 days)
|
||||
### Phase 6 — Pinned checkpoint constant (½ day)
|
||||
|
||||
- Build-time generation of the release-pinned checkpoint and trailing-50K
|
||||
headers blob, packaged as `quartz/src/jvmAndroid/resources/bitcoin/...`.
|
||||
- Forward sync from checkpoint to tip on first launch.
|
||||
- Backward (genesis-direction) sync triggered only when an OTS proof requests
|
||||
a height before the checkpoint.
|
||||
- `bitcoin/store/PinnedCheckpoint.kt` — single hardcoded
|
||||
`Checkpoint(height, blockHash, chainWork, time)`. ~80 bytes of source.
|
||||
- A small Gradle task (`./gradlew bumpBitcoinCheckpoint`) that takes
|
||||
`(height, hash)` arguments and writes the constant. Run by the release
|
||||
process; otherwise no automation.
|
||||
- No backward-sync, no bundled headers blob, no asset resources.
|
||||
Pre-checkpoint heights are routed to the HTTP fallback by the composite
|
||||
resolver (or refused in strict mode).
|
||||
|
||||
### Phase 7 — Android wiring (3–4 days)
|
||||
|
||||
@@ -512,9 +630,14 @@ Each phase produces a shippable artifact with measurable behavior.
|
||||
|
||||
### Phase 9 — Desktop & CLI integration (1–2 days)
|
||||
|
||||
- `desktopApp/` already inherits the explorer because it's in `quartz/`.
|
||||
Confirm it works end-to-end on JVM.
|
||||
- `cli/` (`amy verify-ots`) — confirm CLI sync also works headlessly.
|
||||
- `desktopApp/` inherits the explorer for free (everything is in
|
||||
`quartz/commonMain/`). Confirm headers.db path is plumbed via the same
|
||||
data-dir hook already used by `SQLiteEventStore` on JVM. Confirm sync
|
||||
runs end-to-end on the desktop app.
|
||||
- `cli/` — add an `amy verify-ots <event-json>` command if not present,
|
||||
wired through the same composite resolver. The CLI is a thin assembly
|
||||
layer over Quartz (per `amy-expert`), so the actual sync logic is reused
|
||||
as-is.
|
||||
|
||||
### Phase 10 — Rollout & telemetry (ongoing)
|
||||
|
||||
@@ -523,8 +646,9 @@ Each phase produces a shippable artifact with measurable behavior.
|
||||
fallback rate. Surface in a debug screen.
|
||||
- Default-on in the next release if metrics look healthy.
|
||||
|
||||
Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
(protocol + validation); phases 5–10 are integration.
|
||||
Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
(protocol + validation); phases 5–10 are integration. Phase 6 dropped from
|
||||
2–3 d to ½ d because we ship no bundled data — just one constant.
|
||||
|
||||
## 14. Testing strategy
|
||||
|
||||
@@ -544,8 +668,12 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
(committed as `quartz/src/jvmAndroid/test/resources/btc-handshake.bin`).
|
||||
- Local-network test against a Bitcoin Core regtest node spun up in CI
|
||||
(Docker). Validates full handshake + `getheaders` loop end to end.
|
||||
- Crash-and-resume test: kill mid-sync, restart, confirm we pick up at the
|
||||
correct height.
|
||||
- `SqliteHeaderStore` round-trip on a real on-disk file (jvmTest, in a
|
||||
`tmpdir`): append 5,000 headers in batches, getByHeight, getByHash,
|
||||
reopen, confirm tip survives.
|
||||
- Crash-and-resume test: kill the writer in the middle of a `transaction { }`,
|
||||
reopen, confirm SQLite WAL rolls the partial batch back and sync resumes
|
||||
from the last committed tip.
|
||||
|
||||
### Property tests
|
||||
|
||||
@@ -560,36 +688,132 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
- Tor on — sync completes (slowly) over Orbot.
|
||||
- Disk-full simulation — fail gracefully, don't corrupt store.
|
||||
|
||||
## 14a. Interop test vectors
|
||||
|
||||
Consensus code without external test vectors is "trust me." Every layer
|
||||
that could disagree with the rest of the Bitcoin network gets pinned to
|
||||
authoritative upstream vectors, mirroring how `nip44.vectors.json`,
|
||||
`bip39.vectors.json` and the `mls/*.json` set are already organised in
|
||||
`quartz/src/commonTest/resources/`. The onchain-zaps work uses the same
|
||||
discipline (BIP-341 wallet test vectors at every layer of
|
||||
`nipBCOnchainZaps/taproot/` + `psbt/`).
|
||||
|
||||
### Sources, by component
|
||||
|
||||
| Component | Source(s) | Format | Storage | Effort | Priority |
|
||||
|---|---|---|---|---|---|
|
||||
| **`BlockHeader80` parse/serialize** | Genesis (0), block 1, block 209999 (last pre-halving), 481824 (first post-segwit), 709631 (last pre-taproot), one recent | JSON: `[{height, hash, headerHex, version, prevHash, merkleRoot, time, bits, nonce}, …]` | commit `commonTest/resources/bitcoin/known_headers.json` | 1 d | **High** |
|
||||
| **`DifficultyTarget` compact↔target** | bitcoinj `UtilsTest.testCompactBitsToBigInteger`, btcd `chaincfg/chainhash/compact_test.go`. Vectors: `0x1d00ffff`, `0x1b0404cb`, `0x170398bb`, overflow + negative-bit edges | JSON pairs `(compact, targetHex)` | commit `bitcoin/compact_target.json` | ½ d | **High** |
|
||||
| **Retarget (`CalculateNextWorkRequired`)** | Mainnet retarget heights 2016, 4032, 32256, 60480, ~600000 — extract `(prev_retarget_time, current_time, prev_bits, expected_new_bits)` from any full node | JSON list of retarget cases | commit `bitcoin/retargets.json` (~50 entries, ~5 KB) | 1 d | **High** |
|
||||
| **`MedianTimePast`** | Any 11 consecutive mainnet headers synthesised from `known_headers.json` | inline test | inline | ½ d | Medium |
|
||||
| **HeaderValidator end-to-end on real chain** | First 2016 mainnet headers (~160 KB; covers genesis + first retarget). Larger 50 K-header set lazy-fetched in nightly CI from a pinned-hash mirror, not committed | binary blob | commit `bitcoin/first_2016_headers.bin` | 1–2 d | **High** |
|
||||
| **P2P wire codec — `version`/`verack`/`ping`/`pong`** | btcd `wire/msg*_test.go` has canonical byte strings; Bitcoin Wiki "Protocol documentation" has annotated hex | hex strings in JSON | commit `bitcoin/p2p_messages.json` | 1 d | **High** |
|
||||
| **P2P `getheaders` / `headers` round-trip** | Capture against a real bitcoind once, ~50 KB | raw bytes | commit `bitcoin/p2p_capture.bin` | 1 d | **High** |
|
||||
| **DNS seed list** | Bitcoin Core `chainparams.cpp` (current 9 seeds) | inline constant | n/a | <¼ d | Low |
|
||||
| **Hardcoded fallback IPs** | Bitcoin Core `chainparamsseeds.h` (auto-generated by `contrib/seeds/`) | binary blob | commit `bitcoin/chainparamsseeds.bin` | ½ d | Medium |
|
||||
| **Reorg / chain selection** | Synthetic 2-branch fork — verify higher-cumulative-chainwork wins | inline builder | inline | 1 d | **High** |
|
||||
| **End-to-end OTS proofs** | `python-opentimestamps/tests/test_data/*.ots` + the 3 wild events already inline in `OtsTest.kt`. Import 5–10 more `.ots` for variety (calendar operators, eras) | `.ots` binaries + JSON manifest of expected timestamps | commit `commonTest/resources/ots/*.ots` | 1 d | **High** |
|
||||
| **NIP-03 1040 events from real relays** | Scrape any relay for `kind:1040`; pick a diverse set across calendar operators and years | inline JSON | extend `OtsTest.kt` | ½ d | Medium |
|
||||
| **Differential test: local vs HTTP** | For every `h ∈ [checkpoint, tip]`: assert `LocalHeadersBitcoinExplorer.blockHash(h)` == `OkHttpBitcoinExplorer.blockHash(h)`. Catches consensus drift end-to-end | runtime check, nightly CI | no fixture | 1 d | **High** |
|
||||
| **Regtest end-to-end** | Bitcoin Core in Docker — mine N blocks, sync, assert tip matches | infrastructure | CI script | 2 d | Medium |
|
||||
| **Adversarial peer** | Synthetic peer that lies about retarget, replays old headers, drops mid-batch, sends 2,001-headers messages. Validates ban + reconnect | inline test harness | inline | 2 d | **High** |
|
||||
|
||||
### Committed layout
|
||||
|
||||
```
|
||||
quartz/src/commonTest/resources/bitcoin/
|
||||
├── README.md # provenance + bump procedure for each file
|
||||
├── known_headers.json # ~10 hand-picked mainnet headers across history
|
||||
├── compact_target.json # ~20 compact-bits ↔ target pairs
|
||||
├── retargets.json # ~50 retarget cases from historical mainnet
|
||||
├── first_2016_headers.bin # ~160 KB; genesis + first retarget cycle
|
||||
├── p2p_messages.json # hex of canonical version/verack/ping/pong
|
||||
├── p2p_capture.bin # captured handshake + getheaders/headers (~50 KB)
|
||||
└── chainparamsseeds.bin # current Bitcoin Core fallback IP list
|
||||
|
||||
quartz/src/commonTest/resources/ots/
|
||||
├── manifest.json # {filename → expected unix timestamp | "pending"}
|
||||
└── *.ots # 5–10 fixtures from python-opentimestamps
|
||||
```
|
||||
|
||||
### Provenance discipline
|
||||
|
||||
The `README.md` next to each fixture records, in this order:
|
||||
|
||||
1. **Where it came from** — upstream URL + commit SHA.
|
||||
2. **How to regenerate** — a single command, no manual steps.
|
||||
3. **What it tests** — one sentence.
|
||||
4. **When to bump** — e.g. *"retargets.json is stable; never bumps."* /
|
||||
*"chainparamsseeds.bin bumped each Bitcoin Core major release."*
|
||||
|
||||
Same pattern as the header comment of `nip44.vectors.json`. New committers
|
||||
should never have to ask "where did this 32-byte blob come from."
|
||||
|
||||
### Phase mapping
|
||||
|
||||
These test imports map cleanly onto the existing phases — total extra
|
||||
effort **~5–7 engineer-days** spread across the work, no new top-level
|
||||
phase needed:
|
||||
|
||||
| Phase | Vectors consumed |
|
||||
|---|---|
|
||||
| **Phase 1** (header parsing & validation, no I/O) | `known_headers.json`, `compact_target.json`, `retargets.json`, `first_2016_headers.bin` |
|
||||
| **Phase 3** (P2P codec, no socket) | `p2p_messages.json`, `p2p_capture.bin` |
|
||||
| **Phase 4** (single-peer sync) | adversarial-peer harness + regtest-in-Docker |
|
||||
| **Phase 5** (multi-peer & eclipse) | `chainparamsseeds.bin` |
|
||||
| **Phase 7** (Android wiring) | differential test against `OkHttpBitcoinExplorer` |
|
||||
| **Phase 9** (CLI / Desktop) | OTS `.ots` fixtures via `amy verify-ots --explorer=local` |
|
||||
|
||||
### Why the differential test matters most
|
||||
|
||||
The single highest-value test in the list is the differential one: for
|
||||
every height between the pinned checkpoint and the current tip, the local
|
||||
explorer's answer must equal the HTTP explorer's answer. Run it nightly in
|
||||
CI. Any consensus-rule drift — a wrong retarget formula, a missed
|
||||
overflow case in compact-target decoding, a partial-merkle bug — will
|
||||
surface as a height where the two disagree, and the height is the bug
|
||||
report. This is the test that catches the bug nobody thought to write
|
||||
a unit test for.
|
||||
|
||||
## 15. Risks & open questions
|
||||
|
||||
| # | Risk | Mitigation |
|
||||
|---|------|------------|
|
||||
| R1 | First-run UX cost (1–10 min sync) puts users off | Bundled checkpoint + 50 K trailing headers means most installs need only seconds. Show progress prominently. |
|
||||
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design. |
|
||||
| R1 | First-run UX cost (sync from pinned checkpoint) | Pinned checkpoint ~1 month before release means ~4 K headers, ~340 KB, seconds over WiFi. Show progress in settings; sync runs in background WorkManager job anyway. |
|
||||
| R2 | Mobile OS kills socket mid-sync | Foreground service for first run; resumable design (SQLite WAL handles partial batches automatically). |
|
||||
| R3 | Peer feeds adversarial headers (eclipse) | Multi-peer + checkpoints + chainwork comparison. |
|
||||
| R4 | Disk corruption | Atomic appends, checksum on close, fall back to re-sync from last checkpoint. |
|
||||
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer when sync stalls. |
|
||||
| R6 | APK size growth from bundled headers | Cap bundle at most-recent ~50 K headers (~4 MB). Older headers fetched on demand. |
|
||||
| R4 | Disk corruption | SQLite WAL + `verifyTipIntegrity()` on open; on detected corruption, delete db file and re-sync from pinned checkpoint. |
|
||||
| R5 | Tor + P2P is slow | Increased timeouts; fallback to HTTP-over-Tor explorer (via `BitcoinExplorerEndpoint`) when sync stalls. |
|
||||
| R6 | APK size growth | Zero — no bundled data ships in the APK. |
|
||||
| R7 | Privacy: peers see user's IP | Tor mode supported; `version` user-agent doesn't leak Amethyst install ID. |
|
||||
| R8 | Battery cost on background sync | Default to UNMETERED + BATTERY_OK constraints; back off when idle. |
|
||||
| R9 | Maintenance burden — new soft-forks | Soft-forks don't affect headers-only validation (no script eval). Difficulty algorithm hasn't changed; a future change would require a code update. |
|
||||
| R10 | Test flakiness if CI talks to public Bitcoin nodes | Use Bitcoin Core regtest in Docker for integration tests; public nodes only nightly. |
|
||||
| R11 | SQLite store size on long-running installs | ~10 MB/year typical; ~220 MB at height 1 M which won't happen in any realistic mobile lifetime. Add a "wipe local headers" button in settings for users who want it. |
|
||||
|
||||
### Open questions
|
||||
|
||||
- **Q1.** Do we actually want backward-from-checkpoint sync, or just refuse
|
||||
to verify proofs older than the bundled checkpoint and tell users to use
|
||||
the HTTP fallback? Backward sync is operational complexity for a rare path.
|
||||
- **Q2.** Storage: 80 B/record (recompute chainwork on startup) vs
|
||||
112 B/record (cache it)? At height 1M, 32 MB difference. Probably worth
|
||||
caching.
|
||||
- **Q3.** Should we expose this as a Quartz public API (so 3rd-party apps
|
||||
using Quartz also benefit) or keep it internal to NIP-03? Recommendation:
|
||||
public — it's generally useful.
|
||||
- **Q4.** Do we ship a "headers blob" download URL (HTTPS) as a faster
|
||||
alternative to P2P for first-run? Trade-off: faster but adds a trust point.
|
||||
Could be made cross-checked (download blob, then verify every header's PoW
|
||||
locally — same trust as P2P then). Worth considering as Phase 6b.
|
||||
#### Resolved 2026-05-19
|
||||
|
||||
- **Q1.** *Resolved — HTTP fallback only.* Pre-checkpoint OTS proofs are
|
||||
routed to `OkHttpBitcoinExplorer` via the composite resolver. Strict-mode
|
||||
users (`localHeadersTrustOnly = true`) get a clear error. No backward
|
||||
sync code path.
|
||||
- **Q2.** *Resolved — cache chainwork.* SQLite makes the choice cheap (one
|
||||
extra BLOB column). Reading current tip work without a full scan matters
|
||||
for chain-selection logic; the ~30 MB-at-1M cost is negligible on disk.
|
||||
- **Q4.** *Resolved — no HTTPS headers blob.* The point of this feature is
|
||||
trustless verification, and the pinned-checkpoint approach already gives
|
||||
fast first-run. Adding a second ingestion path (signed blob over HTTPS)
|
||||
doubles the test surface for marginal benefit.
|
||||
|
||||
#### Still open
|
||||
|
||||
- **Q3.** Should `LocalHeadersBitcoinExplorer` (and friends) be a Quartz
|
||||
public API for 3rd-party Quartz consumers, or internal to Amethyst?
|
||||
Recommendation: public — it's a clean reusable primitive. Decide before
|
||||
Phase 0 lands so the package layout doesn't churn. *(See `quartz-integration`
|
||||
skill for the public-API contract Quartz exposes today.)*
|
||||
|
||||
## 16. Migration & rollout
|
||||
|
||||
@@ -605,22 +829,74 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
- A NIP-03 attestation that verifies via Blockstream also verifies via the
|
||||
local explorer (bit-for-bit identical timestamp).
|
||||
- 95th-percentile cold-launch verification of a recent attestation: ≤2 s.
|
||||
- 95th-percentile first-run sync (post-checkpoint): ≤30 s on WiFi.
|
||||
- 95th-percentile first-run sync (from pinned checkpoint to tip): ≤30 s on
|
||||
WiFi for a checkpoint ≤2 months old.
|
||||
- No new crash-rate regression in the verification path.
|
||||
- App size growth: ≤5 MB.
|
||||
- App size growth: **0 MB** (no bundled data).
|
||||
- A user with HTTP explorer disabled can still verify any post-checkpoint
|
||||
attestation.
|
||||
|
||||
## 18. Out of scope (explicit)
|
||||
|
||||
- BIP-157/158 compact filters
|
||||
- BIP-37 merkleblock or full-block fetch
|
||||
- **Trustless verification of NIP-BC onchain zaps** — needs merkleblock or
|
||||
full-block fetch on top of this stack. See §20 for the follow-up plan.
|
||||
- **Trustless wallet operations** for NIP-BC onchain zap *sending* —
|
||||
needs UTXO-by-address and broadcast paths that headers can't provide.
|
||||
Stays on user-configured Esplora.
|
||||
- Lightning, BIP-32 wallet, mempool monitoring
|
||||
- Verifying NIP-03 attestations against altcoin/Liquid timestamps (not in
|
||||
the spec)
|
||||
- Replacing the calendar-server upgrade path (`OtsState.upgrade()`) —
|
||||
unrelated, those are simple HTTPS calls and stay as-is
|
||||
|
||||
## 19. References
|
||||
## 19. Follow-up: trustless NIP-BC onchain zap verification
|
||||
|
||||
*Not in this plan. Tracked here so the design choices above stay aligned.
|
||||
Full implementation plan lives in `amethyst/plans/2026-05-14-onchain-zaps.md`
|
||||
under "Inline SPV proofs (`block` + `proof` tags) — pending".*
|
||||
|
||||
### What the 2026-05-19 spec change buys us
|
||||
|
||||
`nostr-protocol/nips#2332` adds optional `["block", "<hash>", "<height>"]`
|
||||
and `["proof", "<raw-tx-hex>", "<merkle-proof-hex>"]` tags on `kind:8333`.
|
||||
The proof rides inline in the Nostr event, so the trustless-verify path
|
||||
needs **no new P2P messages on top of S1** — no `getdata MSG_BLOCK`, no
|
||||
BIP-37 `filterload`, no full-block parser. Just header lookup + merkle
|
||||
walk against bytes already in the event.
|
||||
|
||||
This collapses the original "follow-up plan" estimate from 10–15 days to
|
||||
**~3–4 days** of work after S1 ships and the spec's merkle-proof
|
||||
encoding is locked.
|
||||
|
||||
### Foundation reused from S1
|
||||
|
||||
- `LocalHeadersBitcoinExplorer.byHash(blockHash)` → validated header (for
|
||||
the `merkleRoot`).
|
||||
- `HeaderStore` for the by-hash index.
|
||||
- `LocalHeadersBitcoinExplorer.tipHeight()` for confirmation count.
|
||||
|
||||
### New work (in `nipBCOnchainZaps/`, not in `bitcoin/`)
|
||||
|
||||
- `MerkleProofVerifier.verify(txid, proofBytes, expectedMerkleRoot)` — small
|
||||
pure-function module. Test-vector-driven.
|
||||
- `OnchainZapVerifier` rewire to prefer the inline proof path when both
|
||||
tags are present; fall back to `backend.getTx` on absence or proof
|
||||
failure (never hard-reject).
|
||||
- `OnchainBackend.getMerkleProof(txid)` on the send side, plus a
|
||||
post-confirmation worker in `OnchainZapSender` that publishes a second
|
||||
`kind:8333` once the tx confirms, this time carrying `block` + `proof`.
|
||||
Receivers dedupe by `(txid, target)` preferring the variant with a
|
||||
valid SPV proof.
|
||||
|
||||
Detailed phase breakdown (G.1–G.7), gap matrix, and design rationale
|
||||
live in the onchain-zaps plan referenced above. The previous text of
|
||||
this section (BIP-37 / full-block / partial-merkle-tree options) is
|
||||
obsolete now that the proof is carried inline — kept in git history if
|
||||
anyone needs to revisit the assumption.
|
||||
|
||||
## 20. References
|
||||
|
||||
- NIP-03: https://github.com/nostr-protocol/nips/blob/master/03.md
|
||||
- OpenTimestamps spec: https://github.com/opentimestamps/python-opentimestamps
|
||||
@@ -632,6 +908,16 @@ Total: **~25–35 engineer-days** end to end. Phases 1–4 are the bulk
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BitcoinExplorer.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/BlockHeader.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/attestation/BitcoinBlockHeaderAttestation.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip03Timestamp/ots/OtsBlockHeightCache.kt`
|
||||
- `quartz/src/jvmAndroid/kotlin/com/vitorpamplona/quartz/nip03Timestamp/okhttp/OkHttpBitcoinExplorer.kt`
|
||||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/TorAwareOkHttpOtsResolverBuilder.kt`
|
||||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/OtsSettings.kt`
|
||||
- `amethyst/src/main/java/com/vitorpamplona/amethyst/model/nip03Timestamp/BitcoinExplorerEndpoint.kt`
|
||||
- Storage precedent (reused as-is):
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IModule.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionExt.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteConnectionPool.kt`
|
||||
- Follow-up area:
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/verify/OnchainZapVerifier.kt`
|
||||
- `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nipBCOnchainZaps/chain/OnchainBackend.kt`
|
||||
|
||||
Reference in New Issue
Block a user