From d19bcf07b07cc78d0a388af90ed1967a825e7c4a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 18:35:30 +0000 Subject: [PATCH] docs(quartz): add interop-test-vectors section to local-headers-explorer plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every consensus-relevant layer of the planned headers explorer (BlockHeader80 parser, DifficultyTarget compact↔target, CalculateNextWorkRequired retarget, MedianTimePast, header validator end-to-end, P2P wire codecs, reorg/chain selection, OTS proofs) is pinned to upstream test vectors committed under quartz/src/commonTest/resources/bitcoin/, matching the existing nip44.vectors.json / bip39.vectors.json / mls/*.json pattern. The single highest-value test is a nightly differential check asserting LocalHeadersBitcoinExplorer.blockHash(h) == OkHttpBitcoinExplorer.blockHash(h) for every height in [checkpoint, tip] — any consensus drift surfaces as a disagreeing height. Maps cleanly onto the existing Phase 1/3/4/5/7/9 work, adding ~5–7 engineer-days total to the plan budget. No new top-level phase needed. --- .../2026-05-08-local-headers-explorer.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/quartz/plans/2026-05-08-local-headers-explorer.md b/quartz/plans/2026-05-08-local-headers-explorer.md index 4244760285..5103663dd8 100644 --- a/quartz/plans/2026-05-08-local-headers-explorer.md +++ b/quartz/plans/2026-05-08-local-headers-explorer.md @@ -29,6 +29,13 @@ intervening 2026-05-14 onchain-zaps work: - **`BitcoinExplorerEndpoint`** (`amethyst/.../model/nip03Timestamp/`) is now the shared Esplora-URL resolver for OTS and onchain zaps. The composite resolver builder must read from it for the HTTP fallback path. +- **Interop test vectors** (§14a) added — every consensus-relevant layer + (header parser, compact-target, retargeter, P2P codec, OTS proofs) is + pinned to upstream vectors committed under + `quartz/src/commonTest/resources/bitcoin/`. A nightly differential test + asserts `LocalHeadersBitcoinExplorer.blockHash(h)` == + `OkHttpBitcoinExplorer.blockHash(h)` for every height in + `[checkpoint, tip]`, catching any consensus-rule drift end-to-end. ## 1. Motivation @@ -681,6 +688,93 @@ Total: **~22–32 engineer-days** end to end. Phases 1–4 are the bulk - Tor on — sync completes (slowly) over Orbot. - Disk-full simulation — fail gracefully, don't corrupt store. +## 14a. Interop test vectors + +Consensus code without external test vectors is "trust me." Every layer +that could disagree with the rest of the Bitcoin network gets pinned to +authoritative upstream vectors, mirroring how `nip44.vectors.json`, +`bip39.vectors.json` and the `mls/*.json` set are already organised in +`quartz/src/commonTest/resources/`. The onchain-zaps work uses the same +discipline (BIP-341 wallet test vectors at every layer of +`nipBCOnchainZaps/taproot/` + `psbt/`). + +### Sources, by component + +| Component | Source(s) | Format | Storage | Effort | Priority | +|---|---|---|---|---|---| +| **`BlockHeader80` parse/serialize** | Genesis (0), block 1, block 209999 (last pre-halving), 481824 (first post-segwit), 709631 (last pre-taproot), one recent | JSON: `[{height, hash, headerHex, version, prevHash, merkleRoot, time, bits, nonce}, …]` | commit `commonTest/resources/bitcoin/known_headers.json` | 1 d | **High** | +| **`DifficultyTarget` compact↔target** | bitcoinj `UtilsTest.testCompactBitsToBigInteger`, btcd `chaincfg/chainhash/compact_test.go`. Vectors: `0x1d00ffff`, `0x1b0404cb`, `0x170398bb`, overflow + negative-bit edges | JSON pairs `(compact, targetHex)` | commit `bitcoin/compact_target.json` | ½ d | **High** | +| **Retarget (`CalculateNextWorkRequired`)** | Mainnet retarget heights 2016, 4032, 32256, 60480, ~600000 — extract `(prev_retarget_time, current_time, prev_bits, expected_new_bits)` from any full node | JSON list of retarget cases | commit `bitcoin/retargets.json` (~50 entries, ~5 KB) | 1 d | **High** | +| **`MedianTimePast`** | Any 11 consecutive mainnet headers synthesised from `known_headers.json` | inline test | inline | ½ d | Medium | +| **HeaderValidator end-to-end on real chain** | First 2016 mainnet headers (~160 KB; covers genesis + first retarget). Larger 50 K-header set lazy-fetched in nightly CI from a pinned-hash mirror, not committed | binary blob | commit `bitcoin/first_2016_headers.bin` | 1–2 d | **High** | +| **P2P wire codec — `version`/`verack`/`ping`/`pong`** | btcd `wire/msg*_test.go` has canonical byte strings; Bitcoin Wiki "Protocol documentation" has annotated hex | hex strings in JSON | commit `bitcoin/p2p_messages.json` | 1 d | **High** | +| **P2P `getheaders` / `headers` round-trip** | Capture against a real bitcoind once, ~50 KB | raw bytes | commit `bitcoin/p2p_capture.bin` | 1 d | **High** | +| **DNS seed list** | Bitcoin Core `chainparams.cpp` (current 9 seeds) | inline constant | n/a | <¼ d | Low | +| **Hardcoded fallback IPs** | Bitcoin Core `chainparamsseeds.h` (auto-generated by `contrib/seeds/`) | binary blob | commit `bitcoin/chainparamsseeds.bin` | ½ d | Medium | +| **Reorg / chain selection** | Synthetic 2-branch fork — verify higher-cumulative-chainwork wins | inline builder | inline | 1 d | **High** | +| **End-to-end OTS proofs** | `python-opentimestamps/tests/test_data/*.ots` + the 3 wild events already inline in `OtsTest.kt`. Import 5–10 more `.ots` for variety (calendar operators, eras) | `.ots` binaries + JSON manifest of expected timestamps | commit `commonTest/resources/ots/*.ots` | 1 d | **High** | +| **NIP-03 1040 events from real relays** | Scrape any relay for `kind:1040`; pick a diverse set across calendar operators and years | inline JSON | extend `OtsTest.kt` | ½ d | Medium | +| **Differential test: local vs HTTP** | For every `h ∈ [checkpoint, tip]`: assert `LocalHeadersBitcoinExplorer.blockHash(h)` == `OkHttpBitcoinExplorer.blockHash(h)`. Catches consensus drift end-to-end | runtime check, nightly CI | no fixture | 1 d | **High** | +| **Regtest end-to-end** | Bitcoin Core in Docker — mine N blocks, sync, assert tip matches | infrastructure | CI script | 2 d | Medium | +| **Adversarial peer** | Synthetic peer that lies about retarget, replays old headers, drops mid-batch, sends 2,001-headers messages. Validates ban + reconnect | inline test harness | inline | 2 d | **High** | + +### Committed layout + +``` +quartz/src/commonTest/resources/bitcoin/ +├── README.md # provenance + bump procedure for each file +├── known_headers.json # ~10 hand-picked mainnet headers across history +├── compact_target.json # ~20 compact-bits ↔ target pairs +├── retargets.json # ~50 retarget cases from historical mainnet +├── first_2016_headers.bin # ~160 KB; genesis + first retarget cycle +├── p2p_messages.json # hex of canonical version/verack/ping/pong +├── p2p_capture.bin # captured handshake + getheaders/headers (~50 KB) +└── chainparamsseeds.bin # current Bitcoin Core fallback IP list + +quartz/src/commonTest/resources/ots/ +├── manifest.json # {filename → expected unix timestamp | "pending"} +└── *.ots # 5–10 fixtures from python-opentimestamps +``` + +### Provenance discipline + +The `README.md` next to each fixture records, in this order: + +1. **Where it came from** — upstream URL + commit SHA. +2. **How to regenerate** — a single command, no manual steps. +3. **What it tests** — one sentence. +4. **When to bump** — e.g. *"retargets.json is stable; never bumps."* / + *"chainparamsseeds.bin bumped each Bitcoin Core major release."* + +Same pattern as the header comment of `nip44.vectors.json`. New committers +should never have to ask "where did this 32-byte blob come from." + +### Phase mapping + +These test imports map cleanly onto the existing phases — total extra +effort **~5–7 engineer-days** spread across the work, no new top-level +phase needed: + +| Phase | Vectors consumed | +|---|---| +| **Phase 1** (header parsing & validation, no I/O) | `known_headers.json`, `compact_target.json`, `retargets.json`, `first_2016_headers.bin` | +| **Phase 3** (P2P codec, no socket) | `p2p_messages.json`, `p2p_capture.bin` | +| **Phase 4** (single-peer sync) | adversarial-peer harness + regtest-in-Docker | +| **Phase 5** (multi-peer & eclipse) | `chainparamsseeds.bin` | +| **Phase 7** (Android wiring) | differential test against `OkHttpBitcoinExplorer` | +| **Phase 9** (CLI / Desktop) | OTS `.ots` fixtures via `amy verify-ots --explorer=local` | + +### Why the differential test matters most + +The single highest-value test in the list is the differential one: for +every height between the pinned checkpoint and the current tip, the local +explorer's answer must equal the HTTP explorer's answer. Run it nightly in +CI. Any consensus-rule drift — a wrong retarget formula, a missed +overflow case in compact-target decoding, a partial-merkle bug — will +surface as a height where the two disagree, and the height is the bug +report. This is the test that catches the bug nobody thought to write +a unit test for. + ## 15. Risks & open questions | # | Risk | Mitigation |