From 6bb2f8045dde030f6316c0832cc081918264d842 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 27 Jun 2026 00:19:26 +0000 Subject: [PATCH] build: pre-merge audit fixes for reproducibility work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bugs / inconsistencies found while reviewing the branch for merge: - dependenciesInfo comment falsely claimed Play "still derives this data server-side, nothing is lost." Not true: includeInBundle=false means the .aab carries no dependency metadata, so Play Console's dependency-insights / SDK-vulnerability alerts go unpopulated (uploads still succeed). Corrected the comment and the BUILDING.md framing (it called the blob "the one remaining blocker" when the Arti .so was the bigger one). - Version-bump workflow was broken: the README told you to run `build-arti.sh --clean` to refresh Cargo.lock, but the build is now --locked (fails on a stale lock) and the clone moved to the canonical /tmp path. Added a dedicated `--regen-lock` mode (clone + cargo generate-lockfile, no NDK needed) and pointed the docs at it. Verified it reproduces the committed lock byte-for-byte. - verify-reproducible.sh: new helper that builds twice and diffs to prove byte-for-byte reproducibility; uses portable sha256 (sha256sum/shasum) and plain `sort` so it runs on macOS too. - README verify recipe referenced paths that only resolved from the repo root while telling you to cd into tools/arti-build — replaced with the helper. - rust-toolchain.toml listed four Android targets but only two ABIs ship a .so; trimmed to match (check_prerequisites adds any other on the fly). - BUILDING.md: documented that the bundled Arti .so is reproducible-from-source and that secp256k1/webrtc are version-pinned Maven prebuilts. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JtjUcSjjpu4auFndw1QKeU --- BUILDING.md | 8 ++- amethyst/build.gradle.kts | 11 ++-- tools/arti-build/README.md | 21 ++++---- tools/arti-build/build-arti.sh | 28 +++++++--- tools/arti-build/rust-toolchain.toml | 5 +- tools/arti-build/verify-reproducible.sh | 69 +++++++++++++++++++++++++ 6 files changed, 119 insertions(+), 23 deletions(-) create mode 100755 tools/arti-build/verify-reproducible.sh diff --git a/BUILDING.md b/BUILDING.md index c0f7522c82..7a85691bdf 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -184,7 +184,13 @@ else. What makes that hold: - **No dependency-metadata blob.** `dependenciesInfo { includeInApk = false; includeInBundle = false }` in `amethyst/build.gradle.kts` stops AGP from embedding the Google-encrypted dependency protobuf in the signing block — that - ciphertext is non-deterministic and was the one remaining blocker. + ciphertext is non-deterministic. +- **Reproducible native library.** The bundled Tor (Arti) `.so` is the one + binary we compile ourselves; it is built reproducibly from source (pinned Rust + toolchain, locked deps, canonical build path). See + [`tools/arti-build/README.md`](tools/arti-build/README.md) → "Reproducible + builds". All other native libs (`secp256k1`, `webrtc`) are version-pinned Maven + prebuilts and so are byte-identical by download. ### Verify a release APK reproduces diff --git a/amethyst/build.gradle.kts b/amethyst/build.gradle.kts index 33bf9655e4..4088c67fd3 100644 --- a/amethyst/build.gradle.kts +++ b/amethyst/build.gradle.kts @@ -268,9 +268,14 @@ android { // in the APK/AAB. That blob is a protobuf of the resolved dependency tree // encrypted with a Google public key; the ciphertext is non-deterministic, // so its presence makes every release artifact impossible to reproduce - // bit-for-bit. Dropping it lets F-Droid / Zapstore independently rebuild and - // verify our developer-signed APKs. (Play still derives this data server-side - // from the upload, so nothing is lost for the Play channel.) + // bit-for-bit. Dropping it (F-Droid's documented recommendation) lets + // F-Droid / Zapstore independently rebuild and verify our developer-signed + // APKs. + // + // Play-channel trade-off: with includeInBundle = false the uploaded .aab no + // longer carries this metadata, so Play Console's app-dependency insights / + // known-vulnerability SDK alerts go unpopulated. Uploads still succeed; only + // that advisory feature is lost. dependenciesInfo { includeInApk = false includeInBundle = false diff --git a/tools/arti-build/README.md b/tools/arti-build/README.md index 382be72237..90f333ebea 100644 --- a/tools/arti-build/README.md +++ b/tools/arti-build/README.md @@ -49,18 +49,17 @@ release profile in `Cargo.toml` (`lto`, `codegen-units = 1`, `strip`, ### Verify the committed binary reproduces +From `tools/arti-build/`, the helper builds twice from clean and diffs the output: + ```bash -# Build, record the hash, then do a clean rebuild and confirm it matches. -# Both runs compile in the canonical /tmp/amethyst-arti-build, so the bytes match -# regardless of where this repo is checked out. -./build-arti.sh --release -sha256sum amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so -./build-arti.sh --clean --release -sha256sum amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so +./verify-reproducible.sh # both ABIs (arm64-v8a + x86_64) +./verify-reproducible.sh --release # arm64-v8a only (faster) ``` -Both hashes match each other and the committed -`amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so`. +It prints `✅ REPRODUCIBLE` when two clean builds produce identical bytes, then +reports whether that matches the committed `.so`. Both builds compile in the +canonical `/tmp/amethyst-arti-build`, so the result is independent of where the +repo is checked out. ## Prerequisites @@ -145,6 +144,7 @@ tools/arti-build/ ├── repro-env.sh # Deterministic build env (path remapping, epoch) — sourced by both scripts ├── build-arti.sh # Build script (Android targets, shipped in APK) ├── build-arti-host.sh # Build script (host target, for JVM integration tests) +├── verify-reproducible.sh # Builds twice + diffs to prove byte-for-byte reproducibility └── src/ └── lib.rs # JNI bridge (Rust → Kotlin) @@ -174,8 +174,7 @@ tools/arti-build/ 4. Regenerate the committed lockfile so the new versions are pinned (builds run `--locked` and will fail until this is refreshed): ```bash - ./build-arti.sh --clean # clones the new tag + sets up the wrapper - cp .arti-source/arti-android-wrapper/Cargo.lock ./Cargo.lock + ./build-arti.sh --regen-lock # re-resolves + rewrites ./Cargo.lock, no compile ``` If you also bump the Rust toolchain, edit `channel` in `rust-toolchain.toml`. diff --git a/tools/arti-build/build-arti.sh b/tools/arti-build/build-arti.sh index 14f50c3b98..33ad3b2583 100755 --- a/tools/arti-build/build-arti.sh +++ b/tools/arti-build/build-arti.sh @@ -43,13 +43,19 @@ MIN_SDK_VERSION=26 TARGETS=("aarch64-linux-android" "x86_64-linux-android") RELEASE_ONLY=false CLEAN=false +REGEN_LOCK=false # Parse arguments for arg in "$@"; do case $arg in --release) RELEASE_ONLY=true; TARGETS=("aarch64-linux-android") ;; --clean) CLEAN=true ;; - --help) echo "Usage: $0 [--release] [--clean] [--help]"; exit 0 ;; + # Refresh the committed Cargo.lock from the pinned Arti tag, then exit + # (no compile — needs only git + cargo, not the NDK). Use after bumping + # ARTI_VERSION / Cargo.toml; the normal build is --locked and will fail + # until the lock is regenerated and committed. + --regen-lock) REGEN_LOCK=true; CLEAN=true ;; + --help) echo "Usage: $0 [--release] [--clean] [--regen-lock] [--help]"; exit 0 ;; esac done @@ -144,14 +150,14 @@ setup_wrapper() { cp "$SCRIPT_DIR/src/lib.rs" "$wrapper_dir/src/lib.rs" # Reproducibility: build against the committed lockfile so transitive - # dependency versions are identical for everyone. `cargo --locked` (below) - # fails if this lock is missing or stale rather than silently re-resolving. + # dependency versions are identical for everyone. `cargo --locked` (in + # build_for_target) fails loudly if this lock is missing or stale rather than + # silently re-resolving. (Missing is only expected during --regen-lock.) if [ -f "$SCRIPT_DIR/Cargo.lock" ]; then cp "$SCRIPT_DIR/Cargo.lock" "$wrapper_dir/Cargo.lock" print_success "Pinned dependencies from committed Cargo.lock" else - print_error "tools/arti-build/Cargo.lock missing — generate it with: cargo generate-lockfile (see README)" - exit 1 + print_info "No committed Cargo.lock yet — run with --regen-lock to create it" fi # Patch Cargo.toml to use local arti-client from the source tree @@ -248,10 +254,20 @@ verify_jni_symbols() { main() { echo -e "${BLUE}Arti Android Build — version $ARTI_VERSION${NC}" - check_prerequisites + # --regen-lock only needs git + cargo, not the NDK/cargo-ndk toolchain. + [ "$REGEN_LOCK" = true ] || check_prerequisites clone_or_update_arti setup_wrapper + if [ "$REGEN_LOCK" = true ]; then + print_header "Regenerating Cargo.lock" + local manifest="$ARTI_SOURCE_DIR/arti-android-wrapper/Cargo.toml" + cargo generate-lockfile --manifest-path "$manifest" + cp "$ARTI_SOURCE_DIR/arti-android-wrapper/Cargo.lock" "$SCRIPT_DIR/Cargo.lock" + print_success "Updated $SCRIPT_DIR/Cargo.lock — commit it, then re-run the build." + exit 0 + fi + # Deterministic build env (needs ARTI_SOURCE_DIR cloned above for SOURCE_DATE_EPOCH). # shellcheck source=repro-env.sh source "$SCRIPT_DIR/repro-env.sh" diff --git a/tools/arti-build/rust-toolchain.toml b/tools/arti-build/rust-toolchain.toml index 794db4e336..cf48bd6cb0 100644 --- a/tools/arti-build/rust-toolchain.toml +++ b/tools/arti-build/rust-toolchain.toml @@ -10,9 +10,10 @@ channel = "1.94.1" profile = "minimal" components = ["rustc", "cargo", "rust-std"] +# Only the two ABIs we actually ship libarti_android.so for (see jniLibs/). If +# build-arti.sh is extended to armv7/i686, add them here too — check_prerequisites +# also adds any missing target on the fly. targets = [ "aarch64-linux-android", "x86_64-linux-android", - "armv7-linux-androideabi", - "i686-linux-android", ] diff --git a/tools/arti-build/verify-reproducible.sh b/tools/arti-build/verify-reproducible.sh new file mode 100755 index 0000000000..f83d12ea7a --- /dev/null +++ b/tools/arti-build/verify-reproducible.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# +# Verify that libarti_android.so builds reproducibly. +# +# Builds the Arti native library twice from a clean state and confirms the two +# outputs are byte-for-byte identical. Both builds compile in the canonical path +# (/tmp/amethyst-arti-build), so a match here means any checkout — ours, +# F-Droid's, an auditor's — produces the same bytes. See README.md → +# "Reproducible builds". +# +# Usage: +# ./verify-reproducible.sh # both ABIs (arm64-v8a + x86_64) +# ./verify-reproducible.sh --release # arm64-v8a only (faster) +# +# Prerequisites are the same as build-arti.sh (rustup, cargo-ndk, Android NDK). +# Exit 0 = reproducible, exit 1 = builds differ. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" +JNILIBS="$PROJECT_ROOT/amethyst/src/main/jniLibs" +PASSTHRU=("$@") + +# Portable sha256 (coreutils sha256sum on Linux, shasum on macOS). +sha256() { + if command -v sha256sum >/dev/null 2>&1; then sha256sum "$@"; else shasum -a 256 "$@"; fi +} + +# sha256 of every built .so, keyed by ABI dir (relative paths → stable keys). +# `find | sort` (plain text sort) is portable across GNU and BSD userlands. +hashes() { + ( cd "$JNILIBS" && find . -name libarti_android.so | sort | while IFS= read -r f; do + sha256 "$f" + done ) +} + +echo "### Reproducibility check for libarti_android.so" +echo "### Canonical build path: ${ARTI_REPRO_DIR:-/tmp/amethyst-arti-build}" +echo + +echo "### Build 1 of 2 (clean)…" +"$SCRIPT_DIR/build-arti.sh" --clean ${PASSTHRU[@]+"${PASSTHRU[@]}"} +H1="$(hashes)" +echo "--- build 1 hashes ---"; echo "$H1"; echo + +echo "### Build 2 of 2 (clean)…" +"$SCRIPT_DIR/build-arti.sh" --clean ${PASSTHRU[@]+"${PASSTHRU[@]}"} +H2="$(hashes)" +echo "--- build 2 hashes ---"; echo "$H2"; echo + +if [ "$H1" = "$H2" ]; then + echo "✅ REPRODUCIBLE — both clean builds produced identical .so bytes." +else + echo "❌ NOT REPRODUCIBLE — the two builds differ:" + diff <(echo "$H1") <(echo "$H2") || true + exit 1 +fi + +# Informational: is the binary committed in git already the reproducible one? +echo +echo "### vs. the committed binaries:" +if git -C "$PROJECT_ROOT" diff --quiet -- amethyst/src/main/jniLibs/; then + echo "✓ The reproducible build matches what's committed — the shipped .so is verifiable as-is." +else + echo "⚠ The reproducible build differs from the committed .so (e.g. the committed one" + echo " predates this toolchain). Commit the rebuilt binaries so the shipped artifact" + echo " is itself a reproducible build:" + echo " git -C \"$PROJECT_ROOT\" add amethyst/src/main/jniLibs && git commit" +fi