From ed3a893d9d9fd528ba0660357c9ad47145dcdd85 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 26 Jun 2026 23:40:18 +0000 Subject: [PATCH] build: build Arti at a canonical path for cross-environment reproducibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirical finding: with the toolchain pin, locked deps, and --remap-path-prefix all in place, two host builds of libarti_android.so at the *same* path are byte-for-byte identical, but two builds at *different* paths still differ — not in any embedded string (no path leaks into the binary) but in the order rustc lays out functions/data, which it derives from the real on-disk artifact paths. --remap-path-prefix only rewrites embedded strings, not that internal ordering. So compile in a fixed location (/tmp/amethyst-arti-build, overridable via ARTI_REPRO_DIR) in both build-arti.sh and build-arti-host.sh. Any checkout then produces matching bytes, which is what lets F-Droid / a verifier build at the same canonical path and reproduce the shipped .so. This mirrors how Rust libraries are reproduced elsewhere (F-Droid builds Rust at a fixed path too). Corrects the README, which previously implied path remapping alone gave path-independent output. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01JtjUcSjjpu4auFndw1QKeU --- tools/arti-build/README.md | 43 ++++++++++++++++++++--------- tools/arti-build/build-arti-host.sh | 3 +- tools/arti-build/build-arti.sh | 14 +++++++++- 3 files changed, 45 insertions(+), 15 deletions(-) diff --git a/tools/arti-build/README.md b/tools/arti-build/README.md index 0f8081e1a9..382be72237 100644 --- a/tools/arti-build/README.md +++ b/tools/arti-build/README.md @@ -20,32 +20,46 @@ rebuild if you want to verify binaries, update the Arti version, or modify the J ## Reproducible builds -The shipped `.so` is **built to be byte-for-byte reproducible** so anyone — -F-Droid, Zapstore, or an independent auditor — can rebuild it from this tag and -confirm the committed binary wasn't tampered with. Three pins make that hold: +The shipped `.so` is **built to be reproducible** so anyone — F-Droid, Zapstore, +or an independent auditor — can rebuild it from this tag and confirm the +committed binary wasn't tampered with. **Four** things have to be fixed: | Source of non-determinism | Pinned by | |---|---| | `rustc` / cargo version | [`rust-toolchain.toml`](rust-toolchain.toml) (rustup auto-installs it) | | transitive dependency versions | committed [`Cargo.lock`](Cargo.lock); builds run `cargo --locked` | -| absolute build paths baked into the binary | `--remap-path-prefix` in [`repro-env.sh`](repro-env.sh) | +| absolute paths *embedded* in the binary | `--remap-path-prefix` in [`repro-env.sh`](repro-env.sh) | +| codegen/link **ordering** keyed on the real build path | **canonical build path** (`build-arti.sh` builds in `/tmp/amethyst-arti-build`) | `repro-env.sh` (sourced by both build scripts) also sets `CARGO_INCREMENTAL=0` and a fixed `SOURCE_DATE_EPOCH` derived from the Arti tag. The size-optimized release profile in `Cargo.toml` (`lto`, `codegen-units = 1`, `strip`, `panic = "abort"`) is itself deterministic for a fixed toolchain. +> **Why the canonical path matters.** Verified empirically: with the toolchain, +> lockfile, and path-remapping all in place, two builds at the **same** path are +> byte-for-byte identical, but two builds at **different** paths still differ — +> not in any embedded string (no path leaks into the binary) but in the *order* +> rustc lays out functions/data, which it derives from the real on-disk artifact +> paths. `--remap-path-prefix` only rewrites embedded strings, not that internal +> ordering. So `build-arti.sh` always compiles in a fixed location +> (`/tmp/amethyst-arti-build`, override with `ARTI_REPRO_DIR`); F-Droid and any +> verifier must use the **same** path to get matching bytes. This is the standard +> way Rust libraries are reproduced (F-Droid builds Rust at a fixed path too). + ### Verify the committed binary reproduces ```bash -# Build twice into different checkout paths and confirm identical bytes. -# (Path remapping is what lets two different directories produce the same .so.) -cp -r tools/arti-build /tmp/arti-a && (cd /tmp/arti-a && ./build-arti.sh --release) -cp -r tools/arti-build /tmp/arti-b && (cd /tmp/arti-b && ./build-arti.sh --release) -sha256sum /tmp/arti-{a,b}/../../amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so +# Build, record the hash, then do a clean rebuild and confirm it matches. +# Both runs compile in the canonical /tmp/amethyst-arti-build, so the bytes match +# regardless of where this repo is checked out. +./build-arti.sh --release +sha256sum amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so +./build-arti.sh --clean --release +sha256sum amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so ``` -A clean run prints the same SHA-256 for both, and matches the committed +Both hashes match each other and the committed `amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so`. ## Prerequisites @@ -131,9 +145,12 @@ tools/arti-build/ ├── repro-env.sh # Deterministic build env (path remapping, epoch) — sourced by both scripts ├── build-arti.sh # Build script (Android targets, shipped in APK) ├── build-arti-host.sh # Build script (host target, for JVM integration tests) -├── src/ -│ └── lib.rs # JNI bridge (Rust → Kotlin) -└── .arti-source/ # [gitignored] Cloned Arti repository +└── src/ + └── lib.rs # JNI bridge (Rust → Kotlin) + +# The Arti source is cloned into the canonical build path +# (/tmp/amethyst-arti-build/.arti-source), not under this dir — see +# "Reproducible builds" for why the build location is fixed. ``` ## Updating Arti version diff --git a/tools/arti-build/build-arti-host.sh b/tools/arti-build/build-arti-host.sh index 3529e05cd9..85dd94a10a 100755 --- a/tools/arti-build/build-arti-host.sh +++ b/tools/arti-build/build-arti-host.sh @@ -26,7 +26,8 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -ARTI_SOURCE_DIR="$SCRIPT_DIR/.arti-source" +# Same canonical build path as build-arti.sh (see its comment for why). +ARTI_SOURCE_DIR="${ARTI_REPRO_DIR:-/tmp/amethyst-arti-build}/.arti-source" WRAPPER_DIR="$ARTI_SOURCE_DIR/arti-android-wrapper" if [ ! -d "$WRAPPER_DIR" ]; then diff --git a/tools/arti-build/build-arti.sh b/tools/arti-build/build-arti.sh index cfcbcfb580..14f50c3b98 100755 --- a/tools/arti-build/build-arti.sh +++ b/tools/arti-build/build-arti.sh @@ -24,8 +24,17 @@ NC='\033[0m' SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" -ARTI_SOURCE_DIR="$SCRIPT_DIR/.arti-source" ARTI_VERSION=$(cat "$SCRIPT_DIR/ARTI_VERSION" | tr -d '[:space:]') + +# Reproducibility: rustc bakes the *real* (un-remapped) absolute paths of the +# build artifacts into its codegen/link ORDERING, so --remap-path-prefix alone +# is not enough — the .so only reproduces byte-for-byte when the compile happens +# at a fixed path. Everyone who needs to reproduce the shipped binary (us, +# F-Droid, an independent verifier) must therefore build at this same canonical +# location. Overriding ARTI_REPRO_DIR changes the output bytes; only do it if +# you don't care about matching the published .so. +ARTI_BUILD_ROOT="${ARTI_REPRO_DIR:-/tmp/amethyst-arti-build}" +ARTI_SOURCE_DIR="$ARTI_BUILD_ROOT/.arti-source" OUTPUT_DIR="$PROJECT_ROOT/amethyst/src/main/jniLibs" LIB_NAME="libarti_android.so" MIN_SDK_VERSION=26 @@ -102,6 +111,9 @@ clone_or_update_arti() { rm -rf "$ARTI_SOURCE_DIR" fi + mkdir -p "$ARTI_BUILD_ROOT" + print_info "Canonical build path: $ARTI_BUILD_ROOT (set ARTI_REPRO_DIR to override)" + if [ ! -d "$ARTI_SOURCE_DIR" ]; then print_info "Cloning Arti repository..." git clone --depth 1 --branch "$ARTI_VERSION" \