diff --git a/BUILDING.md b/BUILDING.md index be8f2879fa..c65217c52a 100644 --- a/BUILDING.md +++ b/BUILDING.md @@ -105,8 +105,9 @@ and each has its own guide: > change. Reusing an existing codepoint needs no regeneration. Both tools have their own prerequisites (`fonttools`/`brotli` for the font; a -Rust toolchain + Android NDK 25+ for Arti) documented in their READMEs — they -are **not** required to build Amethyst from the committed sources. +Rust toolchain + the exact Android NDK revision pinned in +`tools/arti-build/ANDROID_NDK_VERSION` for Arti) documented in their READMEs — +they are **not** required to build Amethyst from the committed sources. --- diff --git a/amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so b/amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so index 8b4a4b828b..9b397b7faa 100755 Binary files a/amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so and b/amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so differ diff --git a/amethyst/src/main/jniLibs/x86_64/libarti_android.so b/amethyst/src/main/jniLibs/x86_64/libarti_android.so index a3825bccd0..1e3c306260 100755 Binary files a/amethyst/src/main/jniLibs/x86_64/libarti_android.so and b/amethyst/src/main/jniLibs/x86_64/libarti_android.so differ diff --git a/tools/arti-build/ANDROID_NDK_VERSION b/tools/arti-build/ANDROID_NDK_VERSION new file mode 100644 index 0000000000..3b572e38a7 --- /dev/null +++ b/tools/arti-build/ANDROID_NDK_VERSION @@ -0,0 +1 @@ +27.3.13750724 diff --git a/tools/arti-build/CARGO_NDK_VERSION b/tools/arti-build/CARGO_NDK_VERSION new file mode 100644 index 0000000000..4d0dcda01c --- /dev/null +++ b/tools/arti-build/CARGO_NDK_VERSION @@ -0,0 +1 @@ +4.1.2 diff --git a/tools/arti-build/README.md b/tools/arti-build/README.md index 90f333ebea..7588c7246b 100644 --- a/tools/arti-build/README.md +++ b/tools/arti-build/README.md @@ -9,7 +9,7 @@ JNI wrapper built directly from Arti source. | | Guardian Project AAR | Custom build | |---|---|---| | **Size** | ~140MB | ~11MB | -| **16KB pages** | No | Yes (NDK 25+) | +| **16KB pages** | No | Yes (rustc aligns Android targets to 16 KiB) | | **Stop/restart** | Broken (state file lock) | Works (TorClient persists, only SOCKS proxy stops) | | **Version** | Behind | Pinned to latest (currently 1.9.0) | @@ -22,15 +22,33 @@ rebuild if you want to verify binaries, update the Arti version, or modify the J The shipped `.so` is **built to be reproducible** so anyone — F-Droid, Zapstore, or an independent auditor — can rebuild it from this tag and confirm the -committed binary wasn't tampered with. **Four** things have to be fixed: +committed binary wasn't tampered with. **Five** things have to be fixed: | Source of non-determinism | Pinned by | |---|---| | `rustc` / cargo version | [`rust-toolchain.toml`](rust-toolchain.toml) (rustup auto-installs it) | +| **Android NDK revision** | [`ANDROID_NDK_VERSION`](ANDROID_NDK_VERSION); `build-arti.sh` refuses to build with any other revision | | transitive dependency versions | committed [`Cargo.lock`](Cargo.lock); builds run `cargo --locked` | | absolute paths *embedded* in the binary | `--remap-path-prefix` in [`repro-env.sh`](repro-env.sh) | | codegen/link **ordering** keyed on the real build path | **canonical build path** (`build-arti.sh` builds in `/tmp/amethyst-arti-build`) | +> **Why the NDK is pinned.** It is not just an SDK detail: the NDK supplies the +> clang that compiles Arti's C dependencies (`ring`, `zstd-sys`, +> `libsqlite3-sys`) and the `lld` that links the final `cdylib`, both of which +> stamp themselves into the binary's `.comment` section next to `rustc`'s own +> version. Swapping the NDK changes the bytes exactly like swapping `rustc` +> would. Before this was pinned the build picked the first directory matching +> `~/Android/Sdk/ndk/*/`, so the committed libraries were produced by r25b while +> this file told everyone to install r27 — two verifiers could both follow the +> README and get different, equally "correct" results. `build-arti.sh` now +> resolves the pinned revision by name, re-reads `source.properties` to confirm +> it, and re-checks the `.note.android.ident` stamp of every `.so` it produced. +> +> [`CARGO_NDK_VERSION`](CARGO_NDK_VERSION) records the `cargo-ndk` release the +> pinned output was verified with. `cargo-ndk` only wraps the NDK, so a mismatch +> is a warning rather than an error — but it is the next thing to check if your +> rebuild does not match. + `repro-env.sh` (sourced by both build scripts) also sets `CARGO_INCREMENTAL=0` and a fixed `SOURCE_DATE_EPOCH` derived from the Arti tag. The size-optimized release profile in `Cargo.toml` (`lto`, `codegen-units = 1`, `strip`, @@ -74,20 +92,23 @@ repo is checked out. rustup target add aarch64-linux-android x86_64-linux-android ``` -3. **cargo-ndk** +3. **cargo-ndk** — the release the pinned output was verified with: ```bash - cargo install cargo-ndk + cargo install cargo-ndk --version "$(cat CARGO_NDK_VERSION)" --locked ``` -4. **Android NDK 25+** (required for 16KB page size support) +4. **Android NDK** — the exact revision in [`ANDROID_NDK_VERSION`](ANDROID_NDK_VERSION) + (currently **27.3.13750724**, r27d). Any other revision is refused: it would + produce a `.so` that does not match the committed one. ```bash # Via Android Studio: SDK Manager → SDK Tools → NDK (Side by side) # Or via command line: - sdkmanager "ndk;27.0.12077973" - - # Set environment variable - export ANDROID_NDK_HOME="$HOME/Android/Sdk/ndk/27.0.12077973" + sdkmanager "ndk;$(cat ANDROID_NDK_VERSION)" ``` + `build-arti.sh` finds it automatically under `$ANDROID_HOME/ndk/`, + `~/Android/Sdk/ndk/`, `~/Library/Android/sdk/ndk/` or + `/usr/local/lib/android/sdk/ndk/`. Set `ANDROID_NDK_HOME` only if yours + lives somewhere else — it is version-checked either way. ## Building @@ -130,7 +151,26 @@ Google Play requires 16KB page-aligned native libraries. Verify with: readelf -l amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so | grep LOAD ``` -The first LOAD segment alignment should be `0x4000` (16384 bytes). +The first LOAD segment alignment should be `0x4000` (16384 bytes). This comes +from rustc's Android target spec (`max-page-size=16384`), not from the NDK, so +it holds for every NDK revision we could build with. + +## Checking which toolchain built a `.so` + +The shipped binaries say so themselves — useful when a rebuild does not match, or +when auditing a `.so` you did not build: + +```bash +# NDK release name + build number (the last component of the pinned revision) +readelf -p .note.android.ident amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so + +# clang / lld (from the NDK) and rustc versions +readelf -p .comment amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so +``` + +For the pinned toolchain that prints `r27d` / `13750724`, clang 18.0.4 and the +`rustc` version from `rust-toolchain.toml`. `build-arti.sh` runs the first check +itself after every build. ## Directory structure @@ -138,6 +178,8 @@ The first LOAD segment alignment should be `0x4000` (16384 bytes). tools/arti-build/ ├── README.md # This file ├── ARTI_VERSION # Pinned Arti git tag (e.g., arti-v1.9.0) +├── ANDROID_NDK_VERSION # Pinned NDK revision — enforced by build-arti.sh (reproducibility) +├── CARGO_NDK_VERSION # cargo-ndk release the pinned output was verified with ├── rust-toolchain.toml # Pinned rustc version + Android targets (reproducibility) ├── Cargo.toml # Rust dependencies and build profile ├── Cargo.lock # Pinned transitive dependency versions (reproducibility) diff --git a/tools/arti-build/build-arti.sh b/tools/arti-build/build-arti.sh index be7d1cacdd..766078d6b5 100755 --- a/tools/arti-build/build-arti.sh +++ b/tools/arti-build/build-arti.sh @@ -6,7 +6,8 @@ # - Rust toolchain: rustup, cargo # - Android targets: rustup target add aarch64-linux-android x86_64-linux-android # - cargo-ndk: cargo install cargo-ndk -# - Android NDK 25+ (for 16KB page size support) +# - Android NDK: the exact revision pinned in ANDROID_NDK_VERSION +# (sdkmanager "ndk;") — see README.md -> "Reproducible builds" # # Usage: # ./build-arti.sh # Build for all targets (arm64 + x86_64) @@ -26,6 +27,22 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" ARTI_VERSION=$(cat "$SCRIPT_DIR/ARTI_VERSION" | tr -d '[:space:]') +# Reproducibility: the NDK ships the clang that compiles Arti's C dependencies +# (ring, zstd-sys, libsqlite3-sys) and the lld that links the whole cdylib, so +# its revision is baked into the output bytes exactly like rustc's is — both +# land in the .comment section of the shipped .so. Pin it here and refuse to +# build with anything else; the old glob over ~/Android/Sdk/ndk/*/ silently +# picked up whatever happened to be installed first. +NDK_VERSION=$(cat "$SCRIPT_DIR/ANDROID_NDK_VERSION" | tr -d '[:space:]') +# The NDK build number (last component of the revision) is what the linker +# stamps into .note.android.ident, so it is how we verify the output afterwards. +NDK_BUILD_NUMBER="${NDK_VERSION##*.}" +# cargo-ndk only wraps the NDK (it sets CC/AR/linker and the --platform flags), +# but those flags reach the linker, so record the version we verified with and +# warn when it differs. Not a hard error: unlike the NDK itself, it has no +# proven effect on the bytes. +CARGO_NDK_VERSION=$(cat "$SCRIPT_DIR/CARGO_NDK_VERSION" | tr -d '[:space:]') + # Reproducibility: rustc bakes the *real* (un-remapped) absolute paths of the # build artifacts into its codegen/link ORDERING, so --remap-path-prefix alone # is not enough — the .so only reproduces byte-for-byte when the compile happens @@ -74,14 +91,30 @@ check_prerequisites() { command -v git >/dev/null 2>&1 || { print_error "git not found"; exit 1; } command -v rustup >/dev/null 2>&1 || { print_error "rustup not found"; exit 1; } command -v cargo >/dev/null 2>&1 || { print_error "cargo not found"; exit 1; } - command -v cargo-ndk >/dev/null 2>&1 || { print_error "cargo-ndk not found. Install: cargo install cargo-ndk"; exit 1; } + command -v cargo-ndk >/dev/null 2>&1 || { print_error "cargo-ndk not found. Install: cargo install cargo-ndk --version $CARGO_NDK_VERSION --locked"; exit 1; } + local found_cargo_ndk + found_cargo_ndk="$(cargo ndk --version 2>/dev/null | awk '{print $2}' || true)" + if [ "$found_cargo_ndk" != "$CARGO_NDK_VERSION" ]; then + print_info "cargo-ndk ${found_cargo_ndk:-unknown} != pinned $CARGO_NDK_VERSION — if the" + print_info " output does not match the committed .so, try: cargo install cargo-ndk --version $CARGO_NDK_VERSION --locked" + else + print_success "cargo-ndk: $CARGO_NDK_VERSION" + fi + + # An explicit ANDROID_NDK_HOME wins (it is verified below like any other); + # otherwise look for the pinned revision by name in the usual SDK layouts. + # Deliberately no wildcard: picking "some NDK" is what let the committed + # binaries be built with r25b while the docs asked for r27. if [ -z "${ANDROID_NDK_HOME:-}" ]; then - # Try common locations for candidate in \ - "$HOME/Android/Sdk/ndk/"*/ \ - "$HOME/Library/Android/sdk/ndk/"*/ \ - "/usr/local/lib/android/sdk/ndk/"*/; do + "${ANDROID_NDK_ROOT:-}" \ + "${ANDROID_HOME:-}/ndk/$NDK_VERSION" \ + "${ANDROID_SDK_ROOT:-}/ndk/$NDK_VERSION" \ + "$HOME/Android/Sdk/ndk/$NDK_VERSION" \ + "$HOME/Library/Android/sdk/ndk/$NDK_VERSION" \ + "/usr/local/lib/android/sdk/ndk/$NDK_VERSION"; do + [ -n "$candidate" ] || continue if [ -d "$candidate" ]; then export ANDROID_NDK_HOME="${candidate%/}" break @@ -90,11 +123,23 @@ check_prerequisites() { fi if [ -z "${ANDROID_NDK_HOME:-}" ]; then - print_error "ANDROID_NDK_HOME not set and NDK not found in common locations" + print_error "Android NDK $NDK_VERSION not found (and ANDROID_NDK_HOME is unset)" + echo " Install it: sdkmanager \"ndk;$NDK_VERSION\"" + echo " Or point ANDROID_NDK_HOME at an existing $NDK_VERSION install." exit 1 fi - print_success "NDK: $ANDROID_NDK_HOME" + local found_ndk + found_ndk="$(sed -n 's/^Pkg\.Revision *= *//p' "$ANDROID_NDK_HOME/source.properties" 2>/dev/null | tr -d '[:space:]' || true)" + if [ "$found_ndk" != "$NDK_VERSION" ]; then + print_error "NDK revision mismatch — this build would not reproduce the shipped .so" + echo " Pinned: $NDK_VERSION (tools/arti-build/ANDROID_NDK_VERSION)" + echo " Found: ${found_ndk:-unknown} at $ANDROID_NDK_HOME" + echo " Install: sdkmanager \"ndk;$NDK_VERSION\"" + exit 1 + fi + + print_success "NDK: $ANDROID_NDK_HOME ($NDK_VERSION)" for target in "${TARGETS[@]}"; do if ! rustup target list --installed | grep -q "$target"; then @@ -180,17 +225,22 @@ PATCH # Build # ============================================================================ +# Android ABI directory (as laid out under jniLibs/) for a Rust target triple. +abi_dir_for() { + case "$1" in + aarch64-linux-android) echo "arm64-v8a" ;; + x86_64-linux-android) echo "x86_64" ;; + armv7-linux-androideabi) echo "armeabi-v7a" ;; + i686-linux-android) echo "x86" ;; + esac +} + build_for_target() { local target="$1" print_header "Building for $target" local arch_dir - case "$target" in - aarch64-linux-android) arch_dir="arm64-v8a" ;; - x86_64-linux-android) arch_dir="x86_64" ;; - armv7-linux-androideabi) arch_dir="armeabi-v7a" ;; - i686-linux-android) arch_dir="x86" ;; - esac + arch_dir="$(abi_dir_for "$target")" local out_dir="$OUTPUT_DIR/$arch_dir" mkdir -p "$out_dir" @@ -236,8 +286,16 @@ verify_jni_symbols() { local arch=$(basename "$arch_dir") local missing=0 + # Read the dynamic symbol table once, into a variable. Piping nm into + # `grep -q` per symbol looks equivalent but is not: grep exits on the + # first match, nm dies of SIGPIPE (141), and `set -o pipefail` then + # reports the pipeline as failed — so every symbol that IS exported gets + # reported as missing. (Reproducible on any build, old or new.) + local syms + syms="$(nm -D "$lib" 2>/dev/null || true)" + for sym in "${expected_symbols[@]}"; do - if ! nm -D "$lib" 2>/dev/null | grep -q "$sym"; then + if [[ "$syms" != *"$sym"* ]]; then print_error "$arch: Missing symbol $sym" missing=1 fi @@ -249,6 +307,35 @@ verify_jni_symbols() { done } +verify_ndk_stamp() { + print_header "Verifying NDK stamp" + + if ! command -v readelf >/dev/null 2>&1; then + print_info "readelf not found — skipping (install binutils to enable this check)" + return 0 + fi + + # Every NDK-linked shared object carries .note.android.ident, which records + # the target API level, the NDK release name (e.g. r27d) and the NDK build + # number. Reading it back proves which toolchain actually produced the + # binary, independently of what the environment claimed — this is how the + # committed r25b libraries were identified in the first place. + for target in "${TARGETS[@]}"; do + local arch + arch="$(abi_dir_for "$target")" + local lib="$OUTPUT_DIR/$arch/$LIB_NAME" + [ -f "$lib" ] || continue + + if readelf -p .note.android.ident "$lib" 2>/dev/null | grep -qw "$NDK_BUILD_NUMBER"; then + print_success "$arch: built by NDK $NDK_VERSION" + else + print_error "$arch: not stamped with NDK build $NDK_BUILD_NUMBER — wrong toolchain?" + readelf -p .note.android.ident "$lib" 2>/dev/null || true + exit 1 + fi + done +} + # ============================================================================ # Main # ============================================================================ @@ -280,6 +367,7 @@ main() { done verify_jni_symbols + verify_ndk_stamp print_header "Build complete" echo "" diff --git a/tools/arti-build/verify-reproducible.sh b/tools/arti-build/verify-reproducible.sh index f83d12ea7a..de7053325e 100755 --- a/tools/arti-build/verify-reproducible.sh +++ b/tools/arti-build/verify-reproducible.sh @@ -12,7 +12,9 @@ # ./verify-reproducible.sh # both ABIs (arm64-v8a + x86_64) # ./verify-reproducible.sh --release # arm64-v8a only (faster) # -# Prerequisites are the same as build-arti.sh (rustup, cargo-ndk, Android NDK). +# Prerequisites are the same as build-arti.sh (rustup, cargo-ndk, and the exact +# Android NDK revision pinned in ANDROID_NDK_VERSION — a different revision is +# refused, because it would change the output bytes). # Exit 0 = reproducible, exit 1 = builds differ. set -euo pipefail