build: pin the Arti NDK and rebuild libarti_android.so on r27d

The committed libraries were built with NDK r25b (25.1.8937393) while the build docs told everyone to install r27. Nothing pinned the NDK, so build-arti.sh took the first directory matching ~/Android/Sdk/ndk/*/. The NDK supplies the clang that compiles Arti's C dependencies (ring, zstd-sys, libsqlite3-sys) and the lld that links the cdylib, so its revision is baked into the output bytes exactly like rustc's is. The reproducible-build promise therefore only held by accident of which NDK a verifier happened to have installed. - Pin the revision in ANDROID_NDK_VERSION (27.3.13750724, r27d) and resolve it by name. A different revision now fails the build with the sdkmanager line that fixes it, instead of silently producing unverifiable bytes. - Record the verified cargo-ndk release in CARGO_NDK_VERSION. Warning only: it wraps the NDK rather than generating code. - Re-read .note.android.ident after each build, so the output has to carry the pinned NDK's stamp to pass. - Rebuild both ABIs on r27d (clang 18.0.4, lld 18.0.4, rustc 1.94.1). verify-reproducible.sh: two clean builds byte-for-byte identical, all 8 JNI symbols exported, 16 KiB LOAD alignment kept, same libc/libm/libdl dependency set as before. - Fix verify_jni_symbols reporting every exported symbol as missing: piping nm into `grep -q` per symbol lets grep exit first, nm dies of SIGPIPE, and `set -o pipefail` fails the pipeline. Pre-existing, reproduces on the old binary too. - Docs: the 16 KiB page alignment comes from rustc's Android target spec, not from "NDK 25+". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cSuXeu4bUTNRAUCZJcLLW
2026-10-06 03:38:23 +00:00 · 2026-09-13 16:43:33 +00:00
parent bd1021c6f4
commit b52f5651de
8 changed files with 163 additions and 28 deletions
@@ -105,8 +105,9 @@ and each has its own guide:
 > change. Reusing an existing codepoint needs no regeneration.

 Both tools have their own prerequisites (`fonttools`/`brotli` for the font; a
-Rust toolchain + Android NDK 25+ for Arti) documented in their READMEs — they
-are **not** required to build Amethyst from the committed sources.
+Rust toolchain + the exact Android NDK revision pinned in
+`tools/arti-build/ANDROID_NDK_VERSION` for Arti) documented in their READMEs —
+they are **not** required to build Amethyst from the committed sources.

 ---

@@ -0,0 +1 @@
+27.3.13750724
@@ -0,0 +1 @@
+4.1.2
@@ -9,7 +9,7 @@ JNI wrapper built directly from Arti source.
 | | Guardian Project AAR | Custom build |
 |---|---|---|
 | **Size** | ~140MB | ~11MB |
-| **16KB pages** | No | Yes (NDK 25+) |
+| **16KB pages** | No | Yes (rustc aligns Android targets to 16 KiB) |
 | **Stop/restart** | Broken (state file lock) | Works (TorClient persists, only SOCKS proxy stops) |
 | **Version** | Behind | Pinned to latest (currently 1.9.0) |

@@ -22,15 +22,33 @@ rebuild if you want to verify binaries, update the Arti version, or modify the J

 The shipped `.so` is **built to be reproducible** so anyone — F-Droid, Zapstore,
 or an independent auditor — can rebuild it from this tag and confirm the
-committed binary wasn't tampered with. **Four** things have to be fixed:
+committed binary wasn't tampered with. **Five** things have to be fixed:

 | Source of non-determinism | Pinned by |
 |---|---|
 | `rustc` / cargo version | [`rust-toolchain.toml`](rust-toolchain.toml) (rustup auto-installs it) |
+| **Android NDK revision** | [`ANDROID_NDK_VERSION`](ANDROID_NDK_VERSION); `build-arti.sh` refuses to build with any other revision |
 | transitive dependency versions | committed [`Cargo.lock`](Cargo.lock); builds run `cargo --locked` |
 | absolute paths *embedded* in the binary | `--remap-path-prefix` in [`repro-env.sh`](repro-env.sh) |
 | codegen/link **ordering** keyed on the real build path | **canonical build path** (`build-arti.sh` builds in `/tmp/amethyst-arti-build`) |

+> **Why the NDK is pinned.** It is not just an SDK detail: the NDK supplies the
+> clang that compiles Arti's C dependencies (`ring`, `zstd-sys`,
+> `libsqlite3-sys`) and the `lld` that links the final `cdylib`, both of which
+> stamp themselves into the binary's `.comment` section next to `rustc`'s own
+> version. Swapping the NDK changes the bytes exactly like swapping `rustc`
+> would. Before this was pinned the build picked the first directory matching
+> `~/Android/Sdk/ndk/*/`, so the committed libraries were produced by r25b while
+> this file told everyone to install r27 — two verifiers could both follow the
+> README and get different, equally "correct" results. `build-arti.sh` now
+> resolves the pinned revision by name, re-reads `source.properties` to confirm
+> it, and re-checks the `.note.android.ident` stamp of every `.so` it produced.
+>
+> [`CARGO_NDK_VERSION`](CARGO_NDK_VERSION) records the `cargo-ndk` release the
+> pinned output was verified with. `cargo-ndk` only wraps the NDK, so a mismatch
+> is a warning rather than an error — but it is the next thing to check if your
+> rebuild does not match.
+
 `repro-env.sh` (sourced by both build scripts) also sets `CARGO_INCREMENTAL=0`
 and a fixed `SOURCE_DATE_EPOCH` derived from the Arti tag. The size-optimized
 release profile in `Cargo.toml` (`lto`, `codegen-units = 1`, `strip`,
@@ -74,20 +92,23 @@ repo is checked out.
   rustup target add aarch64-linux-android x86_64-linux-android
   ```

-3. **cargo-ndk**
+3. **cargo-ndk** — the release the pinned output was verified with:
   ```bash
-   cargo install cargo-ndk
+   cargo install cargo-ndk --version "$(cat CARGO_NDK_VERSION)" --locked
   ```

-4. **Android NDK 25+** (required for 16KB page size support)
+4. **Android NDK** — the exact revision in [`ANDROID_NDK_VERSION`](ANDROID_NDK_VERSION)
+   (currently **27.3.13750724**, r27d). Any other revision is refused: it would
+   produce a `.so` that does not match the committed one.
   ```bash
   # Via Android Studio: SDK Manager → SDK Tools → NDK (Side by side)
   # Or via command line:
-   sdkmanager "ndk;27.0.12077973"
-
-   # Set environment variable
-   export ANDROID_NDK_HOME="$HOME/Android/Sdk/ndk/27.0.12077973"
+   sdkmanager "ndk;$(cat ANDROID_NDK_VERSION)"
   ```
+   `build-arti.sh` finds it automatically under `$ANDROID_HOME/ndk/`,
+   `~/Android/Sdk/ndk/`, `~/Library/Android/sdk/ndk/` or
+   `/usr/local/lib/android/sdk/ndk/`. Set `ANDROID_NDK_HOME` only if yours
+   lives somewhere else — it is version-checked either way.

 ## Building

@@ -130,7 +151,26 @@ Google Play requires 16KB page-aligned native libraries. Verify with:
 readelf -l amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so | grep LOAD
 ```

-The first LOAD segment alignment should be `0x4000` (16384 bytes).
+The first LOAD segment alignment should be `0x4000` (16384 bytes). This comes
+from rustc's Android target spec (`max-page-size=16384`), not from the NDK, so
+it holds for every NDK revision we could build with.
+
+## Checking which toolchain built a `.so`
+
+The shipped binaries say so themselves — useful when a rebuild does not match, or
+when auditing a `.so` you did not build:
+
+```bash
+# NDK release name + build number (the last component of the pinned revision)
+readelf -p .note.android.ident amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so
+
+# clang / lld (from the NDK) and rustc versions
+readelf -p .comment amethyst/src/main/jniLibs/arm64-v8a/libarti_android.so
+```
+
+For the pinned toolchain that prints `r27d` / `13750724`, clang 18.0.4 and the
+`rustc` version from `rust-toolchain.toml`. `build-arti.sh` runs the first check
+itself after every build.

 ## Directory structure

@@ -138,6 +178,8 @@ The first LOAD segment alignment should be `0x4000` (16384 bytes).
 tools/arti-build/
 ├── README.md            # This file
 ├── ARTI_VERSION         # Pinned Arti git tag (e.g., arti-v1.9.0)
+├── ANDROID_NDK_VERSION  # Pinned NDK revision — enforced by build-arti.sh (reproducibility)
+├── CARGO_NDK_VERSION    # cargo-ndk release the pinned output was verified with
 ├── rust-toolchain.toml  # Pinned rustc version + Android targets (reproducibility)
 ├── Cargo.toml           # Rust dependencies and build profile
 ├── Cargo.lock           # Pinned transitive dependency versions (reproducibility)
@@ -6,7 +6,8 @@
 #   - Rust toolchain: rustup, cargo
 #   - Android targets: rustup target add aarch64-linux-android x86_64-linux-android
 #   - cargo-ndk: cargo install cargo-ndk
-#   - Android NDK 25+ (for 16KB page size support)
+#   - Android NDK: the exact revision pinned in ANDROID_NDK_VERSION
+#     (sdkmanager "ndk;<revision>") — see README.md -> "Reproducible builds"
 #
 # Usage:
 #   ./build-arti.sh              # Build for all targets (arm64 + x86_64)
@@ -26,6 +27,22 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
 ARTI_VERSION=$(cat "$SCRIPT_DIR/ARTI_VERSION" | tr -d '[:space:]')

+# Reproducibility: the NDK ships the clang that compiles Arti's C dependencies
+# (ring, zstd-sys, libsqlite3-sys) and the lld that links the whole cdylib, so
+# its revision is baked into the output bytes exactly like rustc's is — both
+# land in the .comment section of the shipped .so. Pin it here and refuse to
+# build with anything else; the old glob over ~/Android/Sdk/ndk/*/ silently
+# picked up whatever happened to be installed first.
+NDK_VERSION=$(cat "$SCRIPT_DIR/ANDROID_NDK_VERSION" | tr -d '[:space:]')
+# The NDK build number (last component of the revision) is what the linker
+# stamps into .note.android.ident, so it is how we verify the output afterwards.
+NDK_BUILD_NUMBER="${NDK_VERSION##*.}"
+# cargo-ndk only wraps the NDK (it sets CC/AR/linker and the --platform flags),
+# but those flags reach the linker, so record the version we verified with and
+# warn when it differs. Not a hard error: unlike the NDK itself, it has no
+# proven effect on the bytes.
+CARGO_NDK_VERSION=$(cat "$SCRIPT_DIR/CARGO_NDK_VERSION" | tr -d '[:space:]')
+
 # Reproducibility: rustc bakes the *real* (un-remapped) absolute paths of the
 # build artifacts into its codegen/link ORDERING, so --remap-path-prefix alone
 # is not enough — the .so only reproduces byte-for-byte when the compile happens
@@ -74,14 +91,30 @@ check_prerequisites() {
    command -v git >/dev/null 2>&1 || { print_error "git not found"; exit 1; }
    command -v rustup >/dev/null 2>&1 || { print_error "rustup not found"; exit 1; }
    command -v cargo >/dev/null 2>&1 || { print_error "cargo not found"; exit 1; }
-    command -v cargo-ndk >/dev/null 2>&1 || { print_error "cargo-ndk not found. Install: cargo install cargo-ndk"; exit 1; }
+    command -v cargo-ndk >/dev/null 2>&1 || { print_error "cargo-ndk not found. Install: cargo install cargo-ndk --version $CARGO_NDK_VERSION --locked"; exit 1; }

+    local found_cargo_ndk
+    found_cargo_ndk="$(cargo ndk --version 2>/dev/null | awk '{print $2}' || true)"
+    if [ "$found_cargo_ndk" != "$CARGO_NDK_VERSION" ]; then
+        print_info "cargo-ndk ${found_cargo_ndk:-unknown} != pinned $CARGO_NDK_VERSION — if the"
+        print_info "  output does not match the committed .so, try: cargo install cargo-ndk --version $CARGO_NDK_VERSION --locked"
+    else
+        print_success "cargo-ndk: $CARGO_NDK_VERSION"
+    fi
+
+    # An explicit ANDROID_NDK_HOME wins (it is verified below like any other);
+    # otherwise look for the pinned revision by name in the usual SDK layouts.
+    # Deliberately no wildcard: picking "some NDK" is what let the committed
+    # binaries be built with r25b while the docs asked for r27.
    if [ -z "${ANDROID_NDK_HOME:-}" ]; then
-        # Try common locations
        for candidate in \
-            "$HOME/Android/Sdk/ndk/"*/ \
-            "$HOME/Library/Android/sdk/ndk/"*/ \
-            "/usr/local/lib/android/sdk/ndk/"*/; do
+            "${ANDROID_NDK_ROOT:-}" \
+            "${ANDROID_HOME:-}/ndk/$NDK_VERSION" \
+            "${ANDROID_SDK_ROOT:-}/ndk/$NDK_VERSION" \
+            "$HOME/Android/Sdk/ndk/$NDK_VERSION" \
+            "$HOME/Library/Android/sdk/ndk/$NDK_VERSION" \
+            "/usr/local/lib/android/sdk/ndk/$NDK_VERSION"; do
+            [ -n "$candidate" ] || continue
            if [ -d "$candidate" ]; then
                export ANDROID_NDK_HOME="${candidate%/}"
                break
@@ -90,11 +123,23 @@ check_prerequisites() {
    fi

    if [ -z "${ANDROID_NDK_HOME:-}" ]; then
-        print_error "ANDROID_NDK_HOME not set and NDK not found in common locations"
+        print_error "Android NDK $NDK_VERSION not found (and ANDROID_NDK_HOME is unset)"
+        echo "  Install it:  sdkmanager \"ndk;$NDK_VERSION\""
+        echo "  Or point ANDROID_NDK_HOME at an existing $NDK_VERSION install."
        exit 1
    fi

-    print_success "NDK: $ANDROID_NDK_HOME"
+    local found_ndk
+    found_ndk="$(sed -n 's/^Pkg\.Revision *= *//p' "$ANDROID_NDK_HOME/source.properties" 2>/dev/null | tr -d '[:space:]' || true)"
+    if [ "$found_ndk" != "$NDK_VERSION" ]; then
+        print_error "NDK revision mismatch — this build would not reproduce the shipped .so"
+        echo "  Pinned:   $NDK_VERSION (tools/arti-build/ANDROID_NDK_VERSION)"
+        echo "  Found:    ${found_ndk:-unknown} at $ANDROID_NDK_HOME"
+        echo "  Install:  sdkmanager \"ndk;$NDK_VERSION\""
+        exit 1
+    fi
+
+    print_success "NDK: $ANDROID_NDK_HOME ($NDK_VERSION)"

    for target in "${TARGETS[@]}"; do
        if ! rustup target list --installed | grep -q "$target"; then
@@ -180,17 +225,22 @@ PATCH
 # Build
 # ============================================================================

+# Android ABI directory (as laid out under jniLibs/) for a Rust target triple.
+abi_dir_for() {
+    case "$1" in
+        aarch64-linux-android) echo "arm64-v8a" ;;
+        x86_64-linux-android) echo "x86_64" ;;
+        armv7-linux-androideabi) echo "armeabi-v7a" ;;
+        i686-linux-android) echo "x86" ;;
+    esac
+}
+
 build_for_target() {
    local target="$1"
    print_header "Building for $target"

    local arch_dir
-    case "$target" in
-        aarch64-linux-android) arch_dir="arm64-v8a" ;;
-        x86_64-linux-android) arch_dir="x86_64" ;;
-        armv7-linux-androideabi) arch_dir="armeabi-v7a" ;;
-        i686-linux-android) arch_dir="x86" ;;
-    esac
+    arch_dir="$(abi_dir_for "$target")"

    local out_dir="$OUTPUT_DIR/$arch_dir"
    mkdir -p "$out_dir"
@@ -236,8 +286,16 @@ verify_jni_symbols() {
        local arch=$(basename "$arch_dir")
        local missing=0

+        # Read the dynamic symbol table once, into a variable. Piping nm into
+        # `grep -q` per symbol looks equivalent but is not: grep exits on the
+        # first match, nm dies of SIGPIPE (141), and `set -o pipefail` then
+        # reports the pipeline as failed — so every symbol that IS exported gets
+        # reported as missing. (Reproducible on any build, old or new.)
+        local syms
+        syms="$(nm -D "$lib" 2>/dev/null || true)"
+
        for sym in "${expected_symbols[@]}"; do
-            if ! nm -D "$lib" 2>/dev/null | grep -q "$sym"; then
+            if [[ "$syms" != *"$sym"* ]]; then
                print_error "$arch: Missing symbol $sym"
                missing=1
            fi
@@ -249,6 +307,35 @@ verify_jni_symbols() {
    done
 }

+verify_ndk_stamp() {
+    print_header "Verifying NDK stamp"
+
+    if ! command -v readelf >/dev/null 2>&1; then
+        print_info "readelf not found — skipping (install binutils to enable this check)"
+        return 0
+    fi
+
+    # Every NDK-linked shared object carries .note.android.ident, which records
+    # the target API level, the NDK release name (e.g. r27d) and the NDK build
+    # number. Reading it back proves which toolchain actually produced the
+    # binary, independently of what the environment claimed — this is how the
+    # committed r25b libraries were identified in the first place.
+    for target in "${TARGETS[@]}"; do
+        local arch
+        arch="$(abi_dir_for "$target")"
+        local lib="$OUTPUT_DIR/$arch/$LIB_NAME"
+        [ -f "$lib" ] || continue
+
+        if readelf -p .note.android.ident "$lib" 2>/dev/null | grep -qw "$NDK_BUILD_NUMBER"; then
+            print_success "$arch: built by NDK $NDK_VERSION"
+        else
+            print_error "$arch: not stamped with NDK build $NDK_BUILD_NUMBER — wrong toolchain?"
+            readelf -p .note.android.ident "$lib" 2>/dev/null || true
+            exit 1
+        fi
+    done
+}
+
 # ============================================================================
 # Main
 # ============================================================================
@@ -280,6 +367,7 @@ main() {
    done

    verify_jni_symbols
+    verify_ndk_stamp

    print_header "Build complete"
    echo ""
@@ -12,7 +12,9 @@
 #   ./verify-reproducible.sh              # both ABIs (arm64-v8a + x86_64)
 #   ./verify-reproducible.sh --release    # arm64-v8a only (faster)
 #
-# Prerequisites are the same as build-arti.sh (rustup, cargo-ndk, Android NDK).
+# Prerequisites are the same as build-arti.sh (rustup, cargo-ndk, and the exact
+# Android NDK revision pinned in ANDROID_NDK_VERSION — a different revision is
+# refused, because it would change the output bytes).
 # Exit 0 = reproducible, exit 1 = builds differ.
 set -euo pipefail