From d9066572cdd9ab2805d0c5c5a3625f4abc217dca Mon Sep 17 00:00:00 2001 From: Vitor Pamplona Date: Tue, 16 Jun 2026 13:44:06 -0400 Subject: [PATCH] test(tor): add repeatable on-device networking scenario suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 60s bootstrap timeout this branch adds is invisible to every JVM test — it only manifests against a real radio. Add a device-driven harness that drives the network transitions that have historically wedged Tor and asserts the lifecycle invariants from logcat, so the behavior can be re-verified whenever Arti is bumped or the Tor management path changes. tools/tor-network-tests/run.sh — adb-driven runner, one function per scenario, PASS/FAIL per check, restores a clean network state on exit: - cold_start Active reached + no pre-ready dial storm (the #3223 gate) - offline_bootstrap empty cache + airplane: asserts the bootstrap is bounded ('bootstrap timed out' logged), then recovers on restore. This is the regression test for this PR — it FAILS on a build without the timeout (create_bootstrapped blocks 95s+). - wifi_cellular WiFi->Cellular handover recovers to Active - airplane offline pauses relays cleanly; restore recovers - pause_resume backgrounding winds relays down (~30s); resume reconnects README.md documents each scenario's rationale, device setup, when to re-run (new Arti version, TorService/TorManager/dial-gate changes), and how the suite relates to TorManagerTest / TorCircuitHealthTrackerTest / the instrumented test. Verified on a Pixel emulator (WiFi + Cellular): all scenarios pass on this branch; offline_bootstrap fails on main (no timeout), confirming the suite discriminates fixed from broken. Co-Authored-By: Claude Opus 4.8 (1M context) --- tools/tor-network-tests/README.md | 129 +++++++++++++++++ tools/tor-network-tests/run.sh | 224 ++++++++++++++++++++++++++++++ 2 files changed, 353 insertions(+) create mode 100644 tools/tor-network-tests/README.md create mode 100755 tools/tor-network-tests/run.sh diff --git a/tools/tor-network-tests/README.md b/tools/tor-network-tests/README.md new file mode 100644 index 0000000000..787d4f47cf --- /dev/null +++ b/tools/tor-network-tests/README.md @@ -0,0 +1,129 @@ +# Tor networking scenario suite + +A repeatable, device-driven test harness for Amethyst's embedded Tor (Arti) +across the network transitions that have historically broken it. These checks +live outside the JVM/instrumented test tree on purpose: they manipulate the +**device's network** (airplane mode, WiFi/cellular, app lifecycle) and assert on +the real `ArtiNative` bootstrap — things a unit test can't reach. + +## When to run this + +Run the suite whenever you touch anything in the Tor management path, and treat +a green run as a release gate for those changes: + +- **Bumping Arti** (`tools/arti-build`, the rebuilt `libarti_android.so`) — a new + Arti version can change bootstrap timing, the SOCKS error mapping, or guard + handling. This is the most important trigger. +- **`TorService.kt`** — lifecycle (`start`/`stop`/`reset`/`resetWithCleanState`), + the bootstrap-timeout handling (`ARTI_ERROR_BOOTSTRAP_TIMEOUT`), or cache/state + wiping. +- **`TorManager.kt`** — the self-heal watchdog, `onNetworkChange`, cooldowns, or + status routing. +- **The dial gate** — `WebsocketBuilder.canConnect` / the `canDial` wiring in + `AppModules.kt` that holds Tor-routed relays until the SOCKS port is ready. +- **The connectivity layer** — `ConnectivityManager` / `RelayProxyClientConnector` + (pause/resume, reconnect-on-change). + +## How it relates to the other tests + +| Layer | Where | Covers | +|-------|-------|--------| +| Pure logic (fast, deterministic) | `amethyst/src/test/.../tor/TorManagerTest.kt` | watchdog, cooldown, `onNetworkChange`, status routing — virtual time, in-memory fakes, no Arti | +| Pure logic (fast) | `amethyst/src/test/.../service/relayClient/TorCircuitHealthTrackerTest.kt` | the Active-but-failing discriminator (companion circuit-health PR) | +| Real Arti, single process | `amethyst/src/androidTest/.../tor/TorBootstrapInstrumentedTest.kt` (`@Ignore`'d) | real `initialize` → `create_bootstrapped` → SOCKS round-trip + `destroy` | +| **Real Arti + real network transitions** | **this suite** | **bootstrap bounding, network change, airplane, pause/resume — end to end** | + +The unit tests are the first line of defense (run in CI). This suite is the +end-to-end backstop you run by hand (or in a device lab) before shipping a Tor +change, because the failures it catches only appear with a real radio. + +## Device setup (once) + +1. Connect **one** device or emulator with internet egress to the Tor network. + An emulator that exposes both WiFi and Cellular (e.g. a stock Pixel AVD) is + ideal — the `wifi_cellular` scenario needs a second transport to fail over to. +2. Install the **debug** build: `./gradlew :amethyst:installPlayDebug`. +3. In the app, set **Tor = INTERNAL** and enable routing relays over Tor + (Settings → Network/Tor). The suite does **not** change Tor settings; it only + manipulates the network and the app lifecycle, so it tests the config the user + actually runs. + +## Running + +```bash +# full suite +tools/tor-network-tests/run.sh + +# a single scenario +tools/tor-network-tests/run.sh offline_bootstrap + +# override the package (e.g. a different flavor) +AMETHYST_PKG=com.vitorpamplona.amethyst.debug tools/tor-network-tests/run.sh +``` + +Exit code is `0` only if every selected scenario passes. The suite restores a +clean network state (airplane off, WiFi+data on) when it finishes. + +## Scenarios + +Each scenario clears logcat, drives a transition, then asserts on Tor's lifecycle +log markers (`SOCKS proxy active`, `bootstrap timed out`, `onNetworkChange`, +`Pausing/Resuming Relay Services`, `OnOpen`). + +### `cold_start` +Force-restart on a healthy network. **Asserts:** Tor reaches `Active`, and the +number of Tor-routed dials issued *before* the SOCKS port was ready stays ~0 +(the #3223 gate). A spike here means the gate regressed and the pool is hammering +a dead proxy during bootstrap. + +### `offline_bootstrap` — the headline regression test for this PR +Wipe Arti state (empty cache), enable airplane mode, then cold-start so +`create_bootstrapped` runs with no network and nothing cached. **Asserts:** the +native bootstrap is *bounded* — a `bootstrap timed out` line appears (the 60s +`tokio::time::timeout` releasing `lifecycleMutex` so the self-heal watchdog can +run), instead of blocking for many minutes. Then restore the network and assert +recovery to `Active`. + +> Without the timeout (`fix: bound Arti bootstrap with a 60s timeout`, this PR), +> `create_bootstrapped` blocks indefinitely holding the lifecycle lock — observed +> wedging at 95s+ on an emulator — and this scenario fails with "no 'bootstrap +> timed out'". That regression is invisible to every JVM test, which is why this +> suite exists. + +### `wifi_cellular` +With Tor Active, disable WiFi (the default route fails over to cellular), wait, +re-enable. **Asserts:** the transport change drives `onNetworkChange` and Tor +returns to `Active`. Catches the "stale guards/circuits from the old network" +class of wedge. + +### `airplane` +Toggle airplane mode on (full offline) then off. **Asserts:** relays pause +cleanly on connectivity loss (no Tor thrash while offline) and Tor + relays +recover on restore. + +### `pause_resume` +Background the app (HOME), wait for the `WhileSubscribed(30s)` wind-down, then +foreground it. **Asserts:** relays pause (`Pausing Relay Services`) while +backgrounded and reconnect (`OnOpen` increases) on resume. The reconnection burst +on resume is the same shape as the post-Active warmup burst, so this also guards +against the circuit-health self-heal firing a false positive on resume. + +## Interpreting failures + +The summary prints `PASS`/`FAIL` per check with a one-line reason. Common ones: + +- `offline_bootstrap` FAIL "no 'bootstrap timed out'" → the native timeout isn't + in the installed `.so` (rebuild `libarti_android.so` from `tools/arti-build`, + or you're on a branch without the fix). +- `cold_start` FAIL "pre-ready doomed dials >5" → the `canDial`/`canConnect` gate + regressed. +- `wifi_cellular` / `airplane` FAIL "did not reach Active" → bootstrap is slow or + wedged on the new network; pull the full log and look for repeated + `ExitTimeout` / `AllGuardsDown`. + +For a deeper look at any run, dump the lifecycle directly: + +```bash +adb logcat -d | grep -E "TorService|TorManager|ManageRelayServices" \ + | grep -iE "Initializing|bootstrap|proxy active|timed out|self-heal|reset|onNetworkChange|Pausing|Resuming" +``` diff --git a/tools/tor-network-tests/run.sh b/tools/tor-network-tests/run.sh new file mode 100755 index 0000000000..2ec088f781 --- /dev/null +++ b/tools/tor-network-tests/run.sh @@ -0,0 +1,224 @@ +#!/usr/bin/env bash +# +# Repeatable Tor networking scenario suite for Amethyst (Android). +# +# Drives a connected device/emulator through the network transitions that have +# historically broken Tor (cold start, hostile/offline bootstrap, WiFi<->Cellular +# handover, airplane mode, app pause/resume) and asserts the Tor lifecycle +# invariants from logcat. See README.md for the rationale of each scenario and +# when to re-run (new Arti version, changes to TorService/TorManager/the dial gate). +# +# Usage: +# tools/tor-network-tests/run.sh [scenario ...] +# +# With no args, runs the full suite. Otherwise runs only the named scenarios: +# cold_start offline_bootstrap wifi_cellular airplane pause_resume +# +# Requirements: +# - adb on PATH with exactly one device/emulator connected and Tor egress. +# - The Amethyst *debug* build installed with Tor set to INTERNAL and routing +# relays over Tor (see README "Device setup"). The suite never changes Tor +# settings — it only manipulates the network and the app lifecycle. +# +# Exit code: 0 if every selected scenario passes, 1 otherwise. + +set -u + +PKG="${AMETHYST_PKG:-com.vitorpamplona.amethyst.debug}" +ACT="${AMETHYST_ACTIVITY:-$PKG/com.vitorpamplona.amethyst.ui.MainActivity}" + +# ---- thresholds (mirror the constants in TorService.kt / TorManager.kt) -------- +BOOTSTRAP_DEADLINE_S=120 # cold start should reach Active within this +TIMEOUT_LOG_DEADLINE_S=90 # empty-cache offline start should log the bootstrap timeout within this +RECOVER_DEADLINE_S=120 # after restoring network, Tor should reach Active within this +PAUSE_WINDDOWN_S=60 # backgrounded relays should pause within this (WhileSubscribed=30s + slack) + +PASS=0 +FAIL=0 +RESULTS=() + +# ---- helpers ------------------------------------------------------------------- + +log() { printf ' %s\n' "$*"; } +banner() { printf '\n=== %s ===\n' "$*"; } + +restart_clean_network() { + adb shell cmd connectivity airplane-mode disable >/dev/null 2>&1 + adb shell svc wifi enable >/dev/null 2>&1 + adb shell svc data enable >/dev/null 2>&1 +} + +force_restart_app() { + adb shell am force-stop "$PKG" >/dev/null 2>&1 + sleep 1 + adb shell am start -n "$ACT" >/dev/null 2>&1 +} + +wipe_arti_state() { + adb shell "run-as $PKG rm -rf /data/data/$PKG/files/arti" >/dev/null 2>&1 +} + +# Wait up to $1 seconds for logcat (current buffer) to contain the regex in $2. +# Returns 0 as soon as it appears, 1 on timeout. +wait_for_log() { + local deadline=$1 pattern=$2 i + for (( i = 0; i < deadline; i += 3 )); do + if adb logcat -d 2>/dev/null | grep -qE "$pattern"; then return 0; fi + sleep 3 + done + return 1 +} + +count_log() { adb logcat -d 2>/dev/null | grep -cE "$1"; } + +# Doomed Tor dials issued BEFORE the SOCKS proxy was ready (the #3223 gate must keep this ~0). +pre_ready_dials() { + local first_active + first_active=$(adb logcat -d 2>/dev/null | grep -m1 "SOCKS proxy active" | awk '{print $2}') + [ -z "$first_active" ] && { echo "-1"; return; } + adb logcat -d 2>/dev/null | grep "SOCKS: Connection refused" \ + | awk -v t="$first_active" '$2 < t' | wc -l | tr -d ' ' +} + +record() { # name pass(0/1) detail + if [ "$2" -eq 0 ]; then PASS=$((PASS+1)); RESULTS+=("PASS $1 — $3"); log "PASS: $3" + else FAIL=$((FAIL+1)); RESULTS+=("FAIL $1 — $3"); log "FAIL: $3"; fi +} + +# ---- scenarios ----------------------------------------------------------------- + +scenario_cold_start() { + banner "cold_start — Tor bootstraps to Active on a healthy network, no pre-ready dial storm" + restart_clean_network; sleep 2 + adb logcat -c + force_restart_app + if wait_for_log "$BOOTSTRAP_DEADLINE_S" "SOCKS proxy active"; then + local pre; pre=$(pre_ready_dials) + if [ "$pre" -le 5 ]; then + record cold_start 0 "reached Active; pre-ready doomed dials=$pre (gate OK)" + else + record cold_start 1 "reached Active but pre-ready doomed dials=$pre (>5 — dial gate regressed)" + fi + else + record cold_start 1 "Tor did not reach Active within ${BOOTSTRAP_DEADLINE_S}s" + fi +} + +scenario_offline_bootstrap() { + banner "offline_bootstrap — empty cache + no network must NOT wedge (PR: 60s bootstrap timeout)" + adb shell am force-stop "$PKG" >/dev/null 2>&1; sleep 1 + wipe_arti_state + adb shell cmd connectivity airplane-mode enable >/dev/null 2>&1 + sleep 3 + adb logcat -c + force_restart_app + log "cold-started offline with empty cache; waiting for the bootstrap-timeout log..." + if wait_for_log "$TIMEOUT_LOG_DEADLINE_S" "bootstrap timed out"; then + record offline_bootstrap 0 "native bootstrap bounded — 'bootstrap timed out' logged (lock released for self-heal)" + else + record offline_bootstrap 1 "no 'bootstrap timed out' within ${TIMEOUT_LOG_DEADLINE_S}s — create_bootstrapped is unbounded (PR #3225 not effective)" + fi + log "restoring network; expecting recovery to Active..." + restart_clean_network + if wait_for_log "$RECOVER_DEADLINE_S" "SOCKS proxy active"; then + record offline_bootstrap_recover 0 "reached Active after network restore" + else + record offline_bootstrap_recover 1 "did not reach Active within ${RECOVER_DEADLINE_S}s after restore" + fi +} + +scenario_wifi_cellular() { + banner "wifi_cellular — WiFi off (failover to cellular) then on triggers onNetworkChange + recovery" + restart_clean_network + if ! wait_for_log "$BOOTSTRAP_DEADLINE_S" "SOCKS proxy active"; then + record wifi_cellular 1 "precondition failed: Tor not Active before the handover"; return + fi + adb logcat -c + adb shell svc wifi disable >/dev/null 2>&1 + sleep 20 + adb shell svc wifi enable >/dev/null 2>&1 + sleep 5 + local changes; changes=$(count_log "onNetworkChange|network identity changed") + if wait_for_log "$RECOVER_DEADLINE_S" "SOCKS proxy active"; then + record wifi_cellular 0 "handover recovered to Active (network-change events seen: $changes)" + else + record wifi_cellular 1 "Tor did not return to Active within ${RECOVER_DEADLINE_S}s after the handover" + fi +} + +scenario_airplane() { + banner "airplane — full offline pauses relays cleanly; restore recovers Tor + resumes relays" + restart_clean_network + wait_for_log "$BOOTSTRAP_DEADLINE_S" "SOCKS proxy active" >/dev/null + adb logcat -c + adb shell cmd connectivity airplane-mode enable >/dev/null 2>&1 + if wait_for_log 30 "Connectivity Off|Pausing Relay Services"; then + log "relays paused on connectivity loss" + else + log "WARN: no 'Pausing Relay Services' seen on airplane-on" + fi + sleep 5 + adb shell cmd connectivity airplane-mode disable >/dev/null 2>&1 + if wait_for_log "$RECOVER_DEADLINE_S" "SOCKS proxy active|Resuming Relay Services"; then + record airplane 0 "recovered after airplane off/on" + else + record airplane 1 "did not recover within ${RECOVER_DEADLINE_S}s after airplane off" + fi +} + +scenario_pause_resume() { + banner "pause_resume — backgrounding winds relays down (~30s); resume reconnects them" + restart_clean_network + wait_for_log "$BOOTSTRAP_DEADLINE_S" "SOCKS proxy active" >/dev/null + adb logcat -c + adb shell input keyevent KEYCODE_HOME + if wait_for_log "$PAUSE_WINDDOWN_S" "Pausing Relay Services"; then + record pause 0 "relays paused after backgrounding (WhileSubscribed wind-down)" + else + record pause 1 "relays did not pause within ${PAUSE_WINDDOWN_S}s of backgrounding" + fi + local before; before=$(count_log "OnOpen") + adb shell am start -n "$ACT" >/dev/null 2>&1 + wait_for_log 30 "Resuming Relay Services" >/dev/null + sleep 15 + local after; after=$(count_log "OnOpen") + if [ "$after" -gt "$before" ]; then + record resume 0 "relays reconnected on resume (OnOpen $before -> $after)" + else + record resume 1 "no relay reconnects observed on resume (OnOpen stuck at $before)" + fi +} + +# ---- driver -------------------------------------------------------------------- + +ndev=$(adb devices | grep -cw "device") +if [ "$ndev" -ne 1 ]; then + echo "ERROR: need exactly one connected device/emulator (found $ndev). Set ANDROID_SERIAL or unplug extras." >&2 + exit 2 +fi +if ! adb shell pm list packages 2>/dev/null | grep -q "$PKG"; then + echo "ERROR: $PKG is not installed. Install the debug build first (see README)." >&2 + exit 2 +fi + +scenarios=("$@") +[ ${#scenarios[@]} -eq 0 ] && scenarios=(cold_start offline_bootstrap wifi_cellular airplane pause_resume) + +echo "Tor network suite — package=$PKG, scenarios: ${scenarios[*]}" +for s in "${scenarios[@]}"; do + case "$s" in + cold_start) scenario_cold_start ;; + offline_bootstrap) scenario_offline_bootstrap ;; + wifi_cellular) scenario_wifi_cellular ;; + airplane) scenario_airplane ;; + pause_resume) scenario_pause_resume ;; + *) echo " unknown scenario: $s" ; FAIL=$((FAIL+1)) ; RESULTS+=("FAIL $s — unknown scenario") ;; + esac +done + +restart_clean_network # leave the device in a good state + +banner "SUMMARY" +for r in "${RESULTS[@]}"; do printf ' %s\n' "$r"; done +printf '\n %d passed, %d failed\n' "$PASS" "$FAIL" +[ "$FAIL" -eq 0 ]