Five protocol-level fixes and a batch of harness correctness fixes to get
the headless Marmot/Whitenoise interop harness from 1/13 to 5/13 passing
cleanly, with the remaining failures all rooted in wn's per-account
serial event-processor retry backoff (which drops undecryptable
pre-membership commits after several minutes) rather than amy behaviour.
quartz + commons
----------------
* MarmotGroupData: hold CURRENT_VERSION at 2. mdk-core (the Rust MLS
engine used by whitenoise-rs) strict-rejects v3 payloads with
`ExtensionFormatError("Trailing bytes in NostrGroupDataExtension")`
— our v3 welcomes and GCE commits never apply, so every cross-client
group flow dies at welcome processing. We still parse v3 happily on
the way in; we just don't emit it until mdk publishes the
forward-compat fix MIP-01 mandates.
* MarmotManager.updateGroupMetadata: MERGE extensions instead of
REPLACING. RFC 9420 §12.1.7 says GCE proposals blow away the old
extension list; callers that pass only [marmot_group_data] dropped
[required_capabilities], which peers then reject. Preserve every slot
except the one we're updating.
* MarmotManager.createGroup: new optional `initialMetadata` parameter
that bakes MarmotGroupData into epoch-0 GroupContext.extensions
directly. Without it, creators had to publish a pre-membership
"bootstrap" commit that no later joiner could decrypt — each such
peer then burned their retry budget on an undecryptable kind:445
before seeing the real state. Threaded through MlsGroup.create /
MlsGroupManager.createGroup.
* MarmotManager.mlsGroupIdHex: new translation helper so any code
juggling the MIP-01 nostr_group_id (what amy indexes on) and the
MLS GroupContext groupId (what mdk indexes on) can cross-reference
them without reaching into MlsGroupManager directly.
amethyst module
---------------
* Account.leaveMarmotGroup: self-demote before SelfRemove per MIP-01,
and promote a surviving member to admin first if the caller is the
sole admin (otherwise we'd throw "admin depletion"). Matches the
cli/GroupMembershipCommands.leave flow.
cli (amy)
---------
* Context.syncIncoming: don't advance `giftWrapSince` on empty polls
(so the first-ever sync doesn't bump the cursor past every
past-timestamped wrap we've ever been sent), subtract 2 days lookback
when filtering (NIP-59 randomWithTwoDays gift wraps can have any
createdAt in the last 48h), and only advance `groupSince` for groups
we actually received events for.
* Context.syncIncoming: after ingest, if any Welcome consumed a
KeyPackage, rotate and publish a fresh one immediately. MIP-00
requires this — a KP can only be welcomed once and leaving the
consumed one on relays just means later senders invite us with a
bundle we no longer have private keys for.
* Context.resolveGroupId: accept either nostr_group_id (amy's primary
key) or the MLS GroupContext groupId (what wn emits) on every verb
that takes a group id. Wired through GroupAdd/Remove/Leave/Metadata
/Read, Message send/list, and all the await* verbs so harness
scripts never have to juggle both forms for a single group.
* GroupCreateCommand: bake initial metadata into epoch 0 (see the
quartz change above). Dropped the now-redundant bootstrap commit
publish and tightened the JSON output to include `mls_group_id`.
* GroupMembershipCommands.leave: self-demote admin before SelfRemove;
promote an heir if we're the only admin, otherwise the GCE would
deplete admins and the leave aborts.
* MarmotIngest.ingestGiftWrap: unwrap the sealed-rumor layer. NIP-59
wrap is gift-wrap(kind:1059) → seal(kind:13) → rumor; the old code
only unwrapped once and then checked `inner.kind == 444`, which is
always false because inner is actually the seal. Unseal once more
before the Welcome check. This single fix is what unsticks every
amy-side Welcome ingestion.
wn harness patches + scripts
----------------------------
* whitenoise-defaults-env.patch: honour $WHITENOISE_DISCOVERY_RELAYS in
`Relay::defaults()` (release builds otherwise bake damus.io / primal
/ nos.lol into every new account's NIP-65 / Inbox / KeyPackage
lists, which breaks publishing and prevents the inbox subscription
plane from ever reaching an operational state in a sandbox).
* setup.sh: sleep 2s after amy's initial kind:30443 publish so
nostr-rs-relay has a chance to fsync before wn's first targeted
discovery query. Without it wn's `keys check` races the relay's
WAL flush and intermittently returns NotFound.
* lib.sh: peel wn's `{"result": …}` wrapper in `jq_group_id`,
`wait_for_invite`, `wait_for_message`, `wait_for_member`. Post-v0.2
wn `--json` output nests everything under `.result` (and
`groups invites[]` nests further under `.group.mls_group_id`) —
these helpers were still pattern-matching on the flat shape, so
they returned empty strings for a perfectly good response.
* tests-{create,manage,extras}.sh: track both group IDs per test
(amy's nostr + wn's MLS), pass each CLI the id it understands, and
bump the post-commit wait timeouts to 90–120s so wn's exponential
retry backoff has time to work through the pre-membership commits
it can't decrypt and get to the ones it can.
https://claude.ai/code/session_016kAxdp6ubB5CnF9URhCEzP
wn now wraps account listings in {"result": [{...}]} so extract_pubkey's
object + array fallbacks both missed it and ensure_identity bailed with
"could not determine \$who npub" immediately after create-identity.
Also: create-identity on a fresh box can exit non-zero with
"failed to connect to any relays" even when the account *was* created
locally. Probe --json whoami afterwards before calling that a fatal.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Containers / CI often block the keyutils syscalls wnd uses by default
to store secret keys (add_key returns EACCES, keyctl_search returns
ENOSYS). The integration-tests feature ships a mock keyring store, but
the stock wnd binary doesn't wire it up. Add a tiny opt-in patch.
Changes:
- headless/patches/whitenoise-mock-keyring.patch: if built with the
integration-tests feature and \$WHITENOISE_MOCK_KEYRING is set, wnd
calls Whitenoise::initialize_mock_keyring_store() before the normal
init path. Harmless on production builds.
- headless/setup.sh: apply both harness patches generically from a
list; build wn/wnd with --features cli,integration-tests; export
WHITENOISE_MOCK_KEYRING=1 alongside WHITENOISE_DISCOVERY_RELAYS when
launching each daemon.
- preflight swaps keyctl for patch in required tools — we no longer
need a real session keyring.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Without this, wnd exits on startup with NoRelayConnections in any
environment that can't reach wnd's baked-in public discovery relay
set (nos.lol, relay.damus.io, …). The headless harness runs entirely
on a loopback relay, so hitting the public internet is never OK.
Changes:
- headless/patches/whitenoise-discovery-env.patch: adds an env-var
override at the top of DiscoveryPlaneConfig::curated_default_relays.
When \$WHITENOISE_DISCOVERY_RELAYS is set (comma-separated) we use
that list instead of the hardcoded public one.
- headless/setup.sh: applies the patch in preflight (idempotent —
checks for a .headless-discovery-patched marker), invalidates the
previous wn/wnd build on first patch, rebuilds, and threads
\$WHITENOISE_DISCOVERY_RELAYS=\$RELAY_URL into every start_daemon
invocation.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
nostr-rs-relay's build.rs needs protoc to compile the nauthz gRPC
definitions. Without it the build fails deep in prost-build with a
panic that's hard to read, so fail up front with a pointer to
`apt-get install protobuf-compiler`.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Every test now flows through one loopback nostr-rs-relay on
ws://127.0.0.1:8080 — no public relays, no egress. Addresses the
repeated 503s we saw from the sandbox and the kind:445 rejections we
hit on public relays even with internet.
Changes:
- Preflight clones https://github.com/scsibug/nostr-rs-relay into
state-headless/nostr-rs-relay and runs `cargo build --release`
(~3 min first run). Cache hits on subsequent runs.
- New start_local_relay / stop_local_relay functions render a minimal
config.toml each run (binds to 127.0.0.1, no kind blacklist, data
under state-headless/relay) and wait up to 20s for the port.
- configure_relays collapsed to a single-relay path — A/B/C all point
at \$RELAY_URL. Kept publish-lists + key-package publish as smoke
signals for the advertise path.
- Flags pared down: drop --local-relays (always local now), add
--port N for when 8080 is taken. --no-build still honoured.
- trap wires relay shutdown alongside daemon shutdown.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Matches the 13 scenarios in marmot-interop.sh but runs A via the new
`amy` CLI instead of prompting the human. Identities B and C still go
through `wn`/`wnd` from whitenoise-rs.
Layout:
- marmot-interop-headless.sh — top-level entrypoint
- headless/setup.sh — preflight, daemon lifecycle, identities
- headless/helpers.sh — amy wrappers + assertion helpers
- headless/tests-create.sh — tests 01..05
- headless/tests-manage.sh — tests 06, 07, 08, 11
- headless/tests-extras.sh — tests 09, 10, 12, 13
Reuses lib.sh for colours, record_result, wait_for_invite/message,
jq_group_id, and dump_daemon_diagnostics.
Keeps the interactive marmot-interop.sh intact for final UI sign-off.
https://claude.ai/code/session_01M6dCKAF5Y1VyHGZPjzwDXq
Adds a dump_daemon_diagnostics helper (relays per type, wn debug health,
relay-control-state, pending invites, wnd stderr tail) and wires it into
Test 02 and Test 04 on both sides of the poll (pre-invite baseline +
post-timeout). wait_for_invite now prints a ~10s heartbeat with the
pending-invite count and the most recent welcome/giftwrap line from wnd
stderr. On timeout the script also prompts the operator to paste the
Amethyst-side MarmotDbg logcat so both ends of the gift-wrap pipe end up
in one log file.
wn prints either JSON (with --json) or a yaml-ish "key: value" pretty
form depending on the command and build. On the user's macOS run,
create-identity printed yaml ("pubkey: npub1..."), the subsequent
wn --json whoami didn't return the expected JSON shape, and
ensure_identity bailed with "could not determine npub for B".
Add a shared extract_pubkey helper in lib.sh that tries, in order:
1. JSON object .pubkey / .npub / .public_key
2. JSON array .[0].pubkey / .[0].npub / .[0].public_key
3. yaml-ish "pubkey: ..." via sed
4. yaml-ish "npub: ..." via sed
ensure_identity and prompt_for_a_npub now use it, and also fall back
to the non-JSON whoami output if --json returns nothing. On total
failure, the raw output is echoed to the log for diagnosis.
Also made npub_to_hex best-effort with the same double-try approach.
When hex lookup fails, it returns the npub unchanged — downstream
expect_contains assertions work against either form since wn's own
group listings may use either encoding.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
wnd only accepts --data-dir and --logs-dir. The socket path is
{data_dir}/release/wnd.sock (release build) or {data_dir}/dev/wnd.sock
(debug build) per src/cli/config.rs in whitenoise-rs. Updated the
script to point B_SOCKET / C_SOCKET at the correct release path and
dropped --socket from the wnd invocation.
Reported by user running the harness on macOS:
error: unexpected argument '--socket' found
Usage: wnd --data-dir <PATH> --logs-dir <PATH>
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
Interactive bash script under tools/marmot-interop/ that validates
Amethyst's Marmot/MLS implementation against the whitenoise-rs wn/wnd
CLI. Builds wn/wnd from source on first run, launches two daemons for
Identities B and C, configures public (or --local-relays) Nostr relays
shared with Amethyst, then walks a human operator through 13 scenarios
covering MIP-00 KeyPackages, MIP-01 metadata, MIP-02 Welcome,
MIP-03 group messages, admin changes, reactions, concurrent-commit race,
leave, offline catch-up, and KeyPackage rotation. Push-notification
(MIP-05) test opt-in via --transponder.
No Amethyst code is modified — black-box testing only. State (daemon
sockets, logs, whitenoise-rs checkout, results TSVs) is kept under
tools/marmot-interop/state/ and gitignored.
https://claude.ai/code/session_01PacmbE1KKWMSRmDw3bnCNo
All log messages go through send_log_to_java() → Kotlin ArtiLogCallback
→ Log.d("TorService"), which already writes to logcat. The android_logger
module was a second FFI call to __android_log_write that duplicated
every line.
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
All features and APIs verified present in 0.41:
- tokio, rustls, compression, onion-service-client, static-sqlite
- TorClient::create_bootstrapped, from_directories, connect()
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
- Set default-features = false on arti-client and tor-rtcompat
- Removed bridge-client (UI doesn't expose bridge config yet)
- Narrowed tokio features from "full" to only what the SOCKS proxy
needs: rt-multi-thread, net, io-util, time, macros
Kept: tokio, rustls, compression, onion-service-client, static-sqlite
(all required for Amethyst's .onion relay support)
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
The Guardian Project's arti-mobile-ex AAR has three problems:
1. No 16KB page-aligned binaries (required for Google Play)
2. ArtiProxy's stop()+start() causes state file lock conflicts
(lock is tied to TorClient object lifetime, released only on GC)
3. ~140MB AAR size
Replace with a custom JNI bridge built from Arti source, following
BitChat's proven approach:
Build tooling (tools/arti-build/):
- build-arti.sh: Clones official Arti, compiles with cargo-ndk
for ARM64 + x86_64, NDK 25+ for 16KB page alignment
- Cargo.toml: Minimal deps with size-optimized release profile
- src/lib.rs: Custom SOCKS5 proxy with proper lifecycle:
- initialize() creates TorClient once (holds state lock forever)
- startSocksProxy() binds port and accepts connections
- stopSocksProxy() aborts listener only (TorClient stays alive)
This cleanly separates "stop routing traffic" from "destroy client"
Kotlin side:
- ArtiNative.kt: JNI declarations + ArtiLogCallback interface
- TorService.kt: Uses ArtiNative directly, start() initializes +
starts proxy, stop() only stops proxy (no lock issues)
- TorManager.kt: Restored stop() calls for OFF/EXTERNAL modes
since our native stop is now safe
Removed: arti-mobile-ex dependency from build.gradle and version catalog
Native libraries must be built separately:
cd tools/arti-build && ./build-arti.sh
https://claude.ai/code/session_01BApgDd5udqBzMqysSRMpZu
Claude
Vitor Pamplona