amethyst/quartz/plans/2026-09-08-marmot-spec-resync.md

Marmot: resync against the adopted spec and current MDK

Status: Stages 0-7 landed. Convergence is on the inbound path, the superseded CommitOrdering tiebreak is deleted, the app layer publishes current-profile KeyPackages and can create current-profile groups, and publish-before-apply is enforced. The MDK interop harness has still never been run against a live MDK build.

Sources checked on 2026-09-08:

• marmot-protocol/marmot @ 4a2bc65 ("Specify durability and restart contract", 2026-08-13)
• marmot-protocol/mdk @ ef73de5 (2026-09-08), released tag v0.9.19 (2026-09-07)
• marmot-protocol/whitenoise-rs @ 95fa0b8 (2026-08-05) — archived/obsolete

1. Executive summary

Our Marmot implementation targets the MIP-era protocol (MIP-00…MIP-05), which upstream deprecated on 2026-07-02 when the "v2" spec was imported and marked adopted. Everything we ship — MarmotGroupData (extension 0xF2EE), the timestamp+event-id commit tiebreak, the kind 10051 KeyPackage relay list, the encoding tag — is either superseded or now explicitly forbidden.

The single hardest break: current MDK rejects our groups outright. MDK classifies a group as Legacy or Current by looking at RequiredCapabilities:

• Legacy ⇔ requires MLS extension 0xf2f1 (marmot.account-identity-proof.v1)
• Current ⇔ requires app component 0x8009 (marmot.member.account-identity-proof.v2)

We require neither — our groups only require 0xF2EE. MDK's protocol_profile_of_group_extensions (crates/cgka-engine/src/account_identity_proof.rs:509) returns an error for (false, false):

"group requires neither legacy proof extension 0xf2f1 nor current proof component 0x8009"

We never implemented the account identity proof at all (grep -ri f2f1 --include=*.kt finds nothing). So we are not even a valid legacy Marmot client by the spec's own definition — foundation/account-identity-proof-v1.md says "a member without a valid v1 proof is not a valid v1 member."

This is not a patch-up job. Reaching the current profile means implementing the MLS extensions draft (app_data_dictionary, app_components, app_data_update) inside our from-scratch Kotlin MLS stack, plus a convergence engine and a group lifecycle state machine we have no equivalent of.

2. Verdict on the Bitcoin++ Insider draft

The draft is a technically accurate description of the current spec. Every checkable protocol claim in it holds up against protocol-core/convergence.md and MDK's source:

Draft claim	Verified against
Five-commit rewind horizon	max_rewind_commits = 5; V1_MAX_REWIND_COMMITS in crates/cgka-engine/src/convergence.rs:14
1s quiescence / 5s absolute pass deadline	settlement_quiescence_ms = 1000, max_convergence_pass_ms = 5000; crates/cgka-engine/src/canonicalization.rs:23,25
Witness quorum = 2 distinct senders on ≥1 epoch, boost capped at 1 commit	witness_quorum_senders_per_epoch = 2, witness_quorum_epochs = 1, max_witness_override_depth = 1
"3-commit branch with quorum ties a 4-commit branch, then wins the next comparison"	selection step 1 (effective_commit_depth) then step 2 (quorum beats no-quorum)
Tiebreak chain ends at tip priority → committer → digest of commit bytes	convergence.md "Branch selection" steps 4–6
Parentage from MLS replay, never relay metadata	convergence.md "Candidate branches"
Publish-before-advancing, ack from ≥1 endpoint in recipient scope	protocol-core/publish-lifecycle.md "Publish obligation"
Earlier implementations broke same-epoch ties on outer Nostr timestamp + event id	exactly our mip03GroupMessages/CommitOrdering.kt:37-40
Conformance simulator drives the production-shaped Nostr peeler	crates/cgka-conformance-simulator/ + crates/transport-nostr-peeler/ both exist

One substantive error. The draft says: "If effective depth ties, the selector compares quorum status, raw commit depth, accumulated witness score, …". The spec explicitly rules out a raw-depth step:

raw_commit_depth has no separate comparison step. It is already part of effective_commit_depth; if effective depth and witness-quorum status are both tied, a further raw-depth comparison is necessarily tied as well.

Harmless for the narrative, wrong if someone implements from the article.

Claims we could not verify here (no non-GitHub network): the White Noise release versions, the fips.network research status, and the downstream apps (Tubestr, AgentNoise, Burrow, Botburrow). The draft's own editorial hold flags most of these as TODO anyway.

What the draft is missing for us — and this is why it reads as vague: it describes the convergence policy but not the surrounding breaking changes that actually dominate our workload. It never mentions that marmot_group_data was dissolved into app_data_dictionary components, that the account identity proof moved from extension 0xf2f1 to component 0x8009, that KeyPackage relay discovery moved from kind 10051 to NIP-65 kind 10002, or that group disbanding (marmot.group.lifecycle.v1) was added. Those are the real work.

3. Upstream changelog since our implementation was written

3a. The structural break (2026-07-02)

The MIP documents were deprecated wholesale. mip-coverage.md is now only a historical map. The spec is reorganized into foundation/, protocol-core/, app-components/, transports/, features/.

MIP-01's monolithic marmot_group_data (0xF2EE) was split into app components carried in the MLS app_data_dictionary extension (0x0006, draft-ietf-mls-extensions-10), mutated via the app_data_update proposal (0x0008):

MIP-era field of MarmotGroupData	New owner
name, description	0x8001 marmot.group.profile.v1
admin_pubkeys	0x8003 marmot.group.admin-policy.v1
nostr_group_id, relays	0x8004 marmot.transport.nostr.routing.v1
image_*	0x8002 marmot.group.blossom.image.v1
disappearing_message_secs	0x8005 marmot.group.message-retention.v1

Plus components with no MIP-era equivalent: 0x8007 avatar-url, 0x8008/0x800b encrypted-media v1/v2, 0x8009 account-identity-proof v2, 0x800c group-lifecycle, 0x8006 agent-text-stream, 0x800a multi-device-join (draft).

3b. Spec commits after adoption

• 2026-07-03 Align admin-policy, membership, and role-change invariants
• 2026-07-05 Tighten wire-boundary validation rules (tag cardinality, pre-peel validation)
• 2026-07-23 Resolve the spec issue sweep
• 2026-07-23 Specify push owner proof signing (kind 451)
• 2026-07-28 Specify terminal MLS group disbanding (marmot.group.lifecycle.v1, Disbanded state)
• 2026-07-30 Define convergence assurance contract (protocol-core/convergence.md as it stands)
• 2026-08-13 Specify durability and restart contract (protocol-core/durability.md)

3c. MDK

MDK is a different codebase from the mdk-core we were byte-matching. It is now a 20+ crate workspace at v0.9.19, on a fork of OpenMLS 0.8.1 (erskingardner/openmls @ 59e7d3b) built with the extensions-draft feature.

whitenoise-rs is archived (obsolescence warning added 2026-08-05, pinned at mdk-core 0.8.0). The wn / wnd binaries our interop harness drives now live in mdk/crates/cli. Our cli/tests/marmot/marmot-interop.sh:101 still clones the dead repo.

4. Gap analysis

Legend: BLOCKER = breaks interop today · BREAK = wire-visible divergence · GAP = required behavior we don't implement.

4.1 Identity — BLOCKER

We have no account identity proof of any kind. Both profiles require one.

• v1 (0xf2f1): BIP-340 Schnorr over a domain-separated preimage, in LeafNode extensions, required in RequiredCapabilities. Test vector in foundation/account-identity-proof-v1.md.
• v2 (0x8009): MarmotAuthorizationProof (104 bytes: signer_pubkey[32] || created_at:u64 || signature[64]) in the LeafNode app_data_dictionary, signing a synthetic kind 450 NIP-01 event. Test vector in app-components/account-identity-proof-v2.md.

Ours: nothing. KeyPackageRotationManager.kt:528 advertises 0xF2EE, 0x000A only.

4.2 Group state carrier — BLOCKER

quartz/.../mip01Groups/MarmotGroupData.kt (TLS struct under extension 0xF2EE) has no counterpart in the current profile. Needs: AppDataDictionary / ComponentData codecs, the app_components (0x0001) and safe_aad (0x0002) upstream components, the AppDataUpdate proposal (0x0008), and per-component encode/decode/validate/authorize for 0x8001–0x800c.

Our Proposal.kt has Add/Update/Remove/SelfRemove/GroupContextExtensions/Psk/ReInit/ ExternalInit — no AppDataUpdate, no AppEphemeral.

Blast radius of MarmotGroupData: 22 files across quartz, commons, amethyst, cli.

4.3 Convergence — GAP (the article's subject)

(As of the gap analysis; CommitOrdering.kt has since been deleted and replaced by MarmotConvergenceEngine — see Stage 6.) We had none of it. CommitOrdering.kt implemented the superseded rule (lowest created_at, then lowest Nostr event id) and was wired into MarmotInboundProcessor as a per-(group, epoch) bucket. The spec forbids using transport arrival order, transport timestamps, or outer event ids in branch selection at all.

Missing, all of protocol-core/convergence.md:

• bounded pass scheduler with pass_base_epoch snapshot, 1s quiescence / 5s hard deadline, frozen input batch, Syncing/Resolving/Settled/Blocked status;
• candidate-graph construction by MLS replay from retained states; deferred-commit handling with epoch-based expiry (canonical_tip_epoch - commit_source_epoch > 5);
• eligibility on pass_base_epoch - fork_epoch <= 5;
• app-payload witnesses counted by distinct Marmot account per branch epoch, capped;
• the six-step branch comparison;
• disposition assignment (accepted/deferred/stale/invalidated/BeyondAnchor) and the withdrawal of app payloads and state notifications from losing branches;
• the fair-scheduling preparation opportunity for a queued admin intent.

4.4 Group lifecycle state machine — GAP

protocol-core/group-state.md defines six states (Stable, PendingPublish, Merging, Recovering, Unrecoverable, Disbanded) plus local gates (Leaving, Disbanding, realized-removal). We have no explicit state machine; publish-before-apply exists only as an ad-hoc awaitCommitAck in MarmotWelcomeSender.kt.

4.5 Durability / restart — GAP

protocol-core/durability.md + foundation/conformance.md §"Crash and restart scenarios" define nine restart boundaries (prepared-not-published, ack-uncertain, confirmed-not-applied, observer-atomic apply, frozen-batch abandonment, …). We have MarmotManagerRestoreTest and nothing resembling this contract.

4.6 Nostr transport — BREAK

Rule (transports/nostr.md)	Ours
KeyPackage relays come from NIP-65 kind 10002 write-capable entries; "there is no dedicated KeyPackage relay list"	we implement kind 10051 KeyPackageRelayListEvent and MarmotSyncPolicy.Relays.keyPackageRelays()
Sender MUST NOT add an encoding tag; receiver MUST NOT switch decoders on one	KeyPackageEvent.build() emits encoding (tags/EncodingTag.kt)
Kind 30443 tag set is d, mls_protocol_version, i, mls_ciphersuite, mls_extensions, mls_proposals, app_components; KeyPackage events do not repeat relays	we emit relays and no app_components
app_components MUST include 0x8009	absent
mls_extensions should carry the profile's extensions (0x0006 current / 0xf2f1 legacy)	we emit 0xf2ee, 0x000a
Last-resort is the empty-data last_resort_key_package component 0x0004 in the KeyPackage-level dictionary, not an extension type	we treat 0x000a as a last-resort extension (tags/MlsProposalsTag.kt:30)
Dedup id is defined over recovered MLS message bytes, never the Nostr event id	MarmotInboundProcessor dedups on processedEventIds (Nostr ids)
Outer decryption tries the bounded retained-candidate key set (canonical epoch, retained epochs in horizon, staged local commit)	single-epoch decrypt
KeyPackage Lifetime MUST exist, be current, and span ≤ 7,261,200 s	KEY_PACKAGE_LIFETIME_SECONDS set but no upper-bound validation on inbound
Kind 445 may carry only h and NIP-40 expiration; commits/proposals MUST NOT carry expiration	builder is clean; the commit/proposal exclusion is unverified

4.7 Group image — BREAK

app-components/group-blossom-image-v1.md now specifies:

• image_key is the ChaCha20-Poly1305 key, image_upload_key is the Nostr secret key — ours are HKDF seeds (MarmotGroupData.kt:91,96);
• AAD is "marmot-group-image-v1" || 0x00 || canonical_media_type — ours is empty (MarmotGroupImageCipher.kt:50,58);
• a media_type<0..128> field exists — we deliberately omitted it to avoid trailing bytes for old mdk-core.

4.8 App payloads — GAP

foundation/application-messages.md + registries.md assign inner kinds we don't handle: 9 default chat, 1009 message edit, 1210 group system event, 1200 agent stream start. grep 1210\|1009 over our marmot tree: no hits. Receiver authentication (inner pubkey == MLS-authenticated account) we do have (MarmotInboundProcessor.kt:438).

4.9 Encrypted media — BREAK

We implement the MIP-04 scheme (mip04EncryptedMedia/). Current is 0x800b marmot.group.encrypted-media.v2 with a policy component (media_format = "encrypted-media-v2", allowed_locator_kinds, default_blob_endpoints); 0x8008 v1 is frozen. Our Mip04ParseResult.DeprecatedV1 path suggests we're on the v1 lineage.

4.10 Push notifications — GAP

Kinds 446–449 exist for us. Missing: the kind 451 push owner proof (spec'd 2026-07-23) and the token-record relay_hint publish-target rules.

4.11 Test/interop infrastructure — was a BLOCKER, addressed in Stage 0

• cli/tests/marmot/ clones the archived whitenoise-rs — repointed at marmot-protocol/mdk (-p wn-cli).
• quartz/tools/mdk-vector-gen pins plain openmls 0.8 — repointed at the extensions-draft fork, and marmot-profile-gen now emits current-profile fixtures.
• Still open: our MIP tests (MarmotMipComplianceTest, MarmotMipBehaviorTest) assert the deprecated rules — e.g. MarmotMipBehaviorTest.kt:793 asserts RequiredCapabilities == [0xF2EE]. They pin us to the old profile and must be re-pointed, not deleted (the legacy bytes still matter for reading our own stored groups).

5. Interop verdict today

• Against current MDK / White Noise: broken. Our groups fail profile classification before any component check. Our KeyPackages carry no 0x8009 data and no app_components tag, and carry a forbidden encoding tag.
• Against old mdk-core 0.8 (archived White Noise): probably still fine, which is what our fixtures prove — and that's now a dead target.
• Our own groups still work with our own clients. Nothing here is urgent for Amethyst-to-Amethyst Marmot chat; it is urgent for cross-client chat.

6. Proposed staging

Each stage is independently shippable and independently testable.

Stage 0 — re-establish a live reference. DONE.

• quartz/tools/mdk-vector-gen now pins erskingardner/openmls at the exact rev MDK's root Cargo.toml names, with the extensions-draft feature. Verified: it builds and runs.
• New generator marmot-profile-gen emits quartz/src/commonTest/resources/mls/marmot-current-profile.json — a real current-profile group with app_data_dictionary state at every location, PublicMessage handshakes, the Add commit, the Welcome, and exporter KATs. It self-checks the account-identity-proof v2 construction against the spec's published fixture at startup, so the emitted proofs are known to match byte-for-byte.
• The interop harness (cli/tests/marmot/) now clones marmot-protocol/mdk and builds -p wn-cli instead of the archived whitenoise-rs. Both source patches are gone: the mock-keyring patch is replaced by MDK's native --secret-store file, and the skip-unprocessable-retry patch targeted a path MDK does not have. The daemon socket is now pinned with wnd --socket rather than guessed from a derived default.

Not yet run end-to-end. The harness changes are derived from reading MDK's DaemonArgs and wn-cli manifest, not from a passing run — building MDK's full workspace needs its pinned toolchain and a local relay. A human run of marmot-interop-headless.sh is the acceptance test, and is likely to surface at least the retry behaviour the old patch used to paper over.

Stage 1 — MLS extensions draft in Quartz. DONE.

• marmot/mls/components/ — ComponentData, AppDataDictionary (extension 0x0006) and the ComponentsList payload shared by app_components (0x0001) and safe_aad (0x0002). Building a dictionary sorts for you; decoding rejects out-of-order or duplicate entries rather than normalizing them, because a receiver that silently sorted would accept two encodings of one dictionary and then disagree with a peer about the signed bytes.
• Proposal.AppDataUpdate (0x0008), including its update/remove operations, wired through MlsGroup on both the committing and receiving paths, plus proposeAppDataUpdate / proposeAppDataRemoval / appDataDictionary().
• Last resort is read as the KeyPackage-level 0x0004 component, not an MLS extension type.

Two application rules were taken from openmls rather than guessed, because both change the resulting GroupContext bytes and therefore the epoch key schedule:

1. AppDataUpdate applies AFTER the rest of the proposal list, so a GroupContextExtensions proposal in the same commit is already reflected — regardless of list order.
2. The dictionary extension is added-or-replaced in place and never dropped, even when the last component is removed. An absent extension and an empty dictionary are different GroupContexts.

MLS deliberately leaves update-payload semantics to the application (openmls hands the proposals back unresolved). Every Marmot component defines its update as a full replacement state, so resolution here is the identity function; a future diff-shaped component would have to resolve before this layer.

Verified: 12 tests parse the MDK-generated KeyPackage in marmot-current-profile.json all the way down — MLSMessage → KeyPackage → LeafNode → dictionary → components — and re-encode the dictionary byte-identically (it sits inside the signed LeafNode, so a one-byte difference would invalidate the signature). 9 more cover the proposal wire format and its group-level application, including two members converging on the same dictionary across the receive path. Full :quartz:jvmTest: 4,507 tests, 0 failures.

Known gap, deliberately left for Stage 3: enforceAuthorizedProposalSet and enforceNoAdminDepletion read MIP-01's marmot_group_data (0xF2EE). A current-profile group has no such extension, so both gates return without enforcing anything — an AppDataUpdate in such a group is currently unauthorized by us. The admin-policy component (0x8003) is what closes it.

Stage 2 — account identity proof v2 (0x8009). DONE.

• marmot/foundation/authorizationProofs/MarmotAuthorizationProof — the 104-byte common envelope from foundation/authorization-proofs.md, with the created_at bounds (1 to 2^53-1, catching a uint64 that reads back negative as a Long) and event-id reconstruction. Deliberately no wall-clock comparison: a proof does not expire, because clock skew must not make two members disagree about the same Commit.
• marmot/appComponents/accountIdentityProof/AccountIdentityProofV2 — the kind-450 signing template, production through NostrSigner (so NIP-46 / NIP-55 signers work), and validation returning a typed reason. create() re-verifies what the signer returned — pubkey, timestamp, kind, tags, content, id and signature — because an external signer is free to substitute.
• marmot/appComponents/AppComponentIds — the component registry from foundation/registries.md, needed by Stages 1 and 3 too.
• MlsCiphersuite gained the RFC 9420 §17.1 signature-scheme mapping, which the proof signs and validates explicitly.

Verified: 16 tests against the spec's published fixture (canonical event serialization, event id, signature, 104-byte layout) plus every signed input's binding, and 6 tests validating the proofs in marmot-current-profile.json — proofs produced by a separate implementation for random keys, which a fixed vector cannot establish. Full :quartz:jvmTest --tests "*marmot*": 395 tests, 0 failures.

Not yet wired: nothing reads or writes these components on a real leaf. The carrier is the app_data_dictionary, which is Stage 1. Until then this is a correct, tested primitive with no call sites — which is exactly what makes Stage 1 mechanical rather than exploratory.

Stage 3 — split MarmotGroupData into components. DONE (codecs + authorization).

All six component codecs in marmot/appComponents/, plus MarmotGroupState — the read view over a GroupContext dictionary that replaces MarmotGroupData:

Component	Notes
0x8001 profile	byte equality, no Unicode normalization; limits are in bytes
0x8003 admin-policy	sorted/unique/non-empty 32-byte account keys; unsigned byte order
0x8004 nostr-routing	raw 32-byte id + sorted relay list; relay-URL profile checked, never rewritten
0x8005 message-retention	fixed uint64, no length prefix; 0 = disabled (MIP-01 rejected 0)
0x800c lifecycle	one byte
0x8002 blossom-image	five var-byte fields incl. the new media_type, plus the domain-separated AAD

The Stage 1 authorization gap is closed. enforceAuthorizedProposalSet and enforceNoAdminDepletion now resolve admins through currentAdminIdentities(), which reads 0x8003 when present and falls back to 0xF2EE. Depletion also resolves an admin-policy change carried by an AppDataUpdate, not just by a GroupContextExtensions proposal.

One design decision worth recording: the admin lookup decodes only 0x8003, never the whole component set. Authorization must not depend on components it does not read — an initial version that went through MarmotGroupState made a malformed profile component freeze the group, which its own tests caught.

The 0xF2EE decoder stays as the legacy read path; nothing that reads it was removed.

Verified: 35 new tests — the MDK-generated GroupContext dictionary decoded component by component and re-encoded byte-identically, per-component validation rules the happy-path fixture cannot reach, and authorization driven through real groups (a non-admin cannot rewrite group state; an admin can; adminship transfers; a phantom admin with no member leaf is rejected; removing the admin policy is rejected). Full :quartz:jvmTest: 4,542 tests, 0 failures. :commons and :cli still compile.

Image crypto — DONE. GroupBlossomImageCrypto implements the current-profile scheme: image_key IS the AEAD key, image_upload_key IS the Blossom-auth secret, and the AAD is "marmot-group-image-v1" || 0x00 || canonical_media_type. MarmotMediaType implements the frozen canonicalization (ASCII case folding only, one alias). The MIP-01 scheme stays in MarmotGroupImageEncryption for groups already on disk, and nothing falls back between them — the component id is the version.

Stage 4 — Nostr transport corrections. MOSTLY DONE.

• KeyPackageEvent.buildCurrentProfile() emits the current tag set: mls_extensions = 0x0006, mls_proposals adds 0x0008, an app_components tag that must name 0x8009, and NO encoding or relays tags. KeyPackageUtils.isValid is profile-aware, told apart by the presence of app_components rather than a version tag; the MIP-era shape stays valid.
• KeyPackage discovery moved to the NIP-65 write set. publishRelaysFor no longer prefers a kind:10051 list — publishing only where a removed list points would make us invisible to a conformant peer, which looks in the NIP-65 set and nowhere else. The 10051 list is unioned in, never substituted, so unmigrated peers still find us.
• Dedup is now SHA-256(mls_message_bytes), never the Nostr event id. The old scheme collapsed nothing: each transport copy of one MLS message carries a fresh ephemeral pubkey and therefore a different event id, and a hostile republisher can mint unlimited ids for one message. OutboundGroupEvent carries the id so self-echo suppression works by MLS identity.
• KeyPackage Lifetime is validated: present, current, and spanning at most 7,261,200 s.
• The embedded account identity proof is validated on inbound current-profile KeyPackages — the app_components tag is only an advertisement, so the decoded LeafNode decides.

Still open: bounded retained-candidate trial decryption for kind:445. The rule ("try the canonical epoch, retained epochs inside the rollback horizon, and any staged-but-unmerged local commit — and no more") is defined in terms of the retained-state set that convergence owns, so it lands with Stage 6 rather than ahead of it.

Stage 5 — lifecycle state machine. CORE DONE.

GroupLifecycleState (the six canonical states with their legal-transition table), ConvergenceStatus (the four derived statuses with the legal-combination table), and LocalOutboundGate for Leaving / Disbanding / realized-removal.

Two table entries are load-bearing and tested as such: there is NO Merging -> Recovering edge — a competing branch seen mid-merge is retained, the merge finishes to Stable, and the bounded pass then triggers Stable -> Recovering, because diverting mid-merge would leave a half-applied epoch — and Disbanded has no outgoing edge at all.

Still open: the publish-obligation record (bytes + recipient scope + prior state + pending state) surviving restart, and wiring these states into MlsGroup/MarmotManager so they actually gate anything. Today they are a correct model with no enforcement behind them.

Stage 6 — convergence engine. SELECTION DONE.

ConvergencePolicy (the v1 constants, with the max_witness_override_depth <= max_rewind_commits bound enforced in the constructor), CandidateBranch (fork/tip epochs, raw depth, tip priority/committer/digest, per-epoch witnesses), BranchSelector (the six-step comparison), and the ConvergenceDisposition / ConvergenceCategory vocabularies.

Things worth knowing about the implementation:

• Byte ordering is UNSIGNED. Account keys and digests are uniformly distributed, so a signed comparison would invert about half of all final ties — and two clients would then disagree that often.
• raw_commit_depth has no comparison step of its own. It is already inside effective_commit_depth. The widely circulated write-up of this algorithm lists it as a step; the spec explicitly does not, and there is a test that fails if it is added.
• Witnesses count DISTINCT sender accounts per branch epoch, capped at the quorum size, and epochs at or before fork_epoch do not count at all.

ConvergencePass implements the bounded window: pass_base_epoch snapshot, the quiescence and absolute deadlines, the frozen batch, and the forced Stable -> Recovering transitions for a fork or an admitted disband candidate. Driven by an injected monotonic clock so the timing rules are tested deterministically rather than flakily.

Two asymmetries there are deliberate and tested: only SELECTION-RELEVANT input restarts quiescence (ordinary chat must not hold a pass open, because outbound work is gated on settling), and neither a fork nor a disband restarts anything — a pass that becomes a recovery is the same pass, and restarting would let a trickle of forks hold it open forever. Deferred commit expiry tracks the LIVE canonical tip while branch eligibility uses the FROZEN pass_base_epoch; using one epoch for both would let an open pass move its own horizon.

CandidateGraphBuilder builds the branches the selector compares, over a CandidateStateEngine<S> so the graph algebra is testable without MLS and the MLS adapter (MlsCandidateStateEngine) is testable on real forks.

The things that make it non-obvious, all of them tested:

• Parentage is derived, never declared. A commit carries no parent pointer, and it must not be believed if it did. The builder finds a parent by asking which retained state the commit's membership tag authenticates against. MlsCandidateGraphTest builds a genuine same-epoch fork (two Alices restored from one snapshot, each adding a different member) and both commits land on the same retained parent.
• A state id is SHA-256 over the serialized GroupContext, not the epoch number. Two states can share an epoch number and be different states — that is what a fork IS — and the GroupContext covers the tree and transcript hashes, so it separates them.
• It is a fixed point, not a sweep. Replaying a commit produces a state that may be the parent of a commit nothing could place a moment earlier, so it keeps sweeping the unplaced set until a pass produces no new edge. A two-commit chain offered child-first still rebuilds as one branch of depth 2.
• "I cannot place this" is not "I caught you misbehaving." An unauthorized commit whose parent IS known is terminal authorization_failed; a commit nothing authenticates is deferred, and only stale once the LIVE canonical tip has passed the rollback horizon. A tampered commit therefore comes out deferred, never authorization_failed.
• An invalid resulting state produces no edge at all, so convergence cannot select it. Validation is "decode the component set" — every decoder is strict, so malformed or unsorted bytes throw rather than yielding a lenient value.
• An unattributable tip is dropped, not zero-filled. Comparison step 5 breaks ties on the tip committer's account pubkey; substituting ByteArray(32) would hand such a tip the lowest possible key and win it a tie-break it never earned. A group whose leaves are not account pubkeys simply produces no selectable branch.
• Every trial replay restores a FRESH group from the retained snapshot, because a candidate parent gets tried by several competing commits and "advance it then roll it back" works until an exception escapes halfway through.

MlsGroup grew three non-mutating helpers for this — resolveCommitProposals, isCommitAuthorized, isSelfOnlyCommit — reusing the same enforceAuthorizedProposalSet / enforceNoAdminDepletion gates the local commit path runs, so an inbound commit and one we authored are held to one rule rather than two that drift.

MarmotConvergenceEngine puts all of it on the inbound path and mip03GroupMessages/ CommitOrdering.kt is deleted — the superseded rule (lowest outer created_at, then lowest Nostr event id) no longer exists anywhere in the tree.

The wiring decision worth recording is that convergence does NOT hold every commit for the quiescence window. A literal reading of the bounded pass would tax the overwhelmingly common single-commit case with a second of latency for nothing. It does not have to, because MLS is its own fork detector: once a commit is applied, a competitor authored against the same parent stops authenticating against the new tip but still authenticates against the RETAINED parent. So linear commits apply eagerly, the state each was applied to is retained, and a commit that authenticates a retained state rather than the tip IS the fork — only then does a pass open.

That is not an optimization that changes the answer, and the reason is the base choice at resolution time. The base is the newest retained state a divergent commit authenticates against, NOT the current tip; the canonical commits applied at or after it are replayed back into the graph, so the incumbent is rebuilt as a branch and scored by the same six-step rule as its challengers instead of winning by being already applied. Eager application only decides which branch is provisionally displayed while a pass runs. MarmotConvergenceWiringTest.twoObserversConvergeRegardlessOfArrivalOrder builds a real same-epoch fork, feeds two observers the same two commits in opposite orders, and asserts they end on the same GroupContext with exactly one of them having rewound.

Supporting pieces: MlsGroupManager.snapshot (state without touching storage) and installState (the rewind primitive — it pushes the outgoing epoch's secrets into the retention window first, so traffic already sent on the abandoned branch still decrypts). MarmotManager records locally-authored commits too, because our own commit is half of any fork we are party to; without it a peer's competitor would look like an unplaceable orphan and be deferred rather than compared.

Still open: app-payload witnesses are plumbed (recordWitness) but nothing feeds them yet — that needs the bounded retained-candidate trial decryption still outstanding from Stage 4. A group that goes quiet mid-pass has no inbound traffic to tick it, so the app layer must drive settleDueConvergence() from a timer while openConvergencePasses() is non-empty; that timer is not wired in commons/amethyst yet.

Stage 7 — app payloads, encrypted media v2, push owner proof. DONE.

The app-payload work turned up a live conformance bug rather than a gap: we were sending inner events WITH a Nostr signature, and a conformant decoder rejects a payload carrying a sig member at all. Every message we sent was refusable by any spec-following peer. MarmotAppEvent is the canonical unsigned shape; the id is unchanged by the switch (NIP-01 never hashed the signature), so existing history still lines up, and the Android pipeline already treated inner events as unsigned rumors, so the empty sig is re-added at the inbound boundary instead of travelling on the wire.

Duplicate-key detection needed its own scanner. Every JSON library here resolves duplicates before the caller sees them, and "last one wins" versus "first one wins" are both defensible — which is the problem, because identical bytes would then yield different ids on two clients.

Kinds 1009 (edits) and 1210 (system rows) are implemented, the latter synthesized from canonical state rather than received, which is what makes it unforgeable by one member. Encrypted-media v2 (0x800b) and the kind-451 push owner proof are implemented and verified against the fixtures the spec publishes — the 1210 example's event id and the push removal vector's owner_sig both reproduce exactly, which is what separates correct from self-consistent.

Stage 7 leftover: durability/restart conformance beyond the publish obligation (which IS durable) — specifically re-emission and reconstruction of application effects after restart.

Settled: Quartz keeps its own MLS

Decided 2026-09-08. We do not bind marmot-c / marmot-uniffi; the pure-Kotlin stack stays, and full MDK interoperability is the target.

Consequences to plan around, since they are now ours to carry:

• Stage 1 means implementing draft-ietf-mls-extensions-10's app_data_dictionary (0x0006), app_components (0x0001), safe_aad (0x0002) and the app_data_update proposal (0x0008) in quartz/.../marmot/mls/, against a draft upstream tracks through a fork of OpenMLS rather than a released crate.
• The OpenMLS rev pinned in mdk-vector-gen/Cargo.toml is a version we now track deliberately. When MDK bumps it, regenerate the vectors in the same change and diff them — a silent bump is how we would drift again.
• Byte-level conformance is the only thing that keeps us honest, so every stage below lands with vectors from marmot-profile-gen, not just unit tests written against our own reading.

Producing current-profile groups

CurrentProfileGroupFactory builds current-profile leaves, KeyPackages and groups. The order it enforces is not stylistic: a leaf must carry an identity proof over its OWN signature key, and only an account signer — possibly a remote bunker — can produce that proof, so the keypair is generated first, authorized, and only then built into a leaf. A proof cannot be added afterwards by code that only sees a finished leaf. MlsGroup.create / createKeyPackage gained leaf-extension, capability and required-capability parameters to make that possible.

Writing the producer side immediately found two bugs the reader-side tests could not:

1. buildLeafNode accepted leaf extensions and then wrote extensions = emptyList(). Every leaf we built would have silently dropped its identity proof.
2. Fresh KeyPackages carried Lifetime(0, Long.MAX_VALUE). That fails the bound Stage 4 had just started enforcing, so every KeyPackage we published would have been rejected by any conformant peer — including, once Stage 4 landed, by us. Now now - 1h to +84 days.

Interop status (2026-09-09)

The MDK 0.9.20 harness runs end to end and is green: 17 of 17, twice in a row from a clean state. It started this pass at 1 passed / 12 failed. The defects it found are below — every one of them a place where two implementations have to agree on something one implementation alone never disagrees with, which is why none of them was visible to any same-implementation test we have.

Defects the harness found in our own code

1. A Commit rebuilt our own leaf from defaults. The UpdatePath leaf replaces OUR leaf — same member, new key material — so its capabilities and extensions must carry over. buildLeafNode was called with neither, so the very first invite we ever sent dropped the account-identity-proof (a LEAF extension no proposal can restore) and stopped advertising the extension and proposal the group's own required_capabilities demanded. MDK reported PublicGroupError(LeafNodeValidation(UnsupportedExtensions)) and dropped the Welcome minted by that same commit; the invitee simply never saw an invite, with nothing logged on either side.

2. We held private keys for our leaf only, not our direct path. RFC 9420 §7.6 lets a committer address us at any node in the copath resolution whose key we hold, and a merged subtree resolves to its PARENT — so from three members on, the ciphertext meant for us stops naming our leaf. Worked for two members, failed for three, which is exactly why it survived every two-party test. MlsGroupState v3 persists them.

3. Three readers only understood the legacy 0xF2EE extension, so they did nothing at all on a current-profile group: the Welcome's nostr_group_id (without which a joiner cannot even subscribe), the admin gate on GroupContextExtensions changes (which skipped rather than failed closed), and disappearing-message expiration. Group metadata reads and writes now go through MarmotManager.groupView / setGroupProfile / setGroupAdmins / setGroupImage, which dispatch on the profile the group actually uses.

4. A non-confirmed publish discarded its obligation, letting a REPLACEMENT commit be prepared for the same epoch. "No OK arrived" is not "no peer took it": a timeout or a dropped connection leaves it unknown, and minting a second commit for an epoch a peer may already hold is the fork the gate exists to prevent. PublishOutcome.UNKNOWN keeps the record and holds the group.

5. We treated a last-resort KeyPackage as single-use. We publish every KeyPackage marked last resort and then dropped its private keys the moment one Welcome consumed it. OpenMLS deletes a consumed bundle only if !key_package.last_resort(); MDK marks all of its own last resort, caches the peer KeyPackage it resolved, and invites from that cached copy every time after. So the first invite addressed to us worked and every one after it died on "No matching KeyPackageBundle". Two things had to be fixed together: the last-resort check had to read the current profile's carrier (a last_resort_key_package component inside the KeyPackage-level app_data_dictionary, not the MIP-era 0x000A extension type), and the Welcome lookup had to trust the MLS KeyPackageRefs over the Nostr e tag, which MDK stamps from its stale cached copy.

6. Every rotation downgraded us off the current profile. MIP-00 replaces a KeyPackage as soon as a Welcome consumes it, so rotation runs right after the first group we are ever invited to — and it minted the replacement through the legacy generator. From that moment the only KeyPackage on relays for us had no account identity proof, MDK refused it outright, and the account was silently uninvitable one join after setup. Rotation and first publication now share one mint path.

7. The harness was parsing a wire format wn no longer speaks. MDK 0.9.x returns {"ok":true,"result":{"invites":[…]}}; iterating .result walked that object's three VALUES, so every poll matched nothing and reported "never received invite" for welcomes that had arrived and been accepted.

8. A hang-up ended the publish wait. A relay that dropped the socket between our EVENT frame and its OK gave no verdict — the event may be stored, it may not — and we recorded that as the relay's answer and stopped waiting. The pool's own outbox would have re-sent on reconnect; nobody was listening by then. publishAndCollectResults now keeps a transport failure provisional for one retry, dials past the backoff, and takes the OK when it arrives; it stays inside the caller's existing timeout, so nothing waits longer than before. This was one lost message per few harness runs, and on mobile it is every publish that races a network change.

Harness defects (not ours)

• Runs inherited each other's state. wnd wipes B's and C's data dirs on start, but A's amy home and the relay's SQLite file survived, and the leftovers are not inert — a KeyPackage from an earlier run is still on the relay to be invited with, and old kind:445 events still arrive undecryptable. Tests 03 and 08 failed on a dirty tree and passed on a clean one. Every run now starts from empty stores; --reuse-state opts out and --tests "…" runs a subset.
• Test 16 asked wn keys publish to rotate. That verb is the idempotent retry of the durable stable-slot replacement, so with nothing pending it republishes the same event id and there is no rotation to observe. wn keys rotate is the one that mints.
• Test 09 polled the wrong surface. wn messages list reads the raw app-event log, where a reaction is its own kind:7 entry with an e tag naming the anchor; reactions.by_emoji is the materialized timeline's aggregate, which that command does not project. The reaction had been arriving and being stored correctly the whole time.

What is NOT done

• Agent-text-stream publishes records but has nowhere to send them. We decode the 0x8006 policy, derive per-stream record keys, open records and fold the transcript, we advertise the 0xF2D1 receive capability, and AgentTextStreamPublisher now seals records under the sequence discipline the spec demands: values reserved durably ahead of use (in windows, so the hot path is not a write per record), never restarted or replayed across a crash, and a publisher that cannot prove which value is next refuses to publish at all rather than colliding a ChaCha20-Poly1305 (key, nonce) pair. Only an in-memory AgentTextStreamSequenceStore exists; a platform-backed one lands with the transport that needs it.

  Our leaf now advertises all three roles — receive (0xF2D1), send (0xF2D2) and fanout (0xF2D4) — which is the same set MDK puts on every KeyPackage it publishes, so a group requiring any of them admits us. A capability is a claim about what the client supports, not a duty to stream: a member that never originates one is a quiet member, not a broken one.

• The QUIC transport binding is implemented and verified against MDK's broker. transports/quic.md is a RAW QUIC binding — its own ALPNs (marmot.quic_broker.v1 / marmot.quic_stream.v1), frames written straight onto QUIC streams — so it does not go through nestsClient's WebTransportSession, which begins above HTTP/3 Extended CONNECT. It sits directly on :quic, which already had everything under that line: the connection, TLS 1.3, ALPN negotiation, stream multiplexing, the UDP socket.

  The pure protocol layer (control envelope, uint32 frame codec with both caps, quic:// candidate parsing, stream-id pinning) is in quartz; the connection layer is the new :marmotQuic module, which mirrors how :nestsClient sits on :quic. MarmotQuicBrokerInteropTest drives our publisher and subscriber through MDK's own marmot-quic-broker and checks that the records come back, open under the group-derived key, and fold to the publisher's transcript hash — plus that the broker keeps rooms apart. Opt in with -DmarmotQuicBroker=host:port; it skips visibly without one.

  The direct path (marmot.quic_stream.v1) is unimplemented — v1 has no start-payload candidate format for it, so it is only usable with an out-of-band endpoint.

• The Android GUI renders live previews. MarmotAgentStreamWatcher in commons follows the newest kind:1200 in a group, folds the QUIC records behind it under the receive discipline, and settles the preview against the durable kind:9 — confirmed when the transcript agrees, dropped when it does not, because a disagreement means we rendered something the publisher did not send. It is deliberately in commons and speaks only quartz's transport port, so it is testable without a UI and the desktop app inherits it the day it grows a Marmot chat screen. Android's chat view shows it as an italic, labelled row between the transcript and the composer: provisional content has to look provisional. Every failure path — no stream, no candidate, an unreachable broker, a platform with no QUIC at all — ends as "no preview", never as a broken group.

• The feature is wired end to end, both directions, against MDK. amy marmot stream start|send|watch|finish mints the kind-1200 anchor, pushes records through a broker, folds a preview under the receive discipline (seq high-water mark, silent replay discard, gap → unverifiable) and publishes the authoritative kind-9 carrying the transcript. Harness tests 18 and 19 run both directions: MDK's wn stream verify confirms our transcript from our own kind-1200 + kind-9, and our subscriber folds MDK's stream to a transcript hash identical to the one wn stream send computed. That equality is the whole key schedule, key context, AEAD, framing and transcript construction agreeing with an implementation that is not ours.

  Two defects only that exercise could have found:

  • The epoch belongs to the stream, not to the clock. The record key context binds mls_epoch, and we were resolving it as "the group's current epoch" at each command. A commit landing between the start and the send (or the watch) put the two sides on different keys and produced an empty preview. The epoch that DELIVERED the kind-1200 is the stream's, so it is persisted with the message now (MarmotMessageStore.recordEpoch, which MDK's own storage has as source_epoch) and read back by both sides.

  • close() dropped the tail of a stream. enqueue only fills the send buffer; tearing the connection down before the driver flushed it lost records silently, because the publisher had already counted them. QUIC ACKs a FIN only after everything ahead of it arrived, so finish() now waits for finAcked and close() gives an unacknowledged FIN a bounded moment. This is why the test passed alone and failed inside a full run.