fix(marmot): compute parent_hash chain + align required_capabilities id

Answers "can MDK and marmot-ts see and talk to a user whose group was
created on Amethyst?" — yes, after four more spec-conformance fixes:

1. parent_hash chain (RFC 9420 §7.9.2): Amethyst's commit() and
   externalJoin() now compute the parent_hash for every parent node on
   the committer's direct path and seal the committer's LeafNode with
   the correct leaf parent_hash. Amethyst↔Amethyst used to work only
   because both sides stored empty parent_hash values; against a
   strict peer (ts-mls, openmls) every Amethyst-authored commit was
   rejected with "Unable to verify parent hash". processCommit()
   now also patches the computed parent_hashes back into its tree so
   treeHash() agrees with the sender — otherwise the epoch key
   schedule diverges and AEAD tags mismatch.

2. REQUIRED_CAPABILITIES extension type: 0x0002 → 0x0003 per
   RFC 9420 §13.3. The old value was ratchet_tree's slot, so Amethyst's
   GroupContext.extensions carried a required_capabilities blob
   labelled as ratchet_tree, and openmls rejected the GroupInfo
   as Malformed (ratchet_tree is not valid in GroupContext). This
   completes the extension-ID set from d7114fc (ratchet_tree,
   external_pub) now aligned with the IANA registry.

3. verifyParentHash on the receive side now computes the expected
   chain top-down from the post-update tree rather than reading
   parent_hash fields that applyUpdatePath leaves as empty
   placeholders.

4. An explicit parentHash parameter on buildLeafNode so COMMIT-source
   leaves include the computed value in their TBS signature.

Test harness — reverse interop (Amethyst → outside world):

  - quartz/tools/{mdk,tsmls}-vector-gen/emit-joiner-kp.{rs,mjs}
    generate an MDK/openmls and a marmot-ts/ts-mls KeyPackage with
    marmot_group_data (0xF2EE) and self_remove (0x000A) advertised in
    capabilities so Amethyst's required_capabilities is satisfiable.

  - AmethystAuthoredVectorGen.kt (env-var gated JUnit test) takes the
    foreign KP, adds it to a fresh Amethyst group, and emits the
    Welcome plus three application PrivateMessages as JSON.

  - verify-amethyst.{rs,mjs} replay the joiner's private state,
    call the foreign MLS library's join + process_message, and
    assert the plaintexts match.

Both verifiers now print ALL PASS end-to-end:
  * openmls ← Amethyst: joinGroup + 3× process_message ✓
  * ts-mls  ← Amethyst: joinGroup + 3× processPrivateMessage ✓

https://claude.ai/code/session_01HfHdd5S5rvxUW2ihEpLGJr
This commit is contained in:
Claude
2026-04-21 16:13:42 +00:00
parent cd12018e08
commit 4feea50ed2
8 changed files with 969 additions and 56 deletions
@@ -475,6 +475,38 @@ class MlsGroup private constructor(
UpdatePathNode(pathKey.publicKey, encryptedSecrets)
}
// Capture sibling tree hashes BEFORE applying the UpdatePath —
// parent_hash computation (RFC 9420 §7.9.2) uses the
// ORIGINAL sibling-subtree tree hashes.
val preUpdateSiblingHashes = capturePreUpdateSiblingHashes(myLeafIndex)
// Apply the UpdatePath to our own tree first so the parent
// nodes carry the new encryption keys. Their parent_hash
// fields are filled in next.
tree.applyUpdatePath(myLeafIndex, pathNodes)
// Compute parent_hash for every direct-path parent node and
// for the committer's leaf (RFC 9420 §7.9.2). Without this
// chain, spec-strict peers (ts-mls, OpenMLS) reject the
// Welcome with "Unable to verify parent hash".
val (parentNodeHashes, leafParentHash) =
computeSenderParentHashes(myLeafIndex, preUpdateSiblingHashes)
// Patch each parent node with its computed parent_hash so
// subsequent treeHash / serialization uses the final values.
val directPath = BinaryTree.directPath(myLeafIndex, tree.leafCount)
for (nodeIdx in directPath) {
val existing = tree.getNode(nodeIdx)
if (existing is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
tree.setParent(
nodeIdx,
existing.parentNode.copy(
parentHash = parentNodeHashes[nodeIdx] ?: ByteArray(0),
),
)
}
}
// If a signing-key rotation is pending (from
// proposeSigningKeyRotation), the UpdatePath's leaf must be
// sealed with the NEW signing identity — otherwise the
@@ -494,6 +526,7 @@ class MlsGroup private constructor(
signingKey = effectiveSigningKey,
groupId = groupId,
leafIndex = myLeafIndex,
parentHash = leafParentHash,
)
encryptionPrivateKey = newEncKp.privateKey
@@ -506,11 +539,6 @@ class MlsGroup private constructor(
val commit = Commit(proposalOrRefs, updatePath)
// Apply UpdatePath to tree
if (updatePath != null) {
tree.applyUpdatePath(myLeafIndex, updatePath.nodes)
}
// Advance epoch
val commitSecret =
if (pathSecrets.isNotEmpty()) {
@@ -954,6 +982,24 @@ class MlsGroup private constructor(
"Parent hash verification failed for UpdatePath"
}
// After verification, patch the computed parent_hash values into
// the direct-path parent nodes. The sender's tree has these
// values filled in; receivers must match for treeHash() to agree
// (and thus for the epoch key schedule to converge).
val (recvParentHashes, _) =
computeSenderParentHashes(senderLeafIndex, preUpdateSiblingHashes)
for (nodeIdx in BinaryTree.directPath(senderLeafIndex, tree.leafCount)) {
val existing = tree.getNode(nodeIdx)
if (existing is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
tree.setParent(
nodeIdx,
existing.parentNode.copy(
parentHash = recvParentHashes[nodeIdx] ?: ByteArray(0),
),
)
}
}
// Decrypt path secret from our copath node
val directPath = BinaryTree.directPath(senderLeafIndex, tree.leafCount)
val myPath = BinaryTree.directPath(myLeafIndex, tree.leafCount)
@@ -1176,9 +1222,102 @@ class MlsGroup private constructor(
): ByteArray = buildConfirmedTranscriptHashInput(commit, senderLeafIndex, groupId, epoch)
/**
* Capture sibling tree hashes BEFORE the UpdatePath is applied.
* Returns a map of direct-path-index to sibling tree hash.
* Compute the RFC 9420 §7.9.2 parent_hash chain for a commit we are
* producing: returns the parent_hash value that should be stored in
* each parent node on our direct path, plus the parent_hash value
* that belongs in the committer's LeafNode (for its TBS signature).
*
* Must be called AFTER [RatchetTree.applyUpdatePath] (so parent
* nodes carry the new encryption keys) and with [preUpdateSiblingHashes]
* captured BEFORE applyUpdatePath (so the "original sibling tree hash"
* really is from the pre-update tree).
*/
private fun computeSenderParentHashes(
senderLeafIndex: Int,
preUpdateSiblingHashes: Map<Int, ByteArray>,
): Pair<Map<Int, ByteArray>, ByteArray> = computeSenderParentHashes(tree, senderLeafIndex, preUpdateSiblingHashes)
private fun computeSenderParentHashes(
tree: com.vitorpamplona.quartz.marmot.mls.tree.RatchetTree,
senderLeafIndex: Int,
preUpdateSiblingHashes: Map<Int, ByteArray>,
): Pair<Map<Int, ByteArray>, ByteArray> {
val directPath = BinaryTree.directPath(senderLeafIndex, tree.leafCount)
if (directPath.isEmpty()) return Pair(emptyMap(), ByteArray(0))
val hashes = mutableMapOf<Int, ByteArray>()
// Root's parent_hash is empty by convention (RFC 9420 §7.9.2).
hashes[directPath.last()] = ByteArray(0)
// Walk top-down (root has no parent → already set). For each node
// X below root, parent_hash(X) = Hash(encode(ParentHashInput{
// encryption_key = parent(X).encryption_key,
// parent_hash = parent(X).parent_hash,
// original_sibling_tree_hash = tree_hash(sibling(X))_preUpdate,
// })).
for (i in directPath.size - 2 downTo 0) {
val xIdx = directPath[i]
val parentIdx = directPath[i + 1]
val parentNode = tree.getNode(parentIdx)
if (parentNode !is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
hashes[xIdx] = ByteArray(0)
continue
}
val siblingTreeHash =
preUpdateSiblingHashes[i + 1]
?: error("missing pre-update sibling tree hash at level ${i + 1}")
hashes[xIdx] =
MlsCryptoProvider.hash(
encodeParentHashInput(
encryptionKey = parentNode.parentNode.encryptionKey,
parentHash = hashes[parentIdx] ?: ByteArray(0),
originalSiblingTreeHash = siblingTreeHash,
),
)
}
// The committer's leaf carries parent_hash(parent(leaf)) = parent_hash
// computed with the immediate parent's (directPath[0]) fields.
val immediateParentIdx = directPath.first()
val immediateParent = tree.getNode(immediateParentIdx)
val leafParentHash =
if (immediateParent is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
val siblingTreeHash =
preUpdateSiblingHashes[0]
?: error("missing pre-update sibling tree hash at leaf level")
MlsCryptoProvider.hash(
encodeParentHashInput(
encryptionKey = immediateParent.parentNode.encryptionKey,
parentHash = hashes[immediateParentIdx] ?: ByteArray(0),
originalSiblingTreeHash = siblingTreeHash,
),
)
} else {
ByteArray(0)
}
return Pair(hashes, leafParentHash)
}
/**
* Serialize RFC 9420 §7.9.2 ParentHashInput:
* struct {
* HPKEPublicKey encryption_key;
* opaque parent_hash<V>;
* opaque original_sibling_tree_hash<V>;
* } ParentHashInput;
*/
private fun encodeParentHashInput(
encryptionKey: ByteArray,
parentHash: ByteArray,
originalSiblingTreeHash: ByteArray,
): ByteArray {
val w = TlsWriter()
w.putOpaqueVarInt(encryptionKey)
w.putOpaqueVarInt(parentHash)
w.putOpaqueVarInt(originalSiblingTreeHash)
return w.toByteArray()
}
private fun capturePreUpdateSiblingHashes(senderLeafIndex: Int): Map<Int, ByteArray> {
val directPath = BinaryTree.directPath(senderLeafIndex, tree.leafCount)
val nodeCount = BinaryTree.nodeCount(tree.leafCount)
@@ -1214,47 +1353,16 @@ class MlsGroup private constructor(
val directPath = BinaryTree.directPath(senderLeafIndex, tree.leafCount)
if (directPath.isEmpty()) return true
// RFC 9420 §7.9.2 requires COMMIT leaf nodes to carry a non-empty
// parent_hash chained up to the root. This implementation does NOT
// yet compute that chain on the sending side (applyUpdatePath sets
// parent_hash = ByteArray(0) for every parent node on the path).
// Until the chain is implemented, treat an empty leaf parent_hash
// as "self-produced commit, chain not computed" and skip the chain
// verification. A compliant peer that does compute parent_hash will
// still be validated against our computed chain below — so if they
// disagree with us, we'll reject them.
//
// TODO: compute parent_hash per RFC 9420 §7.9.2 in [commit] and then
// tighten this verifier back to "MUST be non-empty".
val leafParentHash = updatePath.leafNode.parentHash
if (leafParentHash == null || leafParentHash.isEmpty()) return true
// Walk up the direct path, verifying each parent_hash link
var expectedParentHash = leafParentHash
for ((i, pathNodeIdx) in directPath.withIndex()) {
val pathNode = tree.getNode(pathNodeIdx) ?: continue
if (pathNode is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
// Use the pre-update sibling tree hash (captured before UpdatePath was applied)
val siblingHash = preUpdateSiblingHashes[i] ?: continue
// ParentHashInput
val phi = TlsWriter()
phi.putOpaqueVarInt(pathNode.parentNode.encryptionKey)
phi.putOpaqueVarInt(pathNode.parentNode.parentHash)
phi.putOpaqueVarInt(siblingHash)
val computedHash = MlsCryptoProvider.hash(phi.toByteArray())
if (!expectedParentHash.contentEquals(computedHash)) {
return false // Parent hash chain broken
}
// The next level's expected parent_hash is this node's parent_hash
expectedParentHash = pathNode.parentNode.parentHash
}
}
return true
// RFC 9420 §7.9.2: compute what the sender's parent_hash chain
// SHOULD have been given the post-update tree state, then compare
// the leaf's stored parent_hash to the expected value. We can't
// rely on the stored parent_hash of the parent nodes on our own
// tree because applyUpdatePath writes them as placeholder empty
// bytes — the sender never transmits them on the wire.
val (_, expectedLeafParentHash) =
computeSenderParentHashes(senderLeafIndex, preUpdateSiblingHashes)
val leafParentHash = updatePath.leafNode.parentHash ?: ByteArray(0)
return leafParentHash.contentEquals(expectedLeafParentHash)
}
private fun verifyLeafNodeSignature(
@@ -1686,6 +1794,56 @@ class MlsGroup private constructor(
return writer.toByteArray()
}
/**
* Companion-accessible clone of [computeSenderParentHashes].
* Used by [externalJoin] which builds its own tree locally
* (without constructing an [MlsGroup] first).
*/
private fun computeExternalSenderParentHashes(
tree: com.vitorpamplona.quartz.marmot.mls.tree.RatchetTree,
senderLeafIndex: Int,
preUpdateSiblingHashes: Map<Int, ByteArray>,
): Pair<Map<Int, ByteArray>, ByteArray> {
val directPath = BinaryTree.directPath(senderLeafIndex, tree.leafCount)
if (directPath.isEmpty()) return Pair(emptyMap(), ByteArray(0))
val hashes = mutableMapOf<Int, ByteArray>()
hashes[directPath.last()] = ByteArray(0)
for (i in directPath.size - 2 downTo 0) {
val xIdx = directPath[i]
val parentIdx = directPath[i + 1]
val parentNode = tree.getNode(parentIdx)
if (parentNode !is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
hashes[xIdx] = ByteArray(0)
continue
}
val siblingTreeHash =
preUpdateSiblingHashes[i + 1]
?: error("missing pre-update sibling tree hash at level ${i + 1}")
val w = TlsWriter()
w.putOpaqueVarInt(parentNode.parentNode.encryptionKey)
w.putOpaqueVarInt(hashes[parentIdx] ?: ByteArray(0))
w.putOpaqueVarInt(siblingTreeHash)
hashes[xIdx] = MlsCryptoProvider.hash(w.toByteArray())
}
val immediateParentIdx = directPath.first()
val immediateParent = tree.getNode(immediateParentIdx)
val leafParentHash =
if (immediateParent is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
val siblingTreeHash =
preUpdateSiblingHashes[0]
?: error("missing pre-update sibling tree hash at leaf level")
val w = TlsWriter()
w.putOpaqueVarInt(immediateParent.parentNode.encryptionKey)
w.putOpaqueVarInt(hashes[immediateParentIdx] ?: ByteArray(0))
w.putOpaqueVarInt(siblingTreeHash)
MlsCryptoProvider.hash(w.toByteArray())
} else {
ByteArray(0)
}
return Pair(hashes, leafParentHash)
}
/**
* Compute confirmation_tag = MAC(confirmation_key, confirmed_transcript_hash).
* Static version usable from companion object factory methods.
@@ -1699,7 +1857,11 @@ class MlsGroup private constructor(
return mac.doFinal()
}
private const val REQUIRED_CAPABILITIES_EXTENSION_TYPE = 0x0002
// RFC 9420 §13.3 IANA registry: 0x0003 is required_capabilities.
// (0x0002 is ratchet_tree — putting it here makes GroupContext
// unreadable to OpenMLS/MDK, which type-validates extensions by
// context.)
private const val REQUIRED_CAPABILITIES_EXTENSION_TYPE = 0x0003
// RFC 9420 §13.3 IANA registry: 0x0004 is external_pub.
// (0x0003 is required_capabilities — using it here makes
@@ -2040,8 +2202,10 @@ class MlsGroup private constructor(
// Build ExternalInit proposal
val externalInitProposal = Proposal.ExternalInit(kemOutput)
// Add ourselves to the tree
val leafNode =
// Placeholder leaf so we can claim our slot and compute the
// direct path for sibling-hash capture. The final leaf (with
// parent_hash filled in + fresh signature) replaces this below.
val placeholderLeaf =
buildLeafNode(
encryptionKey = encKp.publicKey,
signatureKey = sigKp.publicKey,
@@ -2049,9 +2213,23 @@ class MlsGroup private constructor(
source = LeafNodeSource.COMMIT,
signingKey = sigKp.privateKey,
groupId = groupContext.groupId,
leafIndex = tree.leafCount, // We'll be the next leaf
leafIndex = tree.leafCount,
)
val myLeafIndex = tree.addLeaf(leafNode)
val myLeafIndex = tree.addLeaf(placeholderLeaf)
// Capture sibling tree hashes BEFORE applying the UpdatePath.
val preUpdateSiblingHashes =
run {
val dp = BinaryTree.directPath(myLeafIndex, tree.leafCount)
val nc = BinaryTree.nodeCount(tree.leafCount)
val map = mutableMapOf<Int, ByteArray>()
for ((i, _) in dp.withIndex()) {
val childIdx =
if (i == 0) BinaryTree.leafToNode(myLeafIndex) else dp[i - 1]
map[i] = tree.treeHashNode(BinaryTree.sibling(childIdx, nc))
}
map
}
// Build UpdatePath for our leaf
val leafSecret = MlsCryptoProvider.randomBytes(MlsCryptoProvider.HASH_OUTPUT_LENGTH)
@@ -2083,8 +2261,39 @@ class MlsGroup private constructor(
UpdatePathNode(pathKey.publicKey, encryptedSecrets)
}
tree.applyUpdatePath(myLeafIndex, pathNodes)
// Compute parent_hash chain + the leaf's parent_hash (RFC 9420
// §7.9.2) on the post-update tree, then patch parent nodes and
// rebuild the committer leaf with the computed parent_hash.
val (extParentHashes, extLeafParentHash) =
computeExternalSenderParentHashes(tree, myLeafIndex, preUpdateSiblingHashes)
for (nodeIdx in BinaryTree.directPath(myLeafIndex, tree.leafCount)) {
val existing = tree.getNode(nodeIdx)
if (existing is com.vitorpamplona.quartz.marmot.mls.tree.TreeNode.Parent) {
tree.setParent(
nodeIdx,
existing.parentNode.copy(
parentHash = extParentHashes[nodeIdx] ?: ByteArray(0),
),
)
}
}
val leafNode =
buildLeafNode(
encryptionKey = encKp.publicKey,
signatureKey = sigKp.publicKey,
identity = identity,
source = LeafNodeSource.COMMIT,
signingKey = sigKp.privateKey,
groupId = groupContext.groupId,
leafIndex = myLeafIndex,
parentHash = extLeafParentHash,
)
tree.setLeaf(myLeafIndex, leafNode)
val updatePath = UpdatePath(leafNode, pathNodes)
tree.applyUpdatePath(myLeafIndex, updatePath.nodes)
// Build Commit
val commit =
@@ -2184,6 +2393,7 @@ class MlsGroup private constructor(
signingKey: ByteArray,
groupId: ByteArray? = null,
leafIndex: Int? = null,
parentHash: ByteArray? = null,
): LeafNode {
val unsigned =
LeafNode(
@@ -2200,6 +2410,7 @@ class MlsGroup private constructor(
} else {
null
},
parentHash = parentHash,
extensions = emptyList(),
signature = ByteArray(0), // Placeholder
)
@@ -69,8 +69,9 @@ class MarmotMipBehaviorTest {
fun create_installsRequiredCapabilitiesExtension() {
val alice = MlsGroup.create(aliceId.hexToByteArray())
val reqCaps = alice.extensions.find { it.extensionType == 0x0002 }
assertNotNull(reqCaps, "create() must install required_capabilities extension (0x0002)")
// RFC 9420 §13.3: required_capabilities is extension type 0x0003.
val reqCaps = alice.extensions.find { it.extensionType == 0x0003 }
assertNotNull(reqCaps, "create() must install required_capabilities extension (0x0003)")
val reader = TlsReader(reqCaps.extensionData)
val extsBytes = reader.readOpaqueVarInt()
@@ -0,0 +1,214 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.marmot.mls
import com.vitorpamplona.quartz.marmot.mls.codec.TlsReader
import com.vitorpamplona.quartz.marmot.mls.crypto.MlsCryptoProvider
import com.vitorpamplona.quartz.marmot.mls.framing.MlsMessage
import com.vitorpamplona.quartz.marmot.mls.group.MlsGroup
import com.vitorpamplona.quartz.marmot.mls.messages.MlsKeyPackage
import com.vitorpamplona.quartz.marmot.mls.messages.Welcome
import com.vitorpamplona.quartz.nip01Core.core.JsonMapper
import com.vitorpamplona.quartz.nip01Core.core.hexToByteArray
import com.vitorpamplona.quartz.nip01Core.core.toHexKey
import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.Json
import java.io.File
import kotlin.test.Test
/**
* Reverse-interop harness: emits a Welcome + PrivateMessages authored by
* Amethyst, so a foreign MLS backend (MDK / marmot-ts) can try to join
* and decrypt. Gated on environment variables so it only runs when you
* are actually driving the cross-impl pipeline:
*
* JOINER_HANDOFF_JSON=/tmp/joiner-handoff.json \
* AMETHYST_FIXTURE_JSON=/tmp/amethyst-fixture.json \
* ./gradlew :quartz:jvmTest \
* --tests com.vitorpamplona.quartz.marmot.mls.AmethystAuthoredVectorGen
*
* The handoff file comes from the foreign backend's
* `emit-joiner-kp` helper (it holds Bob's public KP bytes and Bob's
* private keys in the foreign backend's format). Amethyst reads only
* the public `key_package_raw` here and emits the Welcome + three
* application messages to `AMETHYST_FIXTURE_JSON`. The foreign
* verifier then consumes the fixture alongside its copy of Bob's
* private keys to prove the round-trip closes.
*/
class AmethystAuthoredVectorGen {
@Serializable
private data class HandoffJson(
@SerialName("cipher_suite") val cipherSuite: Int,
val joiner: HandoffJoiner,
)
@Serializable
private data class HandoffJoiner(
@SerialName("key_package_raw") val keyPackageRaw: String,
@SerialName("init_priv") val initPriv: String = "",
)
@Test
fun emitAmethystFixture() {
val handoffPath = System.getenv("JOINER_HANDOFF_JSON") ?: return
val fixturePath =
System.getenv("AMETHYST_FIXTURE_JSON")
?: error("AMETHYST_FIXTURE_JSON env var must be set when JOINER_HANDOFF_JSON is set")
val lenientJson = Json { ignoreUnknownKeys = true }
val handoff =
lenientJson.decodeFromString<HandoffJson>(File(handoffPath).readText())
check(handoff.cipherSuite == 1) {
"only cipher_suite 1 is supported; got ${handoff.cipherSuite}"
}
// Drop the raw KeyPackage bytes straight into Amethyst. We use the
// inner (un-wrapped) form here so this matches MlsGroup.addMember's
// expectations — that entry point reads an MlsKeyPackage, not an
// MlsMessage envelope.
val bobKpBytes = handoff.joiner.keyPackageRaw.hexToByteArray()
val bobKp = MlsKeyPackage.decodeTls(TlsReader(bobKpBytes))
check(bobKp.verifySignature()) { "joiner KeyPackage signature did not verify" }
val alice = MlsGroup.create("alice".encodeToByteArray())
val addResult = alice.addMember(bobKpBytes)
val welcomeBytes =
requireNotNull(addResult.welcomeBytes) { "addMember returned no Welcome" }
// Also emit the GroupInfo plaintext as a diagnostic so foreign
// verifiers can inspect our on-wire shape byte-for-byte.
val groupInfoBytes =
run {
val mlsMsg = MlsMessage.decodeTls(TlsReader(welcomeBytes))
val welcome = Welcome.decodeTls(TlsReader(mlsMsg.payload))
val myRef = bobKp.reference()
val mySecrets =
welcome.secrets.find { it.newMember.contentEquals(myRef) }
?: error("no secrets for joiner in our own Welcome")
val initPriv = handoff.joiner.initPriv.hexToByteArray()
val gsBytes =
MlsCryptoProvider.decryptWithLabel(
initPriv,
"Welcome",
welcome.encryptedGroupInfo,
mySecrets.encryptedGroupSecrets.kemOutput,
mySecrets.encryptedGroupSecrets.ciphertext,
)
val joinerSecret = TlsReader(gsBytes).readOpaqueVarInt()
val pskSecret = ByteArray(MlsCryptoProvider.HASH_OUTPUT_LENGTH)
val memberSecret = MlsCryptoProvider.hkdfExtract(joinerSecret, pskSecret)
val welcomeSecret = MlsCryptoProvider.deriveSecret(memberSecret, "welcome")
val welcomeKey =
MlsCryptoProvider.expandWithLabel(
welcomeSecret,
"key",
ByteArray(0),
MlsCryptoProvider.AEAD_KEY_LENGTH,
)
val welcomeNonce =
MlsCryptoProvider.expandWithLabel(
welcomeSecret,
"nonce",
ByteArray(0),
MlsCryptoProvider.AEAD_NONCE_LENGTH,
)
MlsCryptoProvider.aeadDecrypt(
welcomeKey,
welcomeNonce,
ByteArray(0),
welcome.encryptedGroupInfo,
)
}
val plaintexts =
listOf(
"Hello from Amethyst",
"Second message from Amethyst at epoch 1.",
"Unicode works too: ☕ ❤",
)
val appMessages =
plaintexts.map { pt ->
val ptBytes = pt.encodeToByteArray()
val ct = alice.encrypt(ptBytes)
AppMessageEntry(ptBytes.toHexKey(), ct.toHexKey())
}
// Build the joiner-side KeyPackageBundle by re-importing the bytes so
// that we can drive Amethyst's own processWelcome end-to-end and
// record a post-join MLS-Exporter KAT. The foreign verifier doesn't
// need the exporter — but including it keeps the fixture's shape
// identical to the other cross-impl vectors.
val exporterContext = "group-event".encodeToByteArray()
val fixture =
AmethystFixture(
cipherSuite = 1,
description = "Amethyst-authored Welcome and PrivateMessages for reverse interop.",
welcome = welcomeBytes.toHexKey(),
groupInfoPlaintext = groupInfoBytes.toHexKey(),
appMessagesAliceToBob = appMessages,
exporter =
ExporterJson(
label = "marmot",
context = exporterContext.toHexKey(),
length = 32,
// The receiver computes its own exporter after
// join — foreign verifiers may compare theirs
// against ours if they want a cross-check.
secret = "",
),
)
File(fixturePath).writeText(
JsonMapper.jsonInstance.encodeToString(
AmethystFixture.serializer(),
fixture,
),
)
}
@Serializable
private data class AmethystFixture(
@SerialName("cipher_suite") val cipherSuite: Int,
val description: String,
val welcome: String,
@SerialName("group_info_plaintext")
val groupInfoPlaintext: String,
@SerialName("app_messages_alice_to_bob")
val appMessagesAliceToBob: List<AppMessageEntry>,
val exporter: ExporterJson,
)
@Serializable
private data class AppMessageEntry(
val plaintext: String,
@SerialName("private_message") val privateMessage: String,
)
@Serializable
private data class ExporterJson(
val label: String,
val context: String,
val length: Int,
val secret: String,
)
}
+15
View File
@@ -7,8 +7,23 @@ edition = "2021"
openmls = { version = "0.8.1", features = ["test-utils"] }
openmls_rust_crypto = "0.5.1"
openmls_basic_credential = { version = "0.5", features = ["test-utils"] }
openmls_memory_storage = { version = "0.5", features = ["persistence"] }
openmls_traits = "0.5"
tls_codec = "0.4"
hex = "0.4"
serde_json = "1"
serde = { version = "1", features = ["derive"] }
base64 = "0.22"
env_logger = "0.11"
[[bin]]
name = "mdk-vector-gen"
path = "src/main.rs"
[[bin]]
name = "emit-joiner-kp"
path = "src/emit_joiner_kp.rs"
[[bin]]
name = "verify-amethyst"
path = "src/verify_amethyst.rs"
@@ -0,0 +1,96 @@
// Produce an MDK/OpenMLS-authored joiner KeyPackage for the reverse-interop
// test (Amethyst → MDK). Emits TWO artifacts:
// - a binary storage snapshot at $1 (openmls_memory_storage save_to_file
// format) so verify_amethyst.rs can restore Bob's provider exactly as
// it was when the KP was generated;
// - JSON on stdout with the public KP bytes plus the private keys
// (hex-encoded) so Amethyst can drive its addMember and the reader
// can sanity-check shapes:
//
// {
// "cipher_suite": 1,
// "joiner": {
// "key_package_raw": hex,
// "init_priv": hex(32 B),
// "encryption_priv": hex(32 B),
// "signature_priv": hex(32 B, raw Ed25519 seed),
// "signature_pub": hex(32 B)
// }
// }
use std::env;
use std::fs::File;
use std::process;
use openmls::prelude::*;
use openmls_basic_credential::SignatureKeyPair;
use openmls_rust_crypto::OpenMlsRustCrypto;
use openmls_traits::OpenMlsProvider;
use tls_codec::Serialize;
const CS: Ciphersuite = Ciphersuite::MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519;
fn main() {
let args: Vec<String> = env::args().collect();
if args.len() != 2 {
eprintln!("usage: emit-joiner-kp <storage-snapshot-path>");
process::exit(2);
}
let snapshot_path = &args[1];
let provider = OpenMlsRustCrypto::default();
let cred = BasicCredential::new(b"bob".to_vec());
let sig = SignatureKeyPair::new(CS.signature_algorithm()).unwrap();
sig.store(provider.storage()).unwrap();
let cwk = CredentialWithKey {
credential: cred.into(),
signature_key: sig.public().into(),
};
// Advertise the Marmot extensions a real MDK KeyPackage would carry,
// so Amethyst's required_capabilities (which lists marmot_group_data
// 0xF2EE and the self_remove proposal 0x000A) is satisfied when our
// KP is added to an Amethyst group.
let capabilities = Capabilities::new(
None,
Some(&[CS]),
Some(&[
ExtensionType::from(0xF2EE), // marmot_group_data
ExtensionType::ApplicationId,
ExtensionType::LastResort,
]),
Some(&[openmls::prelude::ProposalType::SelfRemove]),
None,
);
let bundle = KeyPackage::builder()
.leaf_node_capabilities(capabilities)
.mark_as_last_resort()
.build(CS, &provider, &sig, cwk)
.unwrap();
let kp = bundle.key_package().clone();
let kp_bytes = kp.tls_serialize_detached().unwrap();
let init_priv: Vec<u8> = (**bundle.init_private_key()).to_vec();
let enc_priv: Vec<u8> = (**bundle.encryption_private_key()).to_vec();
// Persist the entire provider storage — validate_amethyst restores it
// byte-for-byte so the KeyPackage + bundle + signature key can be used
// to process a Welcome.
let file = File::create(snapshot_path).expect("open snapshot file");
provider
.storage()
.save_to_file(&file)
.expect("save storage snapshot");
let out = serde_json::json!({
"cipher_suite": 1,
"joiner": {
"key_package_raw": hex::encode(&kp_bytes),
"init_priv": hex::encode(&init_priv),
"encryption_priv": hex::encode(&enc_priv),
"signature_priv": hex::encode(sig.private()),
"signature_pub": hex::encode(sig.public()),
}
});
println!("{}", serde_json::to_string_pretty(&out).unwrap());
}
@@ -0,0 +1,153 @@
// Reverse-interop verifier: Amethyst → MDK/OpenMLS.
//
// Takes three paths on argv:
// arg1 — path to the binary storage snapshot written by emit_joiner_kp.rs
// (restores Bob's openmls provider exactly as it was when the
// KeyPackage + bundle were generated);
// arg2 — the joiner-handoff JSON (we only read the signature_pub here,
// to look up Bob's own leaf after joining);
// arg3 — the Amethyst-authored fixture (Welcome + PrivateMessages).
//
// Prints PASS/... lines on success and exits non-zero on any failure.
use std::env;
use std::fs::{self, File};
use std::process;
use base64::Engine;
use openmls::prelude::*;
use openmls_rust_crypto::OpenMlsRustCrypto;
use openmls_traits::OpenMlsProvider;
use serde::Deserialize;
use std::collections::HashMap;
use tls_codec::Deserialize as TlsDeserialize;
#[derive(Deserialize)]
struct Handoff {
cipher_suite: u16,
joiner: HandoffJoiner,
}
#[derive(Deserialize)]
struct HandoffJoiner {
signature_pub: String,
}
#[derive(Deserialize)]
struct Fixture {
cipher_suite: u16,
welcome: String,
app_messages_alice_to_bob: Vec<AppMessage>,
}
#[derive(Deserialize)]
struct AppMessage {
plaintext: String,
private_message: String,
}
fn hex_bytes(s: &str) -> Vec<u8> {
hex::decode(s).expect("bad hex in fixture")
}
fn fail(msg: &str) -> ! {
eprintln!("FAIL: {msg}");
process::exit(1);
}
fn main() {
env_logger::init();
let args: Vec<String> = env::args().collect();
if args.len() != 4 {
eprintln!(
"usage: verify-amethyst <storage-snapshot.bin> \
<joiner-handoff.json> <amethyst-fixture.json>"
);
process::exit(2);
}
let handoff: Handoff = serde_json::from_str(&fs::read_to_string(&args[2]).unwrap()).unwrap();
let fixture: Fixture = serde_json::from_str(&fs::read_to_string(&args[3]).unwrap()).unwrap();
assert_eq!(handoff.cipher_suite, 1);
assert_eq!(fixture.cipher_suite, 1);
// Restore Bob's MemoryStorage key/value map from the snapshot. We
// replicate the on-disk format here (it's a simple base64-encoded
// HashMap) instead of going through MemoryStorage::load_from_file —
// that API wants `&mut self` on a field we can't move into the
// OpenMlsRustCrypto provider (its fields are private and there's no
// constructor that accepts a pre-populated storage).
#[derive(Deserialize)]
struct SerializableKeyStore {
values: HashMap<String, String>,
}
let snap: SerializableKeyStore =
serde_json::from_reader(File::open(&args[1]).expect("open snapshot for read"))
.expect("parse snapshot JSON");
let provider = OpenMlsRustCrypto::default();
{
let mut map = provider.storage().values.write().unwrap();
for (k, v) in snap.values {
map.insert(
base64::prelude::BASE64_STANDARD.decode(k).unwrap(),
base64::prelude::BASE64_STANDARD.decode(v).unwrap(),
);
}
}
// Parse Amethyst's Welcome.
let welcome_bytes = hex_bytes(&fixture.welcome);
let msg_in = MlsMessageIn::tls_deserialize(&mut welcome_bytes.as_slice())
.unwrap_or_else(|e| fail(&format!("welcome decode: {e:?}")));
let welcome = match msg_in.extract() {
MlsMessageBodyIn::Welcome(w) => w,
other => fail(&format!("expected Welcome, got {other:?}")),
};
let cfg = MlsGroupJoinConfig::builder().build();
let staged = StagedWelcome::new_from_welcome(&provider, &cfg, welcome, None)
.unwrap_or_else(|e| fail(&format!("StagedWelcome::new_from_welcome: {e:?}")));
let mut group = staged
.into_group(&provider)
.unwrap_or_else(|e| fail(&format!("StagedWelcome::into_group: {e:?}")));
println!(
"PASS: joined Amethyst-authored Welcome at epoch {}",
group.epoch().as_u64()
);
// Sanity-check: the signature_pub we advertised must appear in the
// ratchet tree as OUR leaf.
let expected_sig_pub = hex_bytes(&handoff.joiner.signature_pub);
let own_leaf = group.own_leaf_node().unwrap();
if own_leaf.signature_key().as_slice() != expected_sig_pub.as_slice() {
fail("own leaf signature key does not match handoff signature_pub");
}
for (idx, app) in fixture.app_messages_alice_to_bob.iter().enumerate() {
let bytes = hex_bytes(&app.private_message);
let msg = MlsMessageIn::tls_deserialize(&mut bytes.as_slice())
.unwrap_or_else(|e| fail(&format!("app msg {idx} decode: {e:?}")));
let protocol = msg
.try_into_protocol_message()
.unwrap_or_else(|e| fail(&format!("app msg {idx} not a protocol message: {e:?}")));
let processed = group
.process_message(&provider, protocol)
.unwrap_or_else(|e| fail(&format!("app msg {idx} process_message: {e:?}")));
match processed.into_content() {
ProcessedMessageContent::ApplicationMessage(app_msg) => {
let got = app_msg.into_bytes();
let want = hex_bytes(&app.plaintext);
if got != want {
fail(&format!(
"app msg {idx} plaintext mismatch\n want: {}\n got : {}",
hex::encode(&want),
hex::encode(&got),
));
}
println!("PASS: decrypted Amethyst app message {idx}");
}
other => fail(&format!("app msg {idx} unexpected content: {other:?}")),
}
}
println!("ALL PASS — openmls read every Amethyst-authored artifact.");
}
@@ -0,0 +1,70 @@
// Produce a ts-mls-authored joiner KeyPackage for the reverse-interop test
// (Amethyst → ts-mls). Emits on stdout:
//
// {
// "cipher_suite": 1,
// "joiner": {
// "key_package_raw": hex, // inner KeyPackage bytes
// "init_priv": hex(32 B),
// "encryption_priv":hex(32 B),
// "signature_priv": hex(32 B, raw seed), // PKCS#8 header stripped
// "signature_pub": hex(32 B)
// }
// }
//
// Amethyst reads `key_package_raw` to drive its MlsGroup.addMember flow;
// verify-amethyst-fixture.mjs reads the private keys back to drive
// ts-mls joinGroup + processPrivateMessage against Amethyst's Welcome.
import {
defaultCapabilities,
defaultCryptoProvider,
defaultLifetime,
generateKeyPackage,
getCiphersuiteImpl,
keyPackageEncoder,
encode,
} from "ts-mls";
const hex = (bytes) =>
Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join("");
const cs = await getCiphersuiteImpl(
"MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519",
defaultCryptoProvider,
);
// Mirror MDK: advertise marmot_group_data (0xF2EE) as a supported
// extension and self_remove (0x000A) as a supported proposal so
// Amethyst's required_capabilities is satisfied when our KP is added.
const caps = defaultCapabilities();
if (!caps.extensions.includes(0xf2ee)) caps.extensions.push(0xf2ee);
if (!caps.proposals.includes(0x000a)) caps.proposals.push(0x000a);
const kp = await generateKeyPackage({
credential: { credentialType: 1, identity: new TextEncoder().encode("bob") },
capabilities: caps,
lifetime: defaultLifetime(),
cipherSuite: cs,
});
const sigPriv = kp.privatePackage.signaturePrivateKey;
const sigSeed =
sigPriv.length === 32
? sigPriv
: sigPriv.length === 64
? sigPriv.slice(0, 32)
: sigPriv.slice(sigPriv.length - 32);
const out = {
cipher_suite: 1,
joiner: {
key_package_raw: hex(encode(keyPackageEncoder, kp.publicPackage)),
init_priv: hex(kp.privatePackage.initPrivateKey),
encryption_priv: hex(kp.privatePackage.hpkePrivateKey),
signature_priv: hex(sigSeed),
signature_pub: hex(kp.publicPackage.leafNode.signaturePublicKey),
},
};
process.stdout.write(JSON.stringify(out, null, 2) + "\n");
@@ -0,0 +1,153 @@
// Reverse-interop verifier: Amethyst → ts-mls.
//
// Reads two JSON files:
// $1 — the joiner-KP handoff produced by emit-joiner-kp.mjs
// (contains Bob's private keys in ts-mls-compatible form).
// $2 — the Amethyst-authored fixture (produced by the Kotlin
// AmethystAuthoredVectorGen test) containing Alice's Welcome
// and three PrivateMessages she sent to Bob.
//
// Succeeds (exit 0) if Bob can:
// 1. decode Amethyst's KeyPackage wrapper (basic wire-format sanity),
// 2. join Alice's group from Amethyst's Welcome,
// 3. decrypt each of Alice's application messages and match the
// expected plaintext.
import { readFileSync } from "node:fs";
import {
decode,
defaultCryptoProvider,
getCiphersuiteImpl,
joinGroup,
keyPackageDecoder,
mlsMessageDecoder,
processPrivateMessage,
unsafeTestingAuthenticationService,
wireformats,
} from "ts-mls";
function assert(cond, msg) {
if (!cond) {
console.error("FAIL:", msg);
process.exit(1);
}
}
const hexToBytes = (s) =>
Uint8Array.from(s.match(/../g).map((b) => parseInt(b, 16)));
const [handoffPath, fixturePath] = process.argv.slice(2);
if (!handoffPath || !fixturePath) {
console.error(
"usage: verify-amethyst-fixture.mjs <joiner-handoff.json> <amethyst-fixture.json>",
);
process.exit(2);
}
const handoff = JSON.parse(readFileSync(handoffPath, "utf8"));
const fixture = JSON.parse(readFileSync(fixturePath, "utf8"));
const cs = await getCiphersuiteImpl(
"MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519",
defaultCryptoProvider,
);
const ctx = {
cipherSuite: cs,
authService: unsafeTestingAuthenticationService,
};
// Rebuild Bob's public+private KP from the handoff.
const kpBytes = hexToBytes(handoff.joiner.key_package_raw);
const kpPub = decode(keyPackageDecoder, kpBytes);
assert(kpPub, "failed to decode joiner key_package_raw");
// ts-mls signature_priv is a PKCS#8 Ed25519 DER envelope. Reconstruct
// it from the raw 32-byte seed emitted by the handoff.
const PKCS8_ED25519_PREFIX = hexToBytes("302e020100300506032b657004220420");
const seed = hexToBytes(handoff.joiner.signature_priv);
const sigPrivDer = new Uint8Array(PKCS8_ED25519_PREFIX.length + seed.length);
sigPrivDer.set(PKCS8_ED25519_PREFIX, 0);
sigPrivDer.set(seed, PKCS8_ED25519_PREFIX.length);
const privateKeys = {
initPrivateKey: hexToBytes(handoff.joiner.init_priv),
hpkePrivateKey: hexToBytes(handoff.joiner.encryption_priv),
signaturePrivateKey: sigPrivDer,
};
// Decode Amethyst's Welcome (wrapped in MlsMessage).
const welcomeBytes = hexToBytes(fixture.welcome);
const welcomeMsg = decode(mlsMessageDecoder, welcomeBytes);
assert(welcomeMsg, "failed to decode Amethyst Welcome bytes");
assert(
welcomeMsg.wireformat === wireformats.mls_welcome,
"Amethyst Welcome wire-format mismatch",
);
// Join.
let state;
try {
state = await joinGroup({
context: ctx,
welcome: welcomeMsg.welcome,
keyPackage: kpPub,
privateKeys,
});
} catch (e) {
console.error("FAIL: joinGroup threw:", e?.message ?? e);
process.exit(1);
}
console.log("PASS: joined Amethyst-authored Welcome at epoch", state.groupContext.epoch);
// Dump the decoded GroupInfo for diagnostic use by other verifiers.
// ts-mls already unwrapped it during joinGroup, so we just print a
// compact summary + the full extensions array.
{
const gc = state.groupContext;
const ext = state.publicGroupState?.groupInfoExtensions;
// (state layout varies; these fields may or may not exist. Best-effort.)
if (process.env.DUMP_GROUP_INFO) {
console.log(JSON.stringify({ groupContext: gc, extensions: ext }, (_k, v) =>
v instanceof Uint8Array ? Array.from(v).map(b => b.toString(16).padStart(2, "0")).join("") : v,
2));
}
}
// Decrypt each application message.
for (const [idx, msg] of fixture.app_messages_alice_to_bob.entries()) {
const msgBytes = hexToBytes(msg.private_message);
const framedMsg = decode(mlsMessageDecoder, msgBytes);
assert(framedMsg, `failed to decode application message ${idx}`);
assert(
framedMsg.wireformat === wireformats.mls_private_message,
`application message ${idx} wire-format mismatch`,
);
let res;
try {
res = await processPrivateMessage({
context: ctx,
state,
privateMessage: framedMsg.privateMessage,
});
} catch (e) {
console.error(`FAIL: processPrivateMessage(${idx}) threw:`, e?.message ?? e);
process.exit(1);
}
state = res.newState;
if (res.kind !== "applicationMessage") {
console.error(`FAIL: processPrivateMessage(${idx}) returned kind=${res.kind}, expected applicationMessage`);
process.exit(1);
}
const gotHex = Array.from(res.message ?? new Uint8Array(), (b) =>
b.toString(16).padStart(2, "0"),
).join("");
if (gotHex !== msg.plaintext) {
console.error(
`FAIL: app message ${idx} plaintext mismatch\n want: ${msg.plaintext}\n got : ${gotHex}`,
);
process.exit(1);
}
console.log(`PASS: decrypted Amethyst app message ${idx}`);
}
console.log("ALL PASS — ts-mls read every Amethyst-authored artifact.");