amethyst/marmotBench/README.md

# marmotBench — quartz Marmot vs MDK, head to head

Measures the same Marmot/MLS operations on both engines so "are we as fast as
the reference" has an answer instead of an opinion.

The quartz half lives here. The MDK half is its own criterion suite:

```sh
# quartz
./gradlew :marmotBench:run                     # table
./gradlew :marmotBench:run --args=--json       # machine-readable

# MDK (the reference)
cd <mdk> && cargo bench -p cgka-engine --bench group_lifecycle
```

## What is compared

| this module              | MDK bench                   |
|--------------------------|-----------------------------|
| `create_group/N`         | `bench_create_group`        |
| `join_welcome`           | `bench_join_welcome`        |
| `send_app_message/N`     | `bench_app_message_send`    |
| `ingest_app_message/N`   | `bench_app_message_ingest`  |
| `ingest_commit/N`        | (no counterpart)            |

`ingest_commit` has no MDK counterpart because neither suite had one. It is the
operation every member pays on every membership or settings change, and the
only one whose cost is supposed to grow with the group, so leaving it
unmeasured left the most load-bearing path in the protocol untested.

Both sides exclude transport crypto and run over in-memory storage, so what is
measured is the engine's own CPU cost. Setup is outside the measured window on
both sides — criterion's `iter_batched(.., PerIteration)` there, an explicit
`setup` lambda here.

**On the shape of `create_group/N`.** An earlier version of this file claimed
we spend "one more commit" than MDK's founding creation. That was wrong, and
`--epoch-probe` exists to keep it honest. It reported, before the founding-add
fix below: `create` publishes nothing and leaves epoch 0, `addMember` produces
exactly one commit, and the invitee joins at epoch 1. MLS permits nothing else
— RFC 9420 section 11 requires a group to be created with a single member — so
MDK commits its founding Adds internally too. Both sides do one commit's worth
of ratchet work. It now reports zero commits published for that same step, for
the reason in the next paragraph; re-run it rather than trusting this text.

What did differ is that we PUBLISHED that founding commit and MDK does not.
`protocol-core/publish-lifecycle.md` gives the founding Add an empty
group-message publication obligation, because the creator is the only
pre-existing member and no peer can be forked by failing to publish it. We
implemented that exception for the epoch-0 creation and missed that it extends
to the Add immediately after. It is fixed now: the founding Add merges locally
and only the Welcomes go out, which is both what the spec says and what the
reference implementation does.

## Why allocation is reported next to latency

The brief is "as fast if not faster, while avoiding GC as much as possible",
and those are two different measurements. A JVM can win a microbenchmark while
allocating tens of times more per operation; the bill arrives later as GC
pauses on a phone, in a frame the benchmark never renders. So every row carries
**bytes allocated per operation** from `com.sun.management.ThreadMXBean`
(in the JDK — no dependency), alongside p50/p90/p99.

That counter is per-thread, which is why every benchmark body runs inline on
the harness thread via `runBlocking`. Work dispatched elsewhere would allocate
off-book and read as free.

The JVM runs with `-XX:+UseSerialGC` on a fixed 4g heap: the point is to keep
allocation attributable to the benchmark thread and to keep a collection from
landing inside a measured sample and corrupting the percentile it falls in.

## Reading the numbers honestly

- Rust has no GC, so `alloc/op` has no MDK counterpart. It is not a
  head-to-head column — it is our own regression signal, and the number to
  drive down.
- Latency across a JVM and a Rust binary on the same host is a fair comparison
  of *this* workload on *this* machine. It is not a language benchmark.
- `create_group/32` builds 32 KeyPackages in setup. That cost is excluded, but
  it makes each iteration expensive to prepare — hence the low iteration count.

## Result: eliminating the field-arithmetic allocation

The first run of this module put **93% of all sampled allocation** (JFR
`jdk.ObjectAllocationSample`) in `Curve25519Field.mul/add/sub`. The pure-Kotlin
Curve25519 returned a fresh `LongArray(16)` from every field operation, and a
Montgomery ladder performs ~18 of them per bit for 255 bits — so a single
X25519 scalar multiplication allocated over a megabyte of garbage.

Each operation now has an in-place `*Into` twin, and both hot paths (the X25519
ladder and Ed25519's extended-coordinate point addition) allocate their working
set once and then run allocation-free. See `Curve25519Field`.

Allocation per operation, before and after. This column reproduces to four
significant figures across runs, so the ratios are real:

| operation           | before       | after       | reduction |
|---------------------|--------------|-------------|-----------|
| `create_group/0`    |   6 958.7 KB |    88.3 KB  | 79x       |
| `create_group/1`    |  27 074.1 KB |   570.5 KB  | 47x       |
| `create_group/8`    |  94 548.6 KB | 3 501.7 KB  | 27x       |
| `create_group/32`   | 333 101.2 KB | 33 216.6 KB | 10x       |
| `join_welcome`      |  10 331.6 KB |   252.2 KB  | 41x       |
| `send_app_message`  |   2 755.1 KB |    71.6 KB  | 38x       |
| `ingest_app_message`|   5 640.3 KB |    70.9 KB  | 80x       |

Latency improved too, though it is the noisier measurement — two post-rewrite
runs are given so the spread is visible rather than averaged away:

| operation           | p50 before | p50 after (run 1 / run 2) |
|---------------------|------------|---------------------------|
| `create_group/0`    |   6 323.7us |    4 184.2 / 3 971.5us     |
| `create_group/1`    |  16 928.2us |   13 197.2 / 13 574.2us    |
| `create_group/8`    |  51 470.7us |   43 244.0 / 43 424.7us    |
| `create_group/32`   | 190 080.0us |  202 261.1 / 172 908.9us   |
| `join_welcome`      |   6 219.3us |    4 919.5 / 4 991.3us     |
| `send_app_message`  |   1 720.9us |    1 314.3 / 1 376.0us     |
| `ingest_app_message`|   3 114.1us |    2 487.5 / 2 545.6us     |

`create_group/32` is the row to distrust: it has the fewest iterations, and its
two runs disagree by 17% at p50 and by nearly 2x at p99 (400.3ms then 210.4ms).
Read it as "no worse"; the other rows are consistent enough to read as gains.

Against MDK this closes most of the `create_group` gap — `create_group/1` goes
from 4.7x slower to about 3.7x — without changing a single protocol behaviour:
the RFC 7748 / RFC 8032 vector suites, the HPKE tests and the full 4833-test
quartz suite all pass unchanged, which is the point of keeping the allocating
functions around to differentially test against.

## Result: 10 limbs instead of 16

The allocation work above left `create_group` still ~3.9x slower than MDK, and
the primitive benchmarks said why: one X25519 scalar multiplication cost 541us,
and `create_group/1` is about two dozen of them. Curve work *was* the operation.

The cause was the representation, not the language. `Curve25519Field` used
TweetNaCl's 16 limbs of radix 2^16, so a schoolbook field multiply spent 256
limb products. SunEC's X25519 — also pure Java, same JIT, same machine — ran
the same operation in 160us using ~26-bit limbs in 10 words, which is 100
products. The ratio of products matched the ratio of times.

So the field was rewritten to 10 limbs of radix 2^25.5, the layout ref10,
curve25519-donna and SunEC all use: 100 products per multiply, 55 per square
(each off-diagonal pair once, doubled), and a dedicated scalar multiply for the
ladder's a24 constant instead of a general multiply against nine zero limbs.

| primitive        | 16 limbs | 10 limbs | speedup |
|------------------|----------|----------|---------|
| `x25519_dh`      |  541us   |  121us   | 4.5x    |
| `x25519_base`    |  535us   |  121us   | 4.4x    |
| `ed25519_sign`   | 1018us   |  259us   | 3.9x    |
| `ed25519_verify` | 2113us   |  536us   | 3.9x    |

At 121us the scalar multiplication is now faster than SunEC's 160us, which is
the useful sanity check on the result: it lands where a good managed-language
implementation should, rather than somewhere suspiciously better.

Against MDK, over the whole suite (two runs shown where they differ; `alloc/op`
reproduces to four significant figures):

| operation            | MDK (Rust) | quartz before | quartz now      | vs MDK       |
|----------------------|------------|---------------|-----------------|--------------|
| `create_group/1`     |  3.61 ms   | 16.93 ms      | 5.25 - 5.85 ms  | 1.5-1.6x slower |
| `create_group/8`     |  9.93 ms   | 51.47 ms      | 16.23 - 16.87 ms| 1.7x slower  |
| `create_group/32`    | 31.64 ms   | 190.08 ms     | 77.5 - 195.7 ms | 2.5x - 6x (see below) |
| `join_welcome`       |  4.77 ms   |  6.22 ms      | 1.89 - 1.93 ms  | **2.5x faster** |
| `send_app_message/0` |  4.28 ms   |  1.72 ms      | 0.55 - 0.60 ms  | **7.1x faster** |
| `ingest_app_message/1`| (n/a)     |  3.11 ms      | 0.88 - 0.95 ms  | —            |

`send_app_message/0` is the row that lines up with MDK, not a flattering pick:
their `prepare_app_send` builds the group with `create_request(vec![])`, so
their sender is alone in it too. The `/1`, `/8` and `/32` rows have no MDK
counterpart. `ingest_app_message` has none either — MDK's bench binary panics
in `bench_deferred_outbound_preflight_matrix` before reaching
`bench_app_message_ingest`, so there is no reference number to compare against
rather than one we chose not to use.

`create_group` remains the weakest row, and `create_group/32` is not just the
noisiest in the suite — it is the one number here that should not be quoted as
a single figure at all. Across five post-rewrite runs on this host its p50 came
out 77.5, 81.5, 82.5, 86.3 and 195.7 ms: four clustered within 11% of each
other and one more than twice the rest. The row has the fewest iterations in
the suite (each one has to build a 32-member group in setup) and this is a
shared cloud vCPU, so a single noisy neighbour moves it in a way it cannot move
the 300-iteration rows. Treat "roughly 2.5x MDK, occasionally much worse" as
the honest reading, and re-run before believing any movement in it.

### What not publishing the founding commit was worth

The founding-add fix removed a commit event that `create_group/N` used to
build, sign, outer-encrypt and publish for an audience of nobody. It is a
correctness fix first (see the commit), but it is also the one change in this
file whose benchmark effect is worth stating precisely, because it is smaller
than it sounds:

| row               | alloc before | alloc now | change |
|-------------------|--------------|-----------|--------|
| `create_group/0`  |    81.3 KB   |  81.3 KB  | none   |
| `create_group/1`  |   543.6 KB   | 482.5 KB  | -11%   |
| `create_group/8`  | ~3 445 KB    | 3 240 KB  | -6%    |
| `create_group/32` |  32 825 KB   | 32 355 KB | -1.4%  |

`create_group/0` is unchanged and must be: with no invitees there is no
founding Add to skip publishing. The saving grows in absolute terms with the
invitee count (~61 KB, ~205 KB, ~470 KB) because the commit not being built
carries N Add proposals, but shrinks as a fraction because everything else in
the operation grows faster. Latency moved within run-to-run noise, so no
latency claim is made for it: the reason to want this change is that a group
creation no longer depends on a relay acknowledgement the spec never asked
for.

### Why the constants can be trusted

Changing the representation re-encodes every curve constant, which is exactly
the kind of change where a single mistyped limb produces code that still runs
and is still wrong. None of them were transcribed by hand: each was re-derived
from its existing 16-bit encoding and then checked against its mathematical
definition — `d == -121665/121666`, `d2 == 2d`, `By == 4/5`, `I^2 == -1` — and
the multiply and square formulas were generated from the representation's
weight bookkeeping and diffed against an independent reference over 20 000
random limb vectors before any Kotlin was written. The carry chain and the
canonical encoder were validated the same way, including at `p`, `p-1`, and on
non-canonical inputs such as `p` itself.

The RFC 7748 and RFC 8032 vector suites, HPKE, the MDK crypto-interop vectors
and the full quartz + commons suites all pass unchanged.

## Group-size scaling: what the one-member benchmarks were hiding

The original `send_app_message` and `ingest_app_message` rows ran on groups of
one and two members. That is the flattering case, and it hid a real asymmetry.

Numbers from one full-suite run:

| benchmark                    | p50     | alloc/op   |
|------------------------------|---------|------------|
| `send_app_message/0 members` |  601 us |   68.7 KB  |
| `send_app_message/1 members` |  553 us |   73.8 KB  |
| `send_app_message/8 members` |  602 us |  117.1 KB  |
| `send_app_message/32 members`|  691 us |  253.1 KB  |
| `ingest_app_message/1`       |  877 us |   61.5 KB  |
| `ingest_app_message/8`       |  951 us |   65.1 KB  |
| `ingest_app_message/32`      |  966 us |   67.4 KB  |

**Latency is flat**, which is what MLS promises: an application message is
sealed under the sender's own ratchet and never touches the tree, so the
cryptography does not care how many members there are. The comparison against
MDK's `send_app_message` therefore survives the parameterisation.

**Allocation is not flat on the send side** — 3.5x from 0 to 32 members, while
the receive side barely moves. The cause is not subtle once looked at:

- `MlsGroupManager.encrypt` calls `persistGroup` **unconditionally**, so the
  whole group state, ratchet tree included, is serialised on every message
  sent.
- `MlsGroupManager.decrypt` calls it **only** when the message was a Commit
  that advanced the epoch. Decrypting an application message persists nothing.

So a 32-member group allocates 252.8 KB to send a message and 67.1 KB to
receive one, and the difference is a full state serialisation performed to
record what amounts to a generation-counter bump. The write itself is
necessary — a sender generation reused after a crash is a nonce-reuse-class
problem — but writing the entire group state for it is heavier than the
invariant requires. Left as a finding rather than a change: send-path
persistence is security-sensitive and deserves its own decision, not a
drive-by.

## `ingest_commit`: logarithmic in time, linear in allocation

Receiving someone else's Commit, from one full-suite run:

| benchmark                  | p50      | alloc/op    |
|----------------------------|----------|-------------|
| `ingest_commit/1 members`  | 1 950 us |   242.8 KB  |
| `ingest_commit/8 members`  | 2 367 us |   523.0 KB  |
| `ingest_commit/32 members` | 3 009 us | 1 390.0 KB  |

Latency grows, but far slower than the member count: 1.5x for a 32x bigger
group. That is the shape MLS predicts. The UpdatePath a Commit carries has one
node per LEVEL of the ratchet tree, so the receiver goes from roughly one HPKE
open at two members to roughly five at thirty-three — a log2 curve, not a
linear one.

Allocation grows 5.7x, tracking the size of the tree being parsed, rebuilt and
persisted rather than the number of curve operations. So `ingest_commit` is
mostly an allocation story, and it is the row to watch on a phone: every member
performs it on every membership or settings change, and at 1.4 MB it is by some
distance the largest single allocator in the suite.

A caution about isolated runs of this row. A `--only=ingest_commit` run
reported 3 995 / 2 905 / 3 182 us — no trend at all, and the one-member case
SLOWEST. That was JIT warm-up: `ingest_commit/1` runs first and pays for
compiling the shared group builder. The monotonic full-suite numbers above are
the trustworthy ones, which is the general rule here — prefer a full run, and
distrust whichever row happens to go first.