From 2d354f1ced2c6bf9c6126670ce0b04779115d0a0 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 15:08:09 +0000 Subject: [PATCH] perf(marmot): re-measure with the founding commit no longer published MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit marmotBench did not compile after the founding-add change — it was outside the modules I rebuilt when the commit event became nullable, so two call sites in it still dereferenced. Both are the same shape as the ones already fixed elsewhere: the ingest_app_message setup re-ingested its own commit as an echo, and the epoch probe printed the commit's kind. Neither has a commit to speak of now, and the probe says so: after createCurrentProfileGroup: epoch=0 after addMember: epoch=1 commits published by addMember: 0 (founding add, merged locally) bob after joining: epoch=1 The measured effect is smaller than "we removed a whole published event" suggests, and the README now says so precisely. Allocation per operation: create_group/0 unchanged at 81.3 KB — with no invitees there is no founding Add to skip — then -11% at one invitee (543.6 -> 482.5 KB), -6% at eight, and -1.4% at 32. The absolute saving grows with the invitee count because the commit that is no longer built carries N Adds; the fraction shrinks because the rest of the operation grows faster. Latency moved within run-to-run noise, so no latency claim is made for it. The reason to want the change is that creating a group with initial members no longer depends on a relay acknowledgement the spec never asked for. Full table refreshed from two runs: against MDK, create_group/1 is 1.5-1.6x slower, create_group/8 1.7x, create_group/32 2.6x, while join_welcome is 2.5x and send_app_message 6.5x FASTER. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016kCuA6tc4JQzHPCDd39GHq --- marmotBench/README.md | 58 +++++++++++++------ .../vitorpamplona/marmotbench/EpochProbe.kt | 5 +- .../marmotbench/MarmotBenchmarks.kt | 5 +- 3 files changed, 48 insertions(+), 20 deletions(-) diff --git a/marmotBench/README.md b/marmotBench/README.md index bf560d7abe..13f8a603b0 100644 --- a/marmotBench/README.md +++ b/marmotBench/README.md @@ -30,11 +30,13 @@ both sides — criterion's `iter_batched(.., PerIteration)` there, an explicit **On the shape of `create_group/N`.** An earlier version of this file claimed we spend "one more commit" than MDK's founding creation. That was wrong, and -`--epoch-probe` exists to keep it honest: our `create` publishes nothing and -leaves epoch 0, `addMember` produces exactly one commit, and the invitee joins -at epoch 1. MLS permits nothing else — RFC 9420 section 11 requires a group to -be created with a single member — so MDK commits its founding Adds internally -too. Both sides do one commit's worth of ratchet work. +`--epoch-probe` exists to keep it honest. It reported, before the founding-add +fix below: `create` publishes nothing and leaves epoch 0, `addMember` produces +exactly one commit, and the invitee joins at epoch 1. MLS permits nothing else +— RFC 9420 section 11 requires a group to be created with a single member — so +MDK commits its founding Adds internally too. Both sides do one commit's worth +of ratchet work. It now reports zero commits published for that same step, for +the reason in the next paragraph; re-run it rather than trusting this text. What did differ is that we PUBLISHED that founding commit and MDK does not. `protocol-core/publish-lifecycle.md` gives the founding Add an empty @@ -148,22 +150,44 @@ At 121us the scalar multiplication is now faster than SunEC's 160us, which is the useful sanity check on the result: it lands where a good managed-language implementation should, rather than somewhere suspiciously better. -Against MDK, over the whole suite (both post-rewrite runs shown where they -differ; `alloc/op` reproduces to four significant figures): +Against MDK, over the whole suite (two runs shown where they differ; `alloc/op` +reproduces to four significant figures): | operation | MDK (Rust) | quartz before | quartz now | vs MDK | |----------------------|------------|---------------|-----------------|--------------| -| `create_group/1` | 3.61 ms | 16.93 ms | 5.42 - 5.88 ms | 1.5-1.6x slower | -| `create_group/8` | 9.93 ms | 51.47 ms | 17.27 - 17.60 ms| 1.8x slower | -| `create_group/32` | 31.64 ms | 190.08 ms | 77.53 - 81.47 ms| 2.5x slower | -| `join_welcome` | 4.77 ms | 6.22 ms | 1.82 - 2.00 ms | **2.5x faster** | -| `send_app_message` | 4.28 ms | 1.72 ms | 0.61 - 0.71 ms | **6.5x faster** | -| `ingest_app_message` | (n/a) | 3.11 ms | 0.88 - 0.90 ms | — | +| `create_group/1` | 3.61 ms | 16.93 ms | 5.25 - 5.85 ms | 1.5-1.6x slower | +| `create_group/8` | 9.93 ms | 51.47 ms | 16.23 - 16.87 ms| 1.7x slower | +| `create_group/32` | 31.64 ms | 190.08 ms | 82.49 - 86.31 ms| 2.6x slower | +| `join_welcome` | 4.77 ms | 6.22 ms | 1.89 - 1.93 ms | **2.5x faster** | +| `send_app_message` | 4.28 ms | 1.72 ms | 0.63 - 0.68 ms | **6.5x faster** | +| `ingest_app_message` | (n/a) | 3.11 ms | 0.90 - 0.95 ms | — | -`create_group` remains the weakest row. Both sides do the same one commit (see -"What is compared"); what we additionally carried was publishing it, which the -founding-add fix has since removed. `create_group/32` is also still the noisiest -row in the suite. +`create_group` remains the weakest row, and `create_group/32` is still the +noisiest in the suite — its two runs here disagree by nearly 2.4x at p99. + +### What not publishing the founding commit was worth + +The founding-add fix removed a commit event that `create_group/N` used to +build, sign, outer-encrypt and publish for an audience of nobody. It is a +correctness fix first (see the commit), but it is also the one change in this +file whose benchmark effect is worth stating precisely, because it is smaller +than it sounds: + +| row | alloc before | alloc now | change | +|-------------------|--------------|-----------|--------| +| `create_group/0` | 81.3 KB | 81.3 KB | none | +| `create_group/1` | 543.6 KB | 482.5 KB | -11% | +| `create_group/8` | ~3 445 KB | 3 240 KB | -6% | +| `create_group/32` | 32 825 KB | 32 355 KB | -1.4% | + +`create_group/0` is unchanged and must be: with no invitees there is no +founding Add to skip publishing. The saving grows in absolute terms with the +invitee count (~61 KB, ~205 KB, ~470 KB) because the commit not being built +carries N Add proposals, but shrinks as a fraction because everything else in +the operation grows faster. Latency moved within run-to-run noise, so no +latency claim is made for it: the reason to want this change is that a group +creation no longer depends on a relay acknowledgement the spec never asked +for. ### Why the constants can be trusted diff --git a/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/EpochProbe.kt b/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/EpochProbe.kt index 4a2b8c1e4b..c41f399ff5 100644 --- a/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/EpochProbe.kt +++ b/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/EpochProbe.kt @@ -40,7 +40,10 @@ fun epochProbe() = val kp = bob.manager.generateKeyPackageEvent(relays = emptyList()) val (commit, welcome) = alice.manager.addMember(groupId, kp, emptyList()) println("after addMember: epoch=${alice.manager.groupEpoch(groupId)}") - println("commits published by addMember: 1 (kind ${commit.signedEvent.kind})") + println( + "commits published by addMember: " + + (commit?.let { "1 (kind ${it.signedEvent.kind})" } ?: "0 (founding add, merged locally)"), + ) println("welcome produced: ${welcome != null}") bob.manager.ingest(welcome!!.giftWrapEvent) diff --git a/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/MarmotBenchmarks.kt b/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/MarmotBenchmarks.kt index d07e0aa65c..0b90775ebe 100644 --- a/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/MarmotBenchmarks.kt +++ b/marmotBench/src/main/kotlin/com/vitorpamplona/marmotbench/MarmotBenchmarks.kt @@ -146,9 +146,10 @@ fun benchIngestAppMessage(): BenchResult = val groupId = newGroupId() alice.manager.createCurrentProfileGroup(groupId, listOf("wss://bench.invalid"), GroupProfileV1("bench", "")) val kp = bob.manager.generateKeyPackageEvent(relays = emptyList()) - val (commit, welcome) = alice.manager.addMember(groupId, kp, emptyList()) + // A founding add publishes no commit, so there is no echo for + // Alice to re-ingest — the Welcome is the whole delivery. + val (_, welcome) = alice.manager.addMember(groupId, kp, emptyList()) bob.manager.ingest(welcome!!.giftWrapEvent) - alice.manager.ingest(commit.signedEvent) val sent = alice.manager.buildTextMessage(groupId, PAYLOAD, persistOwn = false) bob to sent.outbound.signedEvent }