mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-08 23:54:39 +00:00
fix(concord): isolate the Control Plane sub so channels don't starve
A Concord community pinned to the bottom bar folded only a fraction of its channels — Soapbox showed 1 of 12. The live subscription collapsed a community's Control + Guestbook + rekey + every channel plane into ONE kind-1059 filter per relay, and the channel list is folded from the Control Plane. On an AUTH-gated relay that caps a REQ per filter (measured ~100 events/filter on relay.dreamith.to), the chatty Guestbook plane crowded the channel-defining control editions out of the cap, so only a fraction of the channels folded. On the strict relay.ditto.pub the collapsed multi-author filter is refused wholesale until every author is authenticated. - Split the Control Plane into its OWN filter, apart from the Guestbook / rekey / channel planes (ConcordSubscriptionPlanner.controlIsolatedFilters), so it gets an isolated per-filter budget. Both filters still ride the same per-relay REQ (no extra socket). - Add a COMPLETE-mode Control-Plane sweep (Account.syncConcordControlPlanes): re-fetch the whole plane with no `since`, paging past the per-filter cap via fetchAllPagesFromPool, so a forward cursor can never hide an edition and the cap can never truncate the fold. Mounted account-wide, fired on load + membership/held-epoch change + relay reconnect (not a wall-clock poll — the persistent live subscription keeps a connected relay complete). Mirrors Armada's plane-sweep design (one filter per plane scope, COMPLETE-mode control). Verified on-device: Soapbox now folds all 12 channels. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b4e8719ee2
commit
844b0e9803
@@ -223,6 +223,7 @@ import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.PublishResult
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAll
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPagesFromPool
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllWithHooks
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchFirst
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.publishAndCollectResults
|
||||
@@ -3032,6 +3033,44 @@ class Account(
|
||||
client.fetchAll(filters = byRelay, timeoutMs = 20_000L)
|
||||
}
|
||||
|
||||
/**
|
||||
* COMPLETE-mode Control-Plane sync — Armada's plane-sweep discipline for the one plane that must
|
||||
* never fold on a truncated edition set.
|
||||
*
|
||||
* The Control Plane defines the channel list, the roster and the banlist, so a *partial* fold
|
||||
* silently drops channels or mis-renders membership. Two ways that happens, both closed here:
|
||||
* - **Forward-cursor gap:** the live plane subscription advances a `since` cursor, so an edition
|
||||
* with a `created_at` below the high-water mark that we never actually ingested — an unban
|
||||
* published while we were offline, a CORD-06 compaction re-wrap under a newly-held epoch — is
|
||||
* never asked for again and stays invisible. This sweep uses **no `since`**: it re-fetches the
|
||||
* whole plane every run.
|
||||
* - **Per-filter cap:** a relay caps a REQ's result (~100/filter on relay.dreamith.to), which can
|
||||
* crop a busy Control Plane. This **pages past the cap** ([fetchAllPagesFromPool] walks `until`
|
||||
* cursors until a plane is drained), so the fold sees every edition regardless of the cap.
|
||||
*
|
||||
* Current + every held-prior epoch's Control Plane is swept (the anti-rollback floor folds from the
|
||||
* priors). Wraps ingest through the global cache connector → [concordSessions] like every other
|
||||
* Concord drain; AUTH is the shared stream-key handler. Merging communities that share a relay into
|
||||
* one filter is safe here precisely because we page — the cap no longer truncates. The live control
|
||||
* subscription still carries brand-new editions in real time; this is the periodic completeness pass.
|
||||
*/
|
||||
suspend fun syncConcordControlPlanes(entries: List<ConcordCommunityListEntry>) {
|
||||
if (entries.isEmpty()) return
|
||||
val authorsByRelay = HashMap<NormalizedRelayUrl, MutableSet<String>>()
|
||||
for (entry in entries) {
|
||||
for (sub in ConcordSubscriptionPlanner.controlPlaneSubs(listOf(entry))) {
|
||||
for (relay in sub.relays) authorsByRelay.getOrPut(relay) { HashSet() }.add(sub.pubKeyHex)
|
||||
}
|
||||
}
|
||||
if (authorsByRelay.isEmpty()) return
|
||||
// No `since`, no `limit` → fetchAllPages treats each filter as unbounded and pages until a
|
||||
// plane is fully drained (empty page), so the whole Control Plane lands regardless of the cap.
|
||||
val byRelay = authorsByRelay.mapValues { (_, authors) -> listOf(ConcordActions.planeFilterFor(authors.toList())) }
|
||||
var drained = 0
|
||||
client.fetchAllPagesFromPool(filters = byRelay) { _, _ -> drained++ }
|
||||
Log.d("Concord", "syncConcordControlPlanes: paged ${authorsByRelay.size} relay(s), drained $drained control wrap(s)")
|
||||
}
|
||||
|
||||
// ── NIP-29 relay-group actions ───────────────────────────────────────────
|
||||
// All group commands are published ONLY to the group's host relay, where
|
||||
// relay29 authorizes them. The relay is the source of truth; the kind-10009
|
||||
|
||||
+9
-19
@@ -20,7 +20,6 @@
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.publicChannels.concord.datasource
|
||||
|
||||
import com.vitorpamplona.amethyst.commons.actions.ConcordPlaneSub
|
||||
import com.vitorpamplona.amethyst.commons.actions.ConcordSubscriptionPlanner
|
||||
import com.vitorpamplona.amethyst.commons.model.chats.ChatFeedType
|
||||
import com.vitorpamplona.amethyst.commons.relayClient.composeSubscriptionManagers.ComposeSubscriptionManager
|
||||
@@ -77,27 +76,18 @@ class ConcordChannelSubAssembler(
|
||||
val entries = account.concordChannelList.liveCommunities.value
|
||||
if (entries.isEmpty()) return null
|
||||
|
||||
// Control planes for every joined community, plus channel planes for the
|
||||
// ones whose Control Plane has already folded. Deriving the channel planes
|
||||
// is the only account-dependent step; collapsing planes into per-relay
|
||||
// kind-1059 filters lives in the shared planner.
|
||||
val subs = ArrayList<ConcordPlaneSub>()
|
||||
subs += ConcordSubscriptionPlanner.controlPlaneSubs(entries)
|
||||
// The Guestbook (membership) + next-epoch base-rekey planes. Their stream keys derive from
|
||||
// the entry alone, so they AUTH on the initial connection; and since the relay now
|
||||
// re-authenticates on an `auth-required` CLOSED, naming them here no longer starves the
|
||||
// control/channel REQ the way it did before that fix.
|
||||
subs += ConcordSubscriptionPlanner.auxiliaryPlaneSubs(entries)
|
||||
for (entry in entries) {
|
||||
val state =
|
||||
// The Control Plane rides its OWN filter, kept apart from the Guestbook / rekey / channel
|
||||
// planes, so a chatty Guestbook can't crowd its channel-defining editions out of a relay's
|
||||
// per-filter result cap (the "Soapbox shows 1 of 12 channels" bug). See
|
||||
// [ConcordSubscriptionPlanner.controlIsolatedFilters].
|
||||
val all =
|
||||
ConcordSubscriptionPlanner.controlIsolatedFilters(entries, since = since) { entry ->
|
||||
account.concordSessions
|
||||
.sessionFor(entry.id)
|
||||
?.state
|
||||
?.value ?: continue
|
||||
subs += ConcordSubscriptionPlanner.channelPlaneSubs(entry, state)
|
||||
}
|
||||
|
||||
return ConcordSubscriptionPlanner.relayBasedFilters(subs, since)
|
||||
?.value
|
||||
}
|
||||
return all.ifEmpty { null }
|
||||
}
|
||||
|
||||
override fun id(key: ConcordChannelQueryState) = key.account
|
||||
|
||||
+75
@@ -24,11 +24,14 @@ import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.rememberUpdatedState
|
||||
import androidx.lifecycle.compose.collectAsStateWithLifecycle
|
||||
import com.vitorpamplona.amethyst.commons.relayClient.subscriptions.KeyDataSourceSubscription
|
||||
import com.vitorpamplona.amethyst.commons.relayClient.subscriptions.LifecycleAwareKeyDataSourceSubscription
|
||||
import com.vitorpamplona.amethyst.ui.navigation.bottombars.BottomBarEntry
|
||||
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountViewModel
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
|
||||
/**
|
||||
* Mount on any screen that lists the user's joined Concord Channels (the Messages
|
||||
@@ -100,6 +103,10 @@ fun ConcordChannelPreload(accountViewModel: AccountViewModel) {
|
||||
|
||||
bootstrapPinnedCommunities(accountViewModel)
|
||||
|
||||
// COMPLETE-mode Control-Plane sweep: page every joined community's whole Control Plane (no `since`,
|
||||
// past the relay's per-filter cap) so the channel/roster/banlist fold is never silently truncated.
|
||||
ConcordControlPlaneSync(accountViewModel)
|
||||
|
||||
// Warm the last-message preview of every channel of every joined community (one drain per relay),
|
||||
// so the Messages inbox shows a preview for channels the user has never opened.
|
||||
ConcordChannelPreviewAccountPreload(accountViewModel)
|
||||
@@ -107,6 +114,74 @@ fun ConcordChannelPreload(accountViewModel: AccountViewModel) {
|
||||
KeyDataSourceSubscription(state, dataSource)
|
||||
}
|
||||
|
||||
/**
|
||||
* Account-wide COMPLETE-mode sync of every joined community's Control Plane — the completeness pass
|
||||
* that pairs with the always-on live subscription. The live plane subscription
|
||||
* ([ConcordChannelFilterAssembler]) carries brand-new editions in real time, but it advances a
|
||||
* `since` cursor and rides the relay's per-filter cap, so it can miss an edition below the high-water
|
||||
* mark or a cropped initial page — either of which folds a partial channel list / stale roster.
|
||||
* [com.vitorpamplona.amethyst.model.Account.syncConcordControlPlanes] closes both by re-fetching the
|
||||
* whole plane with no `since`, paging past the cap.
|
||||
*
|
||||
* It fires only on the **events** that can create a gap — never on a wall-clock poll, because a
|
||||
* persistent live subscription already keeps a connected relay complete:
|
||||
* 1. **Load / membership / held-epoch change** — keyed on a signature of the joined set + held
|
||||
* epochs (the moments a new Control-Plane address appears: join/leave, Refounding). NOT the fold
|
||||
* revision — a revision-keyed sweep would loop (drain → ingest → fold → revision → drain).
|
||||
* 2. **Reconnect** — a relay hosting a joined community that comes back online may have missed
|
||||
* editions published while it was down; that gap is exactly what the live sub can't backfill. We
|
||||
* re-sweep when the connected-relay set *gains* one of our relays, coalescing flaps with a min
|
||||
* interval.
|
||||
*/
|
||||
@Composable
|
||||
private fun ConcordControlPlaneSync(accountViewModel: AccountViewModel) {
|
||||
val account = accountViewModel.account
|
||||
val communities by account.concordChannelList.liveCommunities.collectAsStateWithLifecycle()
|
||||
// Always-current set for the reconnect collector, whose effect is keyed on relays (which don't
|
||||
// change when a same-relay community leaves) — so it must not close over a stale `communities`.
|
||||
val liveCommunities by rememberUpdatedState(communities)
|
||||
|
||||
val sig =
|
||||
remember(communities) {
|
||||
communities.joinToString(",") { entry ->
|
||||
"${entry.id}@${entry.rootEpoch}:" + entry.heldRoots.joinToString("-") { it.epoch.toString() }
|
||||
}
|
||||
}
|
||||
|
||||
// (1) Load + membership/epoch change: one complete sweep of the whole set.
|
||||
LaunchedEffect(sig) {
|
||||
if (communities.isNotEmpty()) account.syncConcordControlPlanes(communities)
|
||||
}
|
||||
|
||||
// (2) Reconnect: re-sweep when a relay of ours transitions disconnected → connected.
|
||||
val ourRelays =
|
||||
remember(communities) {
|
||||
communities
|
||||
.flatMap { it.relays }
|
||||
.mapNotNullTo(HashSet()) { RelayUrlNormalizer.normalizeOrNull(it) }
|
||||
}
|
||||
LaunchedEffect(ourRelays) {
|
||||
if (ourRelays.isEmpty()) return@LaunchedEffect
|
||||
val connectedFlow = account.client.connectedRelaysFlow()
|
||||
// Baseline = already-open relays; the (1) sweep covered those, so only NEW connections sweep.
|
||||
var open = connectedFlow.value.intersect(ourRelays)
|
||||
var lastSweep = 0L
|
||||
connectedFlow.collect { nowConnected ->
|
||||
val ours = nowConnected.intersect(ourRelays)
|
||||
val newlyUp = ours - open
|
||||
open = ours
|
||||
if (newlyUp.isEmpty()) return@collect
|
||||
val now = TimeUtils.nowMillis()
|
||||
if (now - lastSweep < RECONNECT_RESWEEP_MIN_INTERVAL_MS) return@collect
|
||||
lastSweep = now
|
||||
account.syncConcordControlPlanes(liveCommunities)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Coalesce relay flaps: at most one reconnect-driven completeness sweep per this window. */
|
||||
private const val RECONNECT_RESWEEP_MIN_INTERVAL_MS = 60_000L
|
||||
|
||||
/**
|
||||
* Fetch the private kind-13302 list of any Concord community pinned to the bottom bar whose list we
|
||||
* don't already have, from the relays saved on its tab (see
|
||||
|
||||
+33
@@ -207,6 +207,39 @@ object ConcordSubscriptionPlanner {
|
||||
return authorsByRelay.mapValues { (_, authors) -> listOf(ConcordActions.planeFilterFor(authors)) }
|
||||
}
|
||||
|
||||
/**
|
||||
* The live-subscription filters for every joined community, with the **Control Plane kept in
|
||||
* its own filter**, apart from the Guestbook / next-epoch rekey / channel planes.
|
||||
*
|
||||
* This is the anti-starvation split. The Control Plane is what folds the channel *list* (and
|
||||
* roster/roles); collapsing it into one `authors=[…every plane…]` filter lets a chatty plane —
|
||||
* the Guestbook can carry dozens of membership wraps — crowd its editions out of a relay's
|
||||
* per-filter result cap (measured ~100/filter on relay.dreamith.to), so only a fraction of the
|
||||
* channels fold (the "Soapbox shows 1 of 12 channels" bug). A dedicated filter gives the Control
|
||||
* Plane an isolated per-filter budget, so the whole plane folds no matter how busy the Guestbook
|
||||
* is. Mirrors Armada's plane-sweep design (one filter per plane scope). Both groups ride the same
|
||||
* per-relay REQ downstream (groupByRelay), so no extra socket/subscription is opened.
|
||||
*
|
||||
* [stateOf] supplies each entry's folded state (null → its channels aren't known yet, so only its
|
||||
* Control/aux planes are subscribed). [since] is the per-relay EOSE cursor, applied to every filter.
|
||||
*/
|
||||
fun controlIsolatedFilters(
|
||||
entries: List<ConcordCommunityListEntry>,
|
||||
since: SincePerRelayMap?,
|
||||
stateOf: (ConcordCommunityListEntry) -> ConcordCommunityState?,
|
||||
): List<RelayBasedFilter> {
|
||||
val controlSubs = controlPlaneSubs(entries)
|
||||
|
||||
val otherSubs = ArrayList<ConcordPlaneSub>()
|
||||
otherSubs += auxiliaryPlaneSubs(entries)
|
||||
for (entry in entries) {
|
||||
val state = stateOf(entry) ?: continue
|
||||
otherSubs += channelPlaneSubs(entry, state)
|
||||
}
|
||||
|
||||
return relayBasedFilters(controlSubs, since).orEmpty() + relayBasedFilters(otherSubs, since).orEmpty()
|
||||
}
|
||||
|
||||
/**
|
||||
* Collapses [subs] into one [RelayBasedFilter] per host relay for a live
|
||||
* subscription: each relay gets a single `{kinds:[1059], authors:[…all plane
|
||||
|
||||
+37
@@ -194,4 +194,41 @@ class ConcordSubscriptionPlannerTest {
|
||||
// No planes resolve to a relay -> nothing to subscribe.
|
||||
assertNull(ConcordSubscriptionPlanner.relayBasedFilters(emptyList(), null))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun controlIsolatedFiltersKeepControlPlaneOffTheGuestbookAndChannelFilter() =
|
||||
runTest {
|
||||
// Regression: a Concord community whose Control Plane got collapsed into the same
|
||||
// `authors=[…]` filter as the Guestbook + channel planes folded only a fraction of its
|
||||
// channels — a chatty Guestbook crowded the channel-defining editions out of the relay's
|
||||
// per-filter cap (Soapbox showed 1 of 12 channels). The Control Plane must ride its own filter.
|
||||
val community = ConcordCommunityFactory.create(owner, "Nostrichs", createdAt = 1L, relays = listOf("wss://r.example"))
|
||||
val entry =
|
||||
com.vitorpamplona.quartz.concord.cord02Community.ConcordCommunityListEntry(
|
||||
id = community.communityIdHex,
|
||||
owner = community.ownerPubKey,
|
||||
ownerSalt = community.ownerSalt.toHexKey(),
|
||||
root = community.communityRoot.toHexKey(),
|
||||
rootEpoch = community.rootEpoch,
|
||||
relays = listOf("wss://r.example"),
|
||||
name = "Nostrichs",
|
||||
)
|
||||
val state = ConcordActions.foldCommunity(community.genesisWraps, community.controlPlane, community.ownerPubKey)
|
||||
|
||||
val controlPk = community.controlPlane.publicKeyHex
|
||||
val guestbookPk = ConcordActions.guestbookPlane(community.communityRoot, community.communityId, community.rootEpoch).publicKeyHex
|
||||
val generalPk = ConcordActions.publicChannel(community.communityRoot, community.generalChannelId, community.rootEpoch).publicKeyHex
|
||||
|
||||
val filters = ConcordSubscriptionPlanner.controlIsolatedFilters(listOf(entry), stateOf = { state }, since = null)
|
||||
|
||||
// The filter carrying the Control Plane must NOT also carry the Guestbook or a channel plane.
|
||||
val controlFilter = filters.map { it.filter }.single { controlPk in it.authors.orEmpty() }
|
||||
assertTrue(guestbookPk !in controlFilter.authors.orEmpty(), "Guestbook leaked into the Control Plane filter")
|
||||
assertTrue(generalPk !in controlFilter.authors.orEmpty(), "a channel plane leaked into the Control Plane filter")
|
||||
|
||||
// The Guestbook and the channel plane share the OTHER filter, and it must not carry Control.
|
||||
val otherFilter = filters.map { it.filter }.single { guestbookPk in it.authors.orEmpty() }
|
||||
assertTrue(generalPk in otherFilter.authors.orEmpty(), "channel plane missing from the non-control filter")
|
||||
assertTrue(controlPk !in otherFilter.authors.orEmpty(), "Control Plane leaked into the Guestbook/channel filter")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user