feat: split DM loading into a live tail + bounded history slices

Every widen re-requested the whole DM window (the filters carried `since` only,
no `until`), so a relay re-streamed the entire history from the new floor — a few
pixels of scroll walked the window to the 10-year backstop, re-downloading
exponentially more each step (589 → 1486 → 2609 events in one session). This
splits each DM protocol into two responsibilities:

- Live tail (existing managers, now fixed): a one-week floor with no `until`,
  always open to the future. Never widens, so new messages keep arriving.
- History slices (new managers): load the past in bounded `since`+`until`
  one-shot slices. Widening fetches only the new band `[newFloor, prevFloor]`;
  consecutive slices are disjoint so advancing the filter never re-streams an
  earlier slice — they live in the cache. The NIP-17 2-day wrapper-timestamp
  margin is applied to the slice `since`, overlapping adjacent slices so a
  randomized outer timestamp can't open a gap. NIP-04 (exact timestamps) needs
  no margin.

New: AccountGiftWrapsHistoryEoseManager owns the geometric window and the
bounded slices; ChatroomListNip04HistorySubAssembler / ChatroomNip04History-
SubAssembler follow its slice bounds so both protocols page to the same depth.
The live managers (AccountGiftWrapsEoseManager and the NIP-04 followers) are
reduced to the fixed one-week tail.

Also adds the rooms-list stall-gate: the auto-fill remembers the private-room
count at the last widen (on the history manager, so it survives reopening the
screen) and stops widening once a step brings in no new private room — widening
pulls older messages, not rooms, so a few busy correspondents would otherwise
flood events without ever filling the list. "Fill until full OR nothing new
found", instead of walking to the 10-year backstop.

Design: amethyst/plans/2026-06-01-dm-live-tail-and-history-slices.md

https://claude.ai/code/session_01B1fmmmX8JjQWH3amMLdvcW
This commit is contained in:
Claude
2026-06-01 23:43:02 +00:00
parent 9e2e595cac
commit 793860170f
17 changed files with 651 additions and 238 deletions
@@ -0,0 +1,75 @@
# DM loading: live tail + bounded history slices
## Problem
The DM loaders used a single subscription whose `since` floor grew as the user
scrolled (`loadMore`: 7d → 14d → 28d → …). The filter carried **only `since`,
no `until`**, so every widen re-requested the whole window and the relay
re-streamed the entire history from the new floor. Traces showed this directly:
```
[giftwrap] load summary: 589 event(s) (14d)
[giftwrap] load summary: 1486 event(s) (28d)
[giftwrap] load summary: 2609 event(s) (56d)
```
Each step re-downloaded everything it already had plus the new slice — the
"getting all events over and over again" the window owner reported. It also
cascaded: a few pixels of scroll walked the window to the 10-year backstop,
because widening pulls older *messages* but the rooms list is keyed by
*conversation*, so a handful of busy correspondents flood thousands of events
without adding a single new row, and the "scrolled near the oldest room" trigger
never clears.
## Design (owner's call)
Split each DM protocol into two responsibilities:
1. **Live tail** — keep the existing filters at a fixed ~1-week floor with **no
`until`** (open to the future). Never widens. New messages always arrive.
2. **History slices** — new assemblers that load *the past* in **bounded
`since`+`until` slices**, each fetched once. Widening fetches only the new
band `[newFloor, previousFloor]`; the data already held in `[previousFloor,
now]` is never re-requested.
Because consecutive slices are disjoint, re-issuing the (advanced) historical
filter does not re-stream earlier slices — they live in `LocalCache`. The
NIP-17 ±2-day wrapper-timestamp margin is applied to the slice `since` (via
`filterGiftWrapsToPubkey`), giving a 2-day overlap between adjacent slices so no
gap can open from a randomized outer timestamp. NIP-04 (kind 4) uses exact
timestamps, so its slices need no margin.
### Slice math (gift-wrap history window)
`TimeWindowPagination.since` starts at `now 1week` (= the live-tail floor).
- `loadMore`: `until = window.since` (current floor); `window.loadMore()` moves
`since` back geometrically; new slice = `[window.since, until]`.
- `loadEverything`: `until = window.since`; `window.loadAll()``since = floor`;
slice = `[floor, until]` — one request for the remaining past.
- `updateFilter` returns the **current slice** only (or empty before the first
`loadMore`), so the manager is idle until the user asks for older history.
### Rooms-list cascade stop (stall-gate)
The rooms list auto-fill widens only while it makes progress: it remembers the
private-room count at the last widen and stops once a widen brings in no new
private room (kept on the history manager so it survives leaving/reopening the
screen). "Fill until full **or nothing new found**", instead of walking to the
10-year backstop.
## Touch list
- `commons/.../FilterGiftWrapsToPubkey.kt`, `amethyst/.../FilterNip04DMsToMe.kt`,
`FilterNip04DMsFromMe.kt` — add optional `until`.
- `AccountGiftWrapsEoseManager` — becomes the live tail (fixed week, no until).
- `AccountGiftWrapsHistoryEoseManager` (new) — owns the window, bounded slices,
`loadMore`/`loadEverything`/`exhausted`, per-load instrumentation.
- `ChatroomListNip04SubAssembler` / `ChatroomNip04SubAssembler` — live tail.
- `ChatroomListNip04HistorySubAssembler` / `ChatroomNip04HistorySubAssembler`
(new) — follow the gift-wrap history slice bounds.
- `AccountFilterAssembler`, `ChatroomListFilterAssembler`,
`ChatroomFilterAssembler`, `RelaySubscriptionsCoordinator` — wire the new
managers.
- `ChatroomListFeedView`, `ChatroomView` — point "load older" at the history
managers; combine live+history `loadingMore` for spinners; add the stall-gate.
@@ -90,7 +90,7 @@ class RelaySubscriptionsCoordinator(
// always running, feed assemblers.
val home = HomeFilterAssembler(client)
val chatroomList = ChatroomListFilterAssembler(client, account.giftWraps)
val chatroomList = ChatroomListFilterAssembler(client, account.giftWrapsHistory)
val video = VideoFilterAssembler(client)
val discovery = DiscoveryFilterAssembler(client)
@@ -105,7 +105,7 @@ class RelaySubscriptionsCoordinator(
// active depending on the screen.
val channel = ChannelFilterAssembler(client)
val chatroom = ChatroomFilterAssembler(client, account.giftWraps)
val chatroom = ChatroomFilterAssembler(client, account.giftWrapsHistory)
val community = CommunityFilterAssembler(client)
val gitRepository = RepositoryFilterAssembler(client)
val thread = ThreadFilterAssembler(client)
@@ -28,6 +28,7 @@ import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.marmot.
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.metadata.AccountMetadataEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip01Notifications.AccountNotificationsEoseFromInboxRelaysManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountFeedContentStates
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
@@ -49,12 +50,17 @@ class AccountQueryState(
class AccountFilterAssembler(
client: INostrClient,
) : ComposeSubscriptionManager<AccountQueryState>() {
// Live tail: the recent week of gift wraps, always open at the top for new messages.
val giftWraps = AccountGiftWrapsEoseManager(client, ::allKeys)
// History: older gift wraps, loaded on demand in bounded one-shot slices.
val giftWrapsHistory = AccountGiftWrapsHistoryEoseManager(client, ::allKeys)
val group =
listOf(
AccountMetadataEoseManager(client, ::allKeys),
giftWraps,
giftWrapsHistory,
AccountDraftsEoseManager(client, ::allKeys),
AccountNotificationsEoseFromInboxRelaysManager(client, ::allKeys),
MarmotGroupEventsEoseManager(client, ::allKeys),
@@ -21,36 +21,30 @@
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps
import com.vitorpamplona.amethyst.commons.relayClient.nip17Dm.filterGiftWrapsToPubkey
import com.vitorpamplona.amethyst.commons.relayClient.pagination.TimeWindowPagination
import com.vitorpamplona.amethyst.model.User
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.WindowLoadTracker
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.trackingListener
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.AccountQueryState
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicInteger
/**
* Always-on loader for the account's NIP-17 gift wraps (kind 1059). It owns the single DM time
* window: boot opens a small window so the messages list is usable before the whole history is
* fetched and decrypted, and the screens widen it ([loadMore] / [loadEverything]) as the user
* scrolls. The NIP-04 loaders follow this window's [windowSince] so both DM protocols stay aligned.
* Always-on **live tail** for the account's NIP-17 gift wraps (kind 1059). It keeps a fixed
* one-week floor with no upper bound, so the messages list is usable on boot and new incoming
* messages always stream in. It deliberately never widens: pulling older history is the job of
* [AccountGiftWrapsHistoryEoseManager], which fetches the past in bounded, one-shot slices so a
* widen never re-streams what this tail already holds.
*/
class AccountGiftWrapsEoseManager(
client: INostrClient,
@@ -58,93 +52,11 @@ class AccountGiftWrapsEoseManager(
) : PerUserEoseManager<AccountQueryState>(client, allKeys) {
override fun user(key: AccountQueryState) = key.account.userProfile()
// Per-account window floor. Concurrent: windowFor is reached from the UI thread (loadMore /
// loadEverything) and from Dispatchers.IO (updateFilter), so a plain HashMap would race.
private val windows = ConcurrentHashMap<HexKey, TimeWindowPagination>()
private fun windowFor(user: User) =
windows.computeIfAbsent(user.pubkeyHex) {
TimeWindowPagination(growthFactor = WINDOW_GROWTH_FACTOR).also {
Log.d(TAG) { "[giftwrap] open window since=${it.since} (${daysAgo(it.since)}d back)" }
}
}
/** The current lower bound (epoch seconds) of this account's gift-wrap window. */
fun windowSince(user: User): Long = windowFor(user).since
private fun daysAgo(epochSeconds: Long) = (TimeUtils.now() - epochSeconds) / TimeUtils.ONE_DAY
// A window load is in flight until every dmRelay it was sent to has answered, the event stream
// goes quiet, or a cap fires — never on just the first EOSE (a fast empty relay would trip it).
private val windowLoad = WindowLoadTracker("giftwrap")
// The initial-load tracker drives the boot spinner: it stays true until every DM relay has
// settled (EOSE / CLOSED / cannot-connect) on the one-week tail.
private val windowLoad = WindowLoadTracker("giftwrap.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// True once the window reached the maximum lookback: nothing older to fetch.
private val _exhausted = MutableStateFlow(false)
val exhausted: StateFlow<Boolean> = _exhausted.asStateFlow()
// Account scope for the window-load watchdog. Volatile: written on IO (newSub), read on UI (loadMore).
@Volatile
private var scope: CoroutineScope? = null
// Per-load instrumentation. Each gift-wrap event the relays push during a load is counted, and
// those whose (outer, randomized) created_at falls before the floor the REQ actually asked for are
// counted separately — a relay ignoring `since` re-streams the whole history every widen, so a
// total that keeps growing (and a non-zero out-of-window share) is the fingerprint of "getting all
// events over and over again". [loadSince] is the *margined* floor (window.since 2 days, matching
// the wire REQ from filterGiftWrapsToPubkey), so the deliberate 2-day randomization band reads as
// in-window and only a relay that under-shoots that floor is flagged. Volatile/atomic because the
// event hook runs on the relay IO threads while the summary collector reads on the account scope.
@Volatile
private var loadSince = 0L
private val eventsThisLoad = AtomicInteger(0)
private val outOfWindowThisLoad = AtomicInteger(0)
// The single tracker is shared across every account, so the summary collector is launched once
// (not per newSub) — otherwise a second logged-in account would double every summary line.
@Volatile
private var summaryJob: Job? = null
private fun countEvent(createdAt: Long) {
eventsThisLoad.incrementAndGet()
if (createdAt < loadSince) outOfWindowThisLoad.incrementAndGet()
}
/**
* Starts a window load and resets the per-load counters in the same breath (synchronously, before
* [WindowLoadTracker.startLoading] raises `loading`, so no in-flight event is counted against the
* wrong load). The summary is emitted by a single collector that logs on each load's falling edge.
*/
private fun beginWindowLoad(
user: User,
scope: CoroutineScope,
) {
loadSince = windowFor(user).since - TimeUtils.twoDays()
eventsThisLoad.set(0)
outOfWindowThisLoad.set(0)
ensureSummaryLogger(scope)
windowLoad.startLoading(scope)
}
private fun ensureSummaryLogger(scope: CoroutineScope) {
if (summaryJob?.isActive == true) return
summaryJob =
scope.launch {
var wasLoading = false
windowLoad.loading.collect { loading ->
if (!loading && wasLoading) {
val total = eventsThisLoad.get()
val outOfWindow = outOfWindowThisLoad.get()
Log.d(TAG) {
"[giftwrap] load summary: $total event(s), $outOfWindow before floor " +
"(since=$loadSince, ${daysAgo(loadSince)}d back)"
}
}
wasLoading = loading
}
}
}
override fun updateFilter(
key: AccountQueryState,
since: SincePerRelayMap?,
@@ -155,46 +67,19 @@ class AccountGiftWrapsEoseManager(
}
val relays = key.account.dmRelays.flow.value
windowLoad.setExpectedRelays(relays.toSet())
val windowSince = windowFor(user(key)).since
Log.d(TAG) { "[giftwrap] REQ since=$windowSince (${daysAgo(windowSince)}d) on ${relays.size} relay(s)" }
val sinceTime = TimeUtils.now() - LIVE_TAIL_SECONDS
Log.d(TAG) { "[giftwrap.live] REQ since=$sinceTime (7d, no until) on ${relays.size} relay(s)" }
return relays.flatMap { relay ->
filterGiftWrapsToPubkey(relay = relay, pubkey = user(key).pubkeyHex, since = windowSince)
filterGiftWrapsToPubkey(relay = relay, pubkey = user(key).pubkeyHex, since = sinceTime)
}
}
/** Widens the window one (geometric) step back and re-issues the subscription. No-op if exhausted. */
fun loadMore(user: User) {
val window = windowFor(user)
if (window.isExhausted()) {
Log.d(TAG) { "[giftwrap] loadMore ignored — already exhausted" }
return
}
val before = window.since
window.loadMore()
_exhausted.value = window.isExhausted()
Log.d(TAG) { "[giftwrap] loadMore ${daysAgo(before)}d -> ${daysAgo(window.since)}d back (exhausted=${_exhausted.value})" }
scope?.let { beginWindowLoad(user, it) }
invalidateFilters()
}
/** Jumps the window to the maximum lookback so a single REQ pulls the entire history. */
fun loadEverything(user: User) {
val window = windowFor(user)
if (window.isExhausted()) return
window.loadAll()
_exhausted.value = true
Log.d(TAG) { "[giftwrap] loadEverything — full history (${daysAgo(window.since)}d back)" }
scope?.let { beginWindowLoad(user, it) }
invalidateFilters()
}
private val userJobMap = mutableMapOf<User, List<Job>>()
@OptIn(FlowPreview::class)
override fun newSub(key: AccountQueryState): Subscription {
val user = user(key)
scope = key.account.scope
beginWindowLoad(user, key.account.scope)
windowLoad.startLoading(key.account.scope)
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
listOf(
@@ -205,9 +90,7 @@ class AccountGiftWrapsEoseManager(
)
return requestNewSubscription(
windowLoad.trackingListener(
onEachEvent = { event -> countEvent(event.createdAt) },
) { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
@@ -222,8 +105,7 @@ class AccountGiftWrapsEoseManager(
companion object {
private const val TAG = "DMPagination"
// The window doubles each widen so a sparse history (or confirming nothing older exists)
// reaches the 10-year backstop in ~10 requests instead of crawling a week at a time.
private const val WINDOW_GROWTH_FACTOR = 2L
// The live tail's fixed lower bound: one week of recent history, always open at the top.
val LIVE_TAIL_SECONDS = 7L * TimeUtils.ONE_DAY
}
}
@@ -0,0 +1,233 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps
import com.vitorpamplona.amethyst.commons.relayClient.nip17Dm.filterGiftWrapsToPubkey
import com.vitorpamplona.amethyst.commons.relayClient.pagination.TimeWindowPagination
import com.vitorpamplona.amethyst.model.User
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.WindowLoadTracker
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.trackingListener
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.AccountQueryState
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicInteger
/**
* Loads the account's NIP-17 gift-wrap **history** — everything older than the one-week live tail
* ([AccountGiftWrapsEoseManager]) — in bounded, one-shot `[since, until]` slices.
*
* It is idle until the screens call [loadMore]: each call advances the floor one geometric step and
* requests **only the new band** `[newFloor, previousFloor]`, never the whole window. Consecutive
* slices are disjoint, so re-issuing the (advanced) filter on a reconnect does not re-stream older
* slices — those already live in the cache. The 2-day NIP-17 margin (applied to the slice `since` by
* [filterGiftWrapsToPubkey]) overlaps adjacent slices so a randomized outer timestamp can't open a gap.
*/
class AccountGiftWrapsHistoryEoseManager(
client: INostrClient,
allKeys: () -> Set<AccountQueryState>,
) : PerUserEoseManager<AccountQueryState>(client, allKeys) {
override fun user(key: AccountQueryState) = key.account.userProfile()
// The window's [TimeWindowPagination.since] is the oldest floor reached so far; it starts at the
// live-tail floor (now 1 week) and moves back one geometric step per loadMore.
private val windows = ConcurrentHashMap<HexKey, TimeWindowPagination>()
// The current slice to request per user, or absent until the first loadMore (manager stays idle).
private val slices = ConcurrentHashMap<HexKey, Slice>()
private data class Slice(
val since: Long,
val until: Long,
)
private fun windowFor(user: User) = windows.computeIfAbsent(user.pubkeyHex) { TimeWindowPagination(growthFactor = WINDOW_GROWTH_FACTOR) }
/** The current history slice for [user] (logical, un-margined bounds), or null while idle. */
fun currentSlice(user: User): Pair<Long, Long>? = slices[user.pubkeyHex]?.let { it.since to it.until }
private val windowLoad = WindowLoadTracker("giftwrap.history")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// True once the floor reached the maximum lookback: nothing older to fetch.
private val _exhausted = MutableStateFlow(false)
val exhausted: StateFlow<Boolean> = _exhausted.asStateFlow()
// Rooms-list auto-fill stall mark: the number of distinct private rooms shown the last time the
// list auto-widened. The list stops widening once a step adds no new room (widening only pulls
// older MESSAGES, which for a few busy correspondents can be thousands of events without a single
// new room). Kept here, beside the window it guards, so the stall survives leaving and reopening
// the Messages screen — a fresh UI-local counter would re-widen on every open.
@Volatile
var autoFillPrivateRoomMark: Int = Int.MIN_VALUE
// Account scope for the window-load watchdog. Volatile: written on IO (newSub), read on UI (loadMore).
@Volatile
private var scope: CoroutineScope? = null
override fun updateFilter(
key: AccountQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter> {
val slice = slices[user(key).pubkeyHex]
if (!key.account.isWriteable() || slice == null) {
windowLoad.setExpectedRelays(emptySet())
return emptyList()
}
val relays = key.account.dmRelays.flow.value
windowLoad.setExpectedRelays(relays.toSet())
Log.d(TAG) { "[giftwrap.history] REQ slice [${daysAgo(slice.since)}d, ${daysAgo(slice.until)}d] on ${relays.size} relay(s)" }
return relays.flatMap { relay ->
filterGiftWrapsToPubkey(relay = relay, pubkey = user(key).pubkeyHex, since = slice.since, until = slice.until)
}
}
private fun daysAgo(epochSeconds: Long) = (TimeUtils.now() - epochSeconds) / TimeUtils.ONE_DAY
/** Widens the floor one geometric step back and requests only the new slice. No-op if exhausted. */
fun loadMore(user: User) {
val window = windowFor(user)
if (window.isExhausted()) {
Log.d(TAG) { "[giftwrap.history] loadMore ignored — already exhausted" }
return
}
val until = window.since
window.loadMore()
slices[user.pubkeyHex] = Slice(since = window.since, until = until)
_exhausted.value = window.isExhausted()
Log.d(TAG) { "[giftwrap.history] loadMore slice [${daysAgo(window.since)}d, ${daysAgo(until)}d] (exhausted=${_exhausted.value})" }
scope?.let { beginWindowLoad(user, it) }
invalidateFilters()
}
/** Requests the entire remaining past in one slice `[maxLookback, currentFloor]`. */
fun loadEverything(user: User) {
val window = windowFor(user)
if (window.isExhausted()) return
val until = window.since
window.loadAll()
slices[user.pubkeyHex] = Slice(since = window.since, until = until)
_exhausted.value = true
Log.d(TAG) { "[giftwrap.history] loadEverything — slice [${daysAgo(window.since)}d, ${daysAgo(until)}d]" }
scope?.let { beginWindowLoad(user, it) }
invalidateFilters()
}
// Per-load instrumentation: how many gift wraps a slice pulled, and how many fell outside the band
// the REQ actually asked for ([loadSince], the margined floor) — a non-zero out-of-band share means
// a relay ignored the bounds. Volatile/atomic: the event hook runs on relay IO threads while the
// summary collector reads on the account scope.
@Volatile
private var loadSince = 0L
private val eventsThisLoad = AtomicInteger(0)
private val outOfWindowThisLoad = AtomicInteger(0)
@Volatile
private var summaryJob: Job? = null
private fun countEvent(createdAt: Long) {
eventsThisLoad.incrementAndGet()
if (createdAt < loadSince) outOfWindowThisLoad.incrementAndGet()
}
private fun beginWindowLoad(
user: User,
scope: CoroutineScope,
) {
loadSince = (slices[user.pubkeyHex]?.since ?: windowFor(user).since) - TimeUtils.twoDays()
eventsThisLoad.set(0)
outOfWindowThisLoad.set(0)
ensureSummaryLogger(scope)
windowLoad.startLoading(scope)
}
private fun ensureSummaryLogger(scope: CoroutineScope) {
if (summaryJob?.isActive == true) return
summaryJob =
scope.launch {
var wasLoading = false
windowLoad.loading.collect { loading ->
if (!loading && wasLoading) {
val total = eventsThisLoad.get()
val outOfWindow = outOfWindowThisLoad.get()
Log.d(TAG) {
"[giftwrap.history] load summary: $total event(s), $outOfWindow before floor " +
"(since=$loadSince, ${daysAgo(loadSince)}d back)"
}
}
wasLoading = loading
}
}
}
private val userJobMap = mutableMapOf<User, List<Job>>()
@OptIn(FlowPreview::class)
override fun newSub(key: AccountQueryState): Subscription {
val user = user(key)
scope = key.account.scope
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
listOf(
key.account.scope.launch(Dispatchers.IO) {
key.account.dmRelays.flow
.collectLatest { invalidateFilters() }
},
)
return requestNewSubscription(
windowLoad.trackingListener(
onEachEvent = { event -> countEvent(event.createdAt) },
) { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
override fun endSub(
key: User,
subId: String,
) {
super.endSub(key, subId)
userJobMap[key]?.forEach { it.cancel() }
}
companion object {
private const val TAG = "DMPagination"
// The slice doubles each widen so a sparse history (or confirming nothing older exists)
// reaches the 10-year backstop in ~10 requests instead of crawling a week at a time.
private const val WINDOW_GROWTH_FACTOR = 2L
}
}
@@ -155,10 +155,10 @@ private fun LoadOlderMessagesWhenScrolling(
listState: LazyListState,
accountViewModel: AccountViewModel,
) {
val giftWraps = remember(accountViewModel) { accountViewModel.dataSources().account.giftWraps }
val nip04 = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04 }
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04History }
LaunchedEffect(listState, giftWraps, nip04) {
LaunchedEffect(listState, giftWrapsHistory, nip04History) {
combine(
snapshotFlow {
val info = listState.layoutInfo
@@ -167,17 +167,17 @@ private fun LoadOlderMessagesWhenScrolling(
val overflowsScreen = info.visibleItemsInfo.size < total
overflowsScreen && lastVisible >= total - PREFETCH_OLDER_MESSAGES
},
giftWraps.loadingMore,
nip04.loadingMore,
giftWraps.exhausted,
giftWrapsHistory.loadingMore,
nip04History.loadingMore,
giftWrapsHistory.exhausted,
) { wantMore, loadingGiftWraps, loadingNip04, exhausted ->
wantMore && !loadingGiftWraps && !loadingNip04 && !exhausted
}.distinctUntilChanged()
.filter { it }
.collect {
Log.d("DMPagination") { "convo: widen (scrolled near oldest) → loadMore + reload" }
giftWraps.loadMore(accountViewModel.userProfile())
nip04.reload()
giftWrapsHistory.loadMore(accountViewModel.userProfile())
nip04History.reload()
}
}
}
@@ -198,11 +198,11 @@ fun ChatroomViewUI(
onDispose { Log.d("DMPagination") { "convo: CLOSE room=${room.hashCode()}" } }
}
val giftWraps = remember(accountViewModel) { accountViewModel.dataSources().account.giftWraps }
val nip04 = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04 }
val loadingGiftWraps by giftWraps.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04.loadingMore.collectAsStateWithLifecycle()
val historyExhausted by giftWraps.exhausted.collectAsStateWithLifecycle()
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04History }
val loadingGiftWraps by giftWrapsHistory.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04History.loadingMore.collectAsStateWithLifecycle()
val historyExhausted by giftWrapsHistory.exhausted.collectAsStateWithLifecycle()
Column(Modifier.fillMaxHeight()) {
ObserveRelayListForDMsAndDisplayIfNotFound(accountViewModel, nav)
@@ -235,8 +235,8 @@ fun ChatroomViewUI(
) {
Log.d("DMPagination") { "convo: Load entire history tapped" }
val user = accountViewModel.userProfile()
giftWraps.loadEverything(user)
nip04.reload()
giftWrapsHistory.loadEverything(user)
nip04History.reload()
}
}
},
@@ -22,7 +22,7 @@ package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource
import com.vitorpamplona.amethyst.commons.relayClient.composeSubscriptionManagers.ComposeSubscriptionManager
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKey
@@ -36,13 +36,18 @@ class ChatroomQueryState(
class ChatroomFilterAssembler(
client: INostrClient,
giftWraps: AccountGiftWrapsEoseManager,
giftWrapsHistory: AccountGiftWrapsHistoryEoseManager,
) : ComposeSubscriptionManager<ChatroomQueryState>() {
val nip04 = ChatroomNip04SubAssembler(client, ::allKeys, giftWraps)
// NIP-04 live tail: the recent week, always open at the top.
val nip04 = ChatroomNip04SubAssembler(client, ::allKeys)
// NIP-04 history: older DMs, following the gift-wrap history's bounded slice.
val nip04History = ChatroomNip04HistorySubAssembler(client, ::allKeys, giftWrapsHistory)
val group =
listOf(
nip04,
nip04History,
)
override fun invalidateKeys() = invalidateFilters()
@@ -0,0 +1,87 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserAndFollowListEoseManager
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.WindowLoadTracker
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.trackingListener
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.flow.StateFlow
/**
* Loads older NIP-04 DMs (kind 4) for one conversation, following the gift-wrap history's current
* bounded slice ([AccountGiftWrapsHistoryEoseManager.currentSlice]) so a thread shows both DM
* protocols to the same depth. NIP-04 timestamps are exact, so the slice needs no margin. [reload]
* re-issues at the now-advanced slice; idle until the first slice is opened.
*/
class ChatroomNip04HistorySubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomQueryState>,
private val giftWrapsHistory: AccountGiftWrapsHistoryEoseManager,
) : PerUserAndFollowListEoseManager<ChatroomQueryState, String>(client, allKeys) {
private val windowLoad = WindowLoadTracker("convo.nip04.history")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// Account scope for the watchdog. Volatile: written on IO (newSub), read on UI (reload).
@Volatile
private var scope: CoroutineScope? = null
override fun updateFilter(
key: ChatroomQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? {
val slice = giftWrapsHistory.currentSlice(user(key))
if (!key.account.isWriteable() || slice == null) {
windowLoad.setExpectedRelays(emptySet())
return emptyList()
}
val (sliceSince, sliceUntil) = slice
val filters = filterNip04DMs(key.room.users, key.account, sliceSince, sliceUntil)
windowLoad.setExpectedRelays(filters?.mapTo(mutableSetOf()) { it.relay } ?: emptySet())
Log.d("DMPagination") { "[convo.nip04.history] REQ slice since=$sliceSince until=$sliceUntil on ${filters?.size ?: 0} relay-filter(s)" }
return filters
}
/** Re-issues at the gift-wrap history's now-advanced slice and tracks the load. */
fun reload() {
Log.d("DMPagination") { "[convo.nip04.history] reload" }
scope?.let { windowLoad.startLoading(it) }
invalidateFilters()
}
override fun user(key: ChatroomQueryState) = key.account.userProfile()
override fun list(key: ChatroomQueryState) = key.listId
override fun newSub(key: ChatroomQueryState): Subscription {
scope = key.account.scope
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
}
@@ -30,57 +30,40 @@ import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.flow.StateFlow
/**
* Loads one conversation's NIP-04 DMs (kind 4). Like the rooms-list loader, it follows the account
* gift-wrap window's [AccountGiftWrapsEoseManager.windowSince] floor so a thread shows both DM
* protocols to the same depth. [reload] re-issues at the current floor; [loadingMore] reports when
* this protocol has covered it, which the conversation screen joins with the gift-wrap loader's flag
* to decide how deep the thread is safe to reveal.
* Always-on **live tail** for one conversation's NIP-04 DMs (kind 4). A fixed one-week floor, no
* upper bound, never widens — older history is loaded by [ChatroomNip04HistorySubAssembler] in
* bounded slices that follow the gift-wrap history window.
*/
class ChatroomNip04SubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomQueryState>,
private val giftWraps: AccountGiftWrapsEoseManager,
) : PerUserAndFollowListEoseManager<ChatroomQueryState, String>(client, allKeys) {
private val windowLoad = WindowLoadTracker("convo.nip04")
private val windowLoad = WindowLoadTracker("convo.nip04.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// Account scope for the watchdog. Volatile: written on IO (newSub), read on UI (reload).
@Volatile
private var scope: CoroutineScope? = null
override fun updateFilter(
key: ChatroomQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? =
if (key.account.isWriteable()) {
val windowSince = giftWraps.windowSince(user(key))
val filters = filterNip04DMs(key.room.users, key.account, windowSince)
val sinceTime = TimeUtils.now() - AccountGiftWrapsEoseManager.LIVE_TAIL_SECONDS
val filters = filterNip04DMs(key.room.users, key.account, sinceTime)
windowLoad.setExpectedRelays(filters?.mapTo(mutableSetOf()) { it.relay } ?: emptySet())
val daysAgo = (TimeUtils.now() - windowSince) / TimeUtils.ONE_DAY
Log.d("DMPagination") { "[convo.nip04] REQ since=$windowSince (${daysAgo}d) on ${filters?.size ?: 0} relay-filter(s)" }
Log.d("DMPagination") { "[convo.nip04.live] REQ since=$sinceTime (7d, no until) on ${filters?.size ?: 0} relay-filter(s)" }
filters
} else {
windowLoad.setExpectedRelays(emptySet())
emptyList()
}
/** Re-issues at the (now-wider) shared gift-wrap floor and tracks the load. */
fun reload() {
Log.d("DMPagination") { "[convo.nip04] reload" }
scope?.let { windowLoad.startLoading(it) }
invalidateFilters()
}
override fun user(key: ChatroomQueryState) = key.account.userProfile()
override fun list(key: ChatroomQueryState) = key.listId
override fun newSub(key: ChatroomQueryState): Subscription {
scope = key.account.scope
windowLoad.startLoading(key.account.scope)
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
@@ -33,6 +33,7 @@ fun filterNip04DMs(
group: Set<HexKey>?,
account: Account?,
windowStart: Long,
windowEnd: Long? = null,
): List<RelayBasedFilter>? {
if (group.isNullOrEmpty() || account == null) return null
@@ -75,6 +76,7 @@ fun filterNip04DMs(
authors = group.toList(),
tags = mapOf("p" to listOf(account.userProfile().pubkeyHex)),
since = windowStart,
until = windowEnd,
),
)
} +
@@ -87,6 +89,7 @@ fun filterNip04DMs(
authors = listOf(account.userProfile().pubkeyHex),
tags = mapOf("p" to group.toList()),
since = windowStart,
until = windowEnd,
),
)
}
@@ -23,7 +23,7 @@ package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.rooms.datasource
import androidx.compose.runtime.Stable
import com.vitorpamplona.amethyst.commons.relayClient.composeSubscriptionManagers.ComposeSubscriptionManager
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
// This allows multiple screen to be listening to tags, even the same tag
@@ -35,14 +35,18 @@ class ChatroomListState(
@Stable
class ChatroomListFilterAssembler(
client: INostrClient,
giftWraps: AccountGiftWrapsEoseManager,
giftWrapsHistory: AccountGiftWrapsHistoryEoseManager,
) : ComposeSubscriptionManager<ChatroomListState>() {
// NIP-04 DMs follow the account gift-wrap window's floor (the single source of truth).
val nip04 = ChatroomListNip04SubAssembler(client, ::allKeys, giftWraps)
// NIP-04 live tail: the recent week, always open at the top.
val nip04 = ChatroomListNip04SubAssembler(client, ::allKeys)
// NIP-04 history: older DMs, following the gift-wrap history's bounded slice.
val nip04History = ChatroomListNip04HistorySubAssembler(client, ::allKeys, giftWrapsHistory)
val group =
listOf(
nip04,
nip04History,
FollowingPublicChatSubAssembler(client, ::allKeys),
FollowingEphemeralChatSubAssembler(client, ::allKeys),
)
@@ -0,0 +1,118 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.rooms.datasource
import com.vitorpamplona.amethyst.model.User
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.WindowLoadTracker
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.trackingListener
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
/**
* Loads older NIP-04 DMs (kind 4) for the rooms list, following the gift-wrap history's current
* bounded slice ([AccountGiftWrapsHistoryEoseManager.currentSlice]) so both DM protocols page the
* past to the same depth. NIP-04 timestamps are exact, so the slice needs no 2-day margin. [reload]
* re-issues at the (now-advanced) slice and tracks the load; idle until the first slice is opened.
*/
class ChatroomListNip04HistorySubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomListState>,
private val giftWrapsHistory: AccountGiftWrapsHistoryEoseManager,
) : PerUserEoseManager<ChatroomListState>(client, allKeys) {
private val windowLoad = WindowLoadTracker("rooms.nip04.history")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// Account scope for the watchdog. Volatile: written on IO (newSub), read on UI (reload).
@Volatile
private var scope: CoroutineScope? = null
override fun updateFilter(
key: ChatroomListState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? {
val slice = giftWrapsHistory.currentSlice(user(key))
if (!key.account.isWriteable() || slice == null) {
windowLoad.setExpectedRelays(emptySet())
return emptyList()
}
val (sliceSince, sliceUntil) = slice
val homeRelays = key.account.homeRelays.flow.value
val dmRelays = key.account.dmRelays.flow.value
windowLoad.setExpectedRelays((homeRelays + dmRelays).toSet())
Log.d("DMPagination") { "[rooms.nip04.history] REQ slice since=$sliceSince until=$sliceUntil on ${homeRelays.size + dmRelays.size} relay(s)" }
return homeRelays.map { filterNip04DMsFromMe(key.account.userProfile(), it, sliceSince, sliceUntil) } +
dmRelays.map { filterNip04DMsToMe(key.account.userProfile(), it, sliceSince, sliceUntil) }
}
/** Re-issues at the gift-wrap history's now-advanced slice and tracks the load. */
fun reload() {
Log.d("DMPagination") { "[rooms.nip04.history] reload" }
scope?.let { windowLoad.startLoading(it) }
invalidateFilters()
}
override fun user(key: ChatroomListState) = key.account.userProfile()
private val userJobMap = mutableMapOf<User, List<Job>>()
@OptIn(FlowPreview::class)
override fun newSub(key: ChatroomListState): Subscription {
val user = user(key)
scope = key.account.scope
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
listOf(
key.account.scope.launch(Dispatchers.IO) {
key.account.homeRelays.flow
.collectLatest { invalidateFilters() }
},
key.account.scope.launch(Dispatchers.IO) {
key.account.dmRelays.flow
.collectLatest { invalidateFilters() }
},
)
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
override fun endSub(
key: User,
subId: String,
) {
super.endSub(key, subId)
userJobMap[key]?.forEach { it.cancel() }
}
}
@@ -31,7 +31,6 @@ import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
@@ -40,23 +39,17 @@ import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
/**
* Loads the account's NIP-04 DMs (kind 4) for the rooms list. It does not own a time window: it
* follows the gift-wrap window's [AccountGiftWrapsEoseManager.windowSince] floor, so both DM
* protocols are requested to the same depth. [reload] re-issues at the current floor (called after
* the gift-wrap window widens); [loadingMore] reports when this protocol has covered that floor.
* Always-on **live tail** for the account's NIP-04 DMs (kind 4) in the rooms list. Mirrors the
* gift-wrap live tail: a fixed one-week floor, no upper bound, never widens. Older NIP-04 history is
* loaded by [ChatroomListNip04HistorySubAssembler] in bounded slices.
*/
class ChatroomListNip04SubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomListState>,
private val giftWraps: AccountGiftWrapsEoseManager,
) : PerUserEoseManager<ChatroomListState>(client, allKeys) {
private val windowLoad = WindowLoadTracker("rooms.nip04")
private val windowLoad = WindowLoadTracker("rooms.nip04.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
// Account scope for the watchdog. Volatile: written on IO (newSub), read on UI (reload).
@Volatile
private var scope: CoroutineScope? = null
override fun updateFilter(
key: ChatroomListState,
since: SincePerRelayMap?,
@@ -64,25 +57,16 @@ class ChatroomListNip04SubAssembler(
if (key.account.isWriteable()) {
val homeRelays = key.account.homeRelays.flow.value
val dmRelays = key.account.dmRelays.flow.value
val relays = (homeRelays + dmRelays).toSet()
windowLoad.setExpectedRelays(relays)
val windowSince = giftWraps.windowSince(user(key))
val daysAgo = (TimeUtils.now() - windowSince) / TimeUtils.ONE_DAY
Log.d("DMPagination") { "[rooms.nip04] REQ since=$windowSince (${daysAgo}d) on ${relays.size} relay(s)" }
homeRelays.map { filterNip04DMsFromMe(key.account.userProfile(), it, windowSince) } +
dmRelays.map { filterNip04DMsToMe(key.account.userProfile(), it, windowSince) }
windowLoad.setExpectedRelays((homeRelays + dmRelays).toSet())
val sinceTime = TimeUtils.now() - AccountGiftWrapsEoseManager.LIVE_TAIL_SECONDS
Log.d("DMPagination") { "[rooms.nip04.live] REQ since=$sinceTime (7d, no until) on ${homeRelays.size + dmRelays.size} relay(s)" }
homeRelays.map { filterNip04DMsFromMe(key.account.userProfile(), it, sinceTime) } +
dmRelays.map { filterNip04DMsToMe(key.account.userProfile(), it, sinceTime) }
} else {
windowLoad.setExpectedRelays(emptySet())
emptyList()
}
/** Re-issues at the (now-wider) shared gift-wrap floor and tracks the load. */
fun reload() {
Log.d("DMPagination") { "[rooms.nip04] reload" }
scope?.let { windowLoad.startLoading(it) }
invalidateFilters()
}
override fun user(key: ChatroomListState) = key.account.userProfile()
private val userJobMap = mutableMapOf<User, List<Job>>()
@@ -90,7 +74,6 @@ class ChatroomListNip04SubAssembler(
@OptIn(FlowPreview::class)
override fun newSub(key: ChatroomListState): Subscription {
val user = user(key)
scope = key.account.scope
windowLoad.startLoading(key.account.scope)
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
@@ -30,6 +30,7 @@ fun filterNip04DMsFromMe(
user: User,
relay: NormalizedRelayUrl,
since: Long?,
until: Long? = null,
): RelayBasedFilter =
RelayBasedFilter(
relay = relay,
@@ -38,5 +39,6 @@ fun filterNip04DMsFromMe(
kinds = listOf(PrivateDmEvent.KIND),
authors = listOf(user.pubkeyHex),
since = since,
until = until,
),
)
@@ -30,6 +30,7 @@ fun filterNip04DMsToMe(
user: User,
relay: NormalizedRelayUrl,
since: Long?,
until: Long? = null,
): RelayBasedFilter =
RelayBasedFilter(
relay = relay,
@@ -38,5 +39,6 @@ fun filterNip04DMsToMe(
kinds = listOf(PrivateDmEvent.KIND),
tags = mapOf("p" to listOf(user.pubkeyHex)),
since = since,
until = until,
),
)
@@ -61,7 +61,6 @@ import com.vitorpamplona.quartz.nip28PublicChat.message.ChannelMessageEvent
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.filter
import java.io.Serializable
@Composable
@@ -91,11 +90,11 @@ private fun CrossFadeState(
) {
val feedState by feedContentState.feedContent.collectAsStateWithLifecycle()
// The gift-wrap window is the single DM window (NIP-04 follows it), so once it reaches the
// maximum lookback the rooms list has pulled everything there is. Until then an empty feed means
// "still filling", not "no conversations" — keep the spinner up rather than flash empty.
val giftWraps = remember(accountViewModel) { accountViewModel.dataSources().account.giftWraps }
val historyExhausted by giftWraps.exhausted.collectAsStateWithLifecycle()
// The gift-wrap history window is the DM history window (NIP-04 history follows it), so once it
// reaches the maximum lookback the rooms list has pulled everything there is. Until then an empty
// feed means "still filling", not "no conversations" — keep the spinner up rather than flash empty.
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val historyExhausted by giftWrapsHistory.exhausted.collectAsStateWithLifecycle()
// While the whole list is empty there is no LazyColumn to scroll, so keep widening the private
// DM window here until rooms appear or it is exhausted. (Public / ephemeral / group rooms are
@@ -142,18 +141,24 @@ private fun FeedLoaded(
val myPubKey = accountViewModel.userProfile().pubkeyHex
val giftWraps = remember(accountViewModel) { accountViewModel.dataSources().account.giftWraps }
val nip04 = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04 }
val loadingGiftWraps by giftWraps.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04.loadingMore.collectAsStateWithLifecycle()
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04History }
val loadingGiftWraps by giftWrapsHistory.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04History.loadingMore.collectAsStateWithLifecycle()
val loadingMore = loadingGiftWraps || loadingNip04
val historyExhausted by giftWraps.exhausted.collectAsStateWithLifecycle()
val historyExhausted by giftWrapsHistory.exhausted.collectAsStateWithLifecycle()
// Widen the private DM window only as the user approaches the oldest LOADED private chat —
// ignoring public / group / ephemeral rooms below it. Those are membership-based and can be
// arbitrarily old; counting them would either stall private paging or drag the window back
// years. The lambda reads the live items + scroll state inside the snapshotFlow.
WidenPrivateWindowWhen(accountViewModel, "scroll") {
// years. [privateRoomCount] feeds the stall-gate: widening keeps pulling older MESSAGES, which for
// a few busy correspondents floods events without adding a row — so paging stops once a step
// brings in no new private room. The lambda reads the live items + scroll state in the snapshotFlow.
WidenPrivateWindowWhen(
accountViewModel,
"scroll",
privateRoomCount = { items.list.count { it.event is ChatroomKeyable } },
) {
val info = listState.layoutInfo
val total = info.totalItemsCount
val lastVisible = info.visibleItemsInfo.lastOrNull()?.index ?: -1
@@ -188,8 +193,8 @@ private fun FeedLoaded(
if (index == privateBoundaryIndex && (loadingMore || !historyExhausted)) {
DmLoadMoreIndicator(loadingMore, showLoadAll = !historyExhausted) {
val user = accountViewModel.userProfile()
giftWraps.loadEverything(user)
nip04.reload()
giftWrapsHistory.loadEverything(user)
nip04History.reload()
}
}
}
@@ -199,8 +204,8 @@ private fun FeedLoaded(
item(key = "loadingMoreFooter") {
DmLoadMoreIndicator(loadingMore, showLoadAll = !historyExhausted) {
val user = accountViewModel.userProfile()
giftWraps.loadEverything(user)
nip04.reload()
giftWrapsHistory.loadEverything(user)
nip04History.reload()
}
}
}
@@ -230,29 +235,48 @@ private const val PREFETCH_PRIVATE_CHATS = 5
private fun WidenPrivateWindowWhen(
accountViewModel: AccountViewModel,
trigger: String,
// Number of distinct private rooms currently loaded, or null for callers that should keep widening
// regardless (the empty feed, still hunting for the first room). When provided, the loop stops
// advancing once a widen brings in no new private room — widening only adds older MESSAGES, so a
// few correspondents' history can flood thousands of events without adding a row, and "fill the
// screen" must stop at "no new people" instead of walking the window to the 10-year backstop. The
// mark lives on the history manager so it survives leaving/reopening the screen.
privateRoomCount: (() -> Int)? = null,
wantMore: () -> Boolean,
) {
val giftWraps = remember(accountViewModel) { accountViewModel.dataSources().account.giftWraps }
val nip04 = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04 }
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04History }
LaunchedEffect(giftWraps, nip04) {
LaunchedEffect(giftWrapsHistory, nip04History) {
combine(
snapshotFlow { wantMore() },
giftWraps.loadingMore,
nip04.loadingMore,
giftWraps.exhausted,
) { want, loadingGiftWraps, loadingNip04, exhausted ->
want && !loadingGiftWraps && !loadingNip04 && !exhausted
// Carries the private-room count (>= 0) while a widen is wanted, or NOT_WANTED otherwise.
snapshotFlow { if (wantMore()) (privateRoomCount?.invoke() ?: STILL_SEARCHING) else NOT_WANTED },
giftWrapsHistory.loadingMore,
nip04History.loadingMore,
giftWrapsHistory.exhausted,
) { count, loadingGiftWraps, loadingNip04, exhausted ->
if (count != NOT_WANTED && !loadingGiftWraps && !loadingNip04 && !exhausted) count else NOT_WANTED
}.distinctUntilChanged()
.filter { it }
.collect {
Log.d("DMPagination") { "rooms.list: widen ($trigger) → loadMore + reload" }
giftWraps.loadMore(accountViewModel.userProfile())
nip04.reload()
.collect { count ->
if (count == NOT_WANTED) return@collect
// Stop once a widen adds no new private room (but keep hunting while none are loaded).
if (privateRoomCount != null && count > 0 && count <= giftWrapsHistory.autoFillPrivateRoomMark) {
Log.d("DMPagination") { "rooms.list: widen ($trigger) stop — no new private rooms (count=$count)" }
return@collect
}
if (privateRoomCount != null) giftWrapsHistory.autoFillPrivateRoomMark = count
Log.d("DMPagination") { "rooms.list: widen ($trigger) → loadMore + reload (privateRooms=$count)" }
giftWrapsHistory.loadMore(accountViewModel.userProfile())
nip04History.reload()
}
}
}
// Sentinels for the widen-trigger flow: NOT_WANTED suppresses widening; STILL_SEARCHING is the count a
// caller without a private-room measure reports while it wants to keep widening (e.g. the empty feed).
private const val NOT_WANTED = -1
private const val STILL_SEARCHING = 0
// Stable per-chatroom key — derived from chatroom identity, not the latest
// message id, so reorders move the row instead of recreating it. Compose
// stores LazyColumn item keys in a SaveableStateHolder, which on Android
@@ -32,6 +32,7 @@ fun filterGiftWrapsToPubkey(
relay: NormalizedRelayUrl,
pubkey: HexKey?,
since: Long?,
until: Long? = null,
): List<RelayBasedFilter> {
if (pubkey.isNullOrEmpty()) return emptyList()
@@ -42,7 +43,12 @@ fun filterGiftWrapsToPubkey(
Filter(
kinds = listOf(GiftWrapEvent.KIND, EphemeralGiftWrapEvent.KIND),
tags = mapOf("p" to listOf(pubkey)),
// A gift wrap's outer created_at is randomized up to 2 days before the real
// message time, so widen the lower bound by 2 days to catch wraps for messages
// right at the floor. (The upper bound needs no margin: a slice's `until` is the
// previous slice's un-margined floor, so the 2-day overlap already covers the seam.)
since = since?.minus(TimeUtils.twoDays()),
until = until,
),
),
)