Merge pull request #3151 from vitorpamplona/claude/relay-message-pagination-LMqSQ

DM history: per-relay backward paging, live-tail split + prune-aware window realignment
This commit is contained in:
Vitor Pamplona
2026-06-09 15:05:58 -04:00
committed by GitHub
53 changed files with 4629 additions and 99 deletions
@@ -0,0 +1,332 @@
# DM loading: live tail + per-relay history paging
> **Status:** authoritative as of 2026-06-05. The "Current architecture"
> section below describes the code as it actually stands. The original
> time-slice design and the round-model history are kept at the bottom under
> **Design evolution (historical)** — they are superseded and no longer match
> the code; don't trust them for how it works today.
## Problem
The DM loaders used a single subscription whose `since` floor grew as the user
scrolled (`loadMore`: 7d → 14d → 28d → …). The filter carried **only `since`,
no `until`**, so every widen re-requested the whole window and the relay
re-streamed the entire history from the new floor. Traces showed this directly:
```
[giftwrap] load summary: 589 event(s) (14d)
[giftwrap] load summary: 1486 event(s) (28d)
[giftwrap] load summary: 2609 event(s) (56d)
```
Each step re-downloaded everything it already had plus the new slice — the
"getting all events over and over again" the owner reported. It also cascaded:
a few pixels of scroll walked the window to the 10-year backstop, because
widening pulls older *messages* but the rooms list is keyed by *conversation*,
so a handful of busy correspondents flood thousands of events without adding a
single new row, and the "scrolled near the oldest room" trigger never clears.
---
## Current architecture
### Two layers per protocol
Each DM protocol — **NIP-17** gift wraps (kind 1059) and **NIP-04** legacy DMs
(kind 4) — is split into two independent responsibilities:
1. **Live tail** — a fixed ~1-week floor with **no `until`**, open to the
future. Never widens. New messages always arrive here. Backed by the
**round model** (`WindowLoadTracker`): one REQ fanned to every relay, "done"
when all settle, drives the boot spinner.
2. **History** — everything *older* than the week floor, paged **backward by
`until`+`limit`, per relay, on demand**. Backed by the **per-relay model**
(`RelayLoadingCursors` + `PerRelayLoadTracker`), driven by on-screen markers.
The two are disjoint in time, so re-issuing a history page never re-streams the
live tail, and consecutive history pages never re-stream each other.
| Surface | Live-tail manager (round) | History manager (per-relay) |
|---|---|---|
| Account gift wraps (NIP-17) | `AccountGiftWrapsEoseManager` | `AccountGiftWrapsHistoryEoseManager` |
| Conversation NIP-04 | `ChatroomNip04SubAssembler` | `ChatroomNip04HistorySubAssembler` |
| Rooms-list NIP-04 | `ChatroomListNip04SubAssembler` | `ChatroomListNip04HistorySubAssembler` |
Accessed from the UI via `accountViewModel.dataSources()` as
`.account.giftWrapsHistory`, `.chatroom.nip04History`,
`.chatroomList.nip04History`.
### The history paging primitive: `RelayLoadingCursors`
The time-window model can't tell "this relay is empty" from "this is a gap" — a
`since`/`until` slice that returns nothing might just be a quiet stretch above
older messages. Paging by `until`+`limit` removes that ambiguity: a relay
returns up to `limit` (**10000**) of its newest events older than the cursor,
**skipping gaps**, so an **empty page + EOSE is a gap-proof "nothing older"**.
Per relay, two cursors are kept deliberately decoupled:
- `requestedUntil` — the `until` the REQ carries. Moves **only** in `advance()`.
Leaving it untouched on EOSE is what makes paging demand-driven: a relay that
finished a page just **parks** at the same filter (no re-REQ) until advanced.
- `reachedUntil` — the oldest `created_at` actually delivered. Moves on EOSE.
The in-stream markers sit here; the next page starts at `reachedUntil 1`.
Stop signals: an empty page marks the relay **`done`**. A relay returning fewer
than `limit` is treated as its own cap, **not** exhaustion. A misbehaving relay
that returns events but none older than already reached (echoing its newest
events) is also treated as the bottom, so its marker can't re-request the same
window forever. Tested in `RelayLoadingCursorsTest.kt`.
### The two completion models (and where each lives)
**Round model — `WindowLoadTracker` (live tail only).** One REQ is fanned to all
relays; the window is "done" only when **every** expected relay reaches a
terminal signal (`settled ⊇ expected`), with backstops for stragglers (idle /
silence / connect-grace / absolute cap). `loading` starts **`true`**. This is a
*barrier*: nobody moves on until the cohort answers. It is the right shape for
the one-shot fixed-window backfill the live tail does.
> Note: the silence + connect-grace backstops are gated behind `tracksReqSends`,
> and **none of the three live-tail managers pass `tracksReqSends = true`**, so
> in current use only the settle / idle / cap paths ever fire. The REQ-aware
> machinery is dormant in production — see "Things to scrutinize".
**Per-relay model — `RelayLoadingCursors` + `PerRelayLoadTracker` (all history).**
Each relay advances to its next page the instant *it* EOSEs, independent of the
others; the subscription layer diffs per relay, so re-issuing only re-REQs the
relay whose cursor moved. `loading` starts **`false`** (a `true` start would
wedge the scroll loader's `!loading` gate on first open). Fast relays race to
the bottom in back-to-back pages; slow / auth-walled relays catch up at their
own pace and **none are abandoned** — a stalled relay keeps its subscription
open and resumes when re-advanced. This removes the round model's
slowest-relay coupling, which matters most on the conversation screen where the
fan-out includes correspondents' (often auth-walled, slow) relays.
`exhausted` (per history manager) flips true when **every relay is `done` OR
`stalled`** — "nothing more reachable right now". A merely *parked* relay (more
to load, just not advancing) keeps it false.
> All three history managers (`AccountGiftWrapsHistoryEoseManager`,
> `ChatroomNip04HistorySubAssembler`, `ChatroomListNip04HistorySubAssembler`)
> were structurally the same per-relay loader, so that bookkeeping is now a
> single reusable engine — **`BackwardRelayPager`** (keyless, single-active). It
> does **not** hold the cursors: the per-relay `RelayLoadingCursors` live on the
> scope's own domain object (a `Chatroom` per conversation, a `ChatroomList` per
> account), so they share the cached messages' lifetime and survive an account
> switch. The orchestrator owns only the transient bits — in-flight + silence
> tracking, the stalled set, and the display flows — and
> `bind(cursors, scope, relaysFor)`s to whichever scope is active. Each manager
> supplies its REQ-filter builder, a `relaysFor` lookup, and the subscription
> wiring (forwards relay callbacks via `onEvent`/`onEose`/`onClosed`/`onCannotConnect`,
> re-issues filters after `advance`/`advanceAll`). The earlier round-model history
> (and the rooms-list "stall-gate") was fully removed — see Design evolution.
### What drives `advance()`: on-screen markers, off viewport visibility
History paging is demand-driven by **per-relay window-limit markers** placed in
the message stream, not by a scroll-position trigger:
- **`RelayReachCursor`** — one per (protocol, relay): its `reachedUntil` depth,
its `RelayReachState` (`REACHING ↓` / `STALLED …` / `DONE ✓`), and the
`advance()` that pulls *that relay's* next page. Built in the feed views from
each history manager's `relayProgress` map (gift wraps + NIP-04 combined; a
protocol drops out of the list once `exhausted`).
- **`RelayReachSentinels`** — the load *driver*, **hoisted above the
`LazyColumn`** (via `ChatFeedView`'s `sentinels` slot). Each non-done limit
gets one stable effect (keyed by `protocol:url`) that watches `listState` and
fires `advance()` when its gap is among the **currently visible rows** AND
either it just scrolled into view OR its `reachedUntil` moved (a page landed —
keep paging while visible). Driving off **viewport visibility** instead of row
composition is deliberate: an earlier version placed the sentinel *inside* the
hosting row, so any feed reorder (a live DM, a slow relay dribbling a page)
tore the effect down and re-fired `advance()` on a static screen — re-arming
stalled relays into a silence-watchdog storm. (commit `0394ec2a`)
- **`RelayReachMarkers` / `RelayReachMarker`** — pure UI (via the
`markersInGap` slot): the "Relay sync: ✓ 8 · ↓ 1" divider at each relay's
reached depth. Can be re-placed on every reorder without triggering paging.
- **`BootstrapHistoryWhenEmpty`** — when the feed is genuinely `Empty` (the live
tail came back empty for a thread/list whose newest message is older than a
week) there are no rows to host markers, so this steps every relay one page at
a time (debounced 1200ms, gated per loader on `!loading && !exhausted`) until
messages appear and the markers take over, or the protocol exhausts.
### NIP-04 per-relay filter scoping (`Nip04DmRelayRouting`)
A conversation's NIP-04 filters previously named the whole participant set on
every relay, so a relay belonging to one correspondent was asked about all of
them, and the `from-me` leg (`authors:[me]`) was sent to correspondents' inbox
relays — which auth-walled relays reject outright ("all authors must be
authenticated"), stalling the load.
`Nip04DmRelayRouting` (in `FilterNip04DMs.kt`) is now two **per-relay key maps**
(`relay → which keys to name there`), built from the outbox model:
- **to me** (`#p:[me]`) — my inbox carries the whole group; each correspondent's
outbox carries only that correspondent.
- **from me** (`authors:[me]`) — my outbox carries the whole group; each
correspondent's inbox carries only that correspondent.
So a relay only ever sees the keys that actually own it. The **conversation**
history manager scopes its REQ to the armed relays' key sets this way; the
**rooms-list** and **gift-wrap** history managers query only the account's *own*
relays (home outbox `from-me` + DM inbox `to-me`, via `filterNip04DMsFromMe` /
`filterNip04DMsToMe` and `filterGiftWrapsToPubkey`), which is why their fan-out
stays fast and reachable.
### Status card terminal states (`DmHistoryLoadingCard`)
One card per protocol at its oldest-loaded boundary. While paging it shows the
protocol tag, "N relays" being asked, and the reach-back date; it is tappable
into a per-relay popup (`DmHistoryRelayDialog`) listing every relay with
`✓` done / `…` stalled / `↓` reaching and how far back each paged.
Because `exhausted` conflates `done` and `stalled`, the terminal state is split
on `stalledCount` so it can't overclaim (commit `813110cc`):
- **caught up** (every relay `done`, `stalledCount == 0`) → "All caught up",
lingers ~2.2s then collapses.
- **incomplete** (≥1 stalled) → "Some relays didn't respond · N unreachable",
error-coloured `…`, **stays put** (no collapse), tappable to see which.
### Reply placeholder (`LoadingReplyNote`)
A reply whose target message hasn't been paged in yet isn't *missing*, it's
older than the loaded window (and for gift wraps the rumor id isn't even
queryable — only the outer 1059 wrap is). Instead of the generic `BlankNote`
("post not found"), `LoadingReplyNote` actively walks the relevant protocol's
history backward (kicking `advanceAll` each time a page settles) until the
target decrypts (the surrounding `WatchNoteEvent` crossfades the real message in
and disposes this) or the protocol exhausts. Its terminal state mirrors the
card: "Couldn't find this message" + an honest subtitle ("N relays unreachable ·
tap to see which" when stalled, "Searched every relay · tap to see" when
genuinely done), tappable into the same per-relay popup. Wired via
`ChatMessageCompose.RenderReply``WatchNoteEvent(onBlank = …)`, with the pager
chosen by the parent event's protocol (`DmReplyProtocol.NIP17` / `NIP04`).
### Diagnostics
Everything logs under one tag, **`DMPagination`** (debug builds):
`DmRelayDiagnosticsLogger` folds the per-relay connection timeline (REQ sent,
connect/disconnect, CLOSED/NOTICE/OK-fail) into it; `DmRelayLog` prints the
"relays by source" breakdown (NIP-65 in/out, DM list, private storage, local)
per subscription so an unexpected relay can be traced to the list it leaks in
from; and each assembler logs its milestones (paging start, a relay reaching
the bottom / stalling with the reason, the "window settled" summary of
done-vs-still-trying, each marker fire).
### Related fix: Tor guard-sample self-heal
`TorService` gained a `noUsableGuards()` check that, on init, inspects Arti's
persisted `guards.json` and wipes the on-disk state if a non-empty guard set has
**zero** usable guards (all `disabled` / `unlisted`). This recovers the
long-standing "can't connect to Tor → relays permanently unreachable" wedge
(Arti disables guards past a 0.7 indeterminate-failure ratio, never re-enables
them, and can't replenish once the 60-slot sample is full). Orthogonal to
pagination, but it lived here because unreachable relays were part of the same
"DM history stuck / relays never answer" symptom this branch chased.
---
## Component map (vs `origin/main`)
**Paging primitives (quartz, `nip01Core/relay/client/paging/`, `commonMain`)**
the pure, protocol-level paging *state*; iOS-clean, reusable by any KMP target.
- `RelayLoadingCursors.kt` — per-relay `until`+`limit` cursor state + pinned floor;
held on the scope's domain object. *(+ `RelayLoadingCursorsTest` in amethyst; the
`until`+`limit` wire contract is covered by `UntilLimitPagingRelayTest` against
the quartz `jvmAndroidTest` geode relay)*
- `RelayPagingProgress.kt``(reachedUntil, done, stalled)` per relay.
**Paging orchestrators (commons, `relayClient/paging/`, `jvmAndroid`)** — the
StateFlow-backed, subscription-loading state holders. Moved out of amethyst **and**
out of quartz (per `commons/ARCHITECTURE.md`: the relay-subscription client +
`StateFlow` state holders live in commons) so desktop / CLI / any feed can reuse
them; `jvmAndroid` (uses `java.util.concurrent` + `@Synchronized`) → Android +
Desktop, not iOS.
- `BackwardRelayPager.kt` — the keyless single-active orchestrator the three
history managers `bind` to. *(+ `BackwardRelayPagerTest` state-machine in commons
`jvmTest`)*
- `PerRelayLoadTracker.kt` — per-relay in-flight tracker + silence watchdog.
- `WindowLoadTracker.kt` — round/barrier completion tracker (live tail). *(+ `WindowLoadTrackerIdleTest` in amethyst)*
**Diagnostics (amethyst)**`service/relayClient/eoseManagers/DmRelayLog.kt`,
`service/relayClient/diagnostics/DmRelayDiagnosticsLogger.kt` — the `DMPagination` logs.
**Managers / assemblers**
- `AccountGiftWrapsEoseManager.kt` (live tail) + `AccountGiftWrapsHistoryEoseManager.kt` (new, history).
- `ChatroomNip04SubAssembler.kt` (live tail) + `ChatroomNip04HistorySubAssembler.kt` (new, history).
- `ChatroomListNip04SubAssembler.kt` (live tail) + `ChatroomListNip04HistorySubAssembler.kt` (new, history).
- `FilterNip04DMs.kt` (per-relay `Nip04DmRelayRouting`, live + history builders), `FilterNip04DMsFromMe/ToMe.kt`, `FilterGiftWrapsToPubkey.kt``until`/`limit` added.
- `AccountFilterAssembler`, `ChatroomFilterAssembler`, `ChatroomListFilterAssembler` — wire the new managers.
**Shared UI (commons, `commons/ui/feeds/`)** — extracted from amethyst so Android +
Desktop (and any per-relay feed) render the same widgets; CMP `composeResources`
strings, no app-theme / `java.time` deps.
- `RelayReachMarker.kt``RelayReachCursor` + sentinels (the hoisted, visibility-driven
paging driver) + markers (pure UI) + `RelayReachMarker`/`RelayReachState`.
- `DmHistoryLoadingCard.kt` — the boundary status card + per-relay tap dialog +
`historySubtitle`/`incompleteSubtitle`. Takes a `formatReachDate: (epochSeconds) -> String`
so each platform supplies its locale date formatter.
**Android UI (`amethyst/ui/screen/loggedIn/chats/`)**
- `feed/LoadingReplyNote.kt` — history-walking reply placeholder (uses the shared subtitle helpers/dialog).
- `feed/HistoryDateFormat.kt``formatHistoryReachDate`, the Android locale formatter passed into the shared card.
- `feed/ChatFeedView.kt``markersInGap` + `sentinels` slots.
- `feed/ChatMessageCompose.kt` — reply `onBlank` wiring.
- `privateDM/ChatroomView.kt`, `rooms/feed/ChatroomListFeedView.kt` — assemble cards/markers/sentinels, `BootstrapHistoryWhenEmpty`.
- `res/values/strings.xml``chats_reply_*` (the card's `chats_history_*` now live in commons).
---
## Things to scrutinize (review notes)
1. **`exhausted` conflates `done` + `stalled`** at the manager level. The cards
now distinguish them via `stalledCount`, but other consumers (the scroll
`!loading` gates, `LoadingReplyNote`'s advance loop) treat stalled as
terminal. Intentional (don't hammer dead relays), but confirm it's desired.
2. **`PerRelayLoadTracker.lastActivityMs` is global, not per-relay** — once the
chatty relays finish, a legitimately-slow relay gets the full 15s silence
window and can be marked stalled mid-delivery of a 10000-event page.
3. **"All caught up" can still be technically-true-but-misleading** when
`stalledCount == 0` yet a chat's messages live on a relay *not in the
account's NIP-17 inbox list* — an outbox-coverage gap the card can't detect.
4. **`WindowLoadTracker`'s REQ-aware backstops are dormant** in production
(no live-tail manager sets `tracksReqSends`). Either the live tail should
adopt them or the round model could be slimmer for its current role.
5. **`PAGE_LIMIT = 10000`** caps per-request volume but a single page can still
be a large payload on a dense relay.
---
## Design evolution (historical — superseded, do not trust for current behavior)
These sections describe earlier iterations, kept for context. The code has
moved past all of them.
### v1 — time-slice history (superseded by `RelayLoadingCursors`)
History was first loaded in bounded `since`+`until` **time slices**
(`TimeWindowPagination`, now deleted): `loadMore` fetched only the new band
`[newFloor, previousFloor]`, with a NIP-17 ±2-day wrapper-timestamp margin on
the slice `since` for gift wraps. This bounded re-downloads but still couldn't
tell an empty relay from a gap (an empty slice might sit above older messages),
so the only stop was a 10-year `maxLookback`, and a wide late slice could pull a
20k-event firehose. Replaced by per-relay `until`+`limit` paging.
### v2 — round-model history + rooms-list "stall-gate" (both removed)
History paging once used the **round model** (`WindowLoadTracker`): each
`loadMore` issued one page to all active relays and waited for the slowest to
settle before the next — pacing every relay at the slowest one. The rooms list
additionally had a **stall-gate**: an auto-fill loop that widened only while it
brought in new private rooms, stopping once a widen added none (to avoid the
conversation-keyed cascade).
Both are gone. All history paging is now per-relay independent
(`PerRelayLoadTracker`), and the rooms list pages to exhaustion off marker
visibility like the conversation (commit `98fb8720` dropped the stall-gate;
`60b8629a` / `9f0ecd54` moved gift-wrap and rooms-list history onto the per-relay
model). `WindowLoadTracker` survives **only** as the live-tail completion
barrier. An earlier revision of this doc ("Update 3") still claimed rooms-list
and gift-wrap history used the round model — that is no longer true.
@@ -69,6 +69,7 @@ import com.vitorpamplona.amethyst.service.playback.service.PlaybackServiceClient
import com.vitorpamplona.amethyst.service.relayClient.CacheClientConnector
import com.vitorpamplona.amethyst.service.relayClient.RelayProxyClientConnector
import com.vitorpamplona.amethyst.service.relayClient.authCommand.model.AuthCoordinator
import com.vitorpamplona.amethyst.service.relayClient.diagnostics.DmRelayDiagnosticsLogger
import com.vitorpamplona.amethyst.service.relayClient.notifyCommand.model.NotifyCoordinator
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.RelaySubscriptionsCoordinator
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.EventFinderQueryState
@@ -511,6 +512,9 @@ class AppModules(
val relayReqStats = if (isDebug) RelayReqStats(client) else null
val logger = if (isDebug) RelaySpeedLogger(client) else null
// Focused timeline for the DM / gift-wrap loading path (tag: DMPagination).
val dmDiagnostics = if (isDebug) DmRelayDiagnosticsLogger(client) else null
// Coordinates all subscriptions for the Nostr Client
val sources: RelaySubscriptionsCoordinator =
RelaySubscriptionsCoordinator(
@@ -2645,20 +2645,67 @@ object LocalCache : ILocalCache, ICacheProvider {
}
chatroomList.forEach { userHex, room ->
// History floors are pinned per scope on first advance; null means that window never paged
// history, so its cursors hold no position to misalign and nothing needs rewinding. Only the
// bands strictly BELOW a floor are this window's responsibility — a pruned message newer than
// the floor is the always-on live tail's concern, and rewinding history for it would needlessly
// re-page (and, for a busy room straddling the floor, mis-set the boundary). Hence the per-floor
// filter when accumulating below.
val giftWrapFloor = room.giftWrapHistory.floor
val accountNip04Floor = room.nip04History.floor
room.rooms.map { key, chatroom ->
val toBeRemoved = chatroom.pruneMessagesToTheLatestOnly()
val childrenToBeRemoved = mutableListOf<Note>()
toBeRemoved.forEach {
childrenToBeRemoved.addAll(removeIfWrap(it))
unlinkAndRemove(it)
// Newest pruned `created_at` per relay, in each window's cursor space, capped at < floor.
// Gift wraps page by the OUTER wrap time (from the rumor's host stub); NIP-04 by the event's
// own time, and a kind:4 belongs to BOTH the account (rooms-list) and per-conversation cursor.
val giftWrapPruned = HashMap<NormalizedRelayUrl, Long>()
val accountNip04Pruned = HashMap<NormalizedRelayUrl, Long>()
val roomNip04Pruned = HashMap<NormalizedRelayUrl, Long>()
// chatroom.nip04History is lazy — only touch (allocate) it when this room actually drops a
// kind:4 message, so rooms that never paged conversation history pay nothing.
val roomNip04Floor = if (toBeRemoved.any { it.event is PrivateDmEvent }) chatroom.nip04History.floor else null
childrenToBeRemoved.addAll(it.clearChildLinks())
toBeRemoved.forEach { note ->
when (val ev = note.event) {
is WrappedEvent ->
if (giftWrapFloor != null) {
val outerUntil = ev.host?.createdAt ?: ev.createdAt
if (outerUntil < giftWrapFloor) note.relays.forEach { giftWrapPruned.merge(it, outerUntil, ::maxOf) }
}
is PrivateDmEvent -> {
val until = ev.createdAt
if (accountNip04Floor != null && until < accountNip04Floor) note.relays.forEach { accountNip04Pruned.merge(it, until, ::maxOf) }
if (roomNip04Floor != null && until < roomNip04Floor) note.relays.forEach { roomNip04Pruned.merge(it, until, ::maxOf) }
}
}
childrenToBeRemoved.addAll(removeIfWrap(note))
unlinkAndRemove(note)
childrenToBeRemoved.addAll(note.clearChildLinks())
}
unlinkAndRemove(childrenToBeRemoved)
// Realign the windows so a relay that already paged past (or `done` below) the dropped band
// re-requests it on the next demand-advance instead of skipping the hole.
if (giftWrapPruned.isNotEmpty()) {
room.giftWrapHistory.rewindTo(giftWrapPruned)
Log.d("DMPagination") { "[giftwrap] window rewound after prune: ${giftWrapPruned.size} relay(s), newest pruned wrap @${giftWrapPruned.values.max()}" }
}
if (accountNip04Pruned.isNotEmpty()) {
room.nip04History.rewindTo(accountNip04Pruned)
Log.d("DMPagination") { "[rooms.nip04] window rewound after prune: ${accountNip04Pruned.size} relay(s), newest pruned @${accountNip04Pruned.values.max()}" }
}
if (roomNip04Pruned.isNotEmpty()) {
chatroom.nip04History.rewindTo(roomNip04Pruned)
Log.d("DMPagination") { "[convo.nip04] window rewound after prune of ${key.users.joinToString()}: ${roomNip04Pruned.size} relay(s), newest pruned @${roomNip04Pruned.values.max()}" }
}
if (toBeRemoved.size > 1) {
println(
"PRUNE: ${toBeRemoved.size} private messages from $userHex to ${key.users.joinToString()} removed. ${chatroom.messages.size} kept",
@@ -0,0 +1,190 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.diagnostics
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.AuthMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.ClosedMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.Message
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.NoticeMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.OkMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.AuthCmd
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.Command
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip04Dm.messages.PrivateDmEvent
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.EphemeralGiftWrapEvent
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
import com.vitorpamplona.quartz.utils.Log
/**
* Diagnostic connection listener for the DM loading path both NIP-17 gift wraps
* (kind 1059 / 21059) and NIP-04 legacy DMs (kind 4).
*
* It folds the per-relay timeline REQ sent, connect/disconnect, NOTICE / CLOSED
* rejection and OK failures into the single `DMPagination` log tag with an
* elapsed-time prefix, so a slow cold boot can be attributed (connection? relay
* response?) and a silent failure to load (e.g. a relay answering CLOSED
* "auth-required" / "restricted") becomes visible. Per-event lines are omitted to
* keep the trail readable, but the NIP-42 AUTH handshake (challenge in, AUTH out,
* accept/reject) IS logged: an auth-walled relay's whole load hinges on whether
* that round-trip closes and a re-REQ follows, so it has to be attributable here.
*
* The connection listener fires for EVERY relay the app talks to (hundreds, under
* the outbox model). To keep this readable we only log relays that are part of the
* DM path: a relay is "learned" the first time we send it a kind:1059/21059/4
* REQ, and only those relays' connect/auth/notice lines are emitted thereafter.
*/
class DmRelayDiagnosticsLogger(
val client: INostrClient,
) {
private val startMs = System.currentTimeMillis()
private fun at() = System.currentTimeMillis() - startMs
// Subscription ids whose REQ carried a DM kind, so we can attribute their EOSE/CLOSED.
private val dmSubIds = mutableSetOf<String>()
// Relays we've seen on the DM path, so connect/auth/notice noise from the
// hundreds of unrelated follow/outbox relays is filtered out.
private val dmPathRelays = mutableSetOf<NormalizedRelayUrl>()
// Ids of the AUTH events we've sent to DM relays, so the relay's OK can be tagged "AUTH accepted /
// REJECTED" (the OK that decides whether the post-auth re-REQ will actually be served) instead of
// being lost among ordinary event OKs.
private val authEventIds = mutableSetOf<String>()
private fun isDmRelay(relay: IRelayClient) = relay.url in dmPathRelays
private val listener =
object : RelayConnectionListener {
override fun onConnecting(relay: IRelayClient) {
if (isDmRelay(relay)) Log.d(TAG) { "[+${at()}ms] connecting ${relay.url.url}" }
}
override fun onConnected(
relay: IRelayClient,
pingMillis: Int,
compressed: Boolean,
) {
if (isDmRelay(relay)) {
Log.d(TAG) { "[+${at()}ms] connected ${relay.url.url} (ping ${pingMillis}ms${if (compressed) ", compressed" else ""})" }
}
}
override fun onSent(
relay: IRelayClient,
cmdStr: String,
cmd: Command,
success: Boolean,
) {
if (cmd is AuthCmd) {
// Our reply to a relay's NIP-42 challenge. Remember the id so the relay's OK can be
// tagged as the auth result below. Only for relays already on the DM path.
if (isDmRelay(relay)) {
authEventIds.add(cmd.event.id)
Log.d(TAG) { "[+${at()}ms] AUTH -> ${relay.url.url} success=$success (id ${cmd.event.id.take(8)})" }
}
return
}
if (!isDmReq(cmdStr)) return
dmPathRelays.add(relay.url)
reqSubId(cmdStr)?.let { dmSubIds.add(it) }
Log.d(TAG) { "[+${at()}ms] REQ -> ${relay.url.url} success=$success ${cmdStr.take(400)}" }
}
override fun onIncomingMessage(
relay: IRelayClient,
msgStr: String,
msg: Message,
) {
when (msg) {
is AuthMessage ->
// The relay's NIP-42 challenge. Without it (and the AUTH/OK that follow) an
// auth-walled relay can never serve, so it's the first link to look for.
if (isDmRelay(relay)) Log.d(TAG) { "[+${at()}ms] AUTH challenge <- ${relay.url.url} '${msg.challenge.take(40)}'" }
is NoticeMessage ->
if (isDmRelay(relay)) Log.d(TAG) { "[+${at()}ms] NOTICE <- ${relay.url.url} '${msg.message}'" }
is ClosedMessage ->
if (msg.subId in dmSubIds) {
Log.d(TAG) { "[+${at()}ms] CLOSED <- ${relay.url.url} sub=${msg.subId} reason='${msg.message}'" }
}
is OkMessage ->
if (msg.eventId in authEventIds) {
// The auth result: "accepted" means a syncFilters re-REQ should now be served;
// "REJECTED" means this relay will keep refusing and never serve our DMs.
Log.d(TAG) { "[+${at()}ms] AUTH ${if (msg.success) "accepted" else "REJECTED"} <- ${relay.url.url} '${msg.message}'" }
} else if (!msg.success && isDmRelay(relay)) {
Log.d(TAG) { "[+${at()}ms] OK(fail) <- ${relay.url.url} '${msg.message}'" }
}
else -> {}
}
}
override fun onDisconnected(relay: IRelayClient) {
if (isDmRelay(relay)) Log.d(TAG) { "[+${at()}ms] disconnected ${relay.url.url}" }
}
override fun onCannotConnect(
relay: IRelayClient,
errorMessage: String,
) {
if (isDmRelay(relay)) Log.d(TAG) { "[+${at()}ms] CANNOT CONNECT ${relay.url.url}: $errorMessage" }
}
}
init {
client.addConnectionListener(listener)
}
fun destroy() {
client.removeConnectionListener(listener)
}
companion object {
private const val TAG = "DMPagination"
// The kinds a DM-path REQ carries: NIP-17 gift wraps (1059 + 21059) and NIP-04 legacy DMs
// (4). Matched exactly against the filter's "kinds" array — never as a substring of the whole
// command, since a pubkey hex or timestamp can incidentally contain "1059" or "4".
private val DM_KINDS = setOf(GiftWrapEvent.KIND, EphemeralGiftWrapEvent.KIND, PrivateDmEvent.KIND)
private val KINDS_ARRAY = Regex("\"kinds\":\\[([0-9,\\s]*)]")
// Extracts the subscription id from a `["REQ","<subId>",{...}]` command string.
private val REQ_SUB_ID = Regex("^\\[\"REQ\",\"([^\"]+)\"")
private fun reqSubId(cmdStr: String) = REQ_SUB_ID.find(cmdStr)?.groupValues?.get(1)
/** True only when one of the REQ's `kinds` arrays actually contains a DM kind. */
private fun isDmReq(cmdStr: String): Boolean =
KINDS_ARRAY.findAll(cmdStr).any { match ->
match.groupValues[1]
.split(',')
.any { it.trim().toIntOrNull() in DM_KINDS }
}
}
}
@@ -0,0 +1,63 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.eoseManagers
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
/**
* Diagnostic for "which account is bringing which relays into the DM filters". For each DM
* subscription it prints the account whose relays are used and breaks the relay set down by the
* source list each relay comes from (NIP-65 inbox/outbox, the DM-relay-list, the private-storage
* outbox, local relays). Use it to trace an unexpected relay e.g. a write-only NIP-65 relay that
* only the NIP-04 (home+dm) path queries back to the list it leaks in from.
*/
object DmRelayLog {
private const val TAG = "DMPagination"
fun log(
label: String,
account: Account,
) = Log.d(TAG) {
val pk = account.userProfile().pubkeyHex.take(8)
val inbox = account.nip65RelayList.inboxFlow.value
val outbox = account.nip65RelayList.outboxFlow.value
val dmList = account.dmRelayList.flow.value
val priv = account.privateStorageRelayList.flow.value
val local = account.localRelayList.flow.value
buildString {
append("[$label] account=$pk relays by source:")
appendSource("nip65In", inbox)
appendSource("nip65Out", outbox)
appendSource("dmList", dmList)
appendSource("private", priv)
appendSource("local", local)
}
}
private fun StringBuilder.appendSource(
name: String,
relays: Collection<NormalizedRelayUrl>,
) {
if (relays.isNotEmpty()) append(" $name=${relays.map { it.url }}")
}
}
@@ -28,6 +28,7 @@ import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.marmot.
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.metadata.AccountMetadataEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip01Notifications.AccountNotificationsEoseFromInboxRelaysManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountFeedContentStates
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
@@ -49,10 +50,17 @@ class AccountQueryState(
class AccountFilterAssembler(
client: INostrClient,
) : ComposeSubscriptionManager<AccountQueryState>() {
// Live tail: the recent week of gift wraps, always open at the top for new messages.
val giftWraps = AccountGiftWrapsEoseManager(client, ::allKeys)
// History: older gift wraps, loaded on demand in bounded one-shot slices.
val giftWrapsHistory = AccountGiftWrapsHistoryEoseManager(client, ::allKeys)
val group =
listOf(
AccountMetadataEoseManager(client, ::allKeys),
AccountGiftWrapsEoseManager(client, ::allKeys),
giftWraps,
giftWrapsHistory,
AccountDraftsEoseManager(client, ::allKeys),
AccountNotificationsEoseFromInboxRelaysManager(client, ::allKeys),
MarmotGroupEventsEoseManager(client, ::allKeys),
@@ -20,8 +20,12 @@
*/
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps
import com.vitorpamplona.amethyst.commons.model.privateChats.DmHistoryTuning
import com.vitorpamplona.amethyst.commons.relayClient.nip17Dm.filterGiftWrapsToPubkey
import com.vitorpamplona.amethyst.commons.relayClient.paging.WindowLoadTracker
import com.vitorpamplona.amethyst.commons.relayClient.paging.trackingListener
import com.vitorpamplona.amethyst.model.User
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.AccountQueryState
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
@@ -29,58 +33,68 @@ import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
/**
* Always-on **live tail** for the account's NIP-17 gift wraps (kind 1059). It keeps a fixed
* one-week floor with no upper bound, so the messages list is usable on boot and new incoming
* messages always stream in. It deliberately never widens: pulling older history is the job of
* [AccountGiftWrapsHistoryEoseManager], which fetches the past in bounded, one-shot slices so a
* widen never re-streams what this tail already holds.
*/
class AccountGiftWrapsEoseManager(
client: INostrClient,
allKeys: () -> Set<AccountQueryState>,
) : PerUserEoseManager<AccountQueryState>(client, allKeys) {
override fun user(key: AccountQueryState) = key.account.userProfile()
// The initial-load tracker drives the boot spinner: it stays true until every DM relay has
// settled (EOSE / CLOSED / cannot-connect) on the one-week tail.
private val windowLoad = WindowLoadTracker("giftwrap.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
override fun updateFilter(
key: AccountQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter> {
// Only loads DMs if the account is writeable
return if (key.account.isWriteable()) {
val relays = key.account.dmRelays.flow.value
Log.d("MarmotDbg") {
"AccountGiftWrapsEoseManager.updateFilter: pubkey=${user(key).pubkeyHex.take(8)}" +
"subscribing kind:1059 on ${relays.size} dmRelay(s): ${relays.map { it.url }}"
}
relays.flatMap { relay ->
filterGiftWrapsToPubkey(
relay = relay,
pubkey = user(key).pubkeyHex,
since = since?.get(relay)?.time,
)
}
} else {
Log.d("MarmotDbg") { "AccountGiftWrapsEoseManager.updateFilter: account not writeable, skipping" }
emptyList()
if (!key.account.isWriteable()) {
windowLoad.setExpectedRelays(emptySet())
return emptyList()
}
val relays = key.account.dmRelays.flow.value
windowLoad.setExpectedRelays(relays.toSet())
val sinceTime = DmHistoryTuning.recentBoundary()
DmRelayLog.log("giftwrap.live", key.account)
Log.d(TAG) { "[giftwrap.live] REQ since=$sinceTime (no until) on ${relays.size} relay(s): ${relays.map { it.url }}" }
return relays.flatMap { relay ->
filterGiftWrapsToPubkey(relay = relay, pubkey = user(key).pubkeyHex, since = sinceTime)
}
}
val userJobMap = mutableMapOf<User, List<Job>>()
private val userJobMap = mutableMapOf<User, List<Job>>()
@OptIn(FlowPreview::class)
override fun newSub(key: AccountQueryState): Subscription {
val user = user(key)
windowLoad.startLoading(key.account.scope)
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
listOf(
key.account.scope.launch(Dispatchers.IO) {
key.account.dmRelays.flow.collectLatest {
invalidateFilters()
}
key.account.dmRelays.flow
.collectLatest { invalidateFilters() }
},
)
return super.newSub(key)
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
override fun endSub(
@@ -90,4 +104,8 @@ class AccountGiftWrapsEoseManager(
super.endSub(key, subId)
userJobMap[key]?.forEach { it.cancel() }
}
companion object {
private const val TAG = "DMPagination"
}
}
@@ -0,0 +1,157 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps
import com.vitorpamplona.amethyst.commons.relayClient.nip17Dm.filterGiftWrapsToPubkey
import com.vitorpamplona.amethyst.commons.relayClient.paging.BackwardRelayPager
import com.vitorpamplona.amethyst.commons.relayClient.paging.PagingStatus
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.AccountQueryState
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.flow.StateFlow
/**
* Loads the account's NIP-17 gift-wrap **history** everything older than the one-week live tail
* ([AccountGiftWrapsEoseManager]) by **`until`+`limit` paging, per relay, on demand**.
*
* There is no proactive walk: each relay advances exactly one page when the UI calls [advance] for it,
* and then **parks** at its window limit. The on-screen window-limit markers are the drivers a relay
* pages only while its marker is visible, and keeps paging (page after page) as long as it stays visible
* (see the rooms-list / conversation feed views). So a spam-dense relay never floods: the user has to
* scroll through its messages to pull more, and nothing is fetched while its marker is off screen.
*
* The per-relay cursors live on the account's [ChatroomList][com.vitorpamplona.amethyst.commons.model.privateChats.ChatroomList]
* (so they share the lifetime of the cached gift-wraps); this class binds the single-active
* [BackwardRelayPager] orchestrator to them on [newSub], builds the gift-wrap REQ filters, and forwards
* relay callbacks into the pager. A relay is *done* once it answers an empty page; one that won't answer
* (auth CLOSE, unreachable, or silent) is flagged *stalled* but kept. [exhausted] flips once every relay
* is either done or stalled.
*/
class AccountGiftWrapsHistoryEoseManager(
client: INostrClient,
allKeys: () -> Set<AccountQueryState>,
) : PerUserEoseManager<AccountQueryState>(client, allKeys) {
override fun user(key: AccountQueryState) = key.account.userProfile()
private val pager = BackwardRelayPager("giftwrap.history")
val loadingMore: StateFlow<Boolean> = pager.loadingMore
val status: StateFlow<PagingStatus> = pager.status
private fun daysAgo(epochSeconds: Long) = (TimeUtils.now() - epochSeconds) / TimeUtils.ONE_DAY
override fun updateFilter(
key: AccountQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter> {
if (!key.account.isWriteable()) return emptyList()
// Only relays that have been advanced (armed) and aren't done carry a REQ. A relay that finished a
// page keeps the same `until` here, so re-assembly (triggered when ANOTHER relay advances) doesn't
// re-REQ it — it stays parked until the UI advances it again.
val relays = key.account.dmRelays.flow.value
val armed = pager.armedRelays(relays)
if (armed.isEmpty()) return emptyList()
DmRelayLog.log("giftwrap.history", key.account)
return armed.flatMap { relay ->
val until = pager.requestedUntilFor(relay) ?: return@flatMap emptyList()
Log.d(TAG) { "[giftwrap.history] REQ ${relay.url} until ${daysAgo(until)}d, limit=${pager.pageLimit}" }
filterGiftWrapsToPubkey(relay = relay, pubkey = key.account.userProfile().pubkeyHex, since = null, until = until, limit = pager.pageLimit)
}
}
/** Steps a single [relay] to its next, older page. Driven by that relay's on-screen window-limit marker. */
fun advance(relay: NormalizedRelayUrl) {
if (pager.advance(relay)) invalidateFilters()
}
/** Steps every not-done, not-in-flight relay one page. For the empty/initial boundary (nothing to scroll). */
fun advanceAll() {
if (pager.advanceAll()) {
Log.d(TAG) { "[giftwrap.history] advanceAll (empty-feed bootstrap)" }
invalidateFilters()
}
}
override fun newSub(key: AccountQueryState): Subscription {
// Repoint the single-active orchestrator at this account's gift-wrap cursors (on its ChatroomList)
// and the relays it fans out to, refreshing the display flows from the restored progress.
pager.bind(key.account.chatroomList.giftWrapHistory, key.account.scope) { key.account.dmRelays.flow.value }
return requestNewSubscription(historyListener(key))
}
private fun historyListener(key: AccountQueryState): SubscriptionListener {
// A just-backgrounded account's subscription can still deliver after the orchestrator rebinds to
// another account; gate the pager (single-active) on whether it's still bound to THIS account's
// cursors so a late callback can't move another account's cursors. newEose runs regardless.
val myCursors = key.account.chatroomList.giftWrapHistory
return object : SubscriptionListener {
override fun onEvent(
event: Event,
isLive: Boolean,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onEvent(relay, event.createdAt)
}
override fun onEose(
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors) && pager.onEose(relay)) {
Log.d(TAG) { "[giftwrap.history] ${relay.url} reached the bottom (done)" }
}
// No auto-advance: the relay parks here until its marker asks for the next page.
newEose(key, relay, TimeUtils.now(), forFilters)
}
override fun onClosed(
message: String,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onClosed(relay, message)
}
override fun onCannotConnect(
relay: NormalizedRelayUrl,
message: String,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onCannotConnect(relay, message)
}
}
}
companion object {
private const val TAG = "DMPagination"
}
}
@@ -58,8 +58,23 @@ fun RefreshingChatroomFeedView(
onWantsToEditDraft: (Note) -> Unit,
avoidDraft: DraftTagState? = null,
scrollStateKey: String? = null,
// Opt-in hook handed the feed's scroll state, so a specific screen (e.g. private DMs) can
// attach scroll-driven loading. No-op for the public-chat / channel callers that don't paginate.
listStateObserver: @Composable (LazyListState) -> Unit = {},
// Optional footer rendered at the oldest end of the thread (a "load more" / spinner affordance).
// Null for callers that load their whole history at once (public chats / channels).
olderBoundary: (@Composable () -> Unit)? = null,
// Optional per-gap hook: invoked between each message and its next-older neighbour with their
// createdAt bounds, so a caller (private DMs) can draw per-relay paging markers at the depth each
// relay has reached. No-op for callers without per-relay progress.
markersInGap: (@Composable (newerCreatedAt: Long?, olderCreatedAt: Long?) -> Unit)? = null,
// Optional hoisted load driver: handed the loaded message list and its scroll state once (above the
// LazyColumn), so a caller (private DMs) can drive demand-driven paging off viewport visibility
// rather than per-row composition. No-op for callers that don't paginate.
sentinels: (@Composable (items: List<Note>, listState: LazyListState) -> Unit)? = null,
) {
SaveableFeedState(feedContentState, scrollStateKey) { listState ->
listStateObserver(listState)
RenderChatFeedView(
feedContentState,
accountViewModel,
@@ -69,6 +84,9 @@ fun RefreshingChatroomFeedView(
onWantsToReply,
onWantsToEditDraft,
avoidDraft,
olderBoundary,
markersInGap,
sentinels,
)
}
}
@@ -83,6 +101,9 @@ fun RenderChatFeedView(
onWantsToReply: (Note) -> Unit,
onWantsToEditDraft: (Note) -> Unit,
avoidDraft: DraftTagState? = null,
olderBoundary: (@Composable () -> Unit)? = null,
markersInGap: (@Composable (newerCreatedAt: Long?, olderCreatedAt: Long?) -> Unit)? = null,
sentinels: (@Composable (items: List<Note>, listState: LazyListState) -> Unit)? = null,
) {
val feedState by feed.feedContent.collectAsStateWithLifecycle()
@@ -110,6 +131,9 @@ fun RenderChatFeedView(
onWantsToReply,
onWantsToEditDraft,
avoidDraft,
olderBoundary,
markersInGap,
sentinels,
)
}
}
@@ -126,9 +150,16 @@ fun ChatFeedLoaded(
onWantsToReply: (Note) -> Unit,
onWantsToEditDraft: (Note) -> Unit,
avoidDraft: DraftTagState? = null,
olderBoundary: (@Composable () -> Unit)? = null,
markersInGap: (@Composable (newerCreatedAt: Long?, olderCreatedAt: Long?) -> Unit)? = null,
sentinels: (@Composable (items: List<Note>, listState: LazyListState) -> Unit)? = null,
) {
val items by loaded.feed.collectAsStateWithLifecycle()
// Hoisted load driver (above the LazyColumn): pages each relay off viewport visibility, so feed
// reorders no longer re-fire paging. The per-gap markers below are pure UI.
sentinels?.invoke(items.list, listState)
LaunchedEffect(items.list.firstOrNull()) {
if (listState.firstVisibleItemIndex <= 1) {
listState.animateScrollToItem(0)
@@ -169,7 +200,24 @@ fun ChatFeedLoaded(
)
NewDateOrSubjectDivisor(items.list.getOrNull(index + 1), item)
// Per-relay paging markers belonging in the gap toward the next-older message. With the
// reverse layout this draws just above the message (the older side), so a relay's marker
// appears right below the oldest message it has reached and slides down as it pages.
markersInGap?.invoke(
item.event?.createdAt,
items.list
.getOrNull(index + 1)
?.event
?.createdAt,
)
}
}
// Reverse layout: a trailing item sits at the highest index, i.e. the visual TOP (oldest end).
// That's where the caller's "load more" affordance / spinner lives.
if (olderBoundary != null) {
item(key = "olderBoundary") { olderBoundary() }
}
}
}
@@ -84,6 +84,7 @@ import com.vitorpamplona.quartz.nip04Dm.messages.PrivateDmEvent
import com.vitorpamplona.quartz.nip13Pow.strongPoWOrNull
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKeyable
import com.vitorpamplona.quartz.nip17Dm.files.ChatMessageEncryptedFileHeaderEvent
import com.vitorpamplona.quartz.nip17Dm.messages.ChatMessageEvent
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelCreateEvent
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelMetadataEvent
import com.vitorpamplona.quartz.nip37Drafts.DraftWrapEvent
@@ -106,8 +107,12 @@ fun ChatroomMessageCompose(
onScrollToNote: ((Note) -> Unit)? = null,
shouldHighlight: Boolean = false,
onHighlightFinished: (() -> Unit)? = null,
// Replaces the generic "post not found" blank while baseNote's event hasn't loaded. Used for
// reply quotes inside a DM, where the target is simply older than the loaded window (see
// LoadingReplyNote). Null keeps the default blank for every other caller.
onBlank: (@Composable () -> Unit)? = null,
) {
WatchNoteEvent(baseNote = baseNote, accountViewModel = accountViewModel, nav) {
val onFound: @Composable () -> Unit = {
WatchBlockAndReport(
note = baseNote,
showHiddenWarning = false,
@@ -140,6 +145,12 @@ fun ChatroomMessageCompose(
}
}
}
if (onBlank != null) {
WatchNoteEvent(baseNote = baseNote, onNoteEventFound = onFound, onBlank = onBlank, accountViewModel = accountViewModel)
} else {
WatchNoteEvent(baseNote = baseNote, accountViewModel = accountViewModel, nav = nav, onNoteEventFound = onFound)
}
}
@Composable
@@ -396,9 +407,23 @@ private fun RenderReply(
}
}
replyTo.value?.let { note ->
replyTo.value?.let { replyNote ->
// For a DM, a reply target that hasn't arrived isn't lost — it's older than the loaded
// window (and for gift wraps can't be fetched by id). Swap the generic blank for one that
// walks history backward until it surfaces. Pick the pager by the PARENT's protocol; leave
// public chats / marmot groups (not a DM event here) on the default blank.
val replyBlank: (@Composable () -> Unit)? =
when (note.event) {
is ChatMessageEvent, is ChatMessageEncryptedFileHeaderEvent -> {
{ LoadingReplyNote(DmReplyProtocol.NIP17, accountViewModel) }
}
is PrivateDmEvent -> {
{ LoadingReplyNote(DmReplyProtocol.NIP04, accountViewModel) }
}
else -> null
}
ChatroomMessageCompose(
baseNote = note,
baseNote = replyNote,
routeForLastRead = null,
innerQuote = true,
parentBackgroundColor = bgColor,
@@ -407,6 +432,7 @@ private fun RenderReply(
onWantsToReply = onWantsToReply,
onWantsToEditDraft = onWantsToEditDraft,
onScrollToNote = onScrollToNote,
onBlank = replyBlank,
)
}
}
@@ -0,0 +1,32 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.feed
import java.text.SimpleDateFormat
import java.util.Date
import java.util.Locale
/**
* The Android locale date formatter for the shared [com.vitorpamplona.amethyst.commons.ui.feeds.DmHistoryLoadingCard]
* passed in so the shared (KMP) card carries no `java.time` dependency. Formats a paging reach point
* (epoch seconds) to a short month-year label, e.g. "Jun 2026".
*/
fun formatHistoryReachDate(epochSeconds: Long): String = SimpleDateFormat("MMM yyyy", Locale.getDefault()).format(Date(epochSeconds * 1000))
@@ -0,0 +1,222 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.feed
import androidx.compose.animation.Crossfade
import androidx.compose.animation.core.tween
import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import androidx.lifecycle.compose.collectAsStateWithLifecycle
import com.vitorpamplona.amethyst.R
import com.vitorpamplona.amethyst.commons.relayClient.paging.PagingStatus
import com.vitorpamplona.amethyst.commons.ui.feeds.DmHistoryRelayDialog
import com.vitorpamplona.amethyst.commons.ui.feeds.historySubtitle
import com.vitorpamplona.amethyst.commons.ui.feeds.incompleteSubtitle
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountViewModel
import com.vitorpamplona.amethyst.ui.stringRes
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.filter
/** Which DM history pager backs the conversation an unloaded reply belongs to. */
enum class DmReplyProtocol {
// NIP-17 gift wraps. The rumor id of the reply target is NOT queryable on relays (only the outer
// 1059 wrap id is), so the only way to surface it is to keep paging gift-wrap history until the wrap
// carrying it is decrypted — hence we drive the account-wide gift-wrap history pager.
NIP17,
// NIP-04 legacy DMs (kind 4). Paged per relay for the open conversation.
NIP04,
}
/**
* The inner-quote placeholder for a reply whose target message has not been paged in yet used in
* place of the generic [com.vitorpamplona.amethyst.ui.note.BlankNote] ("post not found") that the main
* feeds show. A reply target inside a conversation isn't *missing*, it's simply older than the window
* loaded so far; for gift wraps it can't even be fetched by id. So instead of declaring it lost, this
* card actively walks the conversation's history backward kicking the protocol's `loadMore` each time
* the previous page settles until either the target decrypts (the surrounding
* [com.vitorpamplona.amethyst.ui.note.WatchNoteEvent] crossfades the real message in and disposes this)
* or that protocol's history runs dry, at which point it settles into the terminal "not found" text.
*
* It runs regardless of scroll position (no oldest-end gate like the scroll-driven loader) precisely so
* that opening a thread and seeing a reply to something off-screen pulls that something in on its own.
* The drive loop is idempotent and gated on the pager's own `loadingMore`/`exhausted`, so several
* unloaded replies on screen and the scroll loader all coalesce onto the same paging window.
*/
@Composable
fun LoadingReplyNote(
protocol: DmReplyProtocol,
accountViewModel: AccountViewModel,
modifier: Modifier = Modifier,
) {
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04History }
val loadingFlow: StateFlow<Boolean> =
when (protocol) {
DmReplyProtocol.NIP17 -> giftWrapsHistory.loadingMore
DmReplyProtocol.NIP04 -> nip04History.loadingMore
}
val statusFlow: StateFlow<PagingStatus> =
when (protocol) {
DmReplyProtocol.NIP17 -> giftWrapsHistory.status
DmReplyProtocol.NIP04 -> nip04History.status
}
val protocolTag =
when (protocol) {
DmReplyProtocol.NIP17 -> "NIP-17"
DmReplyProtocol.NIP04 -> "NIP-04"
}
// One snapshot collector instead of five; the fields below are plain reads off it (downstream unchanged).
val status by statusFlow.collectAsStateWithLifecycle()
val exhausted = status.exhausted
val relayCount = status.relayCount
val stalledCount = status.stalledCount
val reachedBack = status.reachedBack
val relayProgress = status.relayProgress
LaunchedEffect(protocol, loadingFlow, statusFlow) {
// Step the next, older page whenever the previous one has settled and history isn't exhausted.
// The target may surface mid-page (this composable then leaves composition and cancels us); if
// not, we keep walking until the protocol bottoms out and the filter stops passing.
combine(loadingFlow, statusFlow) { loading, s -> !loading && !s.exhausted }
.distinctUntilChanged()
.filter { it }
.collect {
Log.d("DMPagination") { "reply blank: widen → $protocol advanceAll (searching for unloaded reply)" }
when (protocol) {
DmReplyProtocol.NIP17 -> giftWrapsHistory.advanceAll()
DmReplyProtocol.NIP04 -> nip04History.advanceAll()
}
}
}
// Tapping opens the same per-relay popup the history card uses, so when the search gives up the user
// can see exactly which relays were reached and which stalled. Empty progress keeps it non-interactive.
var showRelays by remember { mutableStateOf(false) }
if (showRelays) {
DmHistoryRelayDialog(protocolTag, relayProgress, ::formatHistoryReachDate) { showRelays = false }
}
// Same chrome as DmHistoryLoadingCard (the older-history status card at the oldest end) so an
// unloaded reply reads as the same kind of "reaching back into history" state, just inline in the
// quote: rounded translucent surface, a spinner-in-a-box, then the status line.
Surface(
modifier =
modifier
.fillMaxWidth()
.padding(horizontal = 8.dp, vertical = 4.dp)
.then(if (relayProgress.isNotEmpty()) Modifier.clickable { showRelays = true } else Modifier),
shape = RoundedCornerShape(14.dp),
color = MaterialTheme.colorScheme.surfaceVariant.copy(alpha = 0.55f),
tonalElevation = 2.dp,
) {
// When the walk gives up it splits the same way the history card does: some relays stalled
// (couldn't reach them — the message may still be out there) vs every relay genuinely bottomed
// out (it really isn't in your history). Either way we say what happened instead of a bare glyph.
val stalledOut = exhausted && stalledCount > 0
Crossfade(targetState = exhausted, animationSpec = tween(500), label = "loadingReplyState") { isExhausted ->
Row(
Modifier.fillMaxWidth().padding(horizontal = 14.dp, vertical = 10.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Box(Modifier.size(22.dp), contentAlignment = Alignment.Center) {
when {
stalledOut ->
// Stalled-out: same red "…" the per-relay dialog and history card use for unreachable.
Text(
"",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold,
color = MaterialTheme.colorScheme.error,
)
isExhausted ->
// Genuinely searched everything and it isn't there.
Text(
"",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
else -> CircularProgressIndicator(Modifier.size(18.dp), strokeWidth = 2.dp)
}
}
Spacer(Modifier.width(14.dp))
Column(Modifier.weight(1f)) {
Text(
text =
if (isExhausted) {
stringRes(R.string.chats_reply_not_found)
} else {
stringRes(R.string.chats_reply_searching_history)
},
style = MaterialTheme.typography.bodyMedium,
fontWeight = FontWeight.SemiBold,
color = MaterialTheme.colorScheme.onSurface,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
// While loading: which protocol, how many relays, how far back. When it gives up: either
// "N relays unreachable · tap to see which" (stalled) or "searched every relay · tap to see".
Text(
text =
when {
stalledOut -> incompleteSubtitle(stalledCount)
isExhausted -> stringRes(R.string.chats_reply_searched)
else -> historySubtitle(protocolTag, relayCount, stalledCount, reachedBack, ::formatHistoryReachDate)
},
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
}
}
}
}
}
@@ -26,12 +26,28 @@ import androidx.compose.foundation.layout.fillMaxHeight
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.text.input.setTextAndPlaceCursorAtEnd
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.unit.dp
import androidx.lifecycle.compose.collectAsStateWithLifecycle
import androidx.lifecycle.viewmodel.compose.viewModel
import com.vitorpamplona.amethyst.R
import com.vitorpamplona.amethyst.commons.ui.feeds.DmHistoryLoadingCard
import com.vitorpamplona.amethyst.commons.ui.feeds.FeedContentState
import com.vitorpamplona.amethyst.commons.ui.feeds.FeedState
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachCursor
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachDetailDialog
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachMarkers
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachSentinels
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachState
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.EventFinderFilterAssemblerSubscription
import com.vitorpamplona.amethyst.ui.actions.uploads.resolveSharedMedia
import com.vitorpamplona.amethyst.ui.feeds.WatchLifecycleAndUpdateModel
@@ -40,15 +56,23 @@ import com.vitorpamplona.amethyst.ui.note.LoadAddressableNote
import com.vitorpamplona.amethyst.ui.note.elements.ObserveRelayListForDMsAndDisplayIfNotFound
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountViewModel
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.feed.RefreshingChatroomFeedView
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.feed.formatHistoryReachDate
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.dal.ChatroomFeedViewModel
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource.ChatroomFilterAssemblerSubscription
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.send.ChatNewMessageViewModel
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.send.PrivateMessageEditFieldRow
import com.vitorpamplona.amethyst.ui.theme.DoubleVertSpacer
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayPagingProgress
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKey
import com.vitorpamplona.quartz.nip17Dm.settings.ChatMessageRelayListEvent
import com.vitorpamplona.quartz.utils.Log
import kotlinx.collections.immutable.persistentListOf
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.filter
import kotlinx.coroutines.launch
@Composable
@@ -134,6 +158,47 @@ fun ChatroomView(
)
}
/**
* Bootstraps history when the conversation has no messages yet (the live tail came back empty for a
* thread whose newest message is older than a week). There's nothing on screen to host the per-relay
* window-limit markers that normally drive paging, so while the feed is empty we step every relay one
* page at a time (each protocol independently, gated on its own loader) until messages appear at which
* point the on-screen markers take over or the protocol is exhausted. Once the feed is Loaded this
* does nothing; paging is then purely demand-driven by the markers' visibility.
*/
@Composable
private fun BootstrapHistoryWhenEmpty(
feedContentState: FeedContentState,
accountViewModel: AccountViewModel,
) {
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04History }
val feedState by feedContentState.feedContent.collectAsStateWithLifecycle()
// Empty only (never the transient Loading navigation flashes through), and debounced below, so
// re-opening a conversation that has messages doesn't kick a hunt.
val needsBootstrap = feedState is FeedState.Empty
LaunchedEffect(needsBootstrap, giftWrapsHistory) {
if (!needsBootstrap) return@LaunchedEffect
delay(BOOTSTRAP_DEBOUNCE_MS)
combine(giftWrapsHistory.loadingMore, giftWrapsHistory.status) { loading, s -> !loading && !s.exhausted }
.distinctUntilChanged()
.filter { it }
.collect { giftWrapsHistory.advanceAll() }
}
LaunchedEffect(needsBootstrap, nip04History) {
if (!needsBootstrap) return@LaunchedEffect
delay(BOOTSTRAP_DEBOUNCE_MS)
combine(nip04History.loadingMore, nip04History.status) { loading, s -> !loading && !s.exhausted }
.distinctUntilChanged()
.filter { it }
.collect { nip04History.advanceAll() }
}
}
// Ignore the transient empty feed that navigation flashes through before messages re-appear.
private const val BOOTSTRAP_DEBOUNCE_MS = 1200L
@Composable
fun ChatroomViewUI(
room: ChatroomKey,
@@ -145,6 +210,50 @@ fun ChatroomViewUI(
WatchLifecycleAndUpdateModel(feedViewModel)
ChatroomFilterAssemblerSubscription(room, accountViewModel.dataSources().chatroom, accountViewModel)
DisposableEffect(room) {
Log.d("DMPagination") { "convo: OPEN room=${room.hashCode()}" }
onDispose { Log.d("DMPagination") { "convo: CLOSE room=${room.hashCode()}" } }
}
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroom.nip04History }
val loadingGiftWraps by giftWrapsHistory.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04History.loadingMore.collectAsStateWithLifecycle()
// One atomic snapshot per protocol (exhausted + relays + reached + per-relay progress) instead of six
// separate collectors — the status card and the per-relay markers read all of it together anyway.
val giftWrapsStatus by giftWrapsHistory.status.collectAsStateWithLifecycle()
val nip04Status by nip04History.status.collectAsStateWithLifecycle()
val user = accountViewModel.userProfile()
// Both protocols' per-relay window limits, each carrying the advance() that pulls its own next page.
// Placed in the stream as sentinels (see RelayReachMarkers): a relay pages only while its
// marker is on screen, and keeps paging while it stays there. A protocol drops out once exhausted.
val limits =
remember(nip04Status, giftWrapsStatus, user) {
buildList {
if (!giftWrapsStatus.exhausted) {
giftWrapsStatus.relayProgress.forEach { (relay, p) ->
add(RelayReachCursor("17:${relay.url}", relayShortName(relay), p.reachedUntil, reachState(p), "NIP-17") { giftWrapsHistory.advance(relay) })
}
}
if (!nip04Status.exhausted) {
nip04Status.relayProgress.forEach { (relay, p) ->
add(RelayReachCursor("04:${relay.url}", relayShortName(relay), p.reachedUntil, reachState(p), "NIP-04") { nip04History.advance(relay) })
}
}
}
}
val nip17Name = stringResource(R.string.chats_history_proto_nip17)
val nip04Name = stringResource(R.string.chats_history_proto_nip04)
// The relays behind a tapped in-stream "Relay sync" marker; non-null shows the detail popup.
var syncDetail by remember { mutableStateOf<List<RelayReachCursor>?>(null) }
syncDetail?.let { detail ->
RelayReachDetailDialog(detail, ::formatHistoryReachDate) { syncDetail = null }
}
BootstrapHistoryWhenEmpty(feedViewModel.feedState, accountViewModel)
Column(Modifier.fillMaxHeight()) {
ObserveRelayListForDMsAndDisplayIfNotFound(accountViewModel, nav)
@@ -163,6 +272,32 @@ fun ChatroomViewUI(
avoidDraft = newPostModel.draftTag,
onWantsToReply = newPostModel::reply,
onWantsToEditDraft = newPostModel::editFromDraft,
// One status card per protocol at the oldest end: each shows what it's reaching for
// while it pages and crossfades to "All caught up" when that protocol runs dry.
olderBoundary = {
Column {
DmHistoryLoadingCard(nip17Name, "NIP-17", loadingGiftWraps, giftWrapsStatus.exhausted, giftWrapsStatus.relayCount, giftWrapsStatus.stalledCount, giftWrapsStatus.reachedBack, giftWrapsStatus.relayProgress, ::formatHistoryReachDate)
DmHistoryLoadingCard(nip04Name, "NIP-04", loadingNip04, nip04Status.exhausted, nip04Status.relayCount, nip04Status.stalledCount, nip04Status.reachedBack, nip04Status.relayProgress, ::formatHistoryReachDate)
}
},
// Each relay's window-limit marker, placed at its reached cursor (pure UI). Hidden once
// both protocols are exhausted.
markersInGap =
if (limits.isEmpty()) {
null
} else {
{ newer, older -> RelayReachMarkers(limits, newer, older) { syncDetail = it } }
},
// The hoisted load driver that pulls each relay's next page while its marker is on screen,
// off viewport visibility (see RelayReachSentinels) so feed reorders don't re-page.
sentinels =
if (limits.isEmpty()) {
null
} else {
{ items, listState ->
RelayReachSentinels(limits, listState) { index -> items.getOrNull(index)?.event?.createdAt }
}
},
)
}
@@ -183,3 +318,16 @@ fun ChatroomViewUI(
)
}
}
private fun reachState(p: RelayPagingProgress): RelayReachState =
when {
p.done -> RelayReachState.DONE
p.stalled -> RelayReachState.STALLED
else -> RelayReachState.REACHING
}
private fun relayShortName(relay: NormalizedRelayUrl): String =
relay.url
.substringAfter("://")
.trimEnd('/')
.substringBefore('/')
@@ -36,9 +36,16 @@ class ChatroomQueryState(
class ChatroomFilterAssembler(
client: INostrClient,
) : ComposeSubscriptionManager<ChatroomQueryState>() {
// NIP-04 live tail: the recent week, always open at the top.
val nip04 = ChatroomNip04SubAssembler(client, ::allKeys)
// NIP-04 history: older DMs, paged backward by until+limit, independently of gift wraps.
val nip04History = ChatroomNip04HistorySubAssembler(client, ::allKeys)
val group =
listOf(
ChatroomFilterSubAssembler(client, ::allKeys),
nip04,
nip04History,
)
override fun invalidateKeys() = invalidateFilters()
@@ -0,0 +1,155 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource
import com.vitorpamplona.amethyst.commons.relayClient.paging.BackwardRelayPager
import com.vitorpamplona.amethyst.commons.relayClient.paging.PagingStatus
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserAndFollowListEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.flow.StateFlow
/**
* Loads older NIP-04 DMs (kind 4) for one conversation by **`until`+`limit` paging, per relay, on
* demand**. Each relay advances exactly one page when the conversation's on-screen window-limit marker
* for that relay asks ([advance]); otherwise it parks. Nothing is walked proactively a relay pages
* only while its marker is visible and keeps paging while it stays visible.
*
* The per-relay cursors live on the conversation's [Chatroom][com.vitorpamplona.amethyst.commons.model.privateChats.Chatroom]
* (so reopening the room keeps its progress); this class binds the single-active [BackwardRelayPager]
* orchestrator to the open room's cursors on [newSub], builds the (per-relay scoped) NIP-04 REQ
* filters, and forwards relay callbacks into the pager. A relay is *done* once it answers an empty page;
* one that won't answer (auth CLOSE, unreachable, or silent) is flagged *stalled* but kept. [exhausted]
* flips once every relay is either done or stalled.
*/
class ChatroomNip04HistorySubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomQueryState>,
) : PerUserAndFollowListEoseManager<ChatroomQueryState, String>(client, allKeys) {
private val pager = BackwardRelayPager("convo.nip04.history")
val loadingMore: StateFlow<Boolean> = pager.loadingMore
val status: StateFlow<PagingStatus> = pager.status
override fun user(key: ChatroomQueryState) = key.account.userProfile()
override fun list(key: ChatroomQueryState) = key.listId
// This conversation's persistent paging cursors, held on its Chatroom (per account + room).
private fun cursorsFor(key: ChatroomQueryState) =
key.account.chatroomList
.getOrCreatePrivateChatroom(key.room)
.nip04History
override fun updateFilter(
key: ChatroomQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? {
val relays = nip04DmRelayRouting(key.room.users, key.account)
if (!key.account.isWriteable() || relays == null) return emptyList()
// Only armed (advanced, not done) relays carry a REQ, each at its own requested cursor. A parked
// relay keeps the same filter here, so re-assembly (another relay advancing) doesn't re-REQ it.
val armed = pager.armedRelays(relays.all).toSet()
if (armed.isEmpty()) return emptyList()
DmRelayLog.log("convo.nip04.history", key.account)
val scoped =
Nip04DmRelayRouting(
toMeRelays = relays.toMeRelays.filterKeys { it in armed },
fromMeRelays = relays.fromMeRelays.filterKeys { it in armed },
)
return filterNip04DMsHistory(key.account, scoped, pager.pageLimit) { relay ->
pager.requestedUntilFor(relay)
}
}
/** Steps a single [relay] to its next, older page for the open conversation. Driven by its marker. */
fun advance(relay: NormalizedRelayUrl) {
if (pager.advance(relay)) invalidateFilters()
}
/** Steps every not-done, not-in-flight relay one page. For a thread too short to scroll. */
fun advanceAll() {
if (pager.advanceAll()) {
Log.d("DMPagination") { "[convo.nip04.history] advanceAll (empty-thread bootstrap)" }
invalidateFilters()
}
}
override fun newSub(key: ChatroomQueryState): Subscription {
// Repoint the single-active orchestrator at this conversation's cursors (on its Chatroom) and the
// relays it fans out to, refreshing the display flows from the restored progress.
pager.bind(cursorsFor(key), key.account.scope) { nip04DmRelayRouting(key.room.users, key.account)?.all }
return requestNewSubscription(historyListener(key))
}
private fun historyListener(key: ChatroomQueryState): SubscriptionListener {
// A just-backgrounded room's subscription can still deliver after the orchestrator rebinds to
// another room; gate the pager (single-active) on whether it's still bound to THIS room's cursors
// so a late callback can't move another room's cursors. newEose (framework bookkeeping) runs anyway.
val myCursors = cursorsFor(key)
return object : SubscriptionListener {
override fun onEvent(
event: Event,
isLive: Boolean,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onEvent(relay, event.createdAt)
}
override fun onEose(
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors) && pager.onEose(relay)) {
Log.d("DMPagination") { "[convo.nip04.history] ${relay.url} reached the bottom (done)" }
}
newEose(key, relay, TimeUtils.now(), forFilters)
}
override fun onClosed(
message: String,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onClosed(relay, message)
}
override fun onCannotConnect(
relay: NormalizedRelayUrl,
message: String,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onCannotConnect(relay, message)
}
}
}
}
@@ -20,26 +20,55 @@
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource
import com.vitorpamplona.amethyst.commons.model.privateChats.DmHistoryTuning
import com.vitorpamplona.amethyst.commons.relayClient.paging.WindowLoadTracker
import com.vitorpamplona.amethyst.commons.relayClient.paging.trackingListener
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserAndFollowListEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.flow.StateFlow
class ChatroomFilterSubAssembler(
/**
* Always-on **live tail** for one conversation's NIP-04 DMs (kind 4). A fixed one-week floor, no
* upper bound, never widens older history is loaded by [ChatroomNip04HistorySubAssembler] in
* bounded slices that follow the gift-wrap history window.
*/
class ChatroomNip04SubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomQueryState>,
) : PerUserAndFollowListEoseManager<ChatroomQueryState, String>(client, allKeys) {
private val windowLoad = WindowLoadTracker("convo.nip04.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
override fun updateFilter(
key: ChatroomQueryState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? =
if (key.account.isWriteable()) {
filterNip04DMs(key.room.users, key.account, since)
val sinceTime = DmHistoryTuning.recentBoundary()
val filters = filterNip04DMs(key.room.users, key.account, sinceTime)
windowLoad.setExpectedRelays(filters?.mapTo(mutableSetOf()) { it.relay } ?: emptySet())
DmRelayLog.log("convo.nip04.live", key.account)
Log.d("DMPagination") { "[convo.nip04.live] REQ since=$sinceTime (no until) on ${filters?.size ?: 0} relay-filter(s): ${filters?.map { it.relay.url }?.distinct()}" }
filters
} else {
windowLoad.setExpectedRelays(emptySet())
emptyList()
}
override fun user(key: ChatroomQueryState) = key.account.userProfile()
override fun list(key: ChatroomQueryState) = key.listId
override fun newSub(key: ChatroomQueryState): Subscription {
windowLoad.startLoading(key.account.scope)
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
}
@@ -22,7 +22,6 @@ package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.privateDM.datasource
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.amethyst.model.LocalCache
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
@@ -30,19 +29,55 @@ import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip04Dm.messages.PrivateDmEvent
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
fun filterNip04DMs(
/**
* Where a conversation's NIP-04 DMs flow, resolved via the outbox model and **scoped per relay** so
* every filter only names the keys that actually own the relay it is sent to.
*
* Each map is `relay -> the counterpart keys to name in that relay's filter`:
* - [toMeRelays] queries messages **to me** (`authors=[those keys], #p=[me]`). A relay appears here
* when it is my inbox (then the key set is the whole group) and/or a counterpart's outbox (then
* the key set is just the counterparts who publish there).
* - [fromMeRelays] queries messages **from me** (`authors=[me], #p=[those keys]`). A relay appears
* here when it is my outbox (then the key set is the whole group) and/or a counterpart's inbox
* (then the key set is just the counterparts who read there).
*
* Scoping the key set per relay is what keeps us from sending, e.g., `authors=[bob]` to a relay that
* is only charlie's a filter that relay has no reason to serve.
*/
class Nip04DmRelayRouting(
val toMeRelays: Map<NormalizedRelayUrl, Set<HexKey>>,
val fromMeRelays: Map<NormalizedRelayUrl, Set<HexKey>>,
) {
val all: Set<NormalizedRelayUrl> get() = toMeRelays.keys + fromMeRelays.keys
}
private fun addAll(
map: MutableMap<NormalizedRelayUrl, MutableSet<HexKey>>,
relays: Collection<NormalizedRelayUrl>,
keys: Collection<HexKey>,
) = relays.forEach { map.getOrPut(it) { mutableSetOf() }.addAll(keys) }
fun nip04DmRelayRouting(
group: Set<HexKey>?,
account: Account?,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? {
): Nip04DmRelayRouting? {
if (group.isNullOrEmpty() || account == null) return null
val userOutboxRelays = account.homeRelays.flow.value
val userInboxRelays = account.dmRelays.flow.value
val groupOutboxRelays = mutableSetOf<NormalizedRelayUrl>()
val groupInboxRelays = mutableSetOf<NormalizedRelayUrl>()
// relay -> counterpart keys whose messages-to-me we ask that relay for (authors set, #p=[me]).
val toMe = mutableMapOf<NormalizedRelayUrl, MutableSet<HexKey>>()
// relay -> counterpart keys whose copy of my messages we ask that relay for (#p set, authors=[me]).
val fromMe = mutableMapOf<NormalizedRelayUrl, MutableSet<HexKey>>()
// My own relays carry the whole conversation: my inbox holds everyone's messages to me, my outbox
// holds all of mine. Both filters name the full group on these relays.
addAll(toMe, userInboxRelays, group)
addAll(fromMe, userOutboxRelays, group)
// Each counterpart's own relays only get a filter naming that counterpart: their outbox (where
// they publish their messages to me) and their inbox (where they keep my messages to them).
group.forEach {
val authorHomeRelayEventAddress = AdvertisedRelayListEvent.createAddressTag(it)
val authorHomeRelayEvent = (LocalCache.getAddressableNoteIfExists(authorHomeRelayEventAddress)?.event as? AdvertisedRelayListEvent)
@@ -53,42 +88,81 @@ fun filterNip04DMs(
?: LocalCache.relayHints.hintsForKey(it).ifEmpty { null }
?: emptyList()
groupOutboxRelays.addAll(outbox)
val inbox =
authorHomeRelayEvent?.readRelaysNorm()?.ifEmpty { null }
?: LocalCache.getUserIfExists(it)?.allUsedRelaysOrNull()
?: LocalCache.relayHints.hintsForKey(it).ifEmpty { null }
?: emptyList()
groupInboxRelays.addAll(inbox)
addAll(toMe, outbox, listOf(it))
addAll(fromMe, inbox, listOf(it))
}
val toMeRelays = (userInboxRelays + groupOutboxRelays)
val fromMeRelays = (userOutboxRelays + groupInboxRelays)
return toMeRelays.map {
RelayBasedFilter(
relay = it,
filter =
Filter(
kinds = listOf(PrivateDmEvent.KIND),
authors = group.toList(),
tags = mapOf("p" to listOf(account.userProfile().pubkeyHex)),
since = since?.get(it)?.time,
),
)
} +
fromMeRelays.map {
RelayBasedFilter(
relay = it,
filter =
Filter(
kinds = listOf(PrivateDmEvent.KIND),
authors = listOf(account.userProfile().pubkeyHex),
tags = mapOf("p" to group.toList()),
since = since?.get(it)?.time,
),
)
}
return Nip04DmRelayRouting(toMe, fromMe)
}
private fun toMeFilter(
relay: NormalizedRelayUrl,
authors: Set<HexKey>,
account: Account,
since: Long?,
until: Long?,
limit: Int?,
) = RelayBasedFilter(
relay = relay,
filter =
Filter(
kinds = listOf(PrivateDmEvent.KIND),
authors = authors.toList(),
tags = mapOf("p" to listOf(account.userProfile().pubkeyHex)),
since = since,
until = until,
limit = limit,
),
)
private fun fromMeFilter(
relay: NormalizedRelayUrl,
pTags: Set<HexKey>,
account: Account,
since: Long?,
until: Long?,
limit: Int?,
) = RelayBasedFilter(
relay = relay,
filter =
Filter(
kinds = listOf(PrivateDmEvent.KIND),
authors = listOf(account.userProfile().pubkeyHex),
tags = mapOf("p" to pTags.toList()),
since = since,
until = until,
limit = limit,
),
)
/** Live-tail filters: everything since [windowStart], open-ended at the top (new messages keep arriving). */
fun filterNip04DMs(
group: Set<HexKey>?,
account: Account?,
windowStart: Long,
): List<RelayBasedFilter>? {
if (group.isNullOrEmpty() || account == null) return null
val relays = nip04DmRelayRouting(group, account) ?: return null
return relays.toMeRelays.map { (relay, authors) -> toMeFilter(relay, authors, account, since = windowStart, until = null, limit = null) } +
relays.fromMeRelays.map { (relay, pTags) -> fromMeFilter(relay, pTags, account, since = windowStart, until = null, limit = null) }
}
/**
* History filters: a bounded backward page per relay. Each relay is asked for [limit] events older
* than [untilFor]`(relay)` (no `since`), so it can be paged down to empty independently. The author /
* `#p` key set per relay comes straight from [relays], so each relay only sees the keys it owns.
*/
fun filterNip04DMsHistory(
account: Account,
relays: Nip04DmRelayRouting,
limit: Int,
untilFor: (NormalizedRelayUrl) -> Long?,
): List<RelayBasedFilter> =
relays.toMeRelays.map { (relay, authors) -> toMeFilter(relay, authors, account, since = null, until = untilFor(relay), limit = limit) } +
relays.fromMeRelays.map { (relay, pTags) -> fromMeFilter(relay, pTags, account, since = null, until = untilFor(relay), limit = limit) }
@@ -122,7 +122,7 @@ class ChatroomListNewFeedFilter(
if (room != null &&
(
newNote.author?.pubkeyHex != me.pubkeyHex &&
room.senderIntersects(followingKeySet) &&
!room.senderIntersects(followingKeySet) &&
!account.chatroomList.hasSentMessagesTo(roomKey)
) &&
!account.isAllHidden(roomKey.users)
@@ -35,9 +35,16 @@ class ChatroomListState(
class ChatroomListFilterAssembler(
client: INostrClient,
) : ComposeSubscriptionManager<ChatroomListState>() {
// NIP-04 live tail: the recent week, always open at the top.
val nip04 = ChatroomListNip04SubAssembler(client, ::allKeys)
// NIP-04 history: older DMs, paged backward by until+limit, independently of gift wraps.
val nip04History = ChatroomListNip04HistorySubAssembler(client, ::allKeys)
val group =
listOf(
DMsFromUserFilterSubAssembler(client, ::allKeys),
nip04,
nip04History,
FollowingPublicChatSubAssembler(client, ::allKeys),
FollowingEphemeralChatSubAssembler(client, ::allKeys),
)
@@ -0,0 +1,145 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.rooms.datasource
import com.vitorpamplona.amethyst.commons.relayClient.paging.BackwardRelayPager
import com.vitorpamplona.amethyst.commons.relayClient.paging.PagingStatus
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.flow.StateFlow
/**
* Loads older NIP-04 DMs (kind 4) for the rooms list by **`until`+`limit` paging, per relay, on
* demand** the same model as the gift-wrap history loader
* ([com.vitorpamplona.amethyst.service.relayClient.reqCommand.account.nip59GiftWraps.AccountGiftWrapsHistoryEoseManager]),
* across the account's home (outbox, *from me*) + DM (inbox, *to me*) relays. Each relay advances one
* page when its on-screen window-limit marker asks ([advance]); otherwise it parks. Nothing is walked
* proactively. The per-relay cursors live on the account's
* [ChatroomList][com.vitorpamplona.amethyst.commons.model.privateChats.ChatroomList]; the single-active
* [BackwardRelayPager] orchestrator binds to them on [newSub].
*/
class ChatroomListNip04HistorySubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomListState>,
) : PerUserEoseManager<ChatroomListState>(client, allKeys) {
private fun allRelays(account: Account) = (account.homeRelays.flow.value + account.dmRelays.flow.value).toSet()
private val pager = BackwardRelayPager("rooms.nip04.history")
val loadingMore: StateFlow<Boolean> = pager.loadingMore
val status: StateFlow<PagingStatus> = pager.status
override fun user(key: ChatroomListState) = key.account.userProfile()
override fun updateFilter(
key: ChatroomListState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? {
val user = user(key)
if (!key.account.isWriteable()) return emptyList()
val homeRelays = key.account.homeRelays.flow.value
val dmRelays = key.account.dmRelays.flow.value
val armed = pager.armedRelays((homeRelays + dmRelays).toSet())
if (armed.isEmpty()) return emptyList()
DmRelayLog.log("rooms.nip04.history", key.account)
return armed.flatMap { relay ->
val until = pager.requestedUntilFor(relay) ?: return@flatMap emptyList()
buildList {
if (relay in homeRelays) add(filterNip04DMsFromMe(user, relay, since = null, until = until, limit = pager.pageLimit))
if (relay in dmRelays) add(filterNip04DMsToMe(user, relay, since = null, until = until, limit = pager.pageLimit))
}
}
}
/** Steps a single [relay] to its next, older page. Driven by that relay's on-screen window-limit marker. */
fun advance(relay: NormalizedRelayUrl) {
if (pager.advance(relay)) invalidateFilters()
}
/** Steps every not-done, not-in-flight relay one page. For the empty/initial boundary (nothing to scroll). */
fun advanceAll() {
if (pager.advanceAll()) {
Log.d("DMPagination") { "[rooms.nip04.history] advanceAll (empty-feed bootstrap)" }
invalidateFilters()
}
}
override fun newSub(key: ChatroomListState): Subscription {
// Repoint the single-active orchestrator at this account's rooms-list NIP-04 cursors (on its
// ChatroomList) and the relays it fans out to, refreshing the flows from the restored progress.
pager.bind(key.account.chatroomList.nip04History, key.account.scope) { allRelays(key.account) }
return requestNewSubscription(historyListener(key))
}
private fun historyListener(key: ChatroomListState): SubscriptionListener {
// A just-backgrounded account's subscription can still deliver after the orchestrator rebinds to
// another account; gate the pager (single-active) on whether it's still bound to THIS account's
// cursors so a late callback can't move another account's cursors. newEose runs regardless.
val myCursors = key.account.chatroomList.nip04History
return object : SubscriptionListener {
override fun onEvent(
event: Event,
isLive: Boolean,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onEvent(relay, event.createdAt)
}
override fun onEose(
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors) && pager.onEose(relay)) {
Log.d("DMPagination") { "[rooms.nip04.history] ${relay.url} reached the bottom (done)" }
}
newEose(key, relay, TimeUtils.now(), forFilters)
}
override fun onClosed(
message: String,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onClosed(relay, message)
}
override fun onCannotConnect(
relay: NormalizedRelayUrl,
message: String,
forFilters: List<Filter>?,
) {
if (pager.isBoundTo(myCursors)) pager.onCannotConnect(relay, message)
}
}
}
}
@@ -20,60 +20,79 @@
*/
package com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.rooms.datasource
import com.vitorpamplona.amethyst.commons.model.privateChats.DmHistoryTuning
import com.vitorpamplona.amethyst.commons.relayClient.paging.WindowLoadTracker
import com.vitorpamplona.amethyst.commons.relayClient.paging.trackingListener
import com.vitorpamplona.amethyst.model.User
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.DmRelayLog
import com.vitorpamplona.amethyst.service.relayClient.eoseManagers.PerUserEoseManager
import com.vitorpamplona.amethyst.service.relays.SincePerRelayMap
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.pool.RelayBasedFilter
import com.vitorpamplona.quartz.nip01Core.relay.client.subscriptions.Subscription
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.Job
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.collectLatest
import kotlinx.coroutines.launch
class DMsFromUserFilterSubAssembler(
/**
* Always-on **live tail** for the account's NIP-04 DMs (kind 4) in the rooms list. Mirrors the
* gift-wrap live tail: a fixed one-week floor, no upper bound, never widens. Older NIP-04 history is
* loaded by [ChatroomListNip04HistorySubAssembler] in bounded slices.
*/
class ChatroomListNip04SubAssembler(
client: INostrClient,
allKeys: () -> Set<ChatroomListState>,
) : PerUserEoseManager<ChatroomListState>(client, allKeys) {
private val windowLoad = WindowLoadTracker("rooms.nip04.live")
val loadingMore: StateFlow<Boolean> = windowLoad.loading
override fun updateFilter(
key: ChatroomListState,
since: SincePerRelayMap?,
): List<RelayBasedFilter>? =
if (key.account.isWriteable()) {
key.account.homeRelays.flow.value.map {
filterNip04DMsFromMe(key.account.userProfile(), it, since?.get(it)?.time)
} +
key.account.dmRelays.flow.value.map {
filterNip04DMsToMe(key.account.userProfile(), it, since?.get(it)?.time)
}
val homeRelays = key.account.homeRelays.flow.value
val dmRelays = key.account.dmRelays.flow.value
windowLoad.setExpectedRelays((homeRelays + dmRelays).toSet())
val sinceTime = DmHistoryTuning.recentBoundary()
DmRelayLog.log("rooms.nip04.live", key.account)
Log.d("DMPagination") { "[rooms.nip04.live] REQ since=$sinceTime (no until) fromMe(outbox)=${homeRelays.map { it.url }} toMe(inbox)=${dmRelays.map { it.url }}" }
homeRelays.map { filterNip04DMsFromMe(key.account.userProfile(), it, sinceTime) } +
dmRelays.map { filterNip04DMsToMe(key.account.userProfile(), it, sinceTime) }
} else {
windowLoad.setExpectedRelays(emptySet())
emptyList()
}
override fun user(key: ChatroomListState) = key.account.userProfile()
val userJobMap = mutableMapOf<User, List<Job>>()
private val userJobMap = mutableMapOf<User, List<Job>>()
@OptIn(FlowPreview::class)
override fun newSub(key: ChatroomListState): Subscription {
val user = user(key)
windowLoad.startLoading(key.account.scope)
userJobMap[user]?.forEach { it.cancel() }
userJobMap[user] =
listOf(
key.account.scope.launch(Dispatchers.IO) {
key.account.homeRelays.flow.collectLatest {
invalidateFilters()
}
key.account.homeRelays.flow
.collectLatest { invalidateFilters() }
},
key.account.scope.launch(Dispatchers.IO) {
key.account.dmRelays.flow.collectLatest {
invalidateFilters()
}
key.account.dmRelays.flow
.collectLatest { invalidateFilters() }
},
)
return super.newSub(key)
return requestNewSubscription(
windowLoad.trackingListener { relay, filters -> newEose(key, relay, TimeUtils.now(), filters) },
)
}
override fun endSub(
@@ -30,6 +30,8 @@ fun filterNip04DMsFromMe(
user: User,
relay: NormalizedRelayUrl,
since: Long?,
until: Long? = null,
limit: Int? = null,
): RelayBasedFilter =
RelayBasedFilter(
relay = relay,
@@ -38,5 +40,7 @@ fun filterNip04DMsFromMe(
kinds = listOf(PrivateDmEvent.KIND),
authors = listOf(user.pubkeyHex),
since = since,
until = until,
limit = limit,
),
)
@@ -30,6 +30,8 @@ fun filterNip04DMsToMe(
user: User,
relay: NormalizedRelayUrl,
since: Long?,
until: Long? = null,
limit: Int? = null,
): RelayBasedFilter =
RelayBasedFilter(
relay = relay,
@@ -38,5 +40,7 @@ fun filterNip04DMsToMe(
kinds = listOf(PrivateDmEvent.KIND),
tags = mapOf("p" to listOf(user.pubkeyHex)),
since = since,
until = until,
limit = limit,
),
)
@@ -25,15 +25,29 @@ import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.LazyListState
import androidx.compose.foundation.lazy.items
import androidx.compose.foundation.lazy.itemsIndexed
import androidx.compose.material3.HorizontalDivider
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.res.stringResource
import androidx.lifecycle.compose.collectAsStateWithLifecycle
import com.vitorpamplona.amethyst.R
import com.vitorpamplona.amethyst.commons.model.marmotGroups.MarmotGroupChatroom
import com.vitorpamplona.amethyst.commons.relayClient.paging.PagingStatus
import com.vitorpamplona.amethyst.commons.ui.feeds.DmHistoryLoadingCard
import com.vitorpamplona.amethyst.commons.ui.feeds.FeedContentState
import com.vitorpamplona.amethyst.commons.ui.feeds.FeedState
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachCursor
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachDetailDialog
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachMarkers
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachSentinels
import com.vitorpamplona.amethyst.commons.ui.feeds.RelayReachState
import com.vitorpamplona.amethyst.model.Note
import com.vitorpamplona.amethyst.ui.actions.CrossfadeIfEnabled
import com.vitorpamplona.amethyst.ui.feeds.FeedEmpty
@@ -44,15 +58,25 @@ import com.vitorpamplona.amethyst.ui.feeds.SaveableFeedContentState
import com.vitorpamplona.amethyst.ui.layouts.rememberFeedContentPadding
import com.vitorpamplona.amethyst.ui.navigation.navs.INav
import com.vitorpamplona.amethyst.ui.screen.loggedIn.AccountViewModel
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.feed.formatHistoryReachDate
import com.vitorpamplona.amethyst.ui.screen.loggedIn.chats.rooms.ChatroomHeaderCompose
import com.vitorpamplona.amethyst.ui.theme.DividerThickness
import com.vitorpamplona.amethyst.ui.theme.FeedPadding
import com.vitorpamplona.quartz.experimental.ephemChat.chat.EphemeralChatEvent
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayPagingProgress
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip04Dm.messages.PrivateDmEvent
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKeyable
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelCreateEvent
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelMetadataEvent
import com.vitorpamplona.quartz.nip28PublicChat.message.ChannelMessageEvent
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.combine
import kotlinx.coroutines.flow.distinctUntilChanged
import kotlinx.coroutines.flow.filter
import java.io.Serializable
@Composable
@@ -62,6 +86,10 @@ fun ChatroomListFeedView(
accountViewModel: AccountViewModel,
nav: INav,
) {
DisposableEffect(Unit) {
Log.d("DMPagination") { "rooms.list: OPEN" }
onDispose { Log.d("DMPagination") { "rooms.list: CLOSE" } }
}
RefresheableBox(feedContentState, true) {
SaveableFeedContentState(feedContentState, scrollStateKey) { listState ->
CrossFadeState(feedContentState, listState, accountViewModel, nav)
@@ -78,6 +106,24 @@ private fun CrossFadeState(
) {
val feedState by feedContentState.feedContent.collectAsStateWithLifecycle()
// History is exhausted only once BOTH DM protocols have paged to their end (each stops when a
// round of until+limit pages brings nothing back). Until then an empty feed means "still filling",
// not "no conversations" — keep the spinner up rather than flash empty.
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04History }
val giftWrapsStatus by giftWrapsHistory.status.collectAsStateWithLifecycle()
val nip04Status by nip04History.status.collectAsStateWithLifecycle()
val historyExhausted = giftWrapsStatus.exhausted && nip04Status.exhausted
// A *genuinely* empty list has no rows to host the per-relay window-limit markers, so we step every
// relay one page at a time to hunt for the first rooms. Gated on FeedState.Empty only (never the
// transient Loading that navigation flashes through) and debounced, so re-opening Messages with rooms
// already loaded does NOT kick a hunt. Once rooms appear the markers take over, demand-driven.
val user = accountViewModel.userProfile()
val bootstrap = feedState is FeedState.Empty
BootstrapHistoryWhenEmpty(bootstrap, giftWrapsHistory.loadingMore, giftWrapsHistory.status) { giftWrapsHistory.advanceAll() }
BootstrapHistoryWhenEmpty(bootstrap, nip04History.loadingMore, nip04History.status) { nip04History.advanceAll() }
CrossfadeIfEnabled(
targetState = feedState,
animationSpec = tween(durationMillis = 100),
@@ -85,7 +131,11 @@ private fun CrossFadeState(
) { state ->
when (state) {
is FeedState.Empty -> {
FeedEmpty { feedContentState.invalidateData() }
if (historyExhausted) {
FeedEmpty { feedContentState.invalidateData() }
} else {
LoadingFeed()
}
}
is FeedState.FeedError -> {
@@ -114,14 +164,60 @@ private fun FeedLoaded(
val myPubKey = accountViewModel.userProfile().pubkeyHex
val giftWrapsHistory = remember(accountViewModel) { accountViewModel.dataSources().account.giftWrapsHistory }
val nip04History = remember(accountViewModel) { accountViewModel.dataSources().chatroomList.nip04History }
val loadingGiftWraps by giftWrapsHistory.loadingMore.collectAsStateWithLifecycle()
val loadingNip04 by nip04History.loadingMore.collectAsStateWithLifecycle()
// One atomic snapshot per protocol (exhausted + relays + reached + per-relay progress) instead of six
// separate collectors — the status card and the per-relay markers read all of it together anyway.
val giftWrapsStatus by giftWrapsHistory.status.collectAsStateWithLifecycle()
val nip04Status by nip04History.status.collectAsStateWithLifecycle()
val user = accountViewModel.userProfile()
val nip17Name = stringResource(R.string.chats_history_proto_nip17)
val nip04Name = stringResource(R.string.chats_history_proto_nip04)
val oldestNip17Index = items.list.indexOfLast { it.event is ChatroomKeyable && it.event !is PrivateDmEvent }
val oldestNip04Index = items.list.indexOfLast { it.event is PrivateDmEvent }
// Each relay's window limit, carrying the advance() that pulls its OWN next page. Placed in the list
// at its reached depth as a sentinel (see RelayReachMarkers): a relay pages only while its
// marker is on screen and keeps paging while it stays there, so a spam-dense relay never floods —
// you have to scroll through its messages to pull more. A protocol drops out once exhausted.
val limits =
remember(giftWrapsStatus, nip04Status, user) {
buildList {
if (!giftWrapsStatus.exhausted) {
giftWrapsStatus.relayProgress.forEach { (relay, p) ->
add(RelayReachCursor("17:${relay.url}", relayShortName(relay), p.reachedUntil, reachState(p), "NIP-17") { giftWrapsHistory.advance(relay) })
}
}
if (!nip04Status.exhausted) {
nip04Status.relayProgress.forEach { (relay, p) ->
add(RelayReachCursor("04:${relay.url}", relayShortName(relay), p.reachedUntil, reachState(p), "NIP-04") { nip04History.advance(relay) })
}
}
}
}
// Hoisted load driver: pulls each relay's next page off viewport visibility, so feed reorders
// (a live DM bumping a room) no longer re-fire paging. The markers below are pure UI.
RelayReachSentinels(limits, listState) { index -> items.list.getOrNull(index)?.createdAt() }
// The relays behind a tapped in-stream "Relay sync" marker; non-null shows the per-relay popup so the
// terse divider isn't a dead end — every count/name is one tap from the full breakdown (which relays,
// protocol, how far back each paged).
var syncDetail by remember { mutableStateOf<List<RelayReachCursor>?>(null) }
syncDetail?.let { detail ->
RelayReachDetailDialog(detail, ::formatHistoryReachDate) { syncDetail = null }
}
LazyColumn(
contentPadding = rememberFeedContentPadding(FeedPadding),
state = listState,
) {
items(
itemsIndexed(
items.list,
key = { item -> chatroomLazyKey(item, myPubKey) },
) { item ->
key = { _, item -> chatroomLazyKey(item, myPubKey) },
) { index, item ->
Row(Modifier.fillMaxWidth()) {
ChatroomHeaderCompose(
item,
@@ -133,10 +229,69 @@ private fun FeedLoaded(
HorizontalDivider(
thickness = DividerThickness,
)
// Rendered unconditionally at the protocol's oldest room so the card can run its own
// "All caught up" crossfade-and-collapse when that protocol exhausts.
if (index == oldestNip17Index) {
DmHistoryLoadingCard(nip17Name, "NIP-17", loadingGiftWraps, giftWrapsStatus.exhausted, giftWrapsStatus.relayCount, giftWrapsStatus.stalledCount, giftWrapsStatus.reachedBack, giftWrapsStatus.relayProgress, ::formatHistoryReachDate)
}
if (index == oldestNip04Index) {
DmHistoryLoadingCard(nip04Name, "NIP-04", loadingNip04, nip04Status.exhausted, nip04Status.relayCount, nip04Status.stalledCount, nip04Status.reachedBack, nip04Status.relayProgress, ::formatHistoryReachDate)
}
// Per-relay window-limit markers/sentinels belonging in the gap toward the next-older room:
// each pulls its relay's next page while it's on screen. olderCreatedAt is null past the
// oldest loaded room, so relays that have reached the bottom of the list sit there.
RelayReachMarkers(
limits,
item.createdAt(),
items.list.getOrNull(index + 1)?.createdAt(),
) { syncDetail = it }
}
}
}
/**
* Bootstraps history while the rooms list is genuinely empty: steps every relay one page at a time,
* gated only on its own loader, until rooms appear or the protocol exhausts. Once rooms load this stops
* and the per-relay window-limit markers drive paging on demand.
*
* Leads with a debounce so the brief Empty/Loading flash that navigation passes through does NOT trigger
* a hunt; if [active] drops before it elapses (rooms loaded) the effect cancels and nothing pages.
*/
@Composable
private fun BootstrapHistoryWhenEmpty(
active: Boolean,
loadingMore: StateFlow<Boolean>,
status: StateFlow<PagingStatus>,
advanceAll: () -> Unit,
) {
LaunchedEffect(active, loadingMore, status) {
if (!active) return@LaunchedEffect
delay(BOOTSTRAP_DEBOUNCE_MS)
combine(loadingMore, status) { loading, s -> !loading && !s.exhausted }
.distinctUntilChanged()
.filter { it }
.collect { advanceAll() }
}
}
// Ignore the transient empty feed that navigation flashes through before the rooms re-appear.
private const val BOOTSTRAP_DEBOUNCE_MS = 1200L
private fun reachState(p: RelayPagingProgress) =
when {
p.done -> RelayReachState.DONE
p.stalled -> RelayReachState.STALLED
else -> RelayReachState.REACHING
}
private fun relayShortName(relay: NormalizedRelayUrl): String =
relay.url
.substringAfter("://")
.trimEnd('/')
.substringBefore('/')
// Stable per-chatroom key — derived from chatroom identity, not the latest
// message id, so reorders move the row instead of recreating it. Compose
// stores LazyColumn item keys in a SaveableStateHolder, which on Android
@@ -21,6 +21,7 @@
package com.vitorpamplona.amethyst.ui.tor
import android.content.Context
import com.fasterxml.jackson.module.kotlin.jacksonObjectMapper
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.MutableStateFlow
@@ -70,6 +71,67 @@ class TorService(
}
}
/**
* The on-disk guard sample Arti persists between runs.
* Path: `<filesDir>/arti/state/state/guards.json`.
*/
private fun guardsFile() = File(File(File(artiDataDir(), "state"), "state"), "guards.json")
/**
* Detects the wedged-guard-sample state behind the long-standing "can't
* connect to Tor" bug.
*
* On a flaky network, Arti records circuit failures past the first hop as
* "indeterminate" (it can't tell whether the guard or a later hop was at
* fault). Once a guard's indeterminate ratio crosses 0.7, Arti
* *permanently* disables it (`TooManyIndeterminateFailures`). Disabled
* guards are never re-enabled and never removed from the sample (kept for
* the 60-day confirmed lifetime), and the sample is capped at
* `max_sample_size` (60). Arti normally refills usable guards from the
* network when they drop below `min_filtered_sample_size` (20), but once
* the sample is full of unusable guards there is no room to add more so
* replenishment is permanently wedged and every circuit returns
* `AllGuardsDown`. The state persists in `guards.json`, and bootstrap still
* "succeeds" (it reads cached directory data), so none of the init-failure
* self-heal paths ever fire and Tor is stuck across restarts.
*
* A single usable guard is enough to keep building circuits, so we only
* recover at the last resort: when a non-empty guard set has *zero* usable
* guards. A guard is unusable on disk if it has been permanently
* `disabled` or dropped from the consensus (`unlisted_since` set);
* reachability is in-memory only and not persisted, so it can't be checked
* here. Returns true when at least one non-empty selection has no usable
* guard left.
*/
private fun noUsableGuards(): Boolean {
val file = guardsFile()
if (!file.exists()) return false
return try {
val root = jacksonObjectMapper().readTree(file)
var wedged = false
// Each top-level field is a guard-set selection (e.g. "default").
root.forEach { selection ->
val guards = selection.get("guards") ?: return@forEach
if (guards.isArray && guards.size() > 0) {
val usable =
guards.count { guard ->
val disabled = guard.get("disabled")
val unlisted = guard.get("unlisted_since")
val isDisabled = disabled != null && !disabled.isNull
val isUnlisted = unlisted != null && !unlisted.isNull
!isDisabled && !isUnlisted
}
if (usable == 0) wedged = true
}
}
wedged
} catch (e: Exception) {
Log.w("TorService") { "Could not inspect guards.json: ${e.message}" }
false
}
}
/**
* Clears all Arti persistent data (state + cache). Used as a last resort
* when initialization fails, to recover from corrupted state.
@@ -122,6 +184,16 @@ class TorService(
// fresh network data, preventing stale guards/circuits.
clearArtiCache()
// Self-heal the wedged guard sample (see [noUsableGuards]): if
// the persisted sample has no usable guard left, Arti can
// neither build circuits nor replenish, and would return
// AllGuardsDown forever. Wipe the on-disk state so the next
// bootstrap rebuilds a fresh guard sample.
if (noUsableGuards()) {
Log.w("TorService") { "No usable Arti guards left on disk — wiping state to rebuild the guard sample" }
clearAllArtiData()
}
val dataDir = artiDataDir().absolutePath
Log.d("TorService") { "Initializing Arti with data dir: $dataDir" }
+7
View File
@@ -274,6 +274,13 @@
<string name="generate_a_new_key">Generate a new key</string>
<string name="loading_feed">Loading feed</string>
<string name="loading_account">Loading account</string>
<string name="chats_history_proto_nip17">encrypted</string>
<string name="chats_history_proto_nip04">legacy</string>
<string name="chats_reply_searching_history">Looking for the original message…</string>
<!-- A reply whose target was searched for across all reachable history and never found. -->
<string name="chats_reply_not_found">Couldn\'t find this message</string>
<!-- Reply subtitle when every relay genuinely bottomed out (no stalls) and it still wasn't there. -->
<string name="chats_reply_searched">Searched every relay · tap to see</string>
<string name="error_loading_replies">"Error loading replies: "</string>
<string name="try_again">Try again</string>
<string name="notification_feed_is_empty">No notifications yet.</string>
@@ -0,0 +1,206 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.eoseManagers
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayLoadingCursors
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class RelayLoadingCursorsTest {
private val relayA = RelayUrlNormalizer.normalizeOrNull("wss://a.relay")!!
private val relayB = RelayUrlNormalizer.normalizeOrNull("wss://b.relay")!!
private val start = 1_000L
@Test
fun unarmedRelayIsNotRequestedAndSitsAtTheFloor() {
val cursors = RelayLoadingCursors()
// never advanced, so it carries no REQ
assertEquals(emptyList<Any>(), cursors.armedRelays(listOf(relayA)))
// marker sits at the floor until it delivers
assertEquals(start, cursors.reachedUntilFor(relayA, start))
}
@Test
fun firstAdvanceRequestsTheFloorThenSubsequentPagesStepBelowReached() {
val cursors = RelayLoadingCursors()
assertTrue(cursors.advance(relayA, start))
assertEquals(start, cursors.requestedUntilFor(relayA))
// page returns events; oldest seen = 800
cursors.onEvent(relayA, 900)
cursors.onEvent(relayA, 800)
cursors.onEose(relayA)
assertEquals(800L, cursors.reachedUntilFor(relayA, start))
// EOSE does NOT move the requested cursor — the relay parks at the same filter
assertEquals(start, cursors.requestedUntilFor(relayA))
// next advance steps to reached - 1
assertTrue(cursors.advance(relayA, start))
assertEquals(799L, cursors.requestedUntilFor(relayA))
}
@Test
fun emptyPageMarksRelayDoneAndBlocksFurtherAdvance() {
val cursors = RelayLoadingCursors()
cursors.advance(relayA, start)
cursors.onEose(relayA) // no events
assertTrue(cursors.isDone(relayA))
assertFalse(cursors.advance(relayA, start))
assertEquals(emptyList<Any>(), cursors.armedRelays(listOf(relayA)))
}
@Test
fun aPageThatDoesNotStepOlderEndsTheRelayInsteadOfLooping() {
val cursors = RelayLoadingCursors()
cursors.advance(relayA, start)
cursors.onEvent(relayA, 800)
cursors.onEose(relayA)
assertEquals(800L, cursors.reachedUntilFor(relayA, start))
// misbehaving relay: next page echoes an event no older than what we already reached
cursors.advance(relayA, start) // requested = 799
cursors.onEvent(relayA, 900) // newer than reached(800) — not strictly older
cursors.onEose(relayA)
assertTrue("a non-advancing page should end the relay, not re-loop", cursors.isDone(relayA))
assertEquals(800L, cursors.reachedUntilFor(relayA, start))
}
@Test
fun relaysAreTrackedIndependently() {
val cursors = RelayLoadingCursors()
cursors.advance(relayA, start)
cursors.onEvent(relayA, 500)
cursors.onEose(relayA)
// B never advanced
assertEquals(listOf(relayA), cursors.armedRelays(listOf(relayA, relayB)))
assertEquals(500L, cursors.reachedUntilFor(relayA, start))
assertEquals(start, cursors.reachedUntilFor(relayB, start))
// deepest reached across both = A's 500 (B counts as the floor)
assertEquals(500L, cursors.deepestReached(listOf(relayA, relayB), start))
}
@Test
fun deepestReachedIsNullWhenNoRelays() {
val cursors = RelayLoadingCursors()
assertEquals(null, cursors.deepestReached(emptyList(), start))
}
// ── rewindTo: realign the window after the cache prunes messages out of it ──
@Test
fun rewindReopensThePrunedBandAndResumesFromItOnNextAdvance() {
val cursors = RelayLoadingCursors()
cursors.floor = start
// page deep: floor 1000 → reached 200
cursors.advance(relayA, start)
cursors.onEvent(relayA, 900)
cursors.onEvent(relayA, 200)
cursors.onEose(relayA)
assertEquals(200L, cursors.reachedUntilFor(relayA, start))
// prune drops everything older than 700 (newest pruned = 700)
cursors.rewindTo(mapOf(relayA to 700L))
// reached pulled up to just above the pruned band, not done, and un-armed (demand-driven)
assertEquals(701L, cursors.reachedUntilFor(relayA, start))
assertFalse(cursors.isDone(relayA))
assertEquals(emptyList<Any>(), cursors.armedRelays(listOf(relayA)))
// the next advance resumes at the boundary and re-requests the pruned band (until = 700),
// NOT from the floor (which would re-stream the still-held tail above 700)
assertTrue(cursors.advance(relayA, start))
assertEquals(700L, cursors.requestedUntilFor(relayA))
}
@Test
fun rewindClearsDoneSoAnExhaustedRelayCanReFetch() {
val cursors = RelayLoadingCursors()
cursors.floor = start
cursors.advance(relayA, start)
cursors.onEvent(relayA, 300)
cursors.onEose(relayA) // reached 300
cursors.advance(relayA, start)
cursors.onEose(relayA) // empty page → done
assertTrue(cursors.isDone(relayA))
cursors.rewindTo(mapOf(relayA to 500L))
assertFalse("a pruned relay must be re-fetchable even after it reached the bottom", cursors.isDone(relayA))
assertEquals(501L, cursors.reachedUntilFor(relayA, start))
assertTrue(cursors.advance(relayA, start))
assertEquals(500L, cursors.requestedUntilFor(relayA))
}
@Test
fun rewindNeverClimbsAboveTheFloor() {
val cursors = RelayLoadingCursors()
cursors.floor = start
cursors.advance(relayA, start)
cursors.onEvent(relayA, 300)
cursors.onEose(relayA) // reached 300
// a boundary at/above the floor clamps to the floor (history lives strictly below it)
cursors.rewindTo(mapOf(relayA to start))
assertEquals(start, cursors.reachedUntilFor(relayA, start))
}
@Test
fun rewindSkipsRelaysWithoutACursorOrShallowerThanThePrunedBand() {
val cursors = RelayLoadingCursors()
cursors.floor = start
// A delivered to 200; B never paged
cursors.advance(relayA, start)
cursors.onEvent(relayA, 200)
cursors.onEose(relayA)
// B has no cursor (never paged) → skipped, no entry minted; A's reach (200) is already shallower
// than a boundary of 150 (target 151), so it needs no rewind either.
cursors.rewindTo(mapOf(relayB to 500L, relayA to 150L))
// B: untouched (still at the floor, unarmed)
assertEquals(start, cursors.reachedUntilFor(relayB, start))
assertEquals(emptyList<Any>(), cursors.armedRelays(listOf(relayB)))
// A: unchanged
assertEquals(200L, cursors.reachedUntilFor(relayA, start))
}
@Test
fun rewindIsANoOpWhenTheWindowNeverPagedHistory() {
val cursors = RelayLoadingCursors()
// floor is null (never advanced any history page)
cursors.advance(relayA, start)
cursors.onEvent(relayA, 200)
cursors.onEose(relayA)
cursors.rewindTo(mapOf(relayA to 150L))
// unchanged: with no pinned floor there is no history window to realign
assertEquals(200L, cursors.reachedUntilFor(relayA, start))
}
}
@@ -0,0 +1,84 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.eoseManagers
import com.vitorpamplona.amethyst.commons.relayClient.paging.WindowLoadTracker
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import org.junit.Assert.assertTrue
import org.junit.Test
import kotlin.time.Duration.Companion.milliseconds
/**
* Real-time tests for the **idle** backstop: when a relay streams stored events but never sends EOSE,
* the window can't complete on "every relay settled", so the idle timer finishes it once the stream
* goes quiet but only after every still-pending relay has been *heard from*, so a slow connect (a
* relay that hasn't answered yet) is never mistaken for a stream that has gone quiet.
*/
class WindowLoadTrackerIdleTest {
private val good = NormalizedRelayUrl("wss://vitor.nostr1.com/")
private val streamer = NormalizedRelayUrl("wss://relay.damus.io/")
@Test
fun idleBackstopCompletesARelayThatStreamsButNeverEoses() =
runBlocking {
val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
val tracker = WindowLoadTracker(name = "test", idleTimeout = 50.milliseconds)
tracker.startLoading(scope)
tracker.setExpectedRelays(setOf(good, streamer))
tracker.onRelaySettled(good) // `good` finishes with an EOSE
// `streamer` keeps delivering stored events but never sends EOSE — so "every relay settled"
// can never complete this window. Its events keep it "heard from" (idle gate satisfied).
tracker.onRelayEvent(streamer)
tracker.onRelayEvent(streamer)
// The only way out is the idle backstop, once the stream stays quiet for idleTimeout.
withTimeout(3000) { tracker.loading.first { !it } }
scope.cancel()
}
@Test
fun idleBackstopWaitsUntilEveryPendingRelayHasBeenHeardFrom() =
runBlocking {
val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
val tracker = WindowLoadTracker(name = "test", idleTimeout = 50.milliseconds)
tracker.startLoading(scope)
tracker.setExpectedRelays(setOf(good, streamer))
tracker.onRelaySettled(good)
// `streamer` has NOT been heard from yet (still connecting). The idle gate must hold the window
// open — a connection gap is not a quiet stream. Wait well past idleTimeout AND a watchdog tick.
Thread.sleep(800)
assertTrue("idle must not fire while a pending relay has never been heard from", tracker.loading.value)
// Once it delivers something (now heard-from) and the stream goes quiet, idle completes it.
tracker.onRelayEvent(streamer)
withTimeout(3000) { tracker.loading.first { !it } }
scope.cancel()
}
}
@@ -55,6 +55,28 @@
<string name="accessibility_user_avatar">User avatar</string>
<string name="accessibility_navigate">Navigate</string>
<!-- Relay history paging (shared feed markers + status card) -->
<string name="chats_history_loading_label">Loading:</string>
<string name="chats_history_fully_loaded_label">Fully loaded:</string>
<string name="chats_history_fully_loaded">(fully loaded)</string>
<string name="chats_history_by_relay">History by relay</string>
<string name="chats_history_stalled_retry">Will retry when you reopen this screen</string>
<string name="chats_history_older">Older %1$s messages</string>
<string name="chats_history_all_caught_up">All caught up</string>
<string name="chats_history_reached_start">Reached the start of your %1$s messages</string>
<string name="chats_history_subtitle">%1$s · %2$s · loaded since %3$s</string>
<string name="chats_history_subtitle_no_date">%1$s · %2$s</string>
<string name="chats_history_waiting">waiting on %1$s</string>
<string name="chats_history_incomplete">Some relays didn\'t respond</string>
<string name="chats_history_incomplete_sub">%1$s unreachable · tap to see which</string>
<string name="chats_history_relays_title">%1$s · history by relay</string>
<string name="chats_history_relay_since">since %1$s</string>
<string name="action_dismiss">Dismiss</string>
<plurals name="chats_history_relays">
<item quantity="one">%1$d relay</item>
<item quantity="other">%1$d relays</item>
</plurals>
<!-- Notes & Replies -->
<string name="replying_to">replying to </string>
</resources>
@@ -30,9 +30,10 @@ import com.vitorpamplona.amethyst.commons.util.KmpLock
import com.vitorpamplona.amethyst.commons.util.WeakReference
import com.vitorpamplona.amethyst.commons.util.withLock
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayLoadingCursors
import com.vitorpamplona.quartz.nip04Dm.messages.PrivateDmEvent
import com.vitorpamplona.quartz.nip14Subject.subject
import com.vitorpamplona.quartz.utils.TimeUtils
import com.vitorpamplona.quartz.nip59Giftwrap.WrappedEvent
import kotlinx.coroutines.channels.BufferOverflow
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.MutableStateFlow
@@ -46,6 +47,12 @@ class Chatroom : NotesGatherer {
var ownerSentMessage: Boolean = false
var newestMessage: Note? = null
// Per-conversation NIP-04 history paging cursors, held here so reopening this room keeps its
// progress and the cursors share the lifetime of the cached messages. The conversation history
// loader binds its (single-active) orchestrator to this. Lazy — most rooms in the rooms list are
// never opened for history paging, so they never allocate it.
val nip04History by lazy { RelayLoadingCursors() }
// Per-instance lock shared by previously @Synchronized methods.
private val syncLock = KmpLock()
@@ -132,13 +139,17 @@ class Chatroom : NotesGatherer {
val sorted = messages.sortedWith(DefaultFeedOrder)
val toKeep =
if ((sorted.firstOrNull()?.createdAt() ?: 0L) > TimeUtils.oneWeekAgo()) {
// Recent messages, keep last 100
sorted.take(100).toSet()
if ((sorted.firstOrNull()?.createdAt() ?: 0L) > DmHistoryTuning.recentBoundary()) {
// Recent conversation, keep its newest N
sorted.take(DmHistoryTuning.recentKeepCount).toSet()
} else {
// Old messages, keep the last one.
// Old conversation, keep the last one.
sorted.take(1).toSet()
} + sorted.filter { it.flowSet?.isInUse() ?: false } + sorted.filter { it.event !is PrivateDmEvent }
} + sorted.filter { it.flowSet?.isInUse() ?: false } + sorted.filter { it.event !is PrivateDmEvent && it.event !is WrappedEvent }
// Both DM protocols are pruned by the recency rule above: NIP-04 (PrivateDmEvent) and NIP-17
// (WrappedEvent rumors — ChatMessageEvent / file headers). Anything else that ever lands in a
// room is kept. The caller realigns the per-relay download window for the dropped messages so
// they can be paged again later (see LocalCache.pruneOldMessages + RelayLoadingCursors.rewindTo).
val toRemove = messages.minus(toKeep)
messages = toKeep
@@ -23,6 +23,7 @@ package com.vitorpamplona.amethyst.commons.model.privateChats
import com.vitorpamplona.amethyst.commons.model.Note
import com.vitorpamplona.amethyst.commons.model.User
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayLoadingCursors
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKey
import com.vitorpamplona.quartz.nip17Dm.base.ChatroomKeyable
import com.vitorpamplona.quartz.utils.cache.LargeCache
@@ -34,6 +35,13 @@ class ChatroomList(
var rooms = LargeCache<ChatroomKey, Chatroom>()
private set
// Account-level DM history paging cursors (one scope per account), held here so they share the
// lifetime of the cached messages and are dropped when the cache prunes them. The account-level
// history loaders bind their orchestrator to these. (Per-conversation NIP-04 cursors live on the
// individual [Chatroom] instead.)
val giftWrapHistory = RelayLoadingCursors()
val nip04History = RelayLoadingCursors()
private fun getOrCreatePrivateChatroomSync(key: ChatroomKey): Chatroom = rooms.getOrCreate(key) { Chatroom() }
fun getOrCreatePrivateChatroom(user: User): Chatroom {
@@ -0,0 +1,49 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.model.privateChats
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlin.concurrent.Volatile
/**
* One shared boundary for the DM history window, so the three things that must agree on "where the
* live tail ends and paged history begins" actually read the same number:
* - the live-tail subscriptions' `since` floor (e.g. `AccountGiftWrapsEoseManager`),
* - the backward history pager's pinned floor (`BackwardRelayPager`),
* - the memory prune's recent/old split + retention cap (`Chatroom.pruneMessagesToTheLatestOnly`).
*
* Keeping them in one place avoids the live tail and history overlapping (double-loading) or leaving a
* gap when the boundary is tuned. The knobs are plain `@Volatile` vars (overridable once at startup
* e.g. shrinking the window in a test to exercise pruning + the per-relay download-window realignment
* without needing week-old threads); they are not meant to change mid-session.
*/
object DmHistoryTuning {
/** Seconds below "now" where the live tail ends and paged history begins. Production: one week. */
@Volatile
var liveTailSeconds: Long = 7L * TimeUtils.ONE_DAY
/** How many newest messages a still-recent conversation keeps on a prune. Production: 100. */
@Volatile
var recentKeepCount: Int = 100
/** The epoch-seconds boundary `now [liveTailSeconds]` (recomputed each call against the clock). */
fun recentBoundary(): Long = TimeUtils.now() - liveTailSeconds
}
@@ -32,6 +32,8 @@ fun filterGiftWrapsToPubkey(
relay: NormalizedRelayUrl,
pubkey: HexKey?,
since: Long?,
until: Long? = null,
limit: Int? = null,
): List<RelayBasedFilter> {
if (pubkey.isNullOrEmpty()) return emptyList()
@@ -42,7 +44,13 @@ fun filterGiftWrapsToPubkey(
Filter(
kinds = listOf(GiftWrapEvent.KIND, EphemeralGiftWrapEvent.KIND),
tags = mapOf("p" to listOf(pubkey)),
// A gift wrap's outer created_at is randomized up to 2 days before the real
// message time, so widen the lower bound by 2 days to catch wraps for messages
// right at the floor. (The upper bound needs no margin: a slice's `until` is the
// previous slice's un-margined floor, so the 2-day overlap already covers the seam.)
since = since?.minus(TimeUtils.twoDays()),
until = until,
limit = limit,
),
),
)
@@ -0,0 +1,434 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.ui.feeds
import androidx.compose.animation.AnimatedVisibility
import androidx.compose.animation.Crossfade
import androidx.compose.animation.core.tween
import androidx.compose.animation.fadeIn
import androidx.compose.animation.fadeOut
import androidx.compose.animation.shrinkVertically
import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.CircularProgressIndicator
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import com.vitorpamplona.amethyst.commons.resources.Res
import com.vitorpamplona.amethyst.commons.resources.action_dismiss
import com.vitorpamplona.amethyst.commons.resources.chats_history_all_caught_up
import com.vitorpamplona.amethyst.commons.resources.chats_history_by_relay
import com.vitorpamplona.amethyst.commons.resources.chats_history_incomplete
import com.vitorpamplona.amethyst.commons.resources.chats_history_incomplete_sub
import com.vitorpamplona.amethyst.commons.resources.chats_history_older
import com.vitorpamplona.amethyst.commons.resources.chats_history_reached_start
import com.vitorpamplona.amethyst.commons.resources.chats_history_relay_since
import com.vitorpamplona.amethyst.commons.resources.chats_history_relays
import com.vitorpamplona.amethyst.commons.resources.chats_history_relays_title
import com.vitorpamplona.amethyst.commons.resources.chats_history_stalled_retry
import com.vitorpamplona.amethyst.commons.resources.chats_history_subtitle
import com.vitorpamplona.amethyst.commons.resources.chats_history_subtitle_no_date
import com.vitorpamplona.amethyst.commons.resources.chats_history_waiting
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayPagingProgress
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import kotlinx.coroutines.delay
import org.jetbrains.compose.resources.pluralStringResource
import org.jetbrains.compose.resources.stringResource
// How long the "All caught up" state lingers before the card collapses away.
private const val ALL_DONE_VISIBLE_MS = 2200L
/**
* The "older history" status card for a per-relay [BackwardRelayPager]-backed feed, shown at one
* protocol's oldest-loaded boundary (rooms list and conversation). It tells the user exactly what the
* app is reaching for: which protocol, how many relays it is asking, and how far back it has paged.
* When that protocol runs dry it does NOT just vanish it crossfades to an "All caught up" state,
* holds for a beat, then collapses away. When it stops short because relays stalled it says so and
* stays put (an *incomplete* window messages may still be out there).
*
* Shared across front ends; pass the platform's locale date formatter as [formatReachDate] so the card
* carries no `java.time` / `NSDateFormatter` dependency.
*
* @param protocolName human label woven into sentences, e.g. "encrypted" / "legacy".
* @param protocolTag short technical tag for the subtitle, e.g. "NIP-17" / "NIP-04".
* @param reachedBack epoch seconds of the oldest point reached so far (the deepest `until` cursor).
* @param relayProgress per-relay reach (where each relay's window is, done/stalled). Tapping the card
* opens a popup listing them; pass empty to make the card non-interactive.
* @param formatReachDate formats an epoch-seconds reach point to a short label (e.g. "Jun 2026").
*/
@Composable
fun DmHistoryLoadingCard(
protocolName: String,
protocolTag: String,
loading: Boolean,
exhausted: Boolean,
relayCount: Int,
stalledCount: Int,
reachedBack: Long?,
relayProgress: Map<NormalizedRelayUrl, RelayPagingProgress> = emptyMap(),
formatReachDate: (epochSeconds: Long) -> String,
modifier: Modifier = Modifier,
) {
// Exhausted ("nothing more reachable right now") splits two ways and must NOT read the same:
// - caughtUp: every relay genuinely bottomed out (empty page). This is the real "all caught up".
// - incomplete: we stopped only because some relays are stalled (auth-walled / offline / silent),
// so messages may still be out there. It must say so, stay put, and let the user tap to see which.
val caughtUp = exhausted && stalledCount <= 0
val incomplete = exhausted && stalledCount > 0
// Relays that still have older history to pull: not done and not stalled. Unlike [relayCount] (only
// those fetching a page *right now*), this also counts relays that returned a page and PARKED — the
// `⋯` paused state — so the subtitle doesn't read as a bare tag while the tap-popup lists N relays.
val reaching = remember(relayProgress) { relayProgress.values.count { !it.done && !it.stalled } }
// Only the genuine caught-up state lingers then collapses; an incomplete window stays so it can be acted on.
var collapsed by remember { mutableStateOf(false) }
LaunchedEffect(caughtUp) {
collapsed =
if (caughtUp) {
delay(ALL_DONE_VISIBLE_MS)
true
} else {
false
}
}
var showRelays by remember { mutableStateOf(false) }
if (showRelays) {
DmHistoryRelayDialog(protocolTag, relayProgress, formatReachDate) { showRelays = false }
}
AnimatedVisibility(
visible = !collapsed,
modifier = modifier,
enter = fadeIn(),
exit = shrinkVertically(tween(400)) + fadeOut(tween(250)),
) {
Surface(
modifier =
Modifier
.fillMaxWidth()
.padding(horizontal = 16.dp, vertical = 8.dp)
.then(if (relayProgress.isNotEmpty()) Modifier.clickable { showRelays = true } else Modifier),
shape = RoundedCornerShape(14.dp),
color = MaterialTheme.colorScheme.surfaceVariant.copy(alpha = 0.55f),
tonalElevation = 2.dp,
) {
val phase =
when {
caughtUp -> HistoryPhase.CaughtUp
incomplete -> HistoryPhase.Incomplete
else -> HistoryPhase.Loading
}
Crossfade(targetState = phase, animationSpec = tween(500), label = "dmHistoryState") { state ->
Row(
Modifier
.fillMaxWidth()
.padding(horizontal = 14.dp, vertical = 12.dp),
verticalAlignment = Alignment.CenterVertically,
) {
Box(Modifier.size(22.dp), contentAlignment = Alignment.Center) {
when (state) {
HistoryPhase.CaughtUp ->
Text(
"",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold,
color = MaterialTheme.colorScheme.primary,
)
HistoryPhase.Incomplete ->
// Same glyph the per-relay dialog uses for a stalled relay, same error colour —
// signals "stopped early, some relays didn't answer", not "done".
Text(
"",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold,
color = MaterialTheme.colorScheme.error,
)
HistoryPhase.Loading ->
if (loading) {
CircularProgressIndicator(Modifier.size(18.dp), strokeWidth = 2.dp)
} else {
// Paused: not caught up, but not actively loading (the auto-fill stopped short
// of exhaustion, or we're between pages). Show a static "more" glyph so the
// icon slot is never blank — resumes on scroll.
Text(
"",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
}
}
Spacer(Modifier.width(14.dp))
Column(Modifier.weight(1f)) {
Text(
text =
when (state) {
HistoryPhase.CaughtUp -> stringResource(Res.string.chats_history_all_caught_up)
HistoryPhase.Incomplete -> stringResource(Res.string.chats_history_incomplete)
HistoryPhase.Loading -> stringResource(Res.string.chats_history_older, protocolName)
},
style = MaterialTheme.typography.bodyMedium,
fontWeight = FontWeight.SemiBold,
color = MaterialTheme.colorScheme.onSurface,
)
Text(
text =
when (state) {
HistoryPhase.CaughtUp -> stringResource(Res.string.chats_history_reached_start, protocolName)
HistoryPhase.Incomplete -> incompleteSubtitle(stalledCount)
HistoryPhase.Loading -> historySubtitle(protocolTag, reaching, stalledCount, reachedBack, formatReachDate)
},
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
}
}
}
}
}
/** The three terminal-vs-loading faces of the history card: still loading/paused, genuinely caught up, or
* stopped early because relays stalled. Kept distinct so an incomplete window never reads as "all caught up". */
private enum class HistoryPhase { Loading, CaughtUp, Incomplete }
/** Subtitle for the "stopped early" state: how many relays we couldn't reach, with a hint to tap for the list.
* Shared with the reply placeholder so both read identically. */
@Composable
fun incompleteSubtitle(stalledCount: Int): String =
stringResource(
Res.string.chats_history_incomplete_sub,
pluralStringResource(Res.plurals.chats_history_relays, stalledCount, stalledCount),
)
@Composable
fun historySubtitle(
protocolTag: String,
relayCount: Int,
stalledCount: Int,
reachedBack: Long?,
formatReachDate: (epochSeconds: Long) -> String,
): String {
val backLabel = remember(reachedBack) { reachedBack?.let(formatReachDate) }
// Middle segment: relays still working on it — fetching a page OR parked with more to pull ("N relays")
// — or, when none are reaching but some can't be reached, what we're waiting on ("waiting on N relays").
// With neither (every relay done), just the tag. [relayCount] here is the reaching count, not in-flight.
val middle =
when {
relayCount > 0 -> pluralStringResource(Res.plurals.chats_history_relays, relayCount, relayCount)
stalledCount > 0 ->
stringResource(
Res.string.chats_history_waiting,
pluralStringResource(Res.plurals.chats_history_relays, stalledCount, stalledCount),
)
else -> return protocolTag
}
return if (backLabel != null) {
stringResource(Res.string.chats_history_subtitle, protocolTag, middle, backLabel)
} else {
stringResource(Res.string.chats_history_subtitle_no_date, protocolTag, middle)
}
}
/**
* Popup shown when the history card is tapped: one row per relay with its state glyph ( done, stalled,
* still reaching) and how far back it has paged ("since <date>"), deepest-reaching first. A stalled relay
* also gets a one-line hint that it retries when the screen is reopened.
*/
@Composable
fun DmHistoryRelayDialog(
protocolTag: String,
relayProgress: Map<NormalizedRelayUrl, RelayPagingProgress>,
formatReachDate: (epochSeconds: Long) -> String,
onDismiss: () -> Unit,
) {
val rows = remember(relayProgress) { relayProgress.entries.sortedBy { it.value.reachedUntil } }
AlertDialog(
onDismissRequest = onDismiss,
confirmButton = {
TextButton(onClick = onDismiss) { Text(stringResource(Res.string.action_dismiss)) }
},
title = { Text(stringResource(Res.string.chats_history_relays_title, protocolTag)) },
text = {
Column(
Modifier
.heightIn(max = 360.dp)
.verticalScroll(rememberScrollState()),
) {
rows.forEach { (relay, p) ->
Column(Modifier.fillMaxWidth().padding(vertical = 6.dp)) {
Row(verticalAlignment = Alignment.CenterVertically) {
Text(
text = relayStateGlyph(p),
color = relayStateColor(p),
fontWeight = FontWeight.Bold,
modifier = Modifier.width(22.dp),
)
Text(
text = relayShortName(relay),
modifier = Modifier.weight(1f),
style = MaterialTheme.typography.bodyMedium,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
Spacer(Modifier.width(8.dp))
Text(
text = stringResource(Res.string.chats_history_relay_since, formatReachDate(p.reachedUntil)),
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
maxLines = 1,
)
}
if (p.stalled) StalledRetryHint()
}
}
}
},
)
}
/** The "stopped early, retries on reopen" caption shown under a stalled relay in the per-relay popups.
* Retry is demand-driven (no timer): reopening the screen re-binds and clears the stalled set, which
* retries it so that's what we tell the user rather than a countdown we can't honour. */
@Composable
private fun StalledRetryHint() {
Text(
text = stringResource(Res.string.chats_history_stalled_retry),
style = MaterialTheme.typography.labelSmall,
color = MaterialTheme.colorScheme.error,
modifier = Modifier.padding(start = 22.dp, top = 2.dp),
)
}
private fun relayStateGlyph(p: RelayPagingProgress) =
when {
p.done -> ""
p.stalled -> ""
else -> ""
}
@Composable
private fun relayStateColor(p: RelayPagingProgress): Color =
when {
p.done -> MaterialTheme.colorScheme.primary
p.stalled -> MaterialTheme.colorScheme.error
else -> MaterialTheme.colorScheme.onSurfaceVariant
}
private fun relayShortName(relay: NormalizedRelayUrl): String =
relay.url
.substringAfter("://")
.trimEnd('/')
.substringBefore('/')
/**
* Popup shown when an in-stream "Loading" marker is tapped: the relays whose window sits at that point
* in the stream, each with its protocol tag, state glyph ( done · stalled · reaching) and how far
* back it has paged so the otherwise-terse `Loading: N` divider stops being a dead end and its
* meaning is explorable. A stalled relay also gets the retry-on-reopen hint. Deepest-reaching first.
*/
@Composable
fun RelayReachDetailDialog(
cursors: List<RelayReachCursor>,
formatReachDate: (epochSeconds: Long) -> String,
onDismiss: () -> Unit,
) {
val rows = remember(cursors) { cursors.sortedBy { it.reachedUntil } }
AlertDialog(
onDismissRequest = onDismiss,
confirmButton = {
TextButton(onClick = onDismiss) { Text(stringResource(Res.string.action_dismiss)) }
},
title = { Text(stringResource(Res.string.chats_history_by_relay)) },
text = {
Column(
Modifier
.heightIn(max = 360.dp)
.verticalScroll(rememberScrollState()),
) {
rows.forEach { c ->
Column(Modifier.fillMaxWidth().padding(vertical = 6.dp)) {
Row(verticalAlignment = Alignment.CenterVertically) {
Text(
text = reachGlyph(c.state),
color = reachColor(c.state),
fontWeight = FontWeight.Bold,
modifier = Modifier.width(22.dp),
)
if (c.protocol.isNotEmpty()) {
Text(
text = c.protocol,
style = MaterialTheme.typography.labelSmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
maxLines = 1,
)
Spacer(Modifier.width(6.dp))
}
Text(
text = c.name,
modifier = Modifier.weight(1f),
style = MaterialTheme.typography.bodyMedium,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
Spacer(Modifier.width(8.dp))
Text(
text = stringResource(Res.string.chats_history_relay_since, formatReachDate(c.reachedUntil)),
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
maxLines = 1,
)
}
if (c.state == RelayReachState.STALLED) StalledRetryHint()
}
}
}
},
)
}
@@ -0,0 +1,317 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.ui.feeds
import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.lazy.LazyListState
import androidx.compose.material3.HorizontalDivider
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.key
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberUpdatedState
import androidx.compose.runtime.snapshotFlow
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import com.vitorpamplona.amethyst.commons.resources.Res
import com.vitorpamplona.amethyst.commons.resources.chats_history_fully_loaded
import com.vitorpamplona.amethyst.commons.resources.chats_history_fully_loaded_label
import com.vitorpamplona.amethyst.commons.resources.chats_history_loading_label
import com.vitorpamplona.amethyst.commons.resources.chats_history_relays
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.flow.collect
import kotlinx.coroutines.flow.distinctUntilChanged
import org.jetbrains.compose.resources.pluralStringResource
import org.jetbrains.compose.resources.stringResource
// A relay-reach divider is hair-thin; inlined here so the shared component carries no app-theme dep.
private val DividerThickness = 0.25.dp
/**
* True when [reachedUntil] falls in the gap between a newer message (at [newerCreatedAt]) and its
* next-older neighbour (at [olderCreatedAt], null past the oldest end): the newer side is strictly
* newer than the cursor and the older side is at or below it (or absent). This single predicate both
* places the marker ([RelayReachMarkers]) and decides when its paging sentinel is on screen
* ([RelayReachSentinels]), so the two can never disagree about which gap a cursor lives in.
*/
internal fun reachedFallsInGap(
reachedUntil: Long,
newerCreatedAt: Long?,
olderCreatedAt: Long?,
): Boolean =
newerCreatedAt != null &&
newerCreatedAt > reachedUntil &&
(olderCreatedAt == null || olderCreatedAt <= reachedUntil)
/** How far one relay has paged into a feed's history, for an in-stream progress marker. */
enum class RelayReachState {
// Still paging older — its marker slides down (older) as it advances.
REACHING,
// Accepted but not answering right now (auth CLOSE / unreachable / slow); kept open, still trying.
STALLED,
// Hit an empty page: nothing older on this relay, it has reached the bottom of its window.
DONE,
}
/** One relay's marker entry within a gap. */
data class RelayReach(
val name: String,
val state: RelayReachState,
)
/**
* One relay's window-limit: places a marker and carries the [advance] that pulls that relay's next,
* older page. The marker sits at [reachedUntil] (the oldest point the relay has paged to);
* [RelayReachMarkers] draws it and [RelayReachSentinels] fires [advance] while it is on
* screen.
*
* @param key stable identity (protocol tag + relay url) so the sentinel survives list reorders.
*/
data class RelayReachCursor(
val key: String,
val name: String,
val reachedUntil: Long,
val state: RelayReachState,
// Short protocol tag (e.g. "NIP-17" / "NIP-04") shown in the tap-through detail popup, so a marker
// that mixes protocols in one gap isn't ambiguous. Empty when the feed has a single (implicit) kind.
val protocol: String = "",
val advance: () -> Unit,
)
/**
* Drives demand-driven paging for every limit, **hoisted above the list** so its identity does not ride
* on which row currently hosts the marker. Each non-done limit gets one stable effect (keyed by
* [RelayReachCursor.key]) that watches the [listState] and pulls that relay's next page when its marker
* is on screen.
*
* Why hoisted: the marker for a limit lives in exactly one gap (between the two rows straddling its
* reached cursor). Placing the sentinel *inside* that row made its effect's identity ride the hosting
* row so any feed reorder (a live message, or a slow relay dribbling a history page) moved the gap to a
* different row, tore the effect down and recreated it, and re-fired `advance()` on a static screen.
* That re-armed stalled/auth relays into a silence-watchdog storm and could walk a delivering relay
* back a window with no scroll. Hoisting the effect and driving it off **viewport visibility** instead
* of composition presence removes that coupling.
*
* Fires `advance()` when (and only when) the marker's gap is among the currently visible rows AND either
* it just scrolled into view OR its reached cursor moved (a page landed keep paging while visible).
* A reorder that keeps the marker on the same side of the fold changes neither, so it no longer re-fires.
* A done relay drives nothing.
*
* @param createdAtAt createdAt of the list item at an index (null past the ends / for non-message rows),
* so the visible-gap test mirrors [RelayReachMarkers]'s placement against only the on-screen rows.
*/
@Composable
fun RelayReachSentinels(
limits: List<RelayReachCursor>,
listState: LazyListState,
createdAtAt: (index: Int) -> Long?,
) {
limits.forEach { lim ->
if (lim.state == RelayReachState.DONE) return@forEach
key(lim.key) {
val reached = rememberUpdatedState(lim.reachedUntil)
val advance = rememberUpdatedState(lim.advance)
val getAt = rememberUpdatedState(createdAtAt)
LaunchedEffect(Unit) {
snapshotFlow {
val r = reached.value
val at = getAt.value
// Visible if any on-screen row is the "newer" side of the gap holding this cursor —
// the same predicate RelayReachMarkers uses to place the marker, but over the
// visible rows only.
val onScreen =
listState.layoutInfo.visibleItemsInfo.any { info ->
reachedFallsInGap(r, at(info.index), at(info.index + 1))
}
// Pair so distinctUntilChanged also lets a landed page (r moved) re-fire while visible,
// not just the off→on-screen transition.
onScreen to r
}.distinctUntilChanged()
.collect { (onScreen, r) ->
if (onScreen) {
// One line per sentinel fire — a re-fire LOOP would show the same key firing
// over and over (and whether its reached cursor is drifting).
Log.d("DMPagination") { "marker fire ${lim.key} reachedUntil=$r" }
advance.value()
}
}
}
}
}
}
/**
* Renders the window-limit markers for the relays whose limit falls in the gap between a newer message
* (at [newerCreatedAt]) and its next-older neighbour (at [olderCreatedAt], null at the oldest end). Pure
* UI: the load driving lives in [RelayReachSentinels], so this can be (re)placed freely per row on
* every feed reorder without triggering any paging.
*
* A [DONE][RelayReachState.DONE] relay has no incompleteness frontier it has loaded everything it has
* so it does NOT mark its own history bottom mid-stream (which would read like a false "incomplete below
* here" line). Instead every done relay sinks to the **oldest-end gap** ([olderCreatedAt] null), where it
* renders as one "fully loaded" marker. Only [REACHING][RelayReachState.REACHING] /
* [STALLED][RelayReachState.STALLED] relays the genuine "below here may still be incomplete" frontiers
* are placed at their reached cursor.
*/
@Composable
fun RelayReachMarkers(
limits: List<RelayReachCursor>,
newerCreatedAt: Long?,
olderCreatedAt: Long?,
// Tapped with the relays in this gap, so the caller can open a detail popup (which relays, how far
// back, per protocol). Null leaves the marker a passive, non-interactive divider.
onShowDetail: ((List<RelayReachCursor>) -> Unit)? = null,
) {
val here =
remember(limits, newerCreatedAt, olderCreatedAt) {
limits.filter {
if (it.state == RelayReachState.DONE) {
// Fully loaded → sink to the oldest end rather than mark a frontier it doesn't have.
newerCreatedAt != null && olderCreatedAt == null
} else {
reachedFallsInGap(it.reachedUntil, newerCreatedAt, olderCreatedAt)
}
}
}
if (here.isEmpty()) return
RelayReachMarker(here.map { RelayReach(it.name, it.state) }, onClick = onShowDetail?.let { cb -> { cb(here) } })
}
/**
* A thin divider drawn between two messages marking the point one or more relays have paged down to.
* As a relay loads older history its reached cursor drops, so the caller places this marker further
* down (older) in the stream relays that race ahead leave their marker deep while slower relays'
* markers trail higher up, converging as they catch up.
*
* The line is always captioned so it's never a bare glyph cluster: the live frontiers
* ([REACHING][RelayReachState.REACHING] / [STALLED][RelayReachState.STALLED]) read "Loading:"; the
* oldest-end pile of [DONE][RelayReachState.DONE] relays reads "Fully loaded:". Each state then renders
* one compact label: the host name(s) when one or two short-named relays sit at that state (the usual
* converged case, where each relay rests at its own depth), or just a count when several pile up at the
* same depth (e.g. all nine clustered at the oldest-end floor) so the line can't grow into an unreadable
* comma list. In the rare mixed line (an active frontier sharing the oldest-end gap with done relays) the
* caption is "Loading:", so the done chip is suffixed "(fully loaded)" to keep its meaning clear. Either
* way the whole marker is tappable for the full per-relay breakdown. Reads e.g. "Loading: ↓ nostr.wine",
* "Loading: ↓ 8 relays" or "Fully loaded: ✓ 8 relays".
*/
@Composable
private fun RelayReachMarker(
entries: List<RelayReach>,
onClick: (() -> Unit)? = null,
) {
if (entries.isEmpty()) return
val hasActiveFrontier = entries.any { it.state != RelayReachState.DONE }
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(6.dp),
modifier = Modifier.padding(5.dp).then(if (onClick != null) Modifier.clickable { onClick() } else Modifier),
) {
HorizontalDivider(modifier = Modifier.weight(1f), thickness = DividerThickness)
// Always caption the line so it's never a bare glyph cluster: live frontiers are "Loading:"; the
// oldest-end pile of only-done relays is "Fully loaded:".
Text(
text =
stringResource(
if (hasActiveFrontier) Res.string.chats_history_loading_label else Res.string.chats_history_fully_loaded_label,
),
color = MaterialTheme.colorScheme.onSurfaceVariant,
fontSize = 11.sp,
fontWeight = FontWeight.Medium,
maxLines = 1,
)
// Present states in enum order. Written with an explicit `sortedBy` + `entry.key`/`entry.value`
// (not `toSortedMap(compareBy { it.ordinal })` + a destructured `(state, list)`) because
// Kotlin/Native's Compose compiler can't infer those inside this inline @Composable lambda
// (commons iOS).
entries
.groupBy { it.state }
.entries
.sortedBy { it.key.ordinal }
.forEachIndexed { index, entry ->
val state = entry.key
val list = entry.value
if (index > 0) {
Text("·", color = MaterialTheme.colorScheme.onSurfaceVariant, fontSize = 11.sp)
}
// Spell out 12 short host names; otherwise "N relays" for every state, so a count chip
// always reads as a sentence ("↓ 8 relays", "✓ 8 relays") rather than a bare number. Only a
// mixed line (caption "Loading:") needs the done chip tagged "(fully loaded)" — a pure-done
// line already says so in its "Fully loaded:" caption.
val names = list.map { it.name }
val label = reachInlineNames(names) ?: pluralStringResource(Res.plurals.chats_history_relays, names.size, names.size)
val chip = reachGlyph(state) + " " + label
Text(
text = if (state == RelayReachState.DONE && hasActiveFrontier) chip + " " + stringResource(Res.string.chats_history_fully_loaded) else chip,
color = reachColor(state),
fontSize = 11.sp,
fontWeight = FontWeight.Medium,
maxLines = 1,
overflow = TextOverflow.Ellipsis,
)
}
HorizontalDivider(modifier = Modifier.weight(1f), thickness = DividerThickness)
}
}
// Host names short enough to spell out inline on the single-line divider instead of collapsing to a
// bare count: a name up to [INLINE_NAME_MAX] when it's the lone relay of its state, or two names each
// up to [INLINE_TWO_NAMES_MAX] when a pair shares it. Longer hosts, or 3+ relays at one state, return
// null so the caller renders a count instead and the line can't grow unbounded — the tap-through dialog
// always lists them all.
private const val INLINE_NAME_MAX = 16
private const val INLINE_TWO_NAMES_MAX = 12
internal fun reachInlineNames(names: List<String>): String? =
when {
names.size == 1 && names[0].length <= INLINE_NAME_MAX -> names[0]
names.size == 2 && names.all { it.length <= INLINE_TWO_NAMES_MAX } -> names.joinToString(", ")
else -> null
}
internal fun reachGlyph(state: RelayReachState) =
when (state) {
RelayReachState.REACHING -> ""
RelayReachState.STALLED -> ""
RelayReachState.DONE -> ""
}
@Composable
internal fun reachColor(state: RelayReachState): Color =
when (state) {
RelayReachState.REACHING -> MaterialTheme.colorScheme.onSurfaceVariant
RelayReachState.STALLED -> MaterialTheme.colorScheme.error
RelayReachState.DONE -> MaterialTheme.colorScheme.primary
}
@@ -0,0 +1,73 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.ui.feeds
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* The gap predicate places the per-relay reach marker AND gates its paging sentinel, so an off-by-one
* here would either render a marker in the wrong gap or fire (or never fire) paging. The boundaries are
* deliberately asymmetric newer side strictly `>`, older side `<=` so each edge is pinned here.
*/
class RelayReachMarkerTest {
@Test
fun cursorStrictlyBetweenTwoMessagesIsInTheGap() {
// gap is (older=80, newer=100]; a cursor reached down to 90 sits in it.
assertTrue(reachedFallsInGap(reachedUntil = 90, newerCreatedAt = 100, olderCreatedAt = 80))
}
@Test
fun cursorAtTheOldestEndWithNoOlderNeighbourIsInTheGap() {
// Past the oldest loaded row (olderCreatedAt null): any cursor below the last message sits here.
assertTrue(reachedFallsInGap(reachedUntil = 50, newerCreatedAt = 100, olderCreatedAt = null))
}
@Test
fun noNewerRowMeansNotInThisGap() {
// newerCreatedAt null = nothing on the newer side (e.g. a non-message row) → never placed here.
assertFalse(reachedFallsInGap(reachedUntil = 50, newerCreatedAt = null, olderCreatedAt = 20))
}
@Test
fun newerSideIsStrictlyNewer_equalDoesNotCount() {
// The marker belongs in the gap *below* the message it reached, not at the message itself.
assertFalse(reachedFallsInGap(reachedUntil = 100, newerCreatedAt = 100, olderCreatedAt = 50))
}
@Test
fun olderSideIsInclusive_equalCounts() {
// older == reached: the cursor sits exactly at the older message → still this gap (`<=`).
assertTrue(reachedFallsInGap(reachedUntil = 80, newerCreatedAt = 100, olderCreatedAt = 80))
}
@Test
fun gapWhoseOlderNeighbourIsStillNewerThanTheCursorIsNotIt() {
// older=90 is newer than the cursor (70): the cursor lives in a deeper gap, not this one.
assertFalse(reachedFallsInGap(reachedUntil = 70, newerCreatedAt = 100, olderCreatedAt = 90))
}
@Test
fun cursorNewerThanTheNewerRowIsNotInThisGap() {
assertFalse(reachedFallsInGap(reachedUntil = 150, newerCreatedAt = 100, olderCreatedAt = 80))
}
}
@@ -0,0 +1,309 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.relayClient.paging
import com.vitorpamplona.amethyst.commons.model.privateChats.DmHistoryTuning
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayLoadingCursors
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayPagingProgress
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import java.util.concurrent.ConcurrentHashMap
/**
* Atomic snapshot of a [BackwardRelayPager]'s display state. Every field is recomputed together in one
* pass ([BackwardRelayPager.status]'s producer), so a consumer collects ONE flow and never sees a torn
* mix (e.g. an updated [relayCount] against a still-stale [relayProgress]) or pays for several separate
* recompositions per page settle. [BackwardRelayPager.loadingMore] is deliberately NOT folded in here: it
* is debounced on its own timer in the load tracker, decoupled from this recompute.
*/
data class PagingStatus(
// Nothing more reachable right now: every relay is done or stalled. See the pager doc — not "caught up".
val exhausted: Boolean = false,
// Relays currently fetching a page (for an "asking N relays" status line).
val relayCount: Int = 0,
// Not-done relays that can't be reached right now (auth CLOSE / unreachable / silent).
val stalledCount: Int = 0,
// Oldest `createdAt` reached across all relays (the deepest cursor), or null before any delivery.
val reachedBack: Long? = null,
// Per-relay window position (reached / done / stalled) — what a caller's per-relay progress UI renders.
val relayProgress: Map<NormalizedRelayUrl, RelayPagingProgress> = emptyMap(),
)
/**
* Reusable **per-relay backward pagination** engine: pages a set of relays back through history,
* **one page at a time, per relay, on demand**, by `until`+`limit` ([RelayLoadingCursors]) with each
* relay advancing independently the moment *it* settles, never paced by the slowest one.
*
* This is the **single-active orchestrator** around the paging state. The state itself (the per-relay
* [RelayLoadingCursors]) does NOT live here the caller holds it on whatever object owns the scope, so
* its lifetime matches that object. One orchestrator drives whichever scope is currently bound; calling
* [bind] repoints it at that scope's cursors. This is safe as long as the caller only advances the bound
* scope (e.g. the one the user is viewing), so a backgrounded scope produces no callbacks to mis-route.
*
* What it owns (all transient, recomputed on each [bind]): the in-flight + silence tracking
* ([PerRelayLoadTracker]), the stalled-relay set, and the display flows one atomic [status] snapshot
* ([PagingStatus]) plus the separately-debounced [loadingMore]. The persistent cursors and the pinned
* history floor live on the bound [RelayLoadingCursors].
*
* What it does NOT own (the caller supplies these they are protocol- and framework-specific):
* - **Building the actual REQ filters.** The caller reads [armedRelays] + [requestedUntilFor] and
* assembles its own `RelayBasedFilter`s (the kinds / authors / `#p` tags differ per query).
* - **The subscription lifecycle.** The caller wires its `INostrClient` subscription and forwards
* relay callbacks here via [onEvent] / [onEose] / [onClosed] / [onCannotConnect], then re-issues
* its filter after [advance] / [advanceAll] return true.
* - **Which scope's cursors + which relays.** Supplied together by [bind].
*
* ### Done vs stalled (read [exhausted] with care)
* A relay is **done** once it answers an empty page (gap-proof: nothing older). A relay that won't
* answer right now auth CLOSE, unreachable, or silent past the tracker's window is flagged
* **stalled** but kept (its subscription stays open; re-[advance] retries it). [exhausted] flips true
* once every relay is *done or stalled* "nothing more reachable right now", which is NOT the same as
* "fully caught up". Callers that render a terminal state should split on [stalledCount]: `exhausted &&
* stalledCount == 0` is genuinely caught up; `exhausted && stalledCount > 0` stopped early and may be
* missing messages.
*
* Not internally synchronized beyond the primitives it composes; intended to be driven from one owning
* scope with relay callbacks serialized per relay (as the relay IO layer delivers them).
*/
class BackwardRelayPager(
// Short label for the DMPagination logs (e.g. "giftwrap.history", "convo.nip04.history").
private val name: String,
// Asked of every relay per page; large on purpose (a whole band in one page), and caps per-request
// volume. A relay returning fewer is its own cap, NOT exhaustion — only an empty page ends a relay.
val pageLimit: Int = DEFAULT_PAGE_LIMIT,
// How far below "now" the history floor sits — paging starts here and walks backward. Defaults to
// the shared live-tail boundary ([DmHistoryTuning]): everything newer is the always-on tail's job.
private val liveTailSeconds: Long = DmHistoryTuning.liveTailSeconds,
) {
private val loadTracker = PerRelayLoadTracker(name, onSilenced = ::onSilenced)
// The active scope, set by [bind]: its persistent per-relay cursors (which live on the owning domain
// object) and the lookup for the relay set it fans out to.
@Volatile
private var cursors: RelayLoadingCursors? = null
@Volatile
private var relaysFor: () -> Collection<NormalizedRelayUrl>? = { null }
// Relays not advancing for the active scope (auth CLOSE / unreachable / silent). Transient: cleared
// and recomputed on each [bind]; a stalled relay is kept (its sub stays open) and retried on advance.
private val stalledRelays = ConcurrentHashMap.newKeySet<NormalizedRelayUrl>()
/**
* True while any relay is mid-page. Starts false (an idle engine isn't "loading"). Kept apart from
* [status] on purpose: the load tracker debounces this flow's falling edge on its own timer, decoupled
* from the [publishStatus] recompute, so folding it into the snapshot would miss that delayed flip.
*/
val loadingMore: StateFlow<Boolean> = loadTracker.loading
private val _status = MutableStateFlow(PagingStatus())
/** One atomic snapshot of the display state (exhausted / counts / reached / per-relay progress), all
* recomputed together in [publishStatus] so consumers collect ONE flow and never see a torn mix. */
val status: StateFlow<PagingStatus> = _status.asStateFlow()
// The session-pinned floor for the active scope — kept on its cursors so it persists with the scope
// and does not drift forward on recompute (which would re-trigger an undelivered relay's loader).
private fun floor(): Long {
val c = cursors ?: return TimeUtils.now() - liveTailSeconds
return c.floor ?: (TimeUtils.now() - liveTailSeconds).also { c.floor = it }
}
/**
* Repoints to a scope (call on subscribe / when the active scope changes): its persistent
* [scopeCursors] (held on the caller's scope object), the [scope] for the silence watchdog, and the
* [relaysForScope] lookup. Resets the transient orchestration (in-flight, stalled) and recomputes
* the display flows from the bound cursors so a previously-paged scope restores its progress
* instead of restarting.
*/
fun bind(
scopeCursors: RelayLoadingCursors,
scope: CoroutineScope,
relaysForScope: () -> Collection<NormalizedRelayUrl>?,
) {
cursors = scopeCursors
relaysFor = relaysForScope
loadTracker.bind(scope)
loadTracker.reset()
stalledRelays.clear()
publishStatus()
}
/**
* Whether [c] is the currently-bound scope's cursor object. The orchestrator is single-active: it can
* only correctly process callbacks for the bound scope. A caller whose subscription may still be alive
* for a *just-backgrounded* scope (navigation overlap, a second pane) must gate its forwarded callbacks
* on this otherwise a late EOSE from scope A would move scope B's cursors. The cursor object is the
* scope identity (one per `Chatroom`/`ChatroomList`), so reference identity is the check.
*/
fun isBoundTo(c: RelayLoadingCursors): Boolean = c === cursors
// --- Filter building support: the caller assembles the actual REQ from these. ---
/** Relays of the active scope that have been advanced (armed) and aren't done — i.e. carry a REQ. */
fun armedRelays(relays: Collection<NormalizedRelayUrl>): List<NormalizedRelayUrl> = cursors?.armedRelays(relays) ?: emptyList()
/** The `until` [relay]'s next page should carry (null if it isn't armed / no scope bound). */
fun requestedUntilFor(relay: NormalizedRelayUrl): Long? = cursors?.requestedUntilFor(relay)
// --- Demand-driven advance (the caller re-issues its filter when these return true). ---
/** Steps a single [relay] to its next, older page. @return true if it actually advanced. */
fun advance(relay: NormalizedRelayUrl): Boolean {
if (!arm(relay)) return false
publishStatus()
return true
}
/** Steps every not-done, not-in-flight relay of the active scope one page. For a scope too small to scroll. */
fun advanceAll(): Boolean {
val relays = relaysFor() ?: return false
var any = false
relays.forEach { if (arm(it)) any = true }
if (any) publishStatus()
return any
}
// Moves one relay's cursor to its next page and marks it in-flight. Returns false if it can't advance
// (no scope bound, unknown relay, already fetching, or already done). Caller batches the recompute.
private fun arm(relay: NormalizedRelayUrl): Boolean {
val c = cursors ?: return false
val relays = relaysFor() ?: return false
if (relay !in relays) return false
if (loadTracker.isInFlight(relay)) return false
if (!c.advance(relay, floor())) return false
stalledRelays.remove(relay)
loadTracker.onAdvance(relay)
return true
}
// --- Subscription callbacks: the owner forwards these from its SubscriptionListener. ---
/** Records one delivered event for [relay] (a sign of life + a page tally entry). */
fun onEvent(
relay: NormalizedRelayUrl,
createdAt: Long,
) {
loadTracker.onActivity()
cursors?.onEvent(relay, createdAt)
stalledRelays.remove(relay)
}
/** Finalizes [relay]'s page on EOSE. @return true if this EOSE is the one that marked it done. */
fun onEose(relay: NormalizedRelayUrl): Boolean {
val c = cursors ?: return false
stalledRelays.remove(relay)
c.onEose(relay)
loadTracker.onSettled(relay)
val done = c.isDone(relay)
publishStatus()
return done
}
/** [relay] rejected the REQ (e.g. auth-required): settle it and flag it stalled (kept, retryable). */
fun onClosed(
relay: NormalizedRelayUrl,
message: String,
) {
loadTracker.onSettled(relay)
markStalled(relay, "CLOSED: $message")
publishStatus()
}
/** [relay] is unreachable right now: settle it and flag it stalled (kept, retryable). */
fun onCannotConnect(
relay: NormalizedRelayUrl,
message: String,
) {
loadTracker.onSettled(relay)
markStalled(relay, "cannot connect: $message")
publishStatus()
}
// The tracker's silence watchdog fired: the still-pending relays went quiet after their REQ. Flag them
// stalled but kept, so the window can settle instead of hanging on a dead relay.
private fun onSilenced(relays: Set<NormalizedRelayUrl>) {
relays.forEach { markStalled(it, "no response (silence timeout)") }
publishStatus()
}
private fun markStalled(
relay: NormalizedRelayUrl,
reason: String,
) {
if (stalledRelays.add(relay)) Log.d(TAG) { "[$name] ${relay.url} stalled — $reason (kept, advance to retry)" }
}
// --- Display-state recompute (one atomic snapshot from the bound cursors). ---
/**
* Recomputes the whole [status] snapshot from the active scope's cursors and publishes it in one
* emission, so consumers never see a torn mix of fields nor pay for several recompositions per settle.
*
* `exhausted` is computed here too: nothing more is reachable once every relay is done (empty page) or
* stalled (unreachable) a merely parked relay (more to load, just not advancing) keeps it false. An
* empty / unbound scope leaves `exhausted` at its previous value (mirrors the old recompute's
* early-return), so a transient empty relay set never flips it spuriously.
*/
private fun publishStatus() {
val c = cursors
val relays = relaysFor() ?: emptySet()
val floor = floor()
val prev = _status.value
val exhausted =
if (c == null || relays.isEmpty()) {
prev.exhausted
} else {
relays.none { !c.isDone(it) && it !in stalledRelays }
}
if (exhausted && !prev.exhausted && c != null) {
val done = relays.filter { c.isDone(it) }.map { it.url }
val stuck = relays.filter { it in stalledRelays && !c.isDone(it) }.map { it.url }
Log.d(TAG) { "[$name] window settled (nothing more reachable) — done=$done stalled=$stuck" }
}
_status.value =
PagingStatus(
exhausted = exhausted,
relayCount = loadTracker.count(),
stalledCount = relays.count { it in stalledRelays && c?.isDone(it) != true },
reachedBack = c?.deepestReached(relays, floor),
relayProgress =
relays.associateWith { relay ->
RelayPagingProgress(
reachedUntil = c?.reachedUntilFor(relay, floor) ?: floor,
done = c?.isDone(relay) ?: false,
stalled = relay in stalledRelays && c?.isDone(relay) != true,
)
},
)
}
companion object {
private const val TAG = "DMPagination"
const val DEFAULT_PAGE_LIMIT = 10000
}
}
@@ -0,0 +1,175 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.relayClient.paging
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import java.util.concurrent.ConcurrentHashMap
/**
* Tracks which relays currently have a demand-driven history page **in flight**, so a caller can show a
* spinner while any relay is fetching and clear it the moment they've all answered (or parked). Unlike
* [WindowLoadTracker] this has no notion of a "window" or "round" relays are advanced one page at a
* time, independently, on demand by the caller, so completion is simply "nothing in flight."
*
* A single backstop covers a relay that accepts a REQ and then goes silent (auth-walled / dead): if
* nothing has been heard from ANY in-flight relay for [silenceMs], the still-pending relays are dropped
* from the in-flight set (so the spinner clears) and reported to [onSilenced] so the owner can flag them
* stalled. Relays that answer with CLOSED / cannot-connect are settled directly by the owner and don't
* need the watchdog.
*/
class PerRelayLoadTracker(
private val name: String,
// How long the whole in-flight cohort can go without ANY signal before the still-pending relays are
// dropped + reported stalled. lastActivityMs is global, so any relay delivering keeps it fresh for all
// — this only fires on total dead air. Set well above mobile-over-Tor connect times (which run tens of
// seconds, occasionally past a minute): at 15 s a relay was being flagged "stalled" before its circuit
// even finished connecting. A genuinely dead relay still settles via CLOSED/cannot-connect, not this.
private val silenceMs: Long = 60_000L,
private val onSilenced: (Set<NormalizedRelayUrl>) -> Unit = {},
) {
private val _loading = MutableStateFlow(false)
val loading: StateFlow<Boolean> = _loading.asStateFlow()
private val inFlight = ConcurrentHashMap.newKeySet<NormalizedRelayUrl>()
@Volatile
private var lastActivityMs = 0L
@Volatile
private var watchdog: Job? = null
// Delays dropping the spinner so back-to-back pages don't flicker it off between each one.
@Volatile
private var clearJob: Job? = null
@Volatile
private var scope: CoroutineScope? = null
fun bind(scope: CoroutineScope) {
this.scope = scope
}
fun isInFlight(relay: NormalizedRelayUrl) = inFlight.contains(relay)
fun count() = inFlight.size
/** A relay's next page was just requested. Raises the spinner and (re)arms the silence watchdog. */
@Synchronized
fun onAdvance(relay: NormalizedRelayUrl) {
clearJob?.cancel() // a new page is starting — keep the spinner up, no flicker
clearJob = null
inFlight.add(relay)
lastActivityMs = TimeUtils.nowMillis()
_loading.value = true
ensureWatchdog()
}
/** A sign of life from a relay (an event). Keeps the silence watchdog from firing. */
fun onActivity() {
lastActivityMs = TimeUtils.nowMillis()
}
/**
* A relay answered (EOSE / CLOSED / cannot-connect). Drops it from in-flight. When the last one
* settles, the spinner is dropped after a short linger rather than immediately, so a relay paging
* page-after-page (each page settles, then the caller advances the next) keeps a steady spinner instead
* of flickering it off for the few ms between pages. The linger is cancelled the moment a new page
* starts ([onAdvance]).
*/
@Synchronized
fun onSettled(relay: NormalizedRelayUrl) {
lastActivityMs = TimeUtils.nowMillis()
if (inFlight.remove(relay) && inFlight.isEmpty()) scheduleClear()
}
private fun scheduleClear() {
clearJob?.cancel()
val s = scope
if (s == null) {
_loading.value = false
return
}
clearJob =
s.launch {
delay(LOADING_LINGER_MS)
synchronized(this@PerRelayLoadTracker) {
if (inFlight.isEmpty()) _loading.value = false
}
}
}
/** Drops everything (e.g. the bound scope switched). */
@Synchronized
fun reset() {
inFlight.clear()
clearJob?.cancel()
clearJob = null
_loading.value = false
watchdog?.cancel()
watchdog = null
}
private fun ensureWatchdog() {
if (watchdog?.isActive == true) return
val s = scope ?: return
watchdog =
s.launch {
while (isActive) {
delay(WATCHDOG_TICK_MS)
val silenced =
synchronized(this@PerRelayLoadTracker) {
if (inFlight.isNotEmpty() && TimeUtils.nowMillis() - lastActivityMs > silenceMs) {
val pending = inFlight.toSet()
inFlight.clear()
_loading.value = false
pending
} else {
emptySet()
}
}
if (silenced.isNotEmpty()) {
Log.d(TAG) { "[$name] silenced (no response ${silenceMs}ms): ${silenced.map { it.url }}" }
onSilenced(silenced)
}
if (inFlight.isEmpty()) break
}
}
}
companion object {
private const val TAG = "DMPagination"
private const val WATCHDOG_TICK_MS = 1_000L
// How long to keep the spinner up after the last page settles, to bridge the gap to the next
// back-to-back page so the card doesn't flicker between every page.
private const val LOADING_LINGER_MS = 600L
}
}
@@ -0,0 +1,242 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.relayClient.paging
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import java.util.concurrent.ConcurrentHashMap
import kotlin.time.Duration
import kotlin.time.Duration.Companion.minutes
import kotlin.time.Duration.Companion.seconds
/**
* Tracks when one relay-subscription "window" has finished loading, so a caller (e.g. an auto-fill /
* pagination loop) can wait for the WHOLE response instead of declaring victory on the first EOSE.
*
* A subscription fans a single REQ out to several relays. The first EOSE is a misleading "done"
* signal: a fast but near-empty relay can EOSE in milliseconds while the relay that actually holds
* the data is still connecting, stuck in an auth handshake, or busy streaming thousands of stored
* events. An auto-fill loop driven by the first EOSE or by a fixed wall-clock timeout would
* widen the window again mid-stream, before the events were even processed, re-issuing an ever-wider
* REQ that re-downloads the whole history over and over.
*
* Completion is therefore **per-relay terminal-state** based, not wall-clock based. A relay is
* *settled* once it answers with a terminal signal an EOSE (stored backfill done), a CLOSED (it
* rejected the REQ, e.g. `auth-required`), or a cannot-connect (it is unreachable). The load is done
* when **every targeted relay has settled** ([settled] [expected]). This is the only signal that
* survives the real world: relays connect over a wide spread (tens of seconds on mobile), and some
* answer only with CLOSED a quiet-time heuristic fires in the gap between two relays connecting and
* mistakes a half-loaded window for a finished one, which is exactly how a load reports "1 event"
* when a hundred are still on the way.
*
* Two backstops cover misbehaving relays. If every relay we're still waiting on has at least been
* *heard from* (any event, EOSE, CLOSED, or cannot-connect) but one streamed events without ever
* sending EOSE, an [idleTimeout] of quiet completes the load the "heard from" gate is what keeps
* this from firing in a connection gap. And an [absoluteCap] is the final ceiling on a window that
* somehow defeats the above.
*/
class WindowLoadTracker(
// Short label for the DMPagination logs (e.g. "giftwrap", "rooms.nip04", "convo.nip04").
private val name: String = "dm",
private val idleTimeout: Duration = 3.seconds,
private val absoluteCap: Duration = 5.minutes,
) {
private val _loading = MutableStateFlow(true)
val loading: StateFlow<Boolean> = _loading.asStateFlow()
// Relays the current REQ was sent to. Volatile: written on IO (updateFilter), read on the
// listener threads and the watchdog.
@Volatile
private var expected: Set<NormalizedRelayUrl> = emptySet()
// Relays that have produced any signal at all (event / EOSE / CLOSED / cannot-connect). The idle
// backstop only arms once this covers [expected], so a still-connecting relay can't be skipped.
private val heardFrom = ConcurrentHashMap.newKeySet<NormalizedRelayUrl>()
// Relays that reached a terminal signal (EOSE / CLOSED / cannot-connect). When this covers
// [expected] the stored backfill is complete on every relay and the load is done.
private val settled = ConcurrentHashMap.newKeySet<NormalizedRelayUrl>()
private var watchdog: Job? = null
// Incremented on every (re)start so a stale watchdog that wakes right as a new load begins
// recognizes it has been superseded and bows out instead of completing the new window.
private var generation = 0
// Wall-clock of the last signal for the current window; the idle backstop completes the window
// once this stops advancing for [idleTimeout]. Volatile so the hot per-event path stays lock-free.
@Volatile
private var lastActivityMs = 0L
/** Begins a fresh window load: clears the per-relay sets, raises [loading], and arms the watchdog. */
@Synchronized
fun startLoading(scope: CoroutineScope) {
val gen = ++generation
expected = emptySet()
heardFrom.clear()
settled.clear()
lastActivityMs = TimeUtils.nowMillis()
val wasLoading = _loading.value
_loading.value = true
Log.d(TAG) { "[$name] load start" + if (!wasLoading) "" else " (restart)" }
watchdog?.cancel()
watchdog =
scope.launch {
val deadline = TimeUtils.nowMillis() + absoluteCap.inWholeMilliseconds
while (isActive) {
delay(IDLE_CHECK_MS)
if (!tick(gen, TimeUtils.nowMillis(), deadline)) break
}
}
}
// One watchdog poll. Returns false (stop polling) when this watchdog has been superseded by a
// newer load, the window already finished, or a completion deadline is reached. Synchronized so
// the generation/loading checks and the completion are atomic against startLoading/finish.
@Synchronized
private fun tick(
gen: Int,
now: Long,
deadline: Long,
): Boolean {
if (gen != generation || !_loading.value) return false
if (expected.isNotEmpty()) {
// Once every relay has reached a terminal signal, nothing more is coming for this round.
if (settled.containsAll(expected)) {
finish("settled")
return false
}
// Idle backstop: every relay we're still waiting on has at least streamed something (so this
// isn't a connection gap) and the stream has gone quiet. Settled relays don't count.
val stillWaiting = expected.filterNot { settled.contains(it) }
if (stillWaiting.all { heardFrom.contains(it) } && now - lastActivityMs >= idleTimeout.inWholeMilliseconds) {
finish("idle")
return false
}
}
if (now >= deadline) {
finish("cap")
return false
}
return true
}
/** Records which relays the current REQ was sent to. Completes immediately if there are none. */
@Synchronized
fun setExpectedRelays(relays: Set<NormalizedRelayUrl>) {
expected = relays
if (relays.isEmpty()) {
finish("no relays")
} else if (settled.containsAll(relays)) {
finish("all relays")
}
}
/** A non-terminal sign of life from [relay] (a stored or live event). Keeps the idle timer alive. */
fun onRelayEvent(relay: NormalizedRelayUrl) {
heardFrom.add(relay)
lastActivityMs = TimeUtils.nowMillis()
}
/**
* A terminal signal from [relay] EOSE, CLOSED, or cannot-connect. Once every expected relay has
* settled the stored backfill is complete and the load finishes.
*/
@Synchronized
fun onRelaySettled(relay: NormalizedRelayUrl) {
lastActivityMs = TimeUtils.nowMillis()
heardFrom.add(relay)
settled.add(relay)
if (expected.isNotEmpty() && settled.containsAll(expected)) finish("all relays")
}
// Idempotent: only the first call after a load actually completes (and logs); later calls no-op.
@Synchronized
private fun finish(reason: String) {
if (!_loading.value) return
watchdog?.cancel()
watchdog = null
Log.d(TAG) { "[$name] load done: $reason" }
_loading.value = false
}
companion object {
private const val TAG = "DMPagination"
private const val IDLE_CHECK_MS = 500L
}
}
/**
* Builds the standard [SubscriptionListener] that feeds this tracker. Every event (stored backfill
* included) is a non-terminal sign of life from its relay; an EOSE, CLOSED, or cannot-connect settles
* that relay. [forward] carries the EOSE / live-event signal so the owning EOSE manager can record the
* relay's timestamp (its usual `newEose`).
*/
fun WindowLoadTracker.trackingListener(forward: (NormalizedRelayUrl, List<Filter>?) -> Unit): SubscriptionListener =
object : SubscriptionListener {
override fun onEose(
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
onRelaySettled(relay)
forward(relay, forFilters)
}
override fun onEvent(
event: Event,
isLive: Boolean,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
onRelayEvent(relay)
if (isLive) {
forward(relay, forFilters)
}
}
override fun onClosed(
message: String,
relay: NormalizedRelayUrl,
forFilters: List<Filter>?,
) {
onRelaySettled(relay)
}
override fun onCannotConnect(
relay: NormalizedRelayUrl,
message: String,
forFilters: List<Filter>?,
) {
onRelaySettled(relay)
}
}
@@ -0,0 +1,243 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.relayClient.paging
import com.vitorpamplona.quartz.nip01Core.relay.client.paging.RelayLoadingCursors
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlin.test.AfterTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* State-machine tests for [BackwardRelayPager]: drive its relay callbacks directly (no network) and
* assert the cursor / done / stalled / exhausted bookkeeping the logic that backed the "All caught up
* while messages missing" and the stalled-vs-done bugs. The relay's own `until`+`limit`+EOSE wire
* behaviour is covered separately against the in-process relay in `UntilLimitPagingRelayTest`.
*
* The pager is the **single-active orchestrator**: its per-relay cursors live on a separate
* [RelayLoadingCursors] (in production, on a `Chatroom` / `ChatroomList`), bound in via [bind]. These tests
* supply their own cursor object so they can rebind a previously-paged scope and assert what persists
* (the cursors) versus what is transient and recomputed (the stalled set, the live flows).
*/
class BackwardRelayPagerTest {
private val r1 = NormalizedRelayUrl("wss://r1.example/")
private val r2 = NormalizedRelayUrl("wss://r2.example/")
private val r3 = NormalizedRelayUrl("wss://r3.example/")
private val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())
@AfterTest
fun tearDown() {
scope.cancel()
}
// A pager bound to a fresh scope of [relays]; returns both so tests can read the pinned cursor floor.
private fun pagerOf(vararg relays: NormalizedRelayUrl): Pair<BackwardRelayPager, RelayLoadingCursors> {
val cursors = RelayLoadingCursors()
val p = BackwardRelayPager("test")
p.bind(cursors, scope) { relays.toList() }
return p to cursors
}
@Test
fun firstPageRequestsTheFloorAndAnEmptyPageIsCaughtUp() {
val (p, cursors) = pagerOf(r1)
assertFalse(p.status.value.exhausted)
assertTrue(p.advance(r1))
// The very first page asks `until = floor` (pinned on the bound cursors).
assertEquals(cursors.floor, p.requestedUntilFor(r1))
// Empty page + EOSE → that relay is done; the only relay is done → genuinely caught up.
assertTrue(p.onEose(r1))
assertTrue(
p.status.value.relayProgress
.getValue(r1)
.done,
)
assertTrue(p.status.value.exhausted)
assertEquals(0, p.status.value.stalledCount)
}
@Test
fun nonEmptyPageMovesTheCursorThenBottomsOut() {
val (p, _) = pagerOf(r1)
p.advance(r1)
// A page of three events; the oldest is 80, so the reached cursor drops to 80 (not done).
p.onEvent(r1, 100)
p.onEvent(r1, 80)
p.onEvent(r1, 90)
assertFalse(p.onEose(r1))
assertFalse(
p.status.value.relayProgress
.getValue(r1)
.done,
)
assertEquals(80L, p.status.value.reachedBack)
assertFalse(p.status.value.exhausted)
// The next page must start strictly below the oldest reached (80 → until 79).
assertTrue(p.advance(r1))
assertEquals(79L, p.requestedUntilFor(r1))
// Empty page now → done → caught up.
assertTrue(p.onEose(r1))
assertTrue(p.status.value.exhausted)
assertEquals(0, p.status.value.stalledCount)
}
@Test
fun aStalledRelayMakesExhaustionIncompleteNotCaughtUp() {
val (p, _) = pagerOf(r1, r2)
p.advance(r1)
p.advance(r2)
// r1 genuinely bottoms out; r2 is still pending, so not exhausted yet.
p.onEose(r1)
assertFalse(p.status.value.exhausted)
// r2 auth-walls the REQ → stalled (kept, not done).
p.onClosed(r2, "auth-required")
assertTrue(
p.status.value.relayProgress
.getValue(r2)
.stalled,
)
assertFalse(
p.status.value.relayProgress
.getValue(r2)
.done,
)
// Every relay is now done-or-stalled → exhausted, but it is INCOMPLETE: one relay unreachable.
assertTrue(p.status.value.exhausted)
assertEquals(1, p.status.value.stalledCount)
}
@Test
fun cannotConnectAlsoStalls() {
val (p, _) = pagerOf(r1)
p.advance(r1)
p.onCannotConnect(r1, "offline")
assertTrue(
p.status.value.relayProgress
.getValue(r1)
.stalled,
)
assertTrue(p.status.value.exhausted)
assertEquals(1, p.status.value.stalledCount)
}
@Test
fun reAdvancingAStalledRelayClearsTheStallAndUnExhausts() {
val (p, _) = pagerOf(r1)
p.advance(r1)
p.onClosed(r1, "auth-required")
assertTrue(p.status.value.exhausted)
assertEquals(1, p.status.value.stalledCount)
// Retrying it re-arms the relay: no longer stalled, no longer exhausted.
assertTrue(p.advance(r1))
assertFalse(
p.status.value.relayProgress
.getValue(r1)
.stalled,
)
assertFalse(p.status.value.exhausted)
assertEquals(0, p.status.value.stalledCount)
}
@Test
fun reachedBackIsTheDeepestCursorAcrossRelays() {
val (p, _) = pagerOf(r1, r2)
p.advance(r1)
p.advance(r2)
p.onEvent(r1, 500)
p.onEose(r1) // r1 reached 500
p.onEvent(r2, 300)
p.onEose(r2) // r2 reached 300
// Deepest = the oldest point any relay has reached.
assertEquals(300L, p.status.value.reachedBack)
}
@Test
fun aDoneRelayWillNotAdvanceAgain() {
val (p, _) = pagerOf(r1)
p.advance(r1)
p.onEose(r1) // empty → done
assertFalse(p.advance(r1))
}
@Test
fun advanceAllArmsEveryNotDoneRelay() {
val (p, _) = pagerOf(r1, r2, r3)
// r2 already finished; advanceAll should arm only r1 and r3.
p.advance(r2)
p.onEose(r2)
assertTrue(p.advanceAll())
assertEquals(setOf(r1, r3), p.armedRelays(listOf(r1, r2, r3)).toSet())
}
@Test
fun rebindingRepointsFlowsKeepingDoneCursorsButDroppingTransientStalls() {
val cursorsA = RelayLoadingCursors()
val cursorsB = RelayLoadingCursors()
val p = BackwardRelayPager("test")
// Scope A: r1 bottoms out (done — a persistent cursor fact); r2 auth-walls (stalled — transient).
p.bind(cursorsA, scope) { listOf(r1, r2) }
p.advance(r1)
p.advance(r2)
p.onEose(r1)
p.onClosed(r2, "auth-required")
assertTrue(p.status.value.exhausted)
assertEquals(1, p.status.value.stalledCount)
// Bind to a fresh scope B: the flows reflect B's own (empty) state — nothing stalled, and its
// reach sits at B's floor (no history fetched yet — markers start at the live-tail boundary).
p.bind(cursorsB, scope) { listOf(r3) }
assertFalse(p.status.value.exhausted)
assertEquals(0, p.status.value.stalledCount)
assertEquals(cursorsB.floor, p.status.value.reachedBack)
// Rebind to A: r1 is still DONE (its cursor persisted on cursorsA), but r2's stall is gone — stall
// is transient, so r2 is pending again and A is no longer exhausted (it will retry the auth relay).
p.bind(cursorsA, scope) { listOf(r1, r2) }
assertTrue(
p.status.value.relayProgress
.getValue(r1)
.done,
)
assertEquals(0, p.status.value.stalledCount)
assertFalse(p.status.value.exhausted)
}
}
@@ -23,3 +23,5 @@ package com.vitorpamplona.quartz.utils
actual fun platform() = "Android"
actual fun currentTimeSeconds() = System.currentTimeMillis() / 1000
actual fun currentTimeMillis() = System.currentTimeMillis()
@@ -0,0 +1,211 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.cache.LargeCache
import kotlin.concurrent.Volatile
/**
* Per-relay `until`+`limit` pagination cursors for **one scope** (one account, or one conversation):
* how far back each relay has been *asked* to load, and how far it has actually *delivered*.
*
* Pure paging state, no orchestration the loading / stall / exhaustion / live status a caller shows
* around paging is a separate concern, layered on top by a driver (in this library, `BackwardRelayPager`).
* It carries no key on purpose: the caller holds one instance per scope on whatever object owns that
* scope, so the cursors share that object's lifetime and the object graph is the partition not a map
* kept in here.
*
* Why `until`+`limit` and not a time window: a `since`/`until` slice that comes back empty can't tell
* "nothing older here" from "just a quiet gap". `until`+`limit` returns the N newest events older than
* `until`, skipping gaps so an **empty page + EOSE is the gap-proof stop** ([isDone]). (A short page,
* fewer than the limit, is the relay capping the response, not the bottom.)
*
* Two cursors per relay, kept apart so a relay never pages past what it was asked:
* - [requestedUntilFor] the `until` its REQ carries; moves only in [advance]. Untouched on EOSE, so a
* finished page just parks (no re-REQ) until advanced again this is what makes paging demand-driven.
* - reached ([reachedUntilFor]) the oldest `created_at` it has delivered; moves on EOSE. This is the
* "loaded back to here" point, and the next [advance] starts just below it.
*
* No locks of its own: each relay's callbacks are serialized, the cursor fields are `@Volatile`, and the
* relay map is the thread-safe [LargeCache] (keyed by [NormalizedRelayUrl], which orders consistently
* with `equals`).
*/
class RelayLoadingCursors {
private class RelayCursor {
// The `until` the REQ carries; null until the relay is first advanced. Moves only in advance().
@Volatile var requestedUntil: Long? = null
// The oldest created_at this relay has delivered; null until its first non-empty page. Moves on
// EOSE. This is the "loaded back to here" point; the next page starts just below it.
@Volatile var reachedUntil: Long? = null
// Set once the relay answered an empty page with EOSE: there is nothing older on it.
@Volatile var done: Boolean = false
// Per-page tallies, reset by [advance]: how many events arrived and the oldest among them.
@Volatile var pageCount: Int = 0
@Volatile var pageOldest: Long = Long.MAX_VALUE
}
private val cursors = LargeCache<NormalizedRelayUrl, RelayCursor>()
/**
* The history floor this scope pages down from (e.g. `now liveTail`, pinned by the owner on first
* advance). Kept here so it persists with the scope and doesn't drift on recompute a relay that
* hasn't delivered yet reports this floor as its reached point, and a moving floor would make a
* demand-driven loader re-fire on it.
*/
@Volatile
var floor: Long? = null
private fun cursor(relay: NormalizedRelayUrl) = cursors.getOrCreate(relay) { RelayCursor() }
/** The `until` [relay]'s REQ currently carries. Only meaningful once it has been [advance]d. */
fun requestedUntilFor(relay: NormalizedRelayUrl): Long? = cursor(relay).requestedUntil
/** The oldest point [relay] has reached, or [start] if it hasn't delivered yet. */
fun reachedUntilFor(
relay: NormalizedRelayUrl,
start: Long,
): Long = cursor(relay).reachedUntil ?: start
/** True once [relay] answered an empty page with EOSE — nothing older to ask it for. */
fun isDone(relay: NormalizedRelayUrl): Boolean = cursor(relay).done
/**
* Steps [relay] to its next, older page: points its REQ just below the oldest event it has delivered
* (or [start] for its very first page) and clears the page tally. No-op (returns false) if the relay
* has already paged to the bottom ([isDone]). The owner re-issues the relay's REQ after this.
*/
fun advance(
relay: NormalizedRelayUrl,
start: Long,
): Boolean {
val c = cursor(relay)
if (c.done) return false
val reached = c.reachedUntil
c.requestedUntil =
when {
// Resume just below the oldest event already delivered. Covers both normal page-to-page
// advance and a post-[rewindTo] resume (which un-arms the relay — requestedUntil back to
// null — but keeps the rewound reached point, so the next page picks up at the boundary
// instead of restarting at the floor and re-streaming the still-held tail).
reached != null -> reached - 1
// Very first page for this relay (nothing delivered, nothing requested yet).
c.requestedUntil == null -> start
// Armed but still mid-page (no EOSE yet) — keep asking from the same top.
else -> start
}
c.pageCount = 0
c.pageOldest = Long.MAX_VALUE
return true
}
/** Records one event for [relay] in the current page. */
fun onEvent(
relay: NormalizedRelayUrl,
createdAt: Long,
) {
val c = cursor(relay)
c.pageCount++
if (createdAt < c.pageOldest) c.pageOldest = createdAt
}
/**
* Finalizes [relay] for the page on its EOSE: an empty page marks it [isDone]; otherwise the reached
* cursor drops to the oldest event the page returned. The requested cursor is left alone so the relay
* parks until [advance] is called again.
*/
fun onEose(relay: NormalizedRelayUrl) {
val c = cursor(relay)
if (c.pageCount == 0) {
c.done = true
} else {
// The reached cursor must move strictly older every page (the next page asks `until =
// reached - 1`). A relay that returns events but none older than we already have — a
// misbehaving relay echoing the same newest events — would otherwise pin the cursor and a
// demand-driven loader would re-request the same window forever. Treat that as the bottom.
val prev = c.reachedUntil
if (prev == null || c.pageOldest < prev) {
c.reachedUntil = c.pageOldest
} else {
c.done = true
}
}
}
/**
* Realigns the window after the cache prunes messages out of it: for each `relay newestPrunedUntil`
* entry, rewinds that relay so it no longer claims to hold anything at or below [newestPrunedUntil]
* (the newest cursor-space `created_at` among the messages pruned from that relay for gift wraps the
* **outer-wrap** time, recovered from the rumor's [host][com.vitorpamplona.quartz.nip59Giftwrap.HostStub]).
*
* Without this, a relay that already paged past the pruned band or reached `done` would never
* re-request the dropped messages: its [reachedUntil] still points below them, so the next [advance]
* starts even older and skips the hole entirely.
*
* The rewind pulls [reachedUntil] back up to just above [newestPrunedUntil] (so the next page's
* `until` re-includes it), clears [done] (there *is* older data to re-fetch again), and un-arms the
* relay (requested cursor back to null) so paging stays demand-driven the dropped band comes back
* only when the on-screen marker advances the relay again, not eagerly on the next re-subscribe.
*
* Bounds:
* - A relay with no cursor yet (never paged) is skipped there is no window position to misalign.
* - The rewind never moves [reachedUntil] above the pinned [floor] (history lives strictly below it;
* a pruned message newer than the floor is the live tail's concern, not this window's).
* - A relay whose reached point is already shallower than the pruned band needs no rewind.
*/
fun rewindTo(newestPrunedUntil: Map<NormalizedRelayUrl, Long>) {
val floorAt = floor ?: return
newestPrunedUntil.forEach { (relay, prunedUntil) ->
val c = cursors.get(relay) ?: return@forEach
val reached = c.reachedUntil ?: return@forEach
// Re-include the newest pruned event: the next page asks `until = reached - 1`, so reached must
// sit one tick above it. Never climb above the floor.
val target = minOf(prunedUntil + 1, floorAt)
if (reached < target) {
c.reachedUntil = target
c.requestedUntil = null
c.done = false
c.pageCount = 0
c.pageOldest = Long.MAX_VALUE
}
}
}
/** Relays from [all] that have been armed (advanced at least once) and are not yet [isDone]. */
fun armedRelays(all: Collection<NormalizedRelayUrl>): List<NormalizedRelayUrl> =
all.filter {
val c = cursor(it)
c.requestedUntil != null && !c.done
}
/**
* The oldest point reached across [relays] the minimum reached cursor (how far back paging has
* gone). Relays that haven't delivered count as [start]. Null when [relays] is empty.
*/
fun deepestReached(
relays: Collection<NormalizedRelayUrl>,
start: Long,
): Long? = relays.takeIf { it.isNotEmpty() }?.minOf { cursor(it).reachedUntil ?: start }
}
@@ -0,0 +1,33 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
/** How far back one relay has paged through its history, for a per-relay progress display. */
data class RelayPagingProgress(
// The oldest createdAt this relay has loaded down to (its `until` cursor). This is the "loaded back
// to" point; it slides down (older) as the relay pages further back.
val reachedUntil: Long,
// The relay answered an empty page: it has nothing older, it has reached the bottom of its window.
val done: Boolean,
// The relay isn't answering right now (auth-walled CLOSE / unreachable / slow). It is NOT abandoned
// — its subscription stays open and it keeps trying to catch up — but it isn't currently advancing.
val stalled: Boolean,
)
@@ -22,8 +22,19 @@ package com.vitorpamplona.quartz.nip59Giftwrap
import com.vitorpamplona.quartz.nip01Core.core.HexKey
/**
* A lightweight reference to the host event a [WrappedEvent] was extracted from kept on the inner
* event so callers can broadcast / delete / locate the outer wrap without holding the full event.
*
* [createdAt] is the host's own `created_at` (e.g. the kind:1059 gift-wrap timestamp, randomized per
* NIP-59), carried here so a decrypted rumor self-describes its outer-wrap time. The history pager
* cursors page gift wraps by that outer time, so the prune path uses it to realign the per-relay
* download window when a wrapped message is pruned (the chatroom only keeps the inner rumor, whose
* `created_at` is the real message time, not the wrap time).
*/
class HostStub(
val id: HexKey,
val pubKey: HexKey,
val kind: Int,
val createdAt: Long,
)
@@ -70,7 +70,7 @@ class SealedRumorEvent(
val event = rumor.mergeWith(this)
if (event is WrappedEvent) {
event.host = host ?: HostStub(this.id, this.pubKey, this.kind)
event.host = host ?: HostStub(this.id, this.pubKey, this.kind, this.createdAt)
}
innerEventId = event.id
@@ -74,7 +74,7 @@ open class GiftWrapEvent(
val gift = fromJson(giftStr)
if (gift is WrappedEvent) {
gift.host = HostStub(this.id, this.pubKey, this.kind)
gift.host = HostStub(this.id, this.pubKey, this.kind, this.createdAt)
}
innerEventId = gift.id
@@ -23,3 +23,5 @@ package com.vitorpamplona.quartz.utils
expect fun platform(): String
expect fun currentTimeSeconds(): Long
expect fun currentTimeMillis(): Long
@@ -36,6 +36,8 @@ object TimeUtils {
fun now() = currentTimeSeconds()
fun nowMillis() = currentTimeMillis()
fun tenSecondsFromNow() = now() + TEN_SECONDS
fun tenSecondsAgo() = now() - TEN_SECONDS
@@ -31,3 +31,5 @@ actual fun currentTimeSeconds(): Long {
// NSDate().timeIntervalSince1970 returns seconds since 1970-01-01 00:00:00 UTC
return (NSDate().timeIntervalSince1970).toLong()
}
actual fun currentTimeMillis(): Long = (NSDate().timeIntervalSince1970 * 1000).toLong()
@@ -0,0 +1,100 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
import com.vitorpamplona.geode.fixtures.SyntheticEvents
import com.vitorpamplona.geode.testing.RelayClientTest
import com.vitorpamplona.geode.testing.collectUntilEose
import com.vitorpamplona.geode.testing.preload
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Pins down the relay-side contract the whole [RelayLoadingCursors] / `BackwardRelayPager` design rests on,
* against the in-process relay: a backward `until`+`limit` walk returns each event **exactly once**
* (no re-download), in **newest-first** capped pages, and an **empty page + EOSE** is the gap-proof
* stop. If a relay ever stopped honouring this (e.g. oldest-first, or ignoring `until`), these break
* which is exactly the signal the pager's correctness depends on.
*/
class UntilLimitPagingRelayTest : RelayClientTest() {
@Test
fun backwardUntilLimitWalkCoversEveryEventOnceAndStopsOnEmptyPage() =
runBlocking {
// 250 regular events, createdAt 1..250 (distinct pubkeys so none collapse).
defaultRelay.preload(SyntheticEvents.batch(TOTAL, kind = KIND))
val seenIds = mutableSetOf<String>()
var totalReceived = 0
var pages = 0
var until: Long? = null
while (pages < SAFETY_CAP) {
val (events, eose) =
client.collectUntilEose(
defaultRelayUrl,
Filter(kinds = listOf(KIND), until = until, limit = LIMIT),
)
assertTrue(eose, "every page must end with EOSE")
if (events.isEmpty()) break // gap-proof stop: empty page = nothing older
pages++
assertTrue(events.size <= LIMIT, "page must respect the limit")
// Newest-first + cursor honoured: nothing newer than the cursor leaks into a later page.
until?.let { cursor -> assertTrue(events.all { it.createdAt <= cursor }, "page must be older than the cursor") }
events.forEach { e: Event ->
seenIds.add(e.id)
totalReceived++
}
until = events.minOf { it.createdAt } - 1
}
// No re-download: total delivered equals the corpus, and every id is distinct.
assertEquals(TOTAL, totalReceived, "no event should be delivered twice across pages")
assertEquals(TOTAL, seenIds.size, "every event fetched exactly once")
// 250 / 100 → 100 + 100 + 50, then an empty page stops the walk.
assertEquals(3, pages)
}
@Test
fun anEmptyRelayAnswersOneEmptyPageWithEose() =
runBlocking {
val (events, eose) =
client.collectUntilEose(
defaultRelayUrl,
Filter(kinds = listOf(KIND), until = null, limit = LIMIT),
)
assertTrue(eose)
assertEquals(0, events.size)
}
companion object {
private const val KIND = 1
private const val TOTAL = 250
private const val LIMIT = 100
private const val SAFETY_CAP = 10
}
}
@@ -23,3 +23,5 @@ package com.vitorpamplona.quartz.utils
actual fun platform() = "JVM"
actual fun currentTimeSeconds() = System.currentTimeMillis() / 1000
actual fun currentTimeMillis() = System.currentTimeMillis()
@@ -38,3 +38,12 @@ actual fun currentTimeSeconds(): Long {
return ts.tv_sec
}
}
@OptIn(ExperimentalForeignApi::class)
actual fun currentTimeMillis(): Long {
memScoped {
val ts = alloc<timespec>()
clock_gettime(CLOCK_REALTIME, ts.ptr)
return ts.tv_sec * 1000 + ts.tv_nsec / 1_000_000
}
}
@@ -26,3 +26,5 @@ import platform.Foundation.timeIntervalSince1970
actual fun platform() = "macOS"
actual fun currentTimeSeconds(): Long = (NSDate().timeIntervalSince1970).toLong()
actual fun currentTimeMillis(): Long = (NSDate().timeIntervalSince1970 * 1000).toLong()