From e1cdd40bb5fb8d60f690471e2a1c416b65b60cb5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 3 Jun 2026 17:16:21 +0000 Subject: [PATCH] chore: remove superseded TimeWindowPagination, refresh DM design doc Review prep for the DM pagination branch: - Delete commons TimeWindowPagination + its test: the early since-based time-window approach, referenced only by its own test and fully superseded by UntilLimitPager (until+limit, gap-proof). 212 lines a reviewer would otherwise study for nothing. - Bring the design doc up to the final architecture: NIP-04 per-relay filter scoping, per-relay independent paging (no rounds) + in-stream markers for the convo, the round model still used by rooms/gift-wrap, the WindowLoadTracker backstops and tracksReqSends gating, the loadingMore-starts-false fix, and the DMPagination diagnostics map. Marks the obsolete time-slice section as superseded. https://claude.ai/code/session_01B1fmmmX8JjQWH3amMLdvcW --- ...6-06-01-dm-live-tail-and-history-slices.md | 80 +++++++++++ .../pagination/TimeWindowPagination.kt | 87 ------------ .../pagination/TimeWindowPaginationTest.kt | 125 ------------------ 3 files changed, 80 insertions(+), 212 deletions(-) delete mode 100644 commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPagination.kt delete mode 100644 commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPaginationTest.kt diff --git a/amethyst/plans/2026-06-01-dm-live-tail-and-history-slices.md b/amethyst/plans/2026-06-01-dm-live-tail-and-history-slices.md index fe259a14cc..4be0d089c0 100644 --- a/amethyst/plans/2026-06-01-dm-live-tail-and-history-slices.md +++ b/amethyst/plans/2026-06-01-dm-live-tail-and-history-slices.md @@ -41,6 +41,10 @@ timestamps, so its slices need no margin. ### Slice math (gift-wrap history window) +> Superseded by the two updates below — kept for the history of the design. The +> `TimeWindowPagination` class this described has been removed; the history +> managers now page by `until`+`limit` per relay (`UntilLimitPager`). + `TimeWindowPagination.since` starts at `now − 1week` (= the live-tail floor). - `loadMore`: `until = window.since` (current floor); `window.loadMore()` moves @@ -92,3 +96,79 @@ at all. `limit` also caps per-request volume. Both NIP-04 history managers now paginate themselves (per relay) instead of following the gift-wrap slice; `loadEverything` pages to the end by auto-issuing the next round until exhausted. The live tail and stall-gate are unchanged. + +## Update 2: NIP-04 filters scoped per relay + +A conversation's NIP-04 filters named the whole participant set on every relay, +so a relay that belongs to one correspondent was still asked about all of them +(`{authors:[bob,charlie]}` sent to a relay that is only charlie's), and the +`from-me` leg (`authors:[me]`) was sent to the correspondents' inbox relays — +which auth-walled relays (ditto: "all authors must be authenticated") reject +outright, stalling the load. + +`Nip04DmRelays` is now two **per-relay key maps** (`relay → which keys to name +there`), built from the outbox model: + +- **to me** (`#p:[me]`) — my inbox carries the whole group; each correspondent's + outbox carries only that correspondent. +- **from me** (`authors:[me]`) — my outbox carries the whole group; each + correspondent's inbox carries only that correspondent. + +Relays shared across roles union their key sets, so a relay only ever sees the +keys that actually own it. + +## Update 3: per-relay independent paging + in-stream markers (convo only) + +The round model paced every relay at the slowest one: each `loadMore` issued one +page to all active relays and waited for the slowest to settle before the next. +Fast own-relays that hold the whole conversation were stuck behind a +correspondent's 15 s timeout. + +`ChatroomNip04HistorySubAssembler` was rewritten to page **each relay +independently, no rounds**. A relay continues to its next page the instant *it* +EOSEs (the subscription layer diffs per relay, so re-issuing only re-REQs the +relay whose cursor moved; the others' in-flight REQs are untouched). Fast relays +race to the bottom in back-to-back pages; slow / auth-walled relays catch up at +their own pace — **none are abandoned** (they keep their subscription open and +keep trying), so every relay converges on the same window. + +- A relay is **done** on an empty page; one that won't answer (auth CLOSE, + unreachable, silent) is flagged **stalled** but kept open. +- `loadingMore` reflects "is anything still advancing"; it clears once every + relay is done or stalled. It is exposed as a flow that **starts `false`** (not + `windowLoad.loading`, which starts `true` and would wedge the scroll loader's + `!loading` gate on first open), and the assembler tracks `windowActive` itself + so the first `loadMore` actually starts the window. +- `relayProgress` (`relay → reached-back / done / stalled`) feeds **in-stream + markers** (`RelayReachMarker`, wired through `ChatFeedView.markersInGap`): a + thin divider per relay at the depth it has reached, sliding down as it pages + and converging — `↓` reaching, `…` stalled, `✓` done. + +The **rooms-list and gift-wrap** history managers still use the round model +(`AccountGiftWrapsHistoryEoseManager`, +`ChatroomListNip04HistorySubAssembler`) — they query only the account's own +(fast, reachable) relays, so the lock-step never bites there. Only the +conversation screen, which fans out to correspondents' relays, needed the +per-relay rewrite. + +### Window completion backstops (`WindowLoadTracker`) + +The shared window tracker finishes when every relay reaches a terminal signal +(EOSE / CLOSED / cannot-connect), with three backstops for misbehaving relays: +**idle** (every still-waited relay was heard from and the stream went quiet), +**silence** (a relay that got its REQ but answered nothing for 10 s), and +**connect-grace** (a relay that never even received its REQ within 15 s, stuck +connecting). The two REQ-aware backstops are gated behind `tracksReqSends`, set +only by the convo manager — without it an always-empty `reqSentAt` would make +every relay look connect-stalled and complete the window before its REQs even +went out. Silent relays are reported via `onAbandoned`; the tracker only stops +waiting, the owner decides what to do (the convo keeps them and flags stalled). + +## Diagnostics + +The whole path logs under one tag, **`DMPagination`** (debug builds): +`DmRelayDiagnosticsLogger` folds the per-relay connection timeline (REQ sent, +connect/disconnect, CLOSED/NOTICE/OK-fail) into it; `DmRelayLog` prints the +"relays by source" breakdown per subscription; and each assembler logs its +milestones (paging start, a relay reaching the bottom / stalling with the +reason, the settle summary of done-vs-still-trying). diff --git a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPagination.kt b/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPagination.kt deleted file mode 100644 index 6bbabd8cf5..0000000000 --- a/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPagination.kt +++ /dev/null @@ -1,87 +0,0 @@ -/* - * Copyright (c) 2025 Vitor Pamplona - * - * Permission is hereby granted, free of charge, to any person obtaining a copy of - * this software and associated documentation files (the "Software"), to deal in - * the Software without restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the - * Software, and to permit persons to whom the Software is furnished to do so, - * subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in all - * copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS - * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR - * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN - * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION - * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - */ -package com.vitorpamplona.amethyst.commons.relayClient.pagination - -import com.vitorpamplona.quartz.utils.TimeUtils - -/** - * Tracks how far back in time a relay subscription should reach. - * - * Boot opens a small window (recent-first) so a screen becomes usable before the - * whole history is fetched and decrypted. Each [loadMore] widens the floor backward - * so scrolling pulls older history on demand. - * - * Only the lower bound ([since]) moves: the subscription stays open so new events - * keep streaming live regardless of the window. The floor is requested in full on - * every assembly (the value is small and bounded), which keeps the window robust - * even if the in-memory note store evicts previously-loaded events under memory - * pressure. - * - * The step can grow geometrically ([growthFactor] > 1) so an auto-fill loop that keeps - * widening to fill a screen (or to confirm there is no older history) converges in a - * handful of requests instead of crawling back a fixed slice at a time. [maxLookback] - * is a hard floor: once [since] reaches it, [isExhausted] is true and there is nothing - * older to ask for. - */ -class TimeWindowPagination( - private val initialWindow: Long = ONE_WEEK_IN_SECONDS, - private val step: Long = ONE_WEEK_IN_SECONDS, - private val growthFactor: Long = 1L, - private val maxLookback: Long = TEN_YEARS_IN_SECONDS, -) { - /** Epoch seconds; events older than this are not requested from relays. */ - @Volatile - var since: Long = TimeUtils.now() - initialWindow - private set - - @Volatile - private var currentStep: Long = step - - private fun floor() = TimeUtils.now() - maxLookback - - /** Widens the window backward by the current step, clamped at [maxLookback], then grows the step. */ - fun loadMore() { - since = maxOf(floor(), since - currentStep) - if (growthFactor > 1L) currentStep *= growthFactor - } - - /** Jumps straight to [maxLookback] so a single request pulls the entire history. */ - fun loadAll() { - since = floor() - } - - /** True once the window has reached [maxLookback] — there is no older history to request. */ - fun isExhausted(): Boolean = since <= floor() - - /** Resets the window back to the initial boot size, anchored at the current time. */ - fun reset() { - since = TimeUtils.now() - initialWindow - currentStep = step - } - - companion object { - const val ONE_WEEK_IN_SECONDS = TimeUtils.ONE_WEEK.toLong() - - // Covers the entire history of Nostr (which began ~2021) with margin, so reaching it - // genuinely means "nothing older exists" rather than an arbitrary cutoff. - const val TEN_YEARS_IN_SECONDS = TimeUtils.ONE_YEAR.toLong() * 10 - } -} diff --git a/commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPaginationTest.kt b/commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPaginationTest.kt deleted file mode 100644 index cb79f93940..0000000000 --- a/commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/relayClient/pagination/TimeWindowPaginationTest.kt +++ /dev/null @@ -1,125 +0,0 @@ -/* - * Copyright (c) 2025 Vitor Pamplona - * - * Permission is hereby granted, free of charge, to any person obtaining a copy of - * this software and associated documentation files (the "Software"), to deal in - * the Software without restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the - * Software, and to permit persons to whom the Software is furnished to do so, - * subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in all - * copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS - * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR - * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN - * AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION - * WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - */ -package com.vitorpamplona.amethyst.commons.relayClient.pagination - -import com.vitorpamplona.quartz.utils.TimeUtils -import org.junit.Assert.assertEquals -import org.junit.Assert.assertTrue -import org.junit.Test - -class TimeWindowPaginationTest { - @Test - fun bootOpensAWindowThatStartsRecent() { - val window = 1000L - val pagination = TimeWindowPagination(initialWindow = window, step = 500L) - - // floor is roughly `now - initialWindow`, never unbounded - val expected = TimeUtils.now() - window - assertTrue("floor should be near now - window", kotlin.math.abs(pagination.since - expected) <= 2) - } - - @Test - fun loadMoreWidensTheFloorBackwardByOneStep() { - val pagination = TimeWindowPagination(initialWindow = 1000L, step = 500L) - val before = pagination.since - - pagination.loadMore() - assertEquals(before - 500L, pagination.since) - - pagination.loadMore() - assertEquals(before - 1000L, pagination.since) - } - - @Test - fun resetReturnsToTheInitialBootWindow() { - val window = 1000L - val pagination = TimeWindowPagination(initialWindow = window, step = 500L) - pagination.loadMore() - pagination.loadMore() - - pagination.reset() - - val expected = TimeUtils.now() - window - assertTrue("reset floor should be near now - window", kotlin.math.abs(pagination.since - expected) <= 2) - } - - @Test - fun growingStepDoublesTheReachEachLoadMore() { - val pagination = TimeWindowPagination(initialWindow = 10L, step = 100L, growthFactor = 2L, maxLookback = Long.MAX_VALUE / 2) - val before = pagination.since - - pagination.loadMore() - assertEquals("first step is the base step", before - 100L, pagination.since) - - pagination.loadMore() - assertEquals("second step is doubled", before - 300L, pagination.since) - - pagination.loadMore() - assertEquals("third step is doubled again", before - 700L, pagination.since) - } - - @Test - fun windowIsNotExhaustedWhileWithinLookback() { - val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = 100L) - assertTrue("a fresh window is not exhausted", !pagination.isExhausted()) - - pagination.loadMore() // -> now-20 - assertTrue("still within the 100s lookback", !pagination.isExhausted()) - } - - @Test - fun windowBecomesExhaustedAndClampsAtMaxLookback() { - val maxLookback = 100L - val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = maxLookback) - - // Geometric reach 10,20,40,80 crosses the 100s floor within a handful of steps. - repeat(6) { pagination.loadMore() } - - assertTrue("window should report exhausted at the floor", pagination.isExhausted()) - val floor = TimeUtils.now() - maxLookback - assertTrue("since must not go past the floor", pagination.since >= floor - 2 && pagination.since <= floor + 2) - } - - @Test - fun loadAllJumpsStraightToExhaustion() { - val maxLookback = 100L - val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = maxLookback) - assertTrue("not exhausted before loadAll", !pagination.isExhausted()) - - pagination.loadAll() - - assertTrue("loadAll exhausts the window in one step", pagination.isExhausted()) - val floor = TimeUtils.now() - maxLookback - assertTrue("since lands at the floor", kotlin.math.abs(pagination.since - floor) <= 2) - } - - @Test - fun resetClearsStepGrowth() { - val pagination = TimeWindowPagination(initialWindow = 10L, step = 100L, growthFactor = 2L, maxLookback = Long.MAX_VALUE / 2) - pagination.loadMore() // step grows to 200 - pagination.loadMore() // step grows to 400 - - pagination.reset() - val before = pagination.since - pagination.loadMore() - assertEquals("after reset the step is back to the base", before - 100L, pagination.since) - } -}