mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-10 00:16:59 +00:00
chore: remove superseded TimeWindowPagination, refresh DM design doc
Review prep for the DM pagination branch: - Delete commons TimeWindowPagination + its test: the early since-based time-window approach, referenced only by its own test and fully superseded by UntilLimitPager (until+limit, gap-proof). 212 lines a reviewer would otherwise study for nothing. - Bring the design doc up to the final architecture: NIP-04 per-relay filter scoping, per-relay independent paging (no rounds) + in-stream markers for the convo, the round model still used by rooms/gift-wrap, the WindowLoadTracker backstops and tracksReqSends gating, the loadingMore-starts-false fix, and the DMPagination diagnostics map. Marks the obsolete time-slice section as superseded. https://claude.ai/code/session_01B1fmmmX8JjQWH3amMLdvcW
This commit is contained in:
Claude
@@ -41,6 +41,10 @@ timestamps, so its slices need no margin.
|
||||
|
||||
### Slice math (gift-wrap history window)
|
||||
|
||||
> Superseded by the two updates below — kept for the history of the design. The
|
||||
> `TimeWindowPagination` class this described has been removed; the history
|
||||
> managers now page by `until`+`limit` per relay (`UntilLimitPager`).
|
||||
|
||||
`TimeWindowPagination.since` starts at `now − 1week` (= the live-tail floor).
|
||||
|
||||
- `loadMore`: `until = window.since` (current floor); `window.loadMore()` moves
|
||||
@@ -92,3 +96,79 @@ at all. `limit` also caps per-request volume.
|
||||
Both NIP-04 history managers now paginate themselves (per relay) instead of
|
||||
following the gift-wrap slice; `loadEverything` pages to the end by auto-issuing
|
||||
the next round until exhausted. The live tail and stall-gate are unchanged.
|
||||
|
||||
## Update 2: NIP-04 filters scoped per relay
|
||||
|
||||
A conversation's NIP-04 filters named the whole participant set on every relay,
|
||||
so a relay that belongs to one correspondent was still asked about all of them
|
||||
(`{authors:[bob,charlie]}` sent to a relay that is only charlie's), and the
|
||||
`from-me` leg (`authors:[me]`) was sent to the correspondents' inbox relays —
|
||||
which auth-walled relays (ditto: "all authors must be authenticated") reject
|
||||
outright, stalling the load.
|
||||
|
||||
`Nip04DmRelays` is now two **per-relay key maps** (`relay → which keys to name
|
||||
there`), built from the outbox model:
|
||||
|
||||
- **to me** (`#p:[me]`) — my inbox carries the whole group; each correspondent's
|
||||
outbox carries only that correspondent.
|
||||
- **from me** (`authors:[me]`) — my outbox carries the whole group; each
|
||||
correspondent's inbox carries only that correspondent.
|
||||
|
||||
Relays shared across roles union their key sets, so a relay only ever sees the
|
||||
keys that actually own it.
|
||||
|
||||
## Update 3: per-relay independent paging + in-stream markers (convo only)
|
||||
|
||||
The round model paced every relay at the slowest one: each `loadMore` issued one
|
||||
page to all active relays and waited for the slowest to settle before the next.
|
||||
Fast own-relays that hold the whole conversation were stuck behind a
|
||||
correspondent's 15 s timeout.
|
||||
|
||||
`ChatroomNip04HistorySubAssembler` was rewritten to page **each relay
|
||||
independently, no rounds**. A relay continues to its next page the instant *it*
|
||||
EOSEs (the subscription layer diffs per relay, so re-issuing only re-REQs the
|
||||
relay whose cursor moved; the others' in-flight REQs are untouched). Fast relays
|
||||
race to the bottom in back-to-back pages; slow / auth-walled relays catch up at
|
||||
their own pace — **none are abandoned** (they keep their subscription open and
|
||||
keep trying), so every relay converges on the same window.
|
||||
|
||||
- A relay is **done** on an empty page; one that won't answer (auth CLOSE,
|
||||
unreachable, silent) is flagged **stalled** but kept open.
|
||||
- `loadingMore` reflects "is anything still advancing"; it clears once every
|
||||
relay is done or stalled. It is exposed as a flow that **starts `false`** (not
|
||||
`windowLoad.loading`, which starts `true` and would wedge the scroll loader's
|
||||
`!loading` gate on first open), and the assembler tracks `windowActive` itself
|
||||
so the first `loadMore` actually starts the window.
|
||||
- `relayProgress` (`relay → reached-back / done / stalled`) feeds **in-stream
|
||||
markers** (`RelayReachMarker`, wired through `ChatFeedView.markersInGap`): a
|
||||
thin divider per relay at the depth it has reached, sliding down as it pages
|
||||
and converging — `↓` reaching, `…` stalled, `✓` done.
|
||||
|
||||
The **rooms-list and gift-wrap** history managers still use the round model
|
||||
(`AccountGiftWrapsHistoryEoseManager`,
|
||||
`ChatroomListNip04HistorySubAssembler`) — they query only the account's own
|
||||
(fast, reachable) relays, so the lock-step never bites there. Only the
|
||||
conversation screen, which fans out to correspondents' relays, needed the
|
||||
per-relay rewrite.
|
||||
|
||||
### Window completion backstops (`WindowLoadTracker`)
|
||||
|
||||
The shared window tracker finishes when every relay reaches a terminal signal
|
||||
(EOSE / CLOSED / cannot-connect), with three backstops for misbehaving relays:
|
||||
**idle** (every still-waited relay was heard from and the stream went quiet),
|
||||
**silence** (a relay that got its REQ but answered nothing for 10 s), and
|
||||
**connect-grace** (a relay that never even received its REQ within 15 s, stuck
|
||||
connecting). The two REQ-aware backstops are gated behind `tracksReqSends`, set
|
||||
only by the convo manager — without it an always-empty `reqSentAt` would make
|
||||
every relay look connect-stalled and complete the window before its REQs even
|
||||
went out. Silent relays are reported via `onAbandoned`; the tracker only stops
|
||||
waiting, the owner decides what to do (the convo keeps them and flags stalled).
|
||||
|
||||
## Diagnostics
|
||||
|
||||
The whole path logs under one tag, **`DMPagination`** (debug builds):
|
||||
`DmRelayDiagnosticsLogger` folds the per-relay connection timeline (REQ sent,
|
||||
connect/disconnect, CLOSED/NOTICE/OK-fail) into it; `DmRelayLog` prints the
|
||||
"relays by source" breakdown per subscription; and each assembler logs its
|
||||
milestones (paging start, a relay reaching the bottom / stalling with the
|
||||
reason, the settle summary of done-vs-still-trying).
|
||||
|
||||
-87
@@ -1,87 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.relayClient.pagination
|
||||
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
|
||||
/**
|
||||
* Tracks how far back in time a relay subscription should reach.
|
||||
*
|
||||
* Boot opens a small window (recent-first) so a screen becomes usable before the
|
||||
* whole history is fetched and decrypted. Each [loadMore] widens the floor backward
|
||||
* so scrolling pulls older history on demand.
|
||||
*
|
||||
* Only the lower bound ([since]) moves: the subscription stays open so new events
|
||||
* keep streaming live regardless of the window. The floor is requested in full on
|
||||
* every assembly (the value is small and bounded), which keeps the window robust
|
||||
* even if the in-memory note store evicts previously-loaded events under memory
|
||||
* pressure.
|
||||
*
|
||||
* The step can grow geometrically ([growthFactor] > 1) so an auto-fill loop that keeps
|
||||
* widening to fill a screen (or to confirm there is no older history) converges in a
|
||||
* handful of requests instead of crawling back a fixed slice at a time. [maxLookback]
|
||||
* is a hard floor: once [since] reaches it, [isExhausted] is true and there is nothing
|
||||
* older to ask for.
|
||||
*/
|
||||
class TimeWindowPagination(
|
||||
private val initialWindow: Long = ONE_WEEK_IN_SECONDS,
|
||||
private val step: Long = ONE_WEEK_IN_SECONDS,
|
||||
private val growthFactor: Long = 1L,
|
||||
private val maxLookback: Long = TEN_YEARS_IN_SECONDS,
|
||||
) {
|
||||
/** Epoch seconds; events older than this are not requested from relays. */
|
||||
@Volatile
|
||||
var since: Long = TimeUtils.now() - initialWindow
|
||||
private set
|
||||
|
||||
@Volatile
|
||||
private var currentStep: Long = step
|
||||
|
||||
private fun floor() = TimeUtils.now() - maxLookback
|
||||
|
||||
/** Widens the window backward by the current step, clamped at [maxLookback], then grows the step. */
|
||||
fun loadMore() {
|
||||
since = maxOf(floor(), since - currentStep)
|
||||
if (growthFactor > 1L) currentStep *= growthFactor
|
||||
}
|
||||
|
||||
/** Jumps straight to [maxLookback] so a single request pulls the entire history. */
|
||||
fun loadAll() {
|
||||
since = floor()
|
||||
}
|
||||
|
||||
/** True once the window has reached [maxLookback] — there is no older history to request. */
|
||||
fun isExhausted(): Boolean = since <= floor()
|
||||
|
||||
/** Resets the window back to the initial boot size, anchored at the current time. */
|
||||
fun reset() {
|
||||
since = TimeUtils.now() - initialWindow
|
||||
currentStep = step
|
||||
}
|
||||
|
||||
companion object {
|
||||
const val ONE_WEEK_IN_SECONDS = TimeUtils.ONE_WEEK.toLong()
|
||||
|
||||
// Covers the entire history of Nostr (which began ~2021) with margin, so reaching it
|
||||
// genuinely means "nothing older exists" rather than an arbitrary cutoff.
|
||||
const val TEN_YEARS_IN_SECONDS = TimeUtils.ONE_YEAR.toLong() * 10
|
||||
}
|
||||
}
|
||||
-125
@@ -1,125 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.relayClient.pagination
|
||||
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
|
||||
class TimeWindowPaginationTest {
|
||||
@Test
|
||||
fun bootOpensAWindowThatStartsRecent() {
|
||||
val window = 1000L
|
||||
val pagination = TimeWindowPagination(initialWindow = window, step = 500L)
|
||||
|
||||
// floor is roughly `now - initialWindow`, never unbounded
|
||||
val expected = TimeUtils.now() - window
|
||||
assertTrue("floor should be near now - window", kotlin.math.abs(pagination.since - expected) <= 2)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun loadMoreWidensTheFloorBackwardByOneStep() {
|
||||
val pagination = TimeWindowPagination(initialWindow = 1000L, step = 500L)
|
||||
val before = pagination.since
|
||||
|
||||
pagination.loadMore()
|
||||
assertEquals(before - 500L, pagination.since)
|
||||
|
||||
pagination.loadMore()
|
||||
assertEquals(before - 1000L, pagination.since)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun resetReturnsToTheInitialBootWindow() {
|
||||
val window = 1000L
|
||||
val pagination = TimeWindowPagination(initialWindow = window, step = 500L)
|
||||
pagination.loadMore()
|
||||
pagination.loadMore()
|
||||
|
||||
pagination.reset()
|
||||
|
||||
val expected = TimeUtils.now() - window
|
||||
assertTrue("reset floor should be near now - window", kotlin.math.abs(pagination.since - expected) <= 2)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun growingStepDoublesTheReachEachLoadMore() {
|
||||
val pagination = TimeWindowPagination(initialWindow = 10L, step = 100L, growthFactor = 2L, maxLookback = Long.MAX_VALUE / 2)
|
||||
val before = pagination.since
|
||||
|
||||
pagination.loadMore()
|
||||
assertEquals("first step is the base step", before - 100L, pagination.since)
|
||||
|
||||
pagination.loadMore()
|
||||
assertEquals("second step is doubled", before - 300L, pagination.since)
|
||||
|
||||
pagination.loadMore()
|
||||
assertEquals("third step is doubled again", before - 700L, pagination.since)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun windowIsNotExhaustedWhileWithinLookback() {
|
||||
val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = 100L)
|
||||
assertTrue("a fresh window is not exhausted", !pagination.isExhausted())
|
||||
|
||||
pagination.loadMore() // -> now-20
|
||||
assertTrue("still within the 100s lookback", !pagination.isExhausted())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun windowBecomesExhaustedAndClampsAtMaxLookback() {
|
||||
val maxLookback = 100L
|
||||
val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = maxLookback)
|
||||
|
||||
// Geometric reach 10,20,40,80 crosses the 100s floor within a handful of steps.
|
||||
repeat(6) { pagination.loadMore() }
|
||||
|
||||
assertTrue("window should report exhausted at the floor", pagination.isExhausted())
|
||||
val floor = TimeUtils.now() - maxLookback
|
||||
assertTrue("since must not go past the floor", pagination.since >= floor - 2 && pagination.since <= floor + 2)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun loadAllJumpsStraightToExhaustion() {
|
||||
val maxLookback = 100L
|
||||
val pagination = TimeWindowPagination(initialWindow = 10L, step = 10L, growthFactor = 2L, maxLookback = maxLookback)
|
||||
assertTrue("not exhausted before loadAll", !pagination.isExhausted())
|
||||
|
||||
pagination.loadAll()
|
||||
|
||||
assertTrue("loadAll exhausts the window in one step", pagination.isExhausted())
|
||||
val floor = TimeUtils.now() - maxLookback
|
||||
assertTrue("since lands at the floor", kotlin.math.abs(pagination.since - floor) <= 2)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun resetClearsStepGrowth() {
|
||||
val pagination = TimeWindowPagination(initialWindow = 10L, step = 100L, growthFactor = 2L, maxLookback = Long.MAX_VALUE / 2)
|
||||
pagination.loadMore() // step grows to 200
|
||||
pagination.loadMore() // step grows to 400
|
||||
|
||||
pagination.reset()
|
||||
val before = pagination.since
|
||||
pagination.loadMore()
|
||||
assertEquals("after reset the step is back to the base", before - 100L, pagination.since)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user