diff --git a/plans/feed-outbox-assessment.md b/plans/feed-outbox-assessment.md new file mode 100644 index 0000000..5d24fad --- /dev/null +++ b/plans/feed-outbox-assessment.md @@ -0,0 +1,932 @@ +# Feed Outbox Model — Assessment & Plan + +## Your suspicion is correct, and NDK already has the machinery built in. + +You are missing posts from some follows because the feed only queries the +**relays you subscribe to** (your read/both relays from your kind 10002 list). +If a follow posts exclusively to relays you are *not* connected to, those posts +never arrive. This is exactly the problem the Nostr **outbox model** (NIP-65 + +NIP-17 relay lists) solves: instead of only listening on your relays, you look +up each follow's kind 10002 relay list and fetch their posts from *their* write +relays. + +The good news: **NDK's outbox model is already enabled** in your bundle and is +already wired into the subscription path. The bad news: it is being defeated by +a race condition in how [`www/feed.html`](www/feed.html) bootstraps the feed. + +--- + +## How NDK's outbox model works (already in your bundle) + +Relevant code in [`www/ndk-core.bundle.js`](www/ndk-core.bundle.js): + +1. **Outbox is on by default** — [`NDK`](www/ndk-core.bundle.js:43195) constructor + at line 43295: `if (!(opts.enableOutboxModel === false))` creates an + `outboxPool` (default relays `wss://purplepag.es/`, `wss://nos.lol/`) and an + `outboxTracker`. Your worker never sets `enableOutboxModel: false`, so it is + active. Confirmed: [`www/ndk-worker.js`](www/ndk-worker.js:1506) `ndkOptions` + does not disable it. + +2. **Author-based subscriptions trigger outbox tracking** — + [`NDK.subscribe`](www/ndk-core.bundle.js:43665): when a filter has `authors`, + it calls `this.outboxTracker?.trackUsers(authors)`. This fetches each + author's kind 10002 (and kind 3 fallback) relay list from the outbox pool. + +3. **Relay sets are computed per-author** — + [`calculateRelaySetsFromFilter`](www/ndk-core.bundle.js:33864) calls + [`getRelaysForFilterWithAuthors`](www/ndk-core.bundle.js:31650) → + [`chooseRelayCombinationForPubkeys`](www/ndk-core.bundle.js:31594) which + picks ~2 write relays per author and splits the `authors` array across the + relays each author actually writes to. + +4. **Late outbox data refreshes subscriptions** — when + [`OutboxTracker`](www/ndk-core.bundle.js:42790) resolves a user's relay list, + it emits `user:relay-list-updated`, and the NDK constructor (line 43301) + calls `subscription.refreshRelayConnections()` to add newly-discovered + relays to the live subscription. + +5. **Fallback for unknown authors** — + [`chooseRelayCombinationForPubkeys`](www/ndk-core.bundle.js:31639): authors + with no known relay list are sent to `pool.permanentAndConnectedRelays()` + (your read relays). So outbox never *loses* events vs. the old behavior; it + only *adds* coverage. + +So in principle, a plain `ndk.subscribe({ kinds:[1], authors:[...] })` already +does outbox routing. The feature you want exists. + +--- + +## Why your feed still misses people — the root cause + +The problem is a **race / staleness condition** in +[`www/feed.html`](www/feed.html), not a missing NDK feature. + +### The bootstrap path (lines 481–545) + +[`fetchFeedWindow`](www/feed.html:481) does this for each time window: + +1. `queryCache(filters)` — reads the Dexie cache synchronously. +2. `ndkFetchEvents(filters)` — fires a relay subscription with + `closeOnEose: true` (via [`handleNdkFetchEvents`](www/ndk-worker.js:6316)). + +The filters are `{ kinds:[1], authors, since, limit }`. This *does* go through +NDK's outbox routing. But there are two issues: + +#### Issue A — Outbox data is not pre-warmed + +`outboxTracker.trackUsers` is called inside `NDK.subscribe`, but it is +**fire-and-forget** relative to `EOSE`. The subscription starts immediately +against whatever relays are currently known (initially: your read relays only, +since the tracker is empty). When the outbox data arrives moments later, +`refreshRelayConnections` adds the new relays — but only for **long-lived** +subscriptions. For a `closeOnEose: true` `fetchEvents` call, the subscription +may `EOSE` and close on your read relays *before* the outbox tracker finishes +resolving the follow's kind 10002 lists. Result: you get posts only from +follows who happen to post to your read relays. + +This is the core bug. The bootstrap windows in +[`FEED_BOOTSTRAP_WINDOWS_SECONDS`](www/feed.html:179) fire a burst of +short-lived `fetchEvents` calls that race the outbox tracker. + +#### Issue B — The live subscription is fine, but late + +[`ensureLiveFeedSubscription`](www/feed.html:547) creates a long-lived +`subscribe({ kinds:[1], authors, since })` with `closeOnEose:false`. This one +*does* benefit from `refreshRelayConnections` — once the outbox tracker +resolves, new relays get added and late posts stream in. But: + +- It only covers events **after** `since` (newest known − 30s, or now − 1h). + Historical posts from follows on foreign relays that arrived *before* the + outbox data resolved are never backfilled. +- The bootstrap is what fills the initial feed view, and that's where Issue A + bites. + +#### Issue C — Outbox tracker entries expire in 2 minutes + +[`OutboxTracker`](www/ndk-core.bundle.js:42800): `entryExpirationTimeInMS: 2 * 60 * 1000`. +The tracker is an LRU with a 2-minute TTL. If the live subscription was created +and the tracker entries expired, a later `refreshRelayConnections` won't fire +because there's no `user:relay-list-updated` event to re-trigger it. The +subscription keeps running on whatever relays it had. This is mostly fine for +the live sub, but means any *new* `fetchEvents` call (e.g. "See More" or a new +follow added via [`ingestContactList`](www/feed.html:567)) starts cold again. + +--- + +## What does NOT need to change + +- **Do not** disable the outbox model or build a custom relay-fetching layer. + NDK already does this; you'd be reimplementing + [`getRelayListForUsers`](www/ndk-core.bundle.js:42662) and + [`chooseRelayCombinationForPubkeys`](www/ndk-core.bundle.js:31594). +- **Do not** manually connect to every follow's write relays in the main pool. + NDK's `pool.getRelay(url, true, true)` (temporary relay) already handles + per-subscription temporary connections via + [`calculateRelaySetsFromFilter`](www/ndk-core.bundle.js:33864). Adding them + permanently would bloat the main pool. + +## What DOES need to change + +The fix is to **pre-warm the outbox tracker before issuing the short-lived +bootstrap fetches**, so that by the time `fetchEvents` runs, NDK already knows +each follow's write relays and routes the subscription correctly. + +NDK exposes this via `ndk.outboxTracker.trackUsers(pubkeys)`. Your worker does +not currently call it directly. The cleanest fix is a new worker RPC, +`warmOutbox(pubkeys)`, that calls `ndk.outboxTracker.trackUsers(pubkeys)` and +awaits it, plus a feed.html change to call it before +[`bootstrapFeedPosts`](www/feed.html:507). + +--- + +## Unified Implementation Plan + +The work is organized into phases that build on each other. Each phase is +self-contained and deliverable independently. The phases merge the feed +outbox fix, the relays.html performance fix, the outbox visualization, and +the Amethyst-parity relay management features into one coherent sequence. + +### Phase 1 — Worker: expose outbox pre-warming + relay-event logging toggle + +**Files:** [`www/ndk-worker.js`](www/ndk-worker.js) + +- [ ] **1.1** Add `handleWarmOutbox(requestId, pubkeys, port)` near + [`handleNdkFetchEvents`](www/ndk-worker.js:6316). It should: + - Guard `if (!ndk?.outboxTracker) { respond empty ok }`. + - `await ndk.outboxTracker.trackUsers(pubkeys)`. + - Respond `{ type: 'warmOutboxResult', requestId, ok: true, count: pubkeys.length }`. + - Wrap in try/catch; never reject (pre-warming is best-effort). + +- [ ] **1.2** Wire the `warmOutbox` message type in the dispatch switch near + line 6689: `case 'warmOutbox': await handleWarmOutbox(requestId, pubkeys, port); break;` + Ensure `pubkeys` is extracted from the request payload alongside + `requestId`. + +- [ ] **1.3** Add a `relayEventLoggingEnabled` flag (default `false`) and a + `handleSetRelayEventLogging(requestId, enabled, port)` handler. When + `false`, [`broadcastRelayEvent`](www/ndk-worker.js:917) skips the + `broadcast()` call AND [`logRelayEvent`](www/ndk-worker.js:899) skips + `writeRelayEventToDb`. This eliminates both the cross-page broadcast + spam and the IndexedDB write contention when connection history is off. + +- [ ] **1.4** Wire the `setRelayEventLogging` message type in the dispatch + switch. + +- [ ] **1.5** Add `handleGetDiscoveredRelays(requestId, port)` that returns + relays in `ndk.pool.relays` that are *not* in `relayTypes` (your kind + 10002 list). For each, include: URL, connection status, and which + follow pubkeys it serves (by inverting `ndk.outboxTracker.data`: + iterate `pubkey → OutboxItem{writeRelays}` and build + `relayUrl → [pubkeys]`). + +- [ ] **1.6** Wire the `getDiscoveredRelays` message type in the dispatch + switch. + +### Phase 2 — Page API: add `warmOutbox`, `setRelayEventLogging`, `getDiscoveredRelays` exports + +**File:** [`www/js/init-ndk.mjs`](www/js/init-ndk.mjs) + +- [ ] **2.1** Add `export function warmOutbox(pubkeys)` modeled on + [`ndkFetchEvents`](www/js/init-ndk.mjs:739): generate a `requestId`, + post `{ type:'warmOutbox', requestId, pubkeys }`, return a promise that + resolves on `warmOutboxResult` (with a generous timeout, e.g. 15s, that + resolves rather than rejects — pre-warm is best-effort). + +- [ ] **2.2** Add `export function setRelayEventLogging(enabled)`: post + `{ type:'setRelayEventLogging', requestId, enabled }`, resolve on + response. Fire-and-forget is fine (no need to await). + +- [ ] **2.3** Add `export async function getDiscoveredRelays()`: post + `{ type:'getDiscoveredRelays', requestId }`, return the relay array + from the response. + +### Phase 3 — Feed: pre-warm outbox before bootstrap + +**File:** [`www/feed.html`](www/feed.html) + +- [ ] **3.1** Import `warmOutbox` in the existing import block (line 143). + +- [ ] **3.2** In [`ingestContactList`](www/feed.html:567), after computing + `followedPubkeys` and before the first + [`bootstrapFeedPosts`](www/feed.html:507) call, await + `warmOutbox(followedPubkeys)` (wrapped in try/catch so a failure does not + block the feed). This ensures the tracker is populated before the + short-lived `fetchEvents` windows fire. + +- [ ] **3.3** In [`ingestContactList`](www/feed.html:567), when + `newAuthors.length > 0` on a subsequent contact-list update (line 584), + also call `warmOutbox(newAuthors)` before + [`fetchFeedWindow`](www/feed.html:481) so newly-added follows are + routed correctly. + +- [ ] **3.4** Optional but recommended: in + [`ensureLiveFeedSubscription`](www/feed.html:547), call + `warmOutbox(authors)` fire-and-forget before `subscribe(...)` so the live + sub's `refreshRelayConnections` has data ready. This is belt-and-suspenders + since `NDK.subscribe` already calls `trackUsers`, but doing it explicitly + avoids relying on the internal call's timing. + +### Phase 4 — relays.html: connection history toggle + performance fixes + +**File:** [`www/relays.html`](www/relays.html) + +- [ ] **4.1** Add [`sidenav-sections.css`](www/css/sidenav-sections.css) to + the `` (the page currently only includes `client.css`). + +- [ ] **4.2** Replace the "Relay management options coming soon..." placeholder + in [`openNav`](www/relays.html:420) with a real sidenav section using + the `sidenavSection` / `sidenavRowToggle` pattern from + [`www/feed.html`](www/feed.html:99): + + ```html +
+
Relays
+ +
+ ``` + +- [ ] **4.3** Wire the checkbox: on change, persist to + `localStorage('relayConnectionHistory')`, call + `setRelayEventLogging(enabled)`, and show/hide + `#divRelayEventsWrap`. Default: **off**. + +- [ ] **4.4** When the toggle is off, skip + [`loadRelayEventsFromDb`](www/relays.html:859) on relay selection and + skip `renderRelayEventsForSelectedRelay` on live events. + +- [ ] **4.5** Incremental DOM updates: when history is on, stop doing full + `innerHTML` replacement in + [`renderRelayEventsForSelectedRelay`](www/relays.html:915). Append a + single `.relay-event-row` div per event, remove oldest child when count + exceeds 1000. + +- [ ] **4.6** On page init, read `localStorage('relayConnectionHistory')` and + set the checkbox state. Call `setRelayEventLogging(enabled)` to sync + the worker. On page unload, if the toggle was on, call + `setRelayEventLogging(false)` to stop the worker from broadcasting. + +### Phase 5 — relays.html: discovered relays table (visualize outbox) + +**File:** [`www/relays.html`](www/relays.html) + +- [ ] **5.1** Add a collapsible "Discovered Relays (Outbox)" section below the + main relay table. Columns: Relay URL | Connected | Serving (count of + follows) | Sample Follows (first 3 npubs). Read-only — no toggles, no + remove. + +- [ ] **5.2** Call `getDiscoveredRelays()` on page init and on each + `refreshRelayData` cycle (every 5s). Render the results in the new + section. + +- [ ] **5.3** Add a small "Outbox discovery" status line showing the outbox + pool relays (`purplepag.es`, `nos.lol`) and their connection state. + This is informational — "Outbox discovery: connected to purplepag.es, + nos.lol". + +### Phase 6 — Verification (feed outbox + discovered relays) + +- [ ] **6.1** Open `feed.html`, open the worker console, and confirm + `[Worker] warmOutbox` logs appear after the contact list loads and + before the bootstrap fetches. +- [ ] **6.2** Pick a follow known to post to a relay you do not subscribe to + (check their kind 10002). Confirm their posts now appear in the feed. +- [ ] **6.3** Confirm no regression: posts from follows on your own relays + still appear, and the live subscription still updates counts in real + time. +- [ ] **6.4** Open `relays.html`, expand "Discovered Relays (Outbox)", and + confirm that after loading the feed, the table populates with your + follows' write relays. Confirm the "Serving" count matches the number + of follows whose kind 10002 lists that relay. +- [ ] **6.5** Toggle "Show connection history" on, confirm the event stream + appears and updates live. Toggle off, confirm the stream hides and the + page feels faster. Check the worker console — no relay event + broadcasts when off. + +--- + +## Why this is the minimal, correct fix + +- It uses NDK's existing outbox infrastructure — no custom relay logic. +- It only changes the *timing* of when the tracker is populated, not the + routing algorithm. +- It is best-effort and non-blocking: if pre-warming fails or times out, the + feed falls back to the current behavior (your read relays only), which is + strictly a subset of what outbox routing would return. +- It directly addresses the race between short-lived `fetchEvents` EOSE and + the async `trackUsers` resolution. + +## Risks / notes + +- **Outbox pool relays:** NDK's default outbox pool is `purplepag.es` and + `nos.lol`. These are the relays it queries for kind 10002 lists. If they are + slow or rate-limiting, `trackUsers` can take up to its 1s timeout per batch + (see [`getRelayListForUsers`](www/ndk-core.bundle.js:42662) `timeout = 1e3`). + The 15s page-side timeout in Phase 2.1 accommodates this for ~400-pubkey + batches. +- **Large follow lists:** `trackUsers` batches in slices of 400 + ([`OutboxTracker.trackUsers`](www/ndk-core.bundle.js:42810)). A user with + 1000+ follows will take a few batches. This is fine; the live sub still + works during pre-warm. +- **2-minute TTL:** If a user leaves the feed tab open for a long time, the + tracker entries expire. The live subscription keeps its already-added relays, + so this is not a problem for ongoing streaming. It only matters for *new* + `fetchEvents` calls, which is why Phase 3.3 re-warms for new follows. + +--- + +## Footer & sidenav relay visualization — keeping it unchanged + +You want the footer and sidenav relay indicators (managed by +[`www/js/relay-ui.mjs`](www/js/relay-ui.mjs)) to continue showing only your +subscribed relays, unchanged. I traced the data flow to confirm our changes +won't affect them, and identified one minor edge case to guard against. + +### How the footer/sidenav gets relay data today + +1. [`prepareRelayVisualizationData`](www/js/relay-ui.mjs:235) calls + `getRelayData()` → worker's + [`handleGetRelayData`](www/ndk-worker.js:6401). +2. `handleGetRelayData` builds the relay list from `relayTypes` (your kind + 10002 map) — **not** from `ndk.pool.relays`. This is the key line: + [`relayTypes.size > 0 ? Array.from(relayTypes.entries()) : ...`](www/ndk-worker.js:6415). + So once your kind 10002 is loaded, the footer only ever shows your chosen + relays, regardless of what temporary relays are in the pool. +3. Activity carrots (read/write animations) are driven by + `ndkRelayActivity` window events, which come from + [`trackRelayRead`](www/ndk-worker.js:812) / + [`trackRelayWrite`](www/ndk-worker.js:834). These broadcast + `{ type: 'relayActivity', ... }` — a **separate** broadcast from + `broadcastRelayEvent` (the debug event stream). + +### Impact of our changes on the footer/sidenav + +| Change | Affects footer/sidenav? | Why | +|---|---|---| +| Phase 1: `warmOutbox` | **No** | Only calls `ndk.outboxTracker.trackUsers`. Does not touch `relayTypes` or `handleGetRelayData`. | +| Phase 1: `setRelayEventLogging` | **No** | Only gates `broadcastRelayEvent` / `logRelayEvent`. Does NOT gate `trackRelayRead` / `trackRelayWrite` (the activity broadcasts). Activity carrots continue working. | +| Phase 1: `getDiscoveredRelays` | **No** | New RPC, only called by relays.html. Does not touch `handleGetRelayData`. | +| Phase 3: feed pre-warm | **No** | Adds temporary relays to `ndk.pool`, but `handleGetRelayData` uses `relayTypes`, not the pool. | +| Phase 9: `relayConnectionFilter` | **No** | Filters which relays NDK connects to, but does not change `relayTypes` or `handleGetRelayData`. | + +### The one edge case to guard against + +[`handleGetRelayData`](www/ndk-worker.js:6415) has a fallback: when +`relayTypes.size === 0` (before your kind 10002 is fetched), it uses +`ndk.pool.relays.keys()`. After our outbox fix, temporary outbox relays will +be in `ndk.pool.relays`, so during this brief startup window the footer could +show discovered relays that aren't yours. + +This is **transient** — `relayTypes` populates within seconds of login when +the worker fetches your kind 10002. But to be safe, we should guard the +fallback: + +- [ ] **Guard 1 — Filter the fallback (optional).** In + [`handleGetRelayData`](www/ndk-worker.js:6417), when using the + `ndk.pool.relays.keys()` fallback, filter out temporary relays. NDK + relays added via `useTemporaryRelay` may have a way to detect their + temporary status. If not practical to detect, leave as-is — the + fallback is only used for a few seconds at startup and the visual + impact is minimal (a few extra icons that disappear once `relayTypes` + loads). + +This guard is **optional** — the fallback is already transient and +self-correcting. It's listed here for completeness, not as a required step. + +### What we will NOT change + +- [`www/js/relay-ui.mjs`](www/js/relay-ui.mjs) — no changes. The footer and + sidenav rendering logic stays exactly as-is. +- [`handleGetRelayData`](www/ndk-worker.js:6401) — no changes to the primary + path (the `relayTypes` branch). It will continue to show only your kind + 10002 relays. +- `trackRelayRead` / `trackRelayWrite` — no changes. Activity carrots + continue to fire on your subscribed relays. + +--- + +## Relay UI implications — how outbox relays should appear + +This is an important design question. Once outbox routing is active, NDK will +be connecting to **three categories** of relays, but your current UI only +surfaces one of them. Here is how they map to NDK's internal pools and what +your UI should do about each. + +### The three categories of relays NDK uses + +1. **Main pool — your read/both relays** (from your kind 10002). These are the + relays you chose in `relay.html`. Managed by + [`updateRelays`](www/ndk-worker.js:2088), which only adds read/both relays + to `ndk.pool` (write-only relays are deliberately excluded — line 2131). + These are what [`handleGetRelayData`](www/ndk-worker.js:6401) reports today. + +2. **Outbox pool — relay-list discovery relays** (`purplepag.es`, `nos.lol`). + This is `ndk.outboxPool`, a separate `NDKPool` instance created in the NDK + constructor ([`www/ndk-core.bundle.js:43296`](www/ndk-core.bundle.js:43296)). + It is used *only* to fetch kind 10002 / kind 3 relay lists for authors. It + does **not** carry your feed posts. Your worker never touches it directly. + +3. **Temporary relays — your follows' write relays.** When outbox routing + resolves a follow's kind 10002, NDK calls + [`pool.useTemporaryRelay`](www/ndk-core.bundle.js:35529) to connect to that + follow's write relays on demand, scoped to the subscription that needs them. + These relays are added to `ndk.pool.relays` but marked temporary — they are + removed after 30s of inactivity (`removeIfUnusedAfter = 3e4`). **These are + the relays that actually carry the missing posts.** + +### What your UI currently shows + +[`handleGetRelayData`](www/ndk-worker.js:6401) builds the relay list from +`relayTypes` (your kind 10002 map) or, as a fallback, `ndk.pool.relays.keys()`. +This means: + +- **Category 1** is always shown. ✅ +- **Category 2** (outbox pool) is never shown — it's a separate pool object + the worker doesn't read. This is arguably correct: these relays don't carry + your content, they're just a directory service. Showing them would clutter + the UI with relays the user didn't pick and can't edit. +- **Category 3** (temporary relays) is *partially* visible: they exist in + `ndk.pool.relays`, so the fallback path at line 6417 would include them. + But once `relayTypes` is populated (after the first kind 10002 fetch), the + fallback is not used and temporary relays are **hidden**. They also come and + go every 30s, which would make the footer flicker if shown. + +### Recommended UI treatment + +The cleanest mental model for users is: **"My relays" vs. "Discovered relays."** + +- **Footer + sidenav (relay-ui.mjs):** Keep showing only **Category 1** — the + user's own read/both relays. This is what the user controls and what + [`relay.html`](www/relay.html) edits. Showing temporary outbox relays here + would be noisy (they appear/disappear every 30s) and misleading (the user + can't disconnect or edit them). The footer should answer "are *my* relays + healthy?", not "what is NDK connected to right now?" + + → **No change required to [`www/js/relay-ui.mjs`](www/js/relay-ui.mjs).** + The current `handleGetRelayData` already filters to `relayTypes`, which is + exactly right. + +- **relay.html (the relay management page):** Consider adding a **read-only + "Discovered relays" section** below the editable relay table. This would + surface Category 3 (temporary relays currently in the pool) so a curious + user can see which of their follows' relays NDK is pulling from. This is + informational only — no add/remove/edit. Implementation would be a new + worker RPC `getDiscoveredRelays` that returns + `Array.from(ndk.pool.relays.values()).filter(r => r is temporary && not in + relayTypes)` with their status and which follow(s) they serve. + + This is **optional for the initial outbox fix** and can be deferred. The + core feed fix (Phases 1–3) does not depend on it. + +- **Outbox pool (Category 2):** Do not surface in the UI. These are + infrastructure relays (a directory service), not content relays. If a user + wants to know why relay-list discovery is slow, that's a debug/diagnostic + concern, not a relay-management concern. + +### Why not show temporary relays in the footer + +Concrete reasons, in case this comes up in review: + +1. **Flicker.** Temporary relays are removed after 30s of inactivity + ([`useTemporaryRelay`](www/ndk-core.bundle.js:35529) + `removeIfUnusedAfter = 3e4`). A feed that loads, then goes idle, would show + relays appearing and disappearing in the footer every 30 seconds. +2. **No user control.** The footer relays are clickable and the sidenav relays + are editable. Temporary relays can't be edited — they're driven by follows' + kind 10002 lists. Showing them as if they were user relays violates the + edit affordance. +3. **Scale.** A user following 500 people across 200 distinct relays would + see a footer full of icons they didn't choose. The footer is a health + indicator for *your* setup, not a live connection map. +4. **Redundancy.** [`setRelayActivityState`](www/js/relay-ui.mjs:145) already + shows read/write activity carrots on *your* relays when they carry traffic. + If a follow's post comes through a temporary relay and then gets re-fetched + on your relay (or vice versa), the activity indicator on your relay still + fires. The user sees activity; they don't need to see the temporary relay + itself. + +### Summary + +| Relay category | Pool | Carries feed posts? | Show in footer/sidenav? | Show in relay.html? | +|---|---|---|---|---| +| 1. Your read/both relays | `ndk.pool` (permanent) | Yes | Yes (current behavior) | Yes, editable (current) | +| 2. Outbox discovery relays | `ndk.outboxPool` | No (only kind 10002/3) | No | No | +| 3. Follows' write relays | `ndk.pool` (temporary, 30s TTL) | Yes | No | Optional: read-only section | + +The feed outbox fix (Phases 1–3) requires **no changes** to the relay UI. +The optional "Discovered relays" view in relay.html can be a follow-up task. + +--- + +## Amethyst comparison — what the most comprehensive Nostr client does + +I examined [`~/lt/amethyst`](../amethyst) (both the Android `amethyst/` module and +the `desktopApp/` module, plus the shared `commons/` and `quartz/` libraries). +Amethyst is indeed the most thorough relay implementation in the Nostr +ecosystem. Here is what it does that your client does not, and what it would +take to bring your client to parity. + +### Amethyst's relay set architecture + +Amethyst does not have one relay list. It has **eleven distinct relay lists**, +each backed by a different Nostr event kind, each serving a different purpose. +From [`Account.kt`](../amethyst/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt:330) +and the quartz event definitions: + +| Relay list | NIP / Kind | Purpose | Your client has it? | +|---|---|---|---| +| NIP-65 Outbox/Inbox | 10002 | Where you publish / where others find you | ✅ Yes ([`relay.html`](www/relay.html)) | +| DM Inbox | 10050 (NIP-17) | Where others send you encrypted DMs | ❌ No | +| Search | 10007 (NIP-50) | Relays for full-text search queries | ❌ No | +| Blocked | 10006 (NIP-51) | Relays you refuse to connect to | ❌ No | +| Trusted | NIP-51 private | Relays you trust for Tor routing | ❌ No | +| Proxy | NIP-51 private | Relays to use as proxies | ❌ No | +| Broadcast | NIP-51 private | Extra relays to broadcast every event to | ❌ No | +| Indexer | NIP-51 private | Relays for kind/author indexing lookups | ❌ No | +| Relay Feeds | NIP-51 private | Relays that serve as feed sources | ❌ No | +| Private Storage | 10037 (NIP-37) | Private outbox relays for drafts | ❌ No | +| KeyPackage | 10051 (MIP-00) | MLS key package discovery relays | ❌ No | + +On top of these **eleven published lists**, Amethyst computes **five derived +relay sets** by merging sources ([`Account.kt:415-462`](../amethyst/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt:415)): + +| Derived set | Sources | Used for | +|---|---|---| +| `homeRelays` | NIP-65 + private + local | Home feed subscriptions | +| `outboxRelays` | NIP-65 + private + local + broadcast | Where you publish your events | +| `dmRelays` | DM list + NIP-65 + private + local | DM send/receive | +| `notificationRelays` | NIP-65 read + local | Notifications/zap receipts | +| `followPlusAllMineWithIndex` | Follows' outboxes + your NIP-65 + indexer | Global feed with indexing | +| `followPlusAllMineWithSearch` | Follows' outboxes + your NIP-65 + search | Global feed with search | +| `defaultGlobalRelays` | Follows' outboxes + your NIP-65 | Global feed fallback | + +The key insight: **Amethyst's "outbox model" is not just NDK's +`outboxTracker`**. It is a multi-layer system where: + +1. **Your own relay lists** (11 kinds) define where *you* publish and what + *you* refuse. +2. **Derived sets** merge your lists with your follows' NIP-65 lists to + compute the actual relay set for each subscription type. +3. **Per-event routing** ([`Account.kt:1130-1290`](../amethyst/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt:1130)) + chooses relays dynamically: reactions go to the original author's inbox + relays + your outbox; replies go to the parent author's inbox + tagged + users' inboxes + your outbox; deletions go to your outbox + the original + event's relays. + +### What Amethyst's UI shows + +The desktop relay settings screen +([`RelayConfigTab.kt`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/RelayConfigTab.kt:56)) +has **five collapsible sections**, each independently editable: + +1. **Connected Relays** — the live pool, with add/remove. Collapsed by default. +2. **NIP-65 Inbox/Outbox** (kind 10002) — editable, publishes a kind 10002. +3. **DM Relays** (kind 10050) — editable, publishes a kind 10050. +4. **Search Relays** (kind 10007) — editable, publishes a kind 10007. +5. **Blocked Relays** (kind 10006) — editable, publishes a kind 10006. + +Each section has its own editor component +([`Nip65RelayEditor`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/Nip65RelayEditor.kt:70), +[`DmRelayEditor`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/DmRelayEditor.kt:59), +[`SearchRelayEditor`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/SearchRelayEditor.kt:58), +[`BlockedRelayEditor`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/BlockedRelayEditor.kt:57)) +that signs and publishes the appropriate event kind. + +There is also a **relay health system** +([`RelayHealthStore`](../amethyst/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relays/health/RelayHealthStore.kt:102), +[`ClassifyRelayHealth`](../amethyst/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relays/health/ClassifyRelayHealth.kt:53)) +that monitors which relays in your lists are actually responding, classifies +them as healthy/dead/snoozed, and shows an "Unhealthy Relays" popup with a +one-click "remove from all lists" action via +[`RelayListMutator.removeFromAllUserLists`](../amethyst/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relays/health/RelayListMutator.kt:50). + +### What it would take to add all of this to your client + +This is a large undertaking. Here is a phased breakdown, ordered by value. + +#### Tier 1 — High value, moderate effort (recommended next) + +These directly improve the feed and core UX: + +- [ ] **DM Relay List (kind 10050)** — Your client already does NIP-17 DMs + ([`www/msg.html`](www/msg.html)). Without a kind 10050 list, other + clients don't know where to send you DMs. Add: worker fetch of kind + 10050, a `dmRelayList` state, an editor section in `relay.html`, and + use it in the DM publish path. NDK does not manage this for you — it's + application-level. + +- [ ] **Search Relay List (kind 10007)** — If you have or want NIP-50 search, + this lets the user configure which search relays to query. Add: worker + fetch, state, editor, and use in search queries. + +- [ ] **Blocked Relay List (kind 10006)** — A relay-level blocklist (distinct + from your NIP-51 mute list which is pubkey-based). Add: worker fetch, + state, editor, and a `relayConnectionFilter` on the NDK instance so + NDK refuses to connect to blocked relays. This is the one that + integrates with NDK: set `ndk.relayConnectionFilter = (url) => + !blockedRelays.has(url)` and the outbox tracker will skip blocked + relays ([`www/ndk-core.bundle.js:42835`](www/ndk-core.bundle.js:42835)). + +#### Tier 2 — Medium value, moderate effort + +- [ ] **Relay health monitoring** — Periodic ping/connection checks on your + configured relays, with a UI indicator for dead relays and a "remove" + action. Your worker already has + [`startRelayHealthCheck`](www/ndk-worker.js:1595) but it only + reconnects; it doesn't classify or surface health to the UI. + +- [ ] **Per-event publish routing** — Instead of publishing to all your + relays, route events to the relevant relays: reactions to the original + author's inbox relays, replies to parent author + tagged users, etc. + This requires fetching the recipient's NIP-65 list (NDK's + `getWriteRelaysFor` can help) and is what makes Amethyst's events + reliably reach the right people. + +- [ ] **Broadcast Relay List (NIP-51 private)** — Extra relays to always + include when publishing. Some users want their events on more relays + than their NIP-65 list. Add: NIP-44-encrypted kind 10051 list, state, + editor, merge into `outboxRelays` at publish time. + +#### Tier 3 — Lower value, high effort (only if needed) + +- [ ] **Trusted / Proxy Relay Lists (NIP-51 private)** — Only relevant if you + add Tor support. Amethyst uses these to decide which relays route + through Tor ([`TorRelayEvaluation`](../amethyst/commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/tor/TorRelayEvaluation.kt:27)). + Without Tor, these are unnecessary. + +- [ ] **Indexer / Relay Feeds Lists (NIP-51 private)** — Specialized relay + lists for kind-discovery and feed-source relays. Only useful if you + build features that need them (e.g., "find all kind 30023 authors"). + +- [ ] **Private Storage Relay List (kind 10037)** — For NIP-37 drafts. Only + relevant if you implement draft events. + +- [ ] **KeyPackage Relay List (kind 10051)** — For MLS group messaging + (MIP-00). Only relevant if you implement the marmot/MLS protocol. + +- [ ] **Custom Relay Sets (kind 30002)** — Amethyst's quartz library defines + [`RelaySetEvent`](../amethyst/quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip51Lists/relaySets/RelaySetEvent.kt:44) + (kind 30002, NIP-51 parameterized) for user-defined named relay sets. + These are NIP-44-encrypted private lists with a title/description. This + is the "many different relay sets" feature you mentioned — users can + create arbitrary named sets (e.g., "My backup relays", "High-latency + relays") and reference them. This is a power-user feature; it requires + a full CRUD UI, NIP-44 encryption, and application-level logic to + consume the sets. + +### Architectural recommendation + +**Do not try to replicate Amethyst's architecture wholesale.** Amethyst is a +Kotlin/Compose app with a reactive state-flow system; your client is a +web-worker architecture built on NDK. The key differences: + +1. **NDK already handles outbox routing** (the `outboxTracker` + temporary + relays). Amethyst reimplements this in + [`FollowListOutboxOrProxyRelays`](../amethyst/amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt:455) + because its quartz relay pool does not have NDK's automatic outbox model. + You do **not** need to reimplement follow-outbox computation — you need to + pre-warm it (Phases 1–3 above). + +2. **NDK does not manage application-level relay lists** (DM, search, blocked, + etc.). These are your responsibility. Amethyst has 11 state classes for + these; you would need equivalent worker-side state and relay.html editor + sections. + +3. **The highest-ROI additions** are the Blocked list (because it integrates + with NDK's `relayConnectionFilter` and improves outbox routing safety) and + the DM list (because you already do NIP-17 DMs but don't advertise where + to send them). These two alone would bring your client to functional + parity with Amethyst for most users. + +4. **The "many relay sets" feature** (kind 30002) is the most complex and + lowest-ROI. Defer it unless you have a specific use case. Amethyst itself + does not heavily surface custom relay sets in its main UI — they exist in + the quartz library but the desktop `RelayConfigTab` only shows the 5 + fixed sections. + +### Summary: what to do now vs. later + +| Priority | Task | Effort | Depends on feed fix? | +|---|---|---|---| +| **Now** | Feed outbox pre-warm (Phases 1–3) | Small | — | +| **Now** | Discovered relays read-only view in relay.html | Small | No | +| **Next** | Blocked relay list (kind 10006) + `relayConnectionFilter` | Medium | No | +| **Next** | DM relay list (kind 10050) | Medium | No | +| **Later** | Search relay list (kind 10007) | Medium | No | +| **Later** | Relay health UI | Medium | No | +| **Later** | Per-event publish routing | Large | No | +| **Later** | Broadcast relay list (NIP-51) | Medium | No | +| **Maybe** | Trusted/Proxy/Indexer/RelayFeeds/PrivateStorage/KeyPackage lists | Large | No | +| **Maybe** | Custom relay sets (kind 30002) | Large | No | + +--- + +## relays.html page — current state and redesign plan + +I examined [`www/relays.html`](www/relays.html) (1464 lines). Here is what it +does today, the problems you identified, and a concrete plan for the page. + +### Current structure + +The page has two main areas: + +1. **Relay table** ([`createRelayTable`](www/relays.html:553), lines 553–748) — + A table with columns: Remove | Relay URL | Connected | Read | Write | + DM Inbox | Reads | Writes | Connection Time. Each row is a relay from your + kind 10002 list. The Read/Write checkboxes toggle the NIP-65 marker and + republish kind 10002. The DM Inbox checkbox toggles membership in your + kind 10050 list ([`publishDmInboxRelayList`](www/relays.html:1010)). There + is an add-relay row at the bottom with the same toggles. The table + refreshes every 5 seconds + ([`setInterval(refreshRelayData, 5000)`](www/relays.html:1446)). + +2. **Connection history stream** ([`divRelayEvents`](www/relays.html:260), + lines 859–939, 1200–1272) — A chat-like log of every relay frame + (REQ/EVENT/OK/EOSE/NOTICE/AUTH/etc.) for the selected relay. Events arrive + via `ndkRelayEvent` window events + ([line 1405](www/relays.html:1405)), which the worker broadcasts on every + relay message via [`broadcastRelayEvent`](www/ndk-worker.js:917). The + worker persists each event to IndexedDB + ([`writeRelayEventToDb`](www/ndk-worker.js:709)) and prunes to 200 per + relay every 60s ([`RELAY_EVENT_LOG_LIMIT = 200`](www/ndk-worker.js:307), + [`RELAY_EVENTS_PRUNE_INTERVAL_MS`](www/ndk-worker.js:311)). The page loads + history from IndexedDB on relay selection + ([`loadRelayEventsFromDb`](www/relays.html:859)) and appends live events. + +### Problem 1 — Connection history is heavy and slows the app + +The performance issue has three layers: + +1. **Worker-side: every relay frame is persisted to IndexedDB.** + [`logRelayEvent`](www/ndk-worker.js:899) calls + `void writeRelayEventToDb(entry)` on every single relay message. On an + active feed with 5+ relays, this is dozens of IndexedDB writes per second. + Each write opens a transaction, adds a row, and closes. This is fire-and- + forget but the I/O contention on the `relay-events` IndexedDB database + competes with the NDK cache adapter's own IndexedDB traffic. + +2. **Worker-side: every relay frame is broadcast to all pages.** + [`broadcastRelayEvent`](www/ndk-worker.js:917) sends every event to every + connected port. Pages that don't care about relay events (feed.html, + msg.html, etc.) still receive and dispatch them. On `relays.html` itself, + the `ndkRelayEvent` listener + ([line 1405](www/relays.html:1405)) calls `logRelayEvent` which appends to + the in-memory array and re-renders the entire event stream DOM + ([`renderRelayEventsForSelectedRelay`](www/relays.html:915)) on every + frame. + +3. **Page-side: full DOM re-render on every event.** + `renderRelayEventsForSelectedRelay` does `divRelayEvents.innerHTML = ...` + with up to 1000 entries ([line 1261](www/relays.html:1261) caps at 1000). + On a busy relay, this is a full innerHTML replacement every few + milliseconds. This is the direct cause of the slowdown you see. + +### Solution: a toggle for connection history + +The fix is to make the connection history opt-in, both on the page and in the +worker. The toggle goes in the relays.html sidenav, replacing the placeholder +text "Relay management options coming soon..." +([`openNav`](www/relays.html:420) sets `divSideNavBody.innerHTML` to that +string). It uses the same `sidenavSection` / `sidenavRowToggle` pattern as +[`www/feed.html`](www/feed.html:99) uses for "Autoplay videos": + +```html +
+
Relays
+ +
+``` + +This requires adding the +[`sidenav-sections.css`](www/css/sidenav-sections.css) stylesheet to +`relays.html` (it is not currently included — the page uses only +`client.css`). + +**Implementation:** This is covered by **Phase 1** (worker: +`setRelayEventLogging` RPC + flag that stops both broadcasting and IndexedDB +writes) and **Phase 4** (relays.html: sidenav toggle, show/hide the event +stream, skip DB loads, incremental DOM updates) in the unified plan above. +Default: **off** — the history is a debug tool. + +### Problem 2 — Where to show outbox / discovered relays + +You want to visualize the outbox model working. Today the table only shows +your kind 10002 relays. After the feed outbox fix (Phases 1–3), NDK will be +connecting to your follows' write relays as temporary relays — but they are +invisible on this page. + +The right design is a **second table below the main one**, clearly labeled as +discovered/outbox relays, read-only. + +These are implemented in **Phase 1** (worker RPC), **Phase 2** (page API), +and **Phase 5** (relays.html table) of the unified plan above. The worker +exposes `getDiscoveredRelays` (Phase 1.5–1.6), the page API wraps it (Phase +2.3), and relays.html renders it in a collapsible "Discovered Relays +(Outbox)" section with columns: Relay URL | Connected | Serving (count of +follows) | Sample Follows (first 3 npubs). Read-only — no toggles, no +remove. This is where you see the outbox model working: when you load the +feed, this table populates with your follows' write relays. An optional +"Outbox discovery" status line shows the outbox pool relays +(`purplepag.es`, `nos.lol`) and their connection state. + +### Problem 3 — Setting different relay types (Amethyst-style sections) + +Today the table has Read / Write / DM Inbox columns. To reach Amethyst-level +relay management, the page should grow **collapsible sections** for each +relay list kind, rather than cramming everything into one table's columns. + +Recommended layout (top to bottom): + +1. **Your Relays (NIP-65, kind 10002)** — the current table, with Read/Write + toggles. This is what you have now. Keep as-is. + +2. **DM Inbox Relays (kind 10050)** — Currently a column in the main table. + Move to its own section so it's not conflated with NIP-65 read/write. + Each relay has a remove button. Add-relay input at the bottom. This is a + refactor of the existing DM Inbox column into a dedicated section. + +3. **Search Relays (kind 10007)** — New section. Editable list. Publishes + kind 10007. Requires worker support to fetch/publish kind 10007 (new RPC). + +4. **Blocked Relays (kind 10006)** — New section. Editable list. Publishes + kind 10006. Requires worker support, and setting + `ndk.relayConnectionFilter` so NDK refuses to connect to these relays. + +5. **Discovered Relays (Outbox)** — Read-only, from Phase 5 above. + +6. **Connection History** — The debug stream, behind the toggle from Phase 4. + +Each section is collapsible (like Amethyst's +[`CollapsibleSection`](../amethyst/desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/relay/RelayConfigTab.kt:168)) +so the page doesn't become overwhelming. Default state: NIP-65 expanded, DM +Inbox expanded, others collapsed. + +### Phase 7 — relays.html: refactor DM Inbox into its own section + +**File:** [`www/relays.html`](www/relays.html) + +- [ ] **7.1** Remove the "DM Inbox" column from the main relay table + ([`createRelayTable`](www/relays.html:553), line 591 header and line 642 + body cell). +- [ ] **7.2** Add a new collapsible "DM Inbox Relays (kind 10050)" section + below the main table. Each relay in `dmInboxRelays` gets a row with a + remove button. Add-relay input at the bottom. Reuses the existing + [`publishDmInboxRelayList`](www/relays.html:1010) logic. +- [ ] **7.3** The add-relay row in the main table loses its DM Inbox toggle + (line 658). DM inbox relays are added from the DM Inbox section instead. + +### Phase 8 — relays.html: Search relays section (kind 10007) + +**Files:** [`www/ndk-worker.js`](www/ndk-worker.js), [`www/js/init-ndk.mjs`](www/js/init-ndk.mjs), [`www/relays.html`](www/relays.html) + +- [ ] **8.1** Worker: add fetch/publish for kind 10007. Add a + `handleGetSearchRelayList` handler that fetches kind 10007 for the + current pubkey, and a `handlePublishSearchRelayList` handler that + publishes a kind 10007 event with the relay list. +- [ ] **8.2** Page API: add `getSearchRelayList()` and + `publishSearchRelayList(relays)` exports in init-ndk.mjs. +- [ ] **8.3** relays.html: Add a collapsible "Search Relays (kind 10007)" + section. Editable list with add/remove. Loads from + `getSearchRelayList()` on init, publishes via + `publishSearchRelayList()` on change. + +### Phase 9 — relays.html: Blocked relays section (kind 10006) + relayConnectionFilter + +**Files:** [`www/ndk-worker.js`](www/ndk-worker.js), [`www/js/init-ndk.mjs`](www/js/init-ndk.mjs), [`www/relays.html`](www/relays.html) + +- [ ] **9.1** Worker: add fetch/publish for kind 10006. Add a + `handleGetBlockedRelayList` handler and a + `handlePublishBlockedRelayList` handler. +- [ ] **9.2** Worker: set `ndk.relayConnectionFilter` to skip blocked relays. + When the blocked relay list is loaded/updated, set + `ndk.relayConnectionFilter = (url) => !blockedRelays.has(normalizeRelayUrl(url))`. + This makes NDK's outbox tracker skip blocked relays + ([`www/ndk-core.bundle.js:42835`](www/ndk-core.bundle.js:42835)) and + prevents temporary relay connections to blocked relays. +- [ ] **9.3** Page API: add `getBlockedRelayList()` and + `publishBlockedRelayList(relays)` exports. +- [ ] **9.4** relays.html: Add a collapsible "Blocked Relays (kind 10006)" + section. Editable list with add/remove. + +### Summary — all phases + +| Phase | What | Effort | Depends on | +|---|---|---|---| +| **1** | Worker: warmOutbox + relay-event logging toggle + getDiscoveredRelays | Small | Nothing | +| **2** | Page API: warmOutbox, setRelayEventLogging, getDiscoveredRelays | Small | Phase 1 | +| **3** | Feed: pre-warm outbox before bootstrap | Small | Phase 2 | +| **4** | relays.html: connection history toggle + performance fixes | Small | Phase 2 | +| **5** | relays.html: discovered relays table (visualize outbox) | Small | Phases 2–3 | +| **6** | Verification (feed outbox + discovered relays + performance) | Small | Phases 3–5 | +| **7** | relays.html: refactor DM Inbox into its own section | Medium | Nothing | +| **8** | relays.html: Search relays section (kind 10007) | Medium | Nothing | +| **9** | relays.html: Blocked relays section (kind 10006) + relayConnectionFilter | Medium | Nothing | + +**Phases 1–6 are the core deliverable** — they fix the feed, fix the +performance, and let you see the outbox model working. Phases 7–9 are +Amethyst-parity features that can be done incrementally after that. diff --git a/www/feed.html b/www/feed.html index 023061f..4e7ad01 100644 --- a/www/feed.html +++ b/www/feed.html @@ -163,7 +163,8 @@ onUserSettings, ensureMuteListLoaded, isEventMuted, - addMute + addMute, + warmOutbox } from './js/init-ndk.mjs'; import { HamburgerMorphing } from './hamburger_morphing/hamburger.mjs'; import { initFooterRelayStatus, updateFooterRelayStatus, initSidenavRelaySection, updateSidenavRelaySection, setRelayActivityState } from './js/relay-ui.mjs'; @@ -555,6 +556,10 @@ import { initPostCards } from './js/post-interactions2.mjs'; const now = Math.floor(Date.now() / 1000); const since = newestKnown > 0 ? Math.max(0, newestKnown - 30) : (now - 3600); + // Fire-and-forget: NDK.subscribe already calls trackUsers internally, + // but doing it explicitly avoids relying on internal timing. + void warmOutbox(authors).catch(() => {}); + subscribe( { kinds: [1], authors, since }, { closeOnEose: false, cacheUsage: 'CACHE_FIRST' } @@ -577,11 +582,23 @@ import { initPostCards } from './js/post-interactions2.mjs'; const allAuthors = Array.from(feedPubkeys); if (!hasBootstrappedFeed) { + // Pre-warm the outbox tracker so NDK knows each follow's write relays + // before the short-lived bootstrap fetchEvents calls fire. + try { + await warmOutbox(followedPubkeys); + } catch (e) { + console.warn('[feed.html] warmOutbox failed (non-blocking):', e?.message || e); + } await bootstrapFeedPosts(allAuthors); return; } if (newAuthors.length > 0) { + try { + await warmOutbox(newAuthors); + } catch (e) { + console.warn('[feed.html] warmOutbox for new authors failed (non-blocking):', e?.message || e); + } try { await fetchFeedWindow(newAuthors, { limit: INITIAL_POSTS_LOAD }); renderFeed(); diff --git a/www/js/init-ndk.mjs b/www/js/init-ndk.mjs index 93b5350..bf6f0a9 100644 --- a/www/js/init-ndk.mjs +++ b/www/js/init-ndk.mjs @@ -433,6 +433,37 @@ function handleWorkerMessage(event) { pending.resolve(message.events || []); } } + } else if (message.type === 'warmOutboxResult') { + // Handle warmOutbox response — best-effort pre-warm; always resolves. + const pending = pendingRequests.get(message.requestId); + if (pending) { + pendingRequests.delete(message.requestId); + pending.resolve({ + ok: !!message.ok, + count: message.count || 0, + skipped: !!message.skipped, + error: message.error || null + }); + } + } else if (message.type === 'setRelayEventLoggingResult') { + // Handle setRelayEventLogging response — fire-and-forget, but resolve + // the promise if the caller chose to await it. + const pending = pendingRequests.get(message.requestId); + if (pending) { + pendingRequests.delete(message.requestId); + pending.resolve({ + ok: !!message.ok, + enabled: !!message.enabled, + error: message.error || null + }); + } + } else if (message.type === 'getDiscoveredRelaysResult') { + // Handle getDiscoveredRelays response + const pending = pendingRequests.get(message.requestId); + if (pending) { + pendingRequests.delete(message.requestId); + pending.resolve(message.relays || []); + } } else if (message.type === 'fetchCachedProfileResult') { // Handle cached profile response const pending = pendingRequests.get(message.requestId); @@ -763,6 +794,128 @@ export async function ndkFetchEvents(filters) { }); } +/** + * Pre-warm the NDK outbox tracker for a set of follow pubkeys. + * + * This proactively resolves each follow's kind 10002 relay list via + * ndk.outboxTracker.trackUsers so that subsequent short-lived fetchEvents + * subscriptions route to the correct write relays instead of racing the + * tracker. Pre-warming is best-effort: the returned promise always resolves + * (never rejects). On timeout it resolves with { ok: false, timedOut: true }. + * + * @param {string[]} pubkeys - Array of hex pubkeys to pre-warm. + * @returns {Promise<{ok: boolean, count?: number, skipped?: boolean, timedOut?: boolean, error?: string}>} + */ +export async function warmOutbox(pubkeys) { + if (!ndkWorker) { + throw new Error('NDK worker not initialized. Call initNDKPage() first.'); + } + + const list = Array.isArray(pubkeys) ? pubkeys.filter(Boolean) : []; + if (list.length === 0) { + return { ok: true, count: 0, skipped: true }; + } + + return new Promise((resolve) => { + requestCounter++; + const requestId = `warmOutbox_${Date.now()}_${requestCounter}`; + + pendingRequests.set(requestId, { resolve, reject: resolve }); + + // Best-effort: resolve (not reject) on timeout. + setTimeout(() => { + if (pendingRequests.has(requestId)) { + pendingRequests.delete(requestId); + resolve({ ok: false, timedOut: true }); + } + }, 15000); + + ndkWorker.port.postMessage({ + type: 'warmOutbox', + requestId: requestId, + pubkeys: list + }); + }); +} + +/** + * Toggle relay event logging in the worker. + * + * When enabled, the worker broadcasts relayEvent messages to all ports and + * persists relay events to IndexedDB. When disabled (default), both are + * skipped to avoid cross-page broadcast spam and IndexedDB write contention. + * trackRelayRead/trackRelayWrite (relayActivity events) are not affected. + * + * Fire-and-forget: the returned promise resolves when the worker acknowledges, + * but callers may ignore it. + * + * @param {boolean} enabled - Whether relay event logging should be on. + * @returns {Promise<{ok: boolean, enabled: boolean, error?: string}>} + */ +export function setRelayEventLogging(enabled) { + if (!ndkWorker) { + throw new Error('NDK worker not initialized. Call initNDKPage() first.'); + } + + return new Promise((resolve) => { + requestCounter++; + const requestId = `setRelayEventLogging_${Date.now()}_${requestCounter}`; + + pendingRequests.set(requestId, { resolve, reject: resolve }); + + // Fire-and-forget timeout — resolve gracefully if the worker never + // acknowledges (it always should, but be safe). + setTimeout(() => { + if (pendingRequests.has(requestId)) { + pendingRequests.delete(requestId); + resolve({ ok: false, enabled: !!enabled, error: 'setRelayEventLogging timeout' }); + } + }, 5000); + + ndkWorker.port.postMessage({ + type: 'setRelayEventLogging', + requestId: requestId, + enabled: !!enabled + }); + }); +} + +/** + * Get the set of "discovered" relays currently in the NDK pool that are NOT + * part of the user's own kind 10002 relay list. These are temporary outbox + * relays NDK connected to in order to fetch events from followed authors who + * write to relays the user does not subscribe to. + * + * Each entry is { url, status, connected, servingPubkeys: string[] }. + * + * @returns {Promise>} + */ +export async function getDiscoveredRelays() { + if (!ndkWorker) { + throw new Error('NDK worker not initialized. Call initNDKPage() first.'); + } + + return new Promise((resolve) => { + requestCounter++; + const requestId = `getDiscoveredRelays_${Date.now()}_${requestCounter}`; + + pendingRequests.set(requestId, { resolve, reject: resolve }); + + // Resolve with [] on timeout — this is a read-only diagnostic call. + setTimeout(() => { + if (pendingRequests.has(requestId)) { + pendingRequests.delete(requestId); + resolve([]); + } + }, 5000); + + ndkWorker.port.postMessage({ + type: 'getDiscoveredRelays', + requestId: requestId + }); + }); +} + /** * Fetch events from all relays in parallel * Returns events grouped by relay URL diff --git a/www/js/version.json b/www/js/version.json index a0e0cc1..299d46e 100644 --- a/www/js/version.json +++ b/www/js/version.json @@ -1,5 +1,5 @@ { - "VERSION": "v0.7.39", - "VERSION_NUMBER": "0.7.39", - "BUILD_DATE": "2026-06-26T11:26:09.788Z" + "VERSION": "v0.7.40", + "VERSION_NUMBER": "0.7.40", + "BUILD_DATE": "2026-06-26T13:38:31.838Z" } diff --git a/www/ndk-worker.js b/www/ndk-worker.js index 8ce43cd..815d0dc 100644 --- a/www/ndk-worker.js +++ b/www/ndk-worker.js @@ -304,6 +304,12 @@ const NUTZAP_MINTLIST_PREFETCH_BATCH = 50; // max authors per fetchEvents call const NUTZAP_MINTLIST_PREFETCH_TTL_MS = 6 * 60 * 60 * 1000; // 6h freshness guard // Relay activity tracking +// When false (default), broadcastRelayEvent skips the cross-page broadcast() +// and logRelayEvent skips the IndexedDB write. trackRelayRead/trackRelayWrite +// (which emit relayActivity events for footer/sidenav carrots) are NOT gated +// by this flag and always run. Toggled on by the relays page via the +// setRelayEventLogging RPC when the user opens connection history. +let relayEventLoggingEnabled = false; const RELAY_EVENT_LOG_LIMIT = 200; const RELAY_EVENTS_DB_NAME = 'relay-events'; const RELAY_EVENTS_DB_VERSION = 2; @@ -908,13 +914,24 @@ function logRelayEvent(relayUrl, eventType, data = undefined, side = 'relay', ti data: serializeRelayEventData(data) }; - // Fire-and-forget persistence so relay handling is never blocked by IndexedDB I/O. - void writeRelayEventToDb(entry); + // Only persist to IndexedDB when relay event logging is enabled. + // When the relays page (connection history) is closed, this avoids + // IndexedDB write contention from the high-frequency relay event stream. + if (relayEventLoggingEnabled) { + // Fire-and-forget persistence so relay handling is never blocked by IndexedDB I/O. + void writeRelayEventToDb(entry); + } return entry; } function broadcastRelayEvent(relayUrl, eventType, data = undefined, side = 'relay') { + // Skip the cross-page broadcast entirely when relay event logging is off. + // This eliminates relayEvent message spam to every open tab/page when the + // connection-history UI is not visible. trackRelayRead/trackRelayWrite + // (relayActivity events) are intentionally NOT gated and still broadcast. + if (!relayEventLoggingEnabled) return; + const entry = logRelayEvent(relayUrl, eventType, data, side, Date.now()); if (!entry) return; @@ -6345,6 +6362,196 @@ async function handleNdkFetchEvents(requestId, filters, port) { } } +// Handle outbox pre-warm request — proactively resolves each follow's kind +// 10002 relay list via ndk.outboxTracker.trackUsers so that subsequent +// short-lived fetchEvents subscriptions route to the correct write relays +// instead of racing the tracker. Pre-warming is best-effort: this handler +// never throws; it always responds with an ok/skipped/error payload. +async function handleWarmOutbox(requestId, pubkeys, port) { + try { + if (!ndk?.outboxTracker) { + port.postMessage({ + type: 'warmOutboxResult', + requestId, + ok: true, + count: 0, + skipped: true + }); + return; + } + + const list = Array.isArray(pubkeys) ? pubkeys.filter(Boolean) : []; + if (list.length === 0) { + port.postMessage({ + type: 'warmOutboxResult', + requestId, + ok: true, + count: 0, + skipped: true + }); + return; + } + + await ndk.outboxTracker.trackUsers(list); + + port.postMessage({ + type: 'warmOutboxResult', + requestId, + ok: true, + count: list.length + }); + } catch (err) { + console.warn('[Worker] warmOutbox failed:', err?.message || err); + port.postMessage({ + type: 'warmOutboxResult', + requestId, + ok: false, + error: err?.message || String(err) + }); + } +} + +// Handle relay event logging toggle. When enabled, broadcastRelayEvent will +// emit relayEvent messages to all ports and logRelayEvent will persist entries +// to IndexedDB. When disabled (default), both are skipped to avoid cross-page +// broadcast spam and IndexedDB write contention. trackRelayRead/trackRelayWrite +// are NOT affected by this flag. +async function handleSetRelayEventLogging(requestId, enabled, port) { + try { + relayEventLoggingEnabled = !!enabled; + port.postMessage({ + type: 'setRelayEventLoggingResult', + requestId, + ok: true, + enabled: relayEventLoggingEnabled + }); + } catch (err) { + console.warn('[Worker] setRelayEventLogging failed:', err?.message || err); + port.postMessage({ + type: 'setRelayEventLoggingResult', + requestId, + ok: false, + error: err?.message || String(err), + enabled: relayEventLoggingEnabled + }); + } +} + +// Handle getDiscoveredRelays request — returns relays currently in the NDK +// pool that are NOT part of the user's own kind 10002 relay list (relayTypes). +// These are temporary outbox relays NDK connected to in order to fetch events +// from followed authors who write to relays the user does not subscribe to. +// For each discovered relay we also report which followed pubkeys it serves, +// by inverting ndk.outboxTracker.data (pubkey -> OutboxItem{writeRelays}). +async function handleGetDiscoveredRelays(requestId, port) { + try { + if (!ndk?.pool?.relays) { + port.postMessage({ + type: 'getDiscoveredRelaysResult', + requestId, + relays: [] + }); + return; + } + + // Build the set of the user's own relay URLs (normalized) from + // relayTypes (kind 10002). These are the "configured" relays; anything + // else in the pool is a discovered/temporary outbox relay. + const ownRelays = new Set(); + for (const url of relayTypes.keys()) { + const normalized = normalizeRelayUrl(url); + if (normalized) ownRelays.add(normalized); + } + + // Build relayUrl -> [pubkeys] map by inverting the outbox tracker. + // ndk.outboxTracker.data is an LRUCache (typescript-lru-cache). + // We try a few iteration strategies to be resilient to API differences. + const relayServesPubkeys = new Map(); // normalizedUrl -> Set + try { + const trackerData = ndk?.outboxTracker?.data; + if (trackerData) { + const collectForPubkey = (pubkey) => { + try { + const outboxItem = trackerData.get(pubkey); + const writeRelays = outboxItem?.writeRelays; + if (!writeRelays) return; + for (const r of writeRelays) { + const rUrl = r?.url ? normalizeRelayUrl(r.url) : (typeof r === 'string' ? normalizeRelayUrl(r) : null); + if (!rUrl) continue; + if (!relayServesPubkeys.has(rUrl)) { + relayServesPubkeys.set(rUrl, new Set()); + } + relayServesPubkeys.get(rUrl).add(pubkey); + } + } catch (_) { + // ignore per-pubkey errors + } + }; + + // Strategy 1: standard Map-like entries() + let iterated = false; + try { + if (typeof trackerData.entries === 'function') { + for (const [pubkey] of trackerData.entries()) { + collectForPubkey(pubkey); + } + iterated = true; + } + } catch (_) {} + + // Strategy 2: keys() + get() + if (!iterated) { + try { + if (typeof trackerData.keys === 'function') { + for (const pubkey of trackerData.keys()) { + collectForPubkey(pubkey); + } + iterated = true; + } + } catch (_) {} + } + + // Strategy 3: iterate as Map directly + if (!iterated && trackerData instanceof Map) { + for (const pubkey of trackerData.keys()) { + collectForPubkey(pubkey); + } + } + } + } catch (err) { + console.warn('[Worker] getDiscoveredRelays: outbox tracker inversion failed:', err?.message || err); + } + + const relays = []; + for (const relay of ndk.pool.relays.values()) { + const normalized = normalizeRelayUrl(relay?.url); + if (!normalized) continue; + // Skip relays that are part of the user's own kind 10002 list. + if (ownRelays.has(normalized)) continue; + + const servingSet = relayServesPubkeys.get(normalized); + relays.push({ + url: relay.url, + status: relay.status, + connected: relay.status >= 5, + servingPubkeys: servingSet ? Array.from(servingSet) : [] + }); + } + + port.postMessage({ + type: 'getDiscoveredRelaysResult', + requestId, + relays + }); + } catch (err) { + console.warn('[Worker] getDiscoveredRelays failed:', err?.message || err); + port.postMessage({ + type: 'getDiscoveredRelaysResult', + requestId, + relays: [] + }); + } +} function handleSignResponse(requestId, result, error) { const pending = pendingSignRequests.get(requestId); @@ -6650,7 +6857,9 @@ self.onconnect = (event) => { playlistIdentifier, ownerPubkey, playlistKind, - eventIds + eventIds, + pubkeys, + enabled } = e.data; switch(type) { @@ -6690,6 +6899,18 @@ self.onconnect = (event) => { await handleNdkFetchEvents(requestId, filters, port); break; + case 'warmOutbox': + await handleWarmOutbox(requestId, pubkeys, port); + break; + + case 'setRelayEventLogging': + await handleSetRelayEventLogging(requestId, enabled, port); + break; + + case 'getDiscoveredRelays': + await handleGetDiscoveredRelays(requestId, port); + break; + case 'setActiveSigningPort': activeSigningPort = port; console.log('[Worker] Active signing port set explicitly by page'); diff --git a/www/relays.html b/www/relays.html index ba65c11..09bab34 100644 --- a/www/relays.html +++ b/www/relays.html @@ -7,6 +7,7 @@ RELAYS +