A durable, shareable relay-reachability cache backed by the EventStore as NIP-66
kind:30166 Relay Discovery events, so the crawler, the WoT updater, and future
runs share liveness knowledge instead of each rediscovering dead relays from an
in-memory set wiped at process exit.
- 30166 is addressable by its d-tag (relay URL) → one replaceable status slot per
(monitor, relay), with created_at giving a free TTL.
- Reachable → 30166 with rtt-open; dead → 30166 without (NIP-66 has no explicit
offline field; liveness is inferred from a fresh successful open). Live wins
over dead within the TTL, so third-party monitors' 30166 can be ingested.
- snapshot() loads the fresh set once (not a per-request hot-path query); record()
flushes a run's findings. A relay is only skipped for the TTL, never permanently
— consistent with the outbox rule that every advertised write relay is tried.
Reuses the existing RelayDiscoveryEvent. jvmTest covers record/reload,
live-overrides-dead, TTL expiry, and .onion→Tor network tagging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
- Correct the AUTH-approval persistence check: this JVM uses
MacOSXPreferences (~/Library/Preferences/com.vitorpamplona.amethyst.plist),
not ~/.java/.userPrefs. Updated T3.b/c/d to read it via plutil.
- Record session results (T1,T2,T3a,T3c,T6,T6b,T8,T12 pass) and the three
bugs found+fixed during the run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The relay-hint plumbing existed end to end in quartz (NIP17Factory ->
GiftWrapEvent.create(recipientRelayHint), with GiftWrapRelayHintTest), but
DesktopIAccount called createMessageNIP17/createEncryptedFileNIP17 without
passing recipientRelayHints. The default {null} lambda meant every outgoing
gift wrap shipped a 2-element ["p", pubkey] tag — the hint feature was dead
in production. Manual testing (T8) caught this: wraps on the recipient's
inbox relay had no third element.
Pre-resolve each recipient's primary DM inbox relay (first entry of their
kind:10050, order-preserving) and pass it as the hint, yielding
["p", pubkey, "wss://primary-relay/"]. Recipients with no resolvable
kind:10050 map to null and keep the 2-element shape.
Adds resolveDmInboxRelaysStrictOrdered (order-preserving) as the basis for
both the target-relay set and the primary-relay hint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new-conversation picker rendered a pasted npub as a non-clickable
Surface whenever getUserIfExists returned null — i.e. for any recipient
whose kind:0 metadata the local cache hadn't seen. The npub showed in the
results list but couldn't be selected, so you couldn't start a DM to
anyone new by npub.
A DM recipient is identified purely by pubkey; metadata is not required to
open a conversation. Use getOrCreateUser so a valid npub always resolves to
a selectable UserSearchCard, keeping the non-clickable fallback only for
keys that can't be resolved at all.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round-8 profiling showed the crawl re-querying the same never-had-a-10002
users' outboxes every round they recirculated — ~144k slow kind:10002 drains
(p50 17.4s) against a static discovery set, dragging the round to ~18 users/s.
1. ensureRelayLists guards with `relayListDiscoverySwept`: each user's outbox
discovery runs once. The discovery relay set is static, so a second sweep of
a user still lacking a 10002 cannot find one the first missed.
2. The discovery REQ to the bounded INDEXER set co-fetches [10002, 3]: the
outbox lookup already pays the round-trip and an indexer holding a user's
10002 often holds their kind:3, so we harvest the contact list as a cheap
byproduct. The wide "every live relay" completeness sweep stays 10002-ONLY —
co-fetching kind:3 across thousands of relays downloaded the same big contact
lists repeatedly and inflated the fire-and-forget bgScope sweep the finishing
drain waits on (measured +260s at hop-3; the indexer-only co-fetch keeps
coverage flat at baseline speed).
3. harvestFromStore folds any already-stored kind:3 into the graph at Phase-A
time so Phase B never re-drains a list we hold (also speeds re-runs).
Verified same-session hop-3: pre-fix 685s / narrowed 690s / wide-co-fetch 945s,
coverage 91.74% across all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Covers the review fix — recipient with NIP-65 read relays but no
kind:10050 must be treated as unreachable, not routed to the read relays.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The indexer fan-out already used lists.dmInbox (strict), but the
LocalCache fast-path in DmInboxRelayResolver.resolve() — and the
no-resolver fallback in DesktopIAccount — went through the lenient
User.dmInboxRelays(), which falls back to NIP-65 read relays (kind:10002)
when the recipient has no kind:10050.
Because that fast-path returns first and short-circuits the strict
indexer lookup, a recipient with NIP-65 read relays but no published
DM-inbox would get gift wraps published to relays they never designated
for DMs — re-introducing the metadata leak (recipient pubkey + send
timing) the P0 fix was meant to close. LocalCache commonly holds
kind:10002 but not kind:10050, so this path fired often.
Switch both LocalCache lookups to dmInboxRelaysStrict().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-relay pending maps in RelayLatencyTracker are
Collections.synchronizedMap(LinkedHashMap): individual read/write
ops are thread-safe, but per the synchronizedMap javadoc iteration
is NOT — callers MUST hold the map's monitor while walking its
views. sweep() was iterating directly, so any network-dispatcher
mutation (adding a pending REQ, receiving an OK) during a sweep
would throw ConcurrentModificationException on AWT-EventQueue-0,
killing the Compose renderer while coroutine work kept running.
Pre-existing bug, documented in memory
desktop_relay_health_cme_crash. Ordinarily "not our problem", but
it's actively blocking manual T3 testing of this branch's AUTH
approval banner: adding any new relay triggers a
RelayHealthStore.reclassify sweep, so testers can't get a banner
render in without hitting the crash. Fix it here so the branch is
actually testable end-to-end.
Wrap both iteration loops in synchronized(pending) blocks. Sweep is
O(pending) with typically single-digit entries per relay, so the
hold time is negligible and the network dispatcher just briefly
waits.
Reproduced during manual T3 testing 2026-07-06 when adding
wss://pyramid.fiatjaf.com. Stack: RelayLatencyTracker.sweep:182 →
RelayHealthStore$reclassify$flagged$1.invokeSuspend:268.
DesktopAuthCoordinator.selfApprovedRelaysFor was calling the lenient
User.dmInboxRelays() helper, whose NIP-65-read fallback silently
expanded the tier-1 auto-allow set to include every relay in the
user's read markers. That defeated the tier-2 prompt for any AUTH-
required relay the user happened to have in NIP-65 — such as
wss://pyramid.fiatjaf.com, which never surfaced a banner during
manual testing because the coordinator was auto-signing it as
tier-1.
Switch to dmInboxRelaysStrict() (added in the earlier pre-send
alignment fix) so tier-1 is truly kind:10050 only. The KDoc already
promised strictness ("write/read relays are NOT included") — this
just makes the code match.
Surfaced during manual T3 testing 2026-07-06 with an account whose
NIP-65 outbox included pyramid.fiatjaf.com.
Rewrites the manual testing sheet so every step is an executable action
or a specific observation:
- Numbered steps within each test — no jumping between reference
sections. "Click X", "run Y in a terminal", "watch for Z".
- Each observation records YES/NO/SKIPPED for the sign-off matrix.
- Setup section spells out the wipe-preferences command and the
post-restart smoke check.
- T3 broken into T3.a/b/c/d for each button and its persistence
check separately (previously bundled T3.1–T3.10 was too dense).
- T6/T7 include exact tcpdump/tshark commands for the security
observations that require packet capture.
- T13 shortened to a sanity re-check (already verified).
- Sign-off table lists every test with a checkbox.
Purpose: give the tester a self-contained document they can follow
top to bottom in ~40 min without cross-referencing other sheets.
Updates the manual walkthrough to reflect the full state of the branch
after rebase + banner padding fix + pre-send validation fix:
- Marks what's already verified (T1 startup wiring, T5 lifecycle,
T6 P0 security via upstream UI, T13 pre-send fix).
- Adds T13 covering the alignment + resolver-probe fix from
a139c0fb17 with three sub-scenarios (strict alignment, resolver
unblock, and the real 2-user bootstrap flow).
- Documents the known pre-existing issues surfaced during testing
(RelayLatencyTracker CME, CompressionQuality stale-daemon) so
they don't get filed as regressions of ours.
- Adds a summary table pointing to the automated sheet for the
Phase A greens.
- Explicit out-of-scope list mirroring the deepening synthesis.
No code changes — documentation only.
Two related fixes that surface the same class of bug: the pre-send
"Recipient has no DM relay list" warning could disagree with the
actual send path, causing sends that either fail after the user
clicks Send, or block sends the user would have expected to work.
(a) STRICT ALIGNMENT — User.dmInboxRelays() (commons) is the lenient
"give me a delivery target for a DM" helper: it returns kind:10050
if present else the NIP-65 read marker (kind:10002). The send path
(DesktopIAccount.resolveDmInboxRelaysStrict → DmInboxRelayResolver)
uses the strict variant that returns kind:10050 only, because NIP-17
§Publishing mandates delivery to the recipient's kind:10050
exclusively — routing a wrap through a NIP-65 read relay leaks the
conversation metadata to a relay the recipient did not designate
for DMs.
Add User.dmInboxRelaysStrict() as the kind:10050-only accessor and
switch ChatNewMessageState.updateRecipientRelayStatus() to it so the
UI's "can we deliver" check matches what the send path actually
enforces.
(b) RESOLVER PROBE — pre-send validation was cache-only: if a peer's
kind:10050 hadn't landed in LocalCache yet (e.g. the peer just
published, or their event sits on an indexer relay we don't
subscribe to), the UI reported them unreachable and blocked send
even though the send path's DmInboxRelayResolver would have found
them via indexer fan-out.
Add an optional `dmInboxResolver: suspend (HexKey) -> List<...>?`
callback to ChatNewMessageState. On a synchronous cache miss for any
peer, the state optimistically blocks (preserving the "don't
silent-fail" invariant) and launches a probe. If any peer's relays
turn up, unblock immediately without requiring the user to reopen
the conversation.
Wired at both ChatNewMessageState construction sites in
DesktopMessagesScreen to DesktopIAccount.dmInboxResolver (which
Main.kt injected in the earlier P4 wire-up commit). Android's
ChatNewMessageViewModel is a separate class and keeps
cache-only behaviour — Android UI parity is deferred.
Surfaced during manual testing 2026-07-06 with two accounts where one
had a kind:10050 and one didn't: the UI correctly blocked send, but
the block persisted even after publishing kind:10050 for the missing
account until the conversation was reopened.
Amethyst Desktop uses apple.awt.fullWindowContent = true (see
applyNativeWindowChrome), which draws content edge-to-edge under
the title bar so the macOS traffic-light buttons overlap whatever
sits in the top-left of the App content column.
The AuthApprovalBanner mounts at (0, 0) of the content column,
which put its lock icon + "pyramid.fiatjaf.com" text directly
under the red/yellow/green window buttons. Screenshotted in
manual testing 2026-07-06.
Two-part fix:
1. Bump the row's own padding from horizontal 12dp / vertical 8dp
to horizontal 16dp / vertical 10dp for better breathing room
in general.
2. At the mount site in Main.kt, wrap the banner in a
platform-aware Modifier: on macOS pad 80dp from start (clears
3 traffic lights at 14pt each + spacing) plus 8dp top / bottom
4dp; on other platforms just an 8dp horizontal / 4dp vertical
margin. Non-mac users see the banner flush-ish since their
window chrome doesn't overlap.
Padding lives at the mount site so the banner composable itself
remains reusable inside chat panes or other contexts where
traffic-light clearance isn't needed.
Adds a Progress(current, total, label?) variant to SigningOpState so
multi-step signing operations (NIP-17 group sends via remote signer,
batched zaps) can show "Encrypting via remote signer (3 of 5)" rather
than an opaque indeterminate spinner.
Backwards compatible:
- Pending stays a data object — existing callers' `is Pending` checks
unaffected.
- New helper `isPending()` returns true for both Pending and Progress;
SigningState.execute uses it so a second execute() during Progress
returns null (matching the old single-flight semantics).
- SigningAwareButton renders both Pending and Progress as a spinner;
callers wanting the counter must read the state directly.
- SigningStatusBar adds a Progress branch that shows "<label>
(<current> of <total>)" — uses "Signing" as default label.
New `SigningState.updateProgress(current, total, label?)` lets the
in-flight block emit progress updates between Pending start and
finish. No-op when state is Idle/Error so callers don't have to gate.
Wire-up for NIP-17 bunker sends (publishing per-recipient progress
during NIP17Factory.createWraps' mapNotNullAsync) is a follow-up
that depends on threading the SigningState reference into the
factory's signing lambda; the substrate is here.
Four tests covering the lambda shape that DesktopAuthCoordinator's
signWithAllLoggedInUsers calls into for every NIP-42 challenge:
build RelayAuthEvent template → classify via policy → sign or
block → return List<RelayAuthEvent> for RelayAuthenticator
- tier-1 own-inbox auto-signs a valid kind:22242 event with the
right challenge + relay tags
- tier-2 unknown surfaces a PendingAuthApproval; ONCE resolution
produces a signed event (no persistence)
- tier-2 BLOCKED returns null AND persists the rejection
- tier-2 ALWAYS persists and skips the prompt on subsequent calls
Concurrency: the policy.classify call inside the lambda suspends on
the CompletableDeferred when prompting; tests use coroutineScope +
async + yieldUntilNotNull to model the banner-resolving-from-outside
pattern, mirroring how DesktopAuthCoordinator.resolve() drives the
deferred from a UI click.
Together with the existing PoolEventOutboxStateTest (auth-required
carve-out), AuthApprovalPolicyTest (classifier), and
GiftWrapRelayHintTest (NIP-17 hint placement), this completes
unit-level coverage of the AUTH pipeline. The websocket-level
round-trip stays covered by geode/.../KtorRelayTest.kt against a
real Ktor mock relay; that infra is reusable for a future
desktopApp integration test that combines mock relay + this stack.
Completes Phase 4 end-to-end. DesktopIAccount.resolveDmInboxRelaysStrict
now uses the resolver injected from Main.kt instead of the
LocalCache-only fast path. Three-layer lookup at every call:
1. LocalCache hit (kind:10050 already observed via feed pipeline)
2. Resolver's 1h LRU cache
3. Indexer fan-out via the dedicated unauthenticated NostrClient
The unauthenticated NostrClient is constructed in App() alongside
relayManager and connects on creation; DisposableEffect disconnects
on the App-level dispose. Critically NO RelayAuthenticator is
attached to this client — only the primary relayManager.client has
one (via DesktopAuthCoordinator). This closes security review F-01:
indexer queries no longer extract identity-key signatures during
kind:10050 probes against curated indexers.
resolveDmInboxRelaysStrict is converted from sync to suspend; the
three send paths (sendNip17PrivateMessage, sendNip17EncryptedFile,
sendGiftWraps) already run in suspend context inside DmSendTracker
batches, so the conversion is local. Resolver is plumbed through
MainContent as a new parameter rather than a CompositionLocal —
explicit threading matches the existing pattern for accountRelays
and relayManager.
The legacy LocalCache-only fallback inside resolveDmInboxRelaysStrict
is preserved for the constructor-default case (tests, CLI). When
dmInboxResolver is null, behaviour matches the pre-this-commit
strict-fix from 5293dae65.
Eight tests covering the resolver contract:
- localLookup hit short-circuits indexer fan-out
- empty indexer set returns empty
- empty local + empty indexer (no events arrive) yields empty
- cache hit within TTL skips indexer
- cache expiry triggers fresh indexer call
- clear() wipes all entries
- invalidate(pubkey) removes only the named entry
- localLookup returning an EMPTY list falls through to cache/indexer
(the takeIf { isNotEmpty() } guard — emptyList from localLookup
means "I don't know", not "I know they have nothing")
Uses EmptyNostrClient so RecipientRelayFetcher.fetchRelayLists returns
no events — covers the canonical "indexer found nothing" path without
needing a real mock relay. Tests for the populated-indexer path will
land with the Phase 4 wire-up commit when a Ktor-based mock relay is
plumbed through.
Three-layer resolver for "where do I publish this NIP-17 gift wrap":
1. LocalCache hit — if the caller already saw the user's kind:10050
via the regular feed pipeline, skip I/O entirely.
2. In-memory LRU cache — TTL 1h, 100 entries; avoids re-querying
indexers when opening several conversations in sequence.
3. Indexer fan-out — RecipientRelayFetcher against a curated set
(DefaultDmIndexerRelays: relay.nos.social, relay.damus.io,
nos.lol, relay.nostr.band, purplerelay.com — purplepag.es
deliberately excluded for poor kind:10050 coverage).
Strictness vs. the existing User.dmInboxRelays():
- filters to kind:10050 ONLY; NEVER falls back to NIP-65 read
marker (kind:10002). User.dmInboxRelays() silently substitutes
that, which is the same metadata-leak class fixed by 5293dae65.
- empty list = canonical "unreachable" signal; caller refuses to
publish (DesktopIAccount.resolveDmInboxRelaysStrict already
does this).
Security: the NostrClient passed in MUST be a dedicated
unauthenticated instance — no RelayAuthenticator attached. An
authenticated indexer fan-out (the current state with the primary
client) would extract identity-key signatures during the kind:10050
probe, escalating "indexer learns we want to DM pubkey X" into
"indexer learns user U wants to DM pubkey X". KDoc warning is
explicit; Phase 4 follow-up creates the unauth client in Main.kt
and injects it.
LocalLookup callback is plugged via lambda so CLI / headless
callers (amy) can use this without a Compose LocalCache.
Not yet wired into DesktopIAccount.resolveDmInboxRelaysStrict —
that wire-up is the next commit and converts the sync helper to
suspend, threading through sendNip17* batch construction.
Adds AuthApprovalBanner in commons.relayClient.auth — a Compose-
Multiplatform composable that renders one row per pending tier-2
NIP-42 AUTH challenge with three actions matching the AuthApprovalScope:
[Once] — sign this challenge, don't persist
[Always] — sign + persist ALWAYS via the store
[Never] — drop + persist BLOCKED via the store
Wired into desktop Main.kt as a global top-of-content banner reading
authCoordinator.pendingApprovals and calling authCoordinator.resolve.
Now tier-2 challenges actually have a UI to resolve — desktop AUTH is
end-to-end usable.
Up to 3 rows stack inline; the rest collapse into a "+N more pending"
row (click-to-expand can come later). Each row shows the relay's
display URL plus message-count when multiple challenges from the same
relay have coalesced.
The composable itself is in commons so Android picks it up free when
its AccountAuthApprovals VM wire-up lands — only the Main.kt-level
wiring (where to mount the banner in the layout) is platform-specific.
Lifecycle:
- Banner subscribes to pendingApprovals via collectAsState; recomposes
only when the PersistentMap identity changes (per the substrate
built in earlier commits).
- onResolve calls authCoordinator.resolve(url, scope), which completes
the underlying CompletableDeferred + removes the entry from the
pending map; the suspended signer wakes up and signs (or doesn't).
Until now desktop had no NIP-42 AUTH wiring at all — relays demanding
AUTH from desktop users got silently ignored. This commit closes the
gap, but does it the security-conscious way using the
AuthApprovalPolicy substrate from earlier commits.
DesktopAuthCoordinator binds to AccountState transitions in Main.kt
and per logged-in account:
- constructs a PreferencesAuthApprovalStore scoped by pubkey
- constructs an AuthApprovalPolicy with self-approved relays sourced
from the active account's NIP-17 DM-inbox (kind:10050) cache
- constructs a RelayAuthenticator whose signWithAllLoggedInUsers
lambda routes every AUTH challenge through the policy
Tier 1 (own DM-inbox + persisted ALWAYS) signs automatically. Tier 2
challenges hand back a CompletableDeferred surfaced on
authCoordinator.pendingApprovals. Until the inline banner UI lands
(P2.5 follow-up), tier-2 pending stays unresolved — which means
tier-2 relays don't get an AUTH response, same outcome as the
pre-this-commit world. The improvement here is tier-1: own DM
inbox relays now AUTH automatically without any prompt.
Lifecycle: onLogin attaches the authenticator; onLogout and account-
switch tear it down and complete any pending deferreds with BLOCKED
so suspended signers don't dangle.
Self-approved relays are deliberately scoped to kind:10050 (DM
inbox) only, NOT NIP-65 write/read relays. A user may follow read-
only relays they don't want to AUTH-identify themselves on — and the
common case where AUTH matters most is the user's own DM inbox.
NIP17Factory.createWraps launches all per-recipient seal builds via
mapNotNullAsync, which today runs them fully parallel. Each seal
needs nip44_encrypt + sign — for a NIP-46 (bunker) signer that means
two round-trips per recipient. A 5-recipient group send launches 10
concurrent in-flight requests against the bunker socket, and nsec.app
/ Amber / Keychat typically serialize past ~10 in-flight, so some
requests queue past the 65s timeout and silently fail.
Cap at 4 concurrent when signer is NostrSignerRemote. Local signers
(NostrSignerInternal, NostrSignerSync) bypass the semaphore and stay
fully parallel — no overhead, no behaviour change for nsec users.
The real fix is the batched nip44_get_conversation_keys NIP-46 RPC
(separate spec PR + plan) which collapses N×2 round-trips into ~2.
This commit is the interim throttle until that lands.
Per NIP-17 §Publishing, gift wraps MUST only be published to the relays
advertised in the recipient's kind:10050. Today three send paths in
DesktopIAccount fall through to relayManager.connectedRelays.value
when the recipient has no kind:10050 cached:
sendNip17PrivateMessage (line 200)
sendNip17EncryptedFile (line 231)
sendGiftWraps (line 253)
This is the security-review F-04 metadata leak: at best the wrap never
reaches the recipient (their other clients don't read those relays);
at worst the recipient pubkey + send timestamp leak to general/feed
relays outside their chosen inbox. Same class of bug as the relay-
power-tools work explicitly closed for the relay picker on
2026-04-20 ("block DM fallback to all relays — metadata leak").
Replace the fallback with strict resolution: if the recipient has no
kind:10050 in the cache, return an empty target set. DmSendTracker
already handles total relay count == 0 with a "No relays available"
failure state, so the user gets a visible error instead of a silent
leak.
Indexer fan-out + a UI dialog for the missing-10050 case is the
permanent fix, scoped to Phase 4 (DmInboxRelayResolver). This commit
is the conservative pre-Phase-4 plug — better to fail visibly than
leak silently.
NIP-04 send is unchanged: that path is pre-NIP-17, the encrypted
content sits next to other public events on the sender's outbox by
design.
Three regression tests covering the NIP-17 relay-hint contract just
introduced on GiftWrapEvent.create:
- default (no hint) emits the historical two-element ["p", pubkey]
shape — guards every existing caller against a wire-format
regression.
- with-hint emits ["p", pubkey, relay-url] — the canonical NIP-17
shape with the hint on the public wrap (NOT inside the seal, which
is the encrypted envelope and would hide routing info).
- null-hint must NOT produce ["p", pubkey, ""] — that would broadcast
"this user has no canonical inbox" as a metadata leak.
Per NIP-17 §Publishing, a gift wrap (kind 1059) MAY carry the
recipient's primary DM inbox relay as a third element of the p tag.
Other clients the recipient runs (or relays acting as inbox routers)
can then locate the wrap without performing their own kind:10050
lookup — handy when the recipient is multi-device and the second
device's 10050 cache is cold.
GiftWrapEvent.create gains an optional `recipientRelayHint:
NormalizedRelayUrl?` parameter that flows into PTag.assemble (which
already accepts a relay hint). NIP17Factory.createWraps and the four
public createMessageNIP17 / createEncryptedFileNIP17 /
createReactionWithinGroup entry points gain a matching
`recipientRelayHints: (HexKey) -> NormalizedRelayUrl?` lambda so
multi-recipient sends can pass per-recipient hints in one shot.
All new parameters default to null / { null }, so every existing
caller compiles unchanged and still emits the historical
two-element ["p", recipientPubKey] shape. Callers that resolve
kind:10050 via the (forthcoming) DmInboxRelayResolver can wire the
result through to populate the hint.
While here, document the existing — but undocumented — invariant
that shared rumor created_at falls out naturally because the
rumor is signed once before the per-recipient mapNotNullAsync loop.
This is what anchors cross-recipient reaction/receipt dedupe.
Desktop persistence for the AuthApprovalPolicy in commons. Backs the
`auth_approvals` use case from the plan using java.util.prefs.Preferences
instead of the originally proposed sibling outbox.db SQLite table.
Trade-off rationale: the AUTH approval set per account is small
(typically < 50 relays for any user) and the read pattern is bounded
(one lookup per relay per session, easily cached in memory by the
policy layer). java.util.prefs is already in use elsewhere on desktop
(SearchHistoryStore, DesktopPreferences) and adds zero new
dependencies or schema migrations.
The retry_queue table from the same outbox.db proposal needs the
higher-throughput characteristics SQLite gives us; it remains scoped
to P3 (send visibility), which can introduce a proper sibling DB at
that point.
Per-account scoping by Preferences node — logout/account-delete calls
clear() which removeNode()s the subtree. ONCE scope is never written
to disk, enforced explicitly here in addition to the interface
contract.
Not yet wired into a DesktopAuthCoordinator (today desktop has NO
AUTH wiring at all). That wiring lands in P2.5 alongside the banner
UI.
The current Android-only AuthCoordinator signs every NIP-42 AUTH
challenge from every relay unconditionally (and across every logged-in
account). For desktop there is no AUTH wiring at all — challenges are
ignored, so AUTH-walled relays silently drop DMs.
Both behaviours fail the security review: unconditional signing lets
any relay the user reads (or any malicious relay they touch) extract an
identity-key signature with timestamp, and signing across all accounts
links them under one relay observer.
This commit adds the substrate for a tiered classifier — wire-up will
follow with the desktop AuthCoordinator (P2.5) and SQLite-backed
persistence (P2.4). The policy itself is platform-agnostic and lives in
commons so Android can adopt the same design later.
Two tiers, no third silent-drop path:
- auto-allow when the relay is in the user's own outbox/DM-inbox set,
or has a persisted ALWAYS grant (subject to BLOCKED override)
- prompt-and-suspend via CompletableDeferred for everything else, with
the user's `[Once] [Always] [Never]` choice driving the deferred
Includes InMemoryAuthApprovalStore for tests + the ONCE session cache;
SqliteAuthApprovalStore lands in P2.4 with the sibling outbox.db.
Eight unit tests cover tier-1, persisted ALWAYS, persisted BLOCKED
(including BLOCKED overriding tier-1), unknown-prompt-then-cache,
re-eval of selfApprovedRelays on Account changes, and store.clear().
RelayAuthStatus has to stay mutable — it holds LruCaches addressable from
the per-relay OkHttp dispatcher thread, and replacing the whole holder
on every mutation would be wasteful. But its mutability also makes it
useless as a StateFlow value: mutating an entry doesn't change map
identity, so distinct-until-changed downstream swallows the update and
Compose never recomposes.
Add an immutable view alongside: RelayAuthSnapshot (phase +
lastAuthSuccessAt). RelayAuthStatus.snapshot() derives it from the LRU.
RelayAuthenticator publishes a PersistentMap<NormalizedRelayUrl,
RelayAuthSnapshot> via authStateFlow on every mutation (connect,
disconnect, AUTH-submitted, AUTH-OK, AUTH-fail). PersistentMap gives
O(log32 n) updates and a fresh identity per put, so both StateFlow
equality and Compose strong-skipping work.
This is the substrate for downstream consumers — the AUTH approval
banner, the retry-queue wake on authCompleted, the indexer-fan-out gate
— none of which are wired yet. They will read authStateFlow rather than
querying RelayAuthStatus directly.
Per NIP-17, seal (kind 13) and gift wrap (kind 1059) created_at are
randomized up to 2 days in the past for privacy. A subscription that
applies a `since` window — even with a 2-day adjustment — silently drops
wraps whose randomized timestamp predates the window, losing real DMs
and suppressing the unread badge.
Today only one caller (the desktop subscription coordinator) reaches
FilterDMs.giftWrapsToMe and it already passes no `since`, but the
parameter remained on the function signature as a footgun. Drop it so
the invariant is enforceable by the type, and document why in KDoc.
NIP-42 AUTH challenges arrive as `auth-required:` OK responses. Today
they accumulate via PoolEventOutboxState.newResponse → Tries.addResponse,
and after three of them the relay is silently dropped from the outbox on
the next newTry — even though RelayAuthenticator is concurrently signing
the AUTH event and the relay would have accepted the original publish
once authenticated.
Carve `auth-required:` out of the failure path: it's a "wait, AUTH in
flight" signal, not a rejection. The existing
RelayAuthenticator.checkAuthResults → client.syncFilters hook re-pumps
the outbox after AUTH-OK, so the original event is retried naturally.
Adds PoolEventOutboxStateTest covering the carve-out plus regressions
for regular rejections, terminal rejections, and success.
Audit of the authorsMissingOutbox anti-join surfaced one correctness bug and
one performance win:
- Bug (semantic): kind-1059 giftwraps store a random one-time key in
event_headers.pubkey (the real recipient is only a hash), so the query
returned an unbounded set of ephemeral keys that can never own a 10002 —
junk for the outbox model this feeds. Both the SQLite path and the generic
default now exclude kind 1059 from the "authors" set.
- Performance: replaced the DISTINCT + correlated NOT EXISTS scan with an
index-only EXCEPT (all authors minus 10002 owners). Both sides ride the
unconditional query_by_kind_pubkey_created covering index — so it does NOT
depend on the optional pubkey-alone index — and measured ~3x faster
(44ms vs 137ms at 152k events / 20k authors); the gap widens with author
count, since the old form paid one seek per distinct author. A loose-index
skip-scan was rejected: it needs the pubkey-alone index and degrades to a
full scan per author without it.
Also: corrected the KDocs (the old text implied an efficient index-only
distinct that wasn't guaranteed), added a giftwrap-exclusion test, and added
FsAuthorsMissingOutboxTest — the only coverage of the IEventStore DEFAULT
implementation, which EventStore always overrides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
The network-only WoT data traversal is a crawl, not a sync — and main now
ships negentropy sync (`amy sync`, `graperank update`), so the old verb name
was ambiguous. Rename the subcommand and its handler to `crawl`, keeping
`sync` as a back-compat alias so existing scripts keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Audit follow-ups on the sync engine:
- Perf: syncGroup reconciled against a full store.query<Event>(filter),
decoding the entire local matched set (~1 KB/event) just to read ids +
created_at and to index events for a small residual upload. Reconcile now
uses store.snapshotIdsForNegentropy (id + created_at only, ~40 B/entry) and
the uploader fetches only the residual haves by id. Peak memory drops from
O(all local matches) to O(residual) — matters when a relay hosts a large set.
- Bug: sync() promised best-effort ("one bad relay can't abort the set") but
syncGroup only caught NegentropySyncException, so any other failure (store
I/O, an unexpected throw) escaped async and cancelled every other relay via
awaitAll. Each group now runs under a guard that records the failure instead.
- Bug: the page-fallback catch (Exception) swallowed CancellationException,
breaking cooperative cancellation. Both new catch sites rethrow it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds AuthorsMissingOutboxBenchmark (gated behind -PprodRelayBench=1, like the
other prod benches). It syncs a real sample from relay.damus.io (kind 1 notes +
kind 10002 relay lists), replicates it to 1,000,000 stored rows while preserving
the real author set and outbox-owner set, then times the two shipping
implementations of authorsMissingOutbox() on the same store:
- generic: the IEventStore interface default (decodes every event via
query(Filter()))
- sqlite: EventStore's SELECT DISTINCT pubkey ... NOT EXISTS
Both are asserted to return the same set, matching the seeded ground truth.
Measured on a 4-core container, 1,000,000 events (best of 3):
generic 138,880 ms
sqlite 2,623 ms → ~53x faster
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Extracts GrapeRankUpdater's per-relay sync engine into a standalone
NegentropyStoreSync in the relay-client accessories, so any caller can
two-pass sync an arbitrary `relay -> filters` set against a local store.
Given an INostrClient + IEventStore it syncs each (relay, filter) group:
a bidirectional NIP-77 reconcile into/from the store (down/up), a deletion
settle over the residual (applyDown downloads the relay's kind:5 when an
uploaded record was rejected), and a paged-download fallback when a relay
can't reconcile. sync() runs many groups with relays concurrent and each
relay's own filters sequential (so one relay never exceeds its subscription
budget). Directions and bounds are a Config; every group is best-effort and
its outcome is a GroupResult. This is also the reusable engine `amy sync`
open-codes today.
GrapeRankUpdater now only owns the GrapeRank specifics: it reads kind:10002,
inverts to write-relay -> authors (the outbox model), fans that into one
filter per (relay, author chunk), hands the set to NegentropyStoreSync, and
folds the per-group results back up per relay. Its public Config/Result and
the CLI wrapper are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a whole-store query returning every distinct author with at least
one stored event that has NO NIP-65 relay list (kind 10002 / outbox).
This is a set-difference the positive-only nostr Filter grammar can't
express (there is no "NOT kind 10002"), so it lives as a dedicated
IEventStore method rather than a query(Filter). The interface carries a
correct default (collect authors-with-outbox, then stream events keeping
the rest — O(events)); SQLiteEventStore overrides it with a single
SELECT DISTINCT ... NOT EXISTS that seeks the outbox check on the
(kind, pubkey, created_at) index.
"Missing" is relative to what the store holds: an author whose only
10002 was deleted (NIP-09) or expired (NIP-40) is reported as missing
again, since no row remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuLzfXyVZ16ozG8oJ7hBBc
Moves the `amy graperank update` logic out of the CLI and into quartz as
GrapeRankUpdater, alongside GrapeRankDataCrawler in experimental/graperank,
so Android and any other quartz consumer can run the same refresh.
Given an INostrClient + IEventStore it reads every kind:10002 in the store,
inverts them into a write-relay -> authors map (the outbox model), then runs
one NIP-77 negentropy reconcile per write relay scoped to its authors:
bidirectional content sync into/from the store, deletion settle over the
residual (applyDown downloads the relay's kind:5 when an uploaded record was
rejected because the author retracted it), and a full paged-download fallback
when a relay can't reconcile. Bounds and directions are a Config; per-relay
and aggregate outcomes are returned as a Result.
The CLI `graperank update` is now a thin wrapper: it parses flags, builds the
Config, and renders GrapeRankUpdater.Result as text/JSON — no sync logic left
in cli/ (all reconcile/window/back-pressure/deletion logic lives in quartz's
relay-client accessories, which GrapeRankUpdater composes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a store-driven refresh of the record kinds a GrapeRank score is a
function of (0 profiles / 3 follows / 10002 outbox lists / 1984 reports).
It reads every kind:10002 already in the local store, inverts them into a
write-relay -> authors map (the outbox model), then runs one NIP-77
negentropy reconcile per write relay scoped to exactly the authors who
publish there. Bidirectional by default; each group then settles deletions
over the reconcile residual via quartz's negentropySettleDeletions, whose
applyDown direction downloads the relay's covering kind:5 when an uploaded
record was rejected because the author retracted it.
When negentropy can't reconcile a relay (no NIP-77, an over-cap minimal
window, a mid-sync disconnect), the group falls back to a full paged
download (Context.drainAllPages) of the same authors+kinds so those
records are still refreshed.
Thin assembly only: reconcile, windowing, back-pressure, and deletion
settle all live in the quartz relay-client accessories, mirroring
SyncCommand; this only routes ids to Context.drain / drainAllPages /
publish and inverts the relay list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
The probe does blocking DNS + TCP connect, and dead-domain DNS lookups hang well
past the connect timeout. On the shared Dispatchers.IO those hanging lookups
starved the crawl's own IO: an A/B at hop-3 showed probe-on 981s vs probe-off
517s, the entire +464s landing on the finishing drain (rounds were identical).
Coverage was unchanged (91.84% vs 91.74%), so the probe classification is correct
— it was purely IO contention.
Give the probe its own fixed daemon pool (128 threads) so its blocking work can
never touch the crawl's IO, and align the culler's concurrency to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The culler filtered candidates by !isDead and not-yet-probed, but not by
liveRelays — so it probed relays the WS path had already proven live, wasting a
probe and opening a needless TCP connection to the hot relays the crawl depends
on. Skip any authority already in liveRelays up front. liveRelays becomes a
ConcurrentSet so the background culler can read it while the crawl writes it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
At hop-8 the crawl dials into thousands of dead relay hints from old accounts.
Most fail slowly: a silently-dropping host has no RST to receive, so the WS
connect just hangs to the 7s connectTimeout. First-strike eviction pays that once
per host, but with ~3,000 dead hosts that's ~80s of connect-setup serialized
through the dispatcher.
Add a background reachability culler: a cheap raw TCP connect (one round trip, 2s
timeout) over the learned relays COLD-TAIL FIRST, dropping the unreachable ones
into deadHosts before the WS path pays its 7s. The key property is that a tight
TCP timeout is safe where a tight WS timeout is not — a busy-but-alive relay
accepts the SYN instantly at the kernel level and only stalls at the app layer, so
the probe separates "unreachable" from "slow" and never false-kills the busy. It
only ever marks dead and probes each authority once; a host the WS path already
resolved (isDead) is skipped, and a live host passes the probe, so the WS verdict
always wins. Injected as an optional Config.reachabilityProbe (JVM: java.net.Socket
in the CLI; --no-probe disables); writeRelayFreq becomes concurrent so the culler
can read it while routeByOutbox writes.
Also: when there's no Tor transport (Config.torEnabled=false), isDead skips every
.onion relay on sight — no socket, no wasted connect.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Claude
Vitor PamplonaandGitHub
nrobi144andClaude Opus 4.8
davotoulaandClaude Fable 5