Audit follow-up to the gift-wrap relay-icon fix: Marmot group chat rows
render the decrypted inner note (kind 9/7) via the standard chat feed with
RelayBadgesHorizontal, but nothing ever populated that note's relay list —
OK acceptances and relay deliveries all landed on the kind-445 envelope,
which has no link to its inner event.
- GroupEvent now implements HasInnerEvent (same @Transient @Volatile
innerEventId pattern as GiftWrapEvent/SealedRumorEvent), so
LocalCache.addRelayToNoteAndInners drills 445 -> inner for both OK
confirmations and duplicate EVENT deliveries. RouteMaker is unaffected:
it gates on the concrete wrap types before casting to HasInnerEvent.
- GroupEventHandler sets innerEventId at decrypt time and copies the
envelope's accumulated relays down to the inner note (looked up by
event.id, not the eventNote/publicNote params, which belong to the
triggering event when replayed from retryPendingFor).
- sendMarmotGroupMessage sets innerEventId on the freshly built envelope
before consuming/publishing, so acceptances for sent group messages
reach the rendered note as soon as the inner note is indexed.
Also audited the remaining chat-rendered types: NIP-04 PrivateDmEvent,
ChatMessageEvent rumors, NIP-C7 ChatEvent, ChannelMessage/Ephemeral/
LiveActivities messages all route through consumeRegularEvent (duplicates
covered by the shared helper), and NIP-37 drafts render the wrap note
itself with markAsSeen covering the version/addressable pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YEiq1NMK3q12KGQ2yYhPEp
Adds an owner-only role toggle to the Concord note quick-action menu, alongside
Ban:
- Account.concordAdminTarget gates the action to the owner (only rank 0 strictly
outranks the position-1 Admin role, as the resolver requires), never the
owner's own note or the owner as target, and reports whether the author is
already an admin (via the new AuthorityResolver.rolesOf accessor).
- makeConcordAdmin mints a default Admin role (all management + moderation
permissions, position 1) if the community doesn't have one yet, then grants it;
removeConcordAdmin revokes via an empty grant.
- The menu item flips between "Make admin" and "Remove admin" and fires
toggleConcordAdmin.
Extends ConcordModerationTest to cover rolesOf and revoke-via-empty-grant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Two bugs found in a pre-merge audit:
- quartz PoolEventOutboxState: auth-required NAKs only spared the `responses`
budget, but `tries` (grown by every send/re-pump and NOT auth-aware) still
accumulated across reconnects, so a slow/flapping AUTH handshake could exhaust
Tries.isDone() and drop the event — with a spurious onEventGaveUp — before AUTH
landed. Now an auth-required NAK resets the relay's retry budget (it responded,
so it's up and just wants auth). Regression test added.
- RelayAuthPromptBus used a replay=0 SharedFlow, so a challenge that resolved to
ASK before RelayAuthPromptHost subscribed (cold start / account switch) was
dropped and the auth coroutine stalled the full timeout then DISMISSed. Add
replay so late subscribers recover pending prompts (the host already filters
resolved ones). Regression test added.
Also record the as-built design (Always/Never/Custom + toggles, venues, give-up
toast, known deny-relay-outbox limitation) in the plan doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Instead of silently dropping an event once its per-relay retry budget is
spent, PoolEventOutbox now reports it: PoolEventOutboxState.newTry returns
whether the attempt gave up on the relay, PoolEventOutbox.onSent surfaces
the dropped event, and NostrClient notifies a new (default no-op, so
non-breaking) RelayConnectionListener.onEventGaveUp(relay, event). Lets a
host surface a failed delivery rather than lose it silently; the event may
still be pending on other relays. Unit-tested via the outbox try budget.
Note: timed retry backoff (the other half of this item) is intentionally
deferred — applied in the shared syncState path it would also delay the
post-auth resend and regress the auth-required fix, so it needs
trigger-aware handling designed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Add INostrClient.activeOutboxEvents(url) (backed by
PoolEventOutbox.activeOutboxEventsFor) returning the full events still
pending delivery to a relay, not just their ids like activeOutboxCache.
This lets a host explain *why* a relay is being authenticated with —
e.g. a pending kind-1059 gift wrap means we're sending a DM to its
recipient — by inspecting kind/tags. Combined with the existing
activeRequests(url) filters, it is the generic challenge context the
NIP-42 decision hook needs. Updates the INostrClient test fakes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
PoolEventOutboxState treated an `auth-required` OK-false the same as any
transient failure, recording it against the per-relay Tries budget
(responses > 2 drops the event on the next send attempt). Relays that
NAK every unauthenticated EVENT could therefore exhaust the budget and
drop the message before the AUTH handshake completed — the event was
gone by the time syncFilters re-sent it after the auth OK.
Treat `auth-required` as a deferred state instead: keep the relay in
relaysRemaining and record no failure, so the existing
syncFilters-after-auth path redelivers it. Mirrors the behavior already
present in StandaloneRelayClient. Terminal rejections (invalid/pow/
replaced/deleted) and ordinary transient errors are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Chat rows render the inner rumor note of a NIP-17 message, but relay
attribution only landed there through narrow windows, so accepted relays
often never showed as icons:
- Duplicate deliveries were stranded on the wrap: a gift wrap re-delivered
by a second relay hit the duplicate branch of consumeRegularEvent, which
tagged the outer wrap note only and never re-processed the event. Extract
the OK-path drilling (wrap -> seal -> rumor) into
LocalCache.addRelayToNoteAndInners and call it from both the OK
confirmation path (markAsSeen) and the duplicate EVENT path, replacing
CacheClientConnector's private copy.
- Cross-thread visibility: Note.event, Note.relays, Note.flowSet and the
innerEventId of GiftWrapEvent/SealedRumorEvent are written by decrypt/index
coroutines and read lock-free on relay socket threads; a stale read parks
an acceptance on the outer envelope permanently. Mark them @Volatile.
- Orphaned UI flows: RenderClosedRelayList/RenderAllRelayList and
createMustShowExpandButtonFlows captured note.flow().relays.stateFlow once
in remember/stateIn; MemoryTrimmingService.cleanObservers destroys the
unobserved NoteFlowSet while the lifecycle is stopped, so resumed rows
never saw another relay update. Wrap in a cold flow that re-resolves
flow() on every collection start.
- Indexing latency: sent DMs waited for the ~1s newEventBundles batcher
before the self-wrap was unwrapped and the message reached the chatroom,
parking early OKs on the wrap. broadcastPrivately and
sendNip04PrivateMessage now run the EventProcessor on the freshly
consumed note immediately; the batched re-delivery is idempotent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YEiq1NMK3q12KGQ2yYhPEp
Receiving reactions/replies already worked (they decrypt via channelRumors and
wire to their target Note by id). This adds the send side, which must NOT go
through the generic public/NIP-17 reaction path — a plaintext kind-7 would e-tag
the private rumor id onto public relays, and the NIP-17 path wraps to named
recipients, not the channel plane.
- ChannelChat.reaction/reply build kind-7/kind-9 rumors bound to channel+epoch
(reaction e-tags the target; reply q-tags the parent)
- ConcordActions.buildChannelReaction/buildChannelReply wrap them on the plane
- Account.reactToConcordMessage + sendConcordChannelMessage(replyTo) publish the
wrap with an instant local echo, factored through publishConcordWrap
- AccountViewModel.reactToOrDelete intercepts Concord notes (detected by the
ConcordChannel gatherer) and routes to the plane-wrapped reaction
- ConcordChannelScreen wires onWantsToReply into the composer with a reply banner
Zaps already route correctly: a Concord message is an unsigned rumor, so the
existing isPrivateRumor() path forces a PRIVATE (NIP-57 encrypted) zap, same as
NIP-17 DMs. A fully on-plane nutzap (no public receipt) remains a future upgrade.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
The channel model the shared chat UI renders, mirroring RelayGroupChannel:
- quartz ConcordChannelId(communityId, channelId): the stable channel address
(analog of NIP-29 GroupId, but community-scoped rather than relay-pinned)
- commons ConcordChannel : Channel — name/voice/private from the folded Control
Plane (no single relay-signed metadata event), relays() = the community relay
set (a plane may be mirrored on several), membership from the authority
resolver, canPost(), placeholderNote() for immediate Messages-list rows;
updateFrom(state, relays, myPubKey) refreshes on each re-fold
- LocalCache: concordChannels LargeCache index + getOrCreate/getIfExists
Compiles across :quartz/:commons/:amethyst. Next: the decryption subscription
that folds inbound channel-plane wraps into these, then the Messages inbox hook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Promote the kind-13302 joined-communities list to a proper Event subclass so the
Android LocalCache/Account layer can observe it the way RelayGroupListState
observes SimpleGroupListEvent (10009):
- ConcordCommunityListEvent : Event — createAddress (13302, pubkey, ""), a
suspend create(signer, entries), and decrypt(signer); content stays NIP-44
self-encrypted and carries each community's secrets
- register the kind in EventFactory so inbound events parse to the typed class
- extract encode/decode JSON helpers in ConcordCommunityList (shared by both)
Test: create -> the wire form hides the name, a JSON round-trip resolves to the
typed class via EventFactory, decrypt recovers entries, and the replaceable
address is (13302, pubkey, ""). Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Completes CORD-05: for a known npub, deliver the CommunityInvite as a NIP-59
giftwrap instead of a public bundle — a kind-3313 rumor sealed (kind 13) to the
recipient and wrapped (1059) with ["p", recipient] and a ["k","3313"] index tag
so recipients can query pending invites without decrypting every giftwrap.
Cannot be revoked (recipient holds the keys on arrival).
Reuses SealedRumorEvent + the giftwrap primitives. Tests cover round-trip to the
intended recipient (with p/k tags) and that strangers cannot open it. Green on
:quartz:jvmTest.
With this, the Quartz protocol layer covers CORD-01..07 end to end.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
- VoicePresence: kind-23313 join/left presence rumors bound to the channel/epoch
and carrying the SFU identity + broker, with heartbeat/stale constants and a
verifiedParticipants fold that renders an identity only when exactly one author
claims it (contested identities stay unverified)
- ConcordBrokerToken: the NIP-98-style kind-27235 token request signed by the
channel's derived voice signer key (its pubkey is the SFU room name), the
'Authorization: Concord <base64(event)>' header, and the
/.well-known/concord/av/<room> path
Voice key derivation (voice_signer/voice_media/voice_sender) already lives in
ConcordKeyDerivation. Tests cover presence round-trip, uncontested-only
verification, staleness, and that the broker token is signed by the voice-room
key. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Completes the public invite path (CORD-05), pinned to Concord v2 (Armada
invite.ts):
- CommunityInvite: the bundle contents with exact snake_case field names
(community_id, owner, owner_salt, community_root, root_epoch, channels[],
relays, name, icon, expires_at, creator_npub, label) + ImagePointer/InviteChannel
- ConcordInviteBundle: build/parse the kind-33301 event (content =
nip44(CommunityInvite, inviteBundleKey(token)); tags d="",vsk="6"; signed by a
per-link signer), self-certification validate (owner+salt reproduce
community_id), expiry check, and mintLink (fresh token + link signer -> bundle
event + shareable URL)
End-to-end test: create a community, mint an invite link, a stranger parses the
URL, decrypts the bundle with the fragment token, validates the owner
commitment, reconstructs the root, and reads the genesis #general channel.
Wrong-token and forged-owner rejections covered. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Completes the "create a community" path (CORD-02 Genesis), pinned to Armada
(concord-v2 control.ts/community.ts):
- ConcordKeyDerivation: control/guestbook plane keys and the keyless entity
coordinates — grantCoordinate = hkdf32(communityId, "concord/grant"||member),
banlistCoordinate (ZERO32 id), inviteLinksCoordinate (creator)
- ControlEditionBuilder: assembles kind-3308 edition rumors (vsk/eid/ev/ep/vac),
the inverse of ControlEdition.fromRumor
- MetadataEntity gains relays
- ConcordCommunityFactory.create: mints owner_salt + self-certifying community_id,
an independent community_root, and two owner-signed genesis editions (metadata
with eid=communityId, and a public #general channel) as plaintext-seal wraps on
the Control Plane at epoch 0
Test creates a community, verifies the id commitment, opens the genesis wraps
(20014 seals, owner-authored), folds them into live ConcordCommunityState with a
#general channel and owner authority, and confirms one owner yields distinct
communities. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
The NIP-51 analog for returning to signed-up Concord communities (CORD-05):
- ConcordCommunityListEntry: per-community credentials needed to re-derive planes
on any device (id, owner, ownerSalt, current root + rootEpoch, past heldRoots,
privateChannels keys, relays, cached name)
- ConcordCommunityList: build/parse the replaceable kind-13302 event, NIP-44
self-encrypted so relays store only ciphertext, plus a cross-device merge that
keeps the freshest root epoch per community
Channels are intentionally not listed — holding the root and folding the Control
Plane yields them. Tests cover self-encrypted round-trip, that only the owner can
decrypt, and epoch-wins merge. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
The Messages-list feed already filtered kind-9/1068/11/1111 content when
selecting a group's representative note, but the ChatroomEntry render fallback
still rendered ANY group-scoped note — including a kind-7 reaction lingering in
the in-memory list — as the group row, using the reaction's content and time.
Add a shared quartz helper `Event.isGroupChatContent()` and use it in both
places: the feed filter and the render fallback. A non-content group-scoped note
now falls back to the channel placeholder ("No messages yet") instead of showing
the reaction as the room's last message.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Non-ratcheted async key rotation to remove members from a channel or (root
scope) the whole community, pinned to Concord v2 (Armada rekey.ts):
- RekeyPayload: the 72-byte scope_id||epoch_be8||new_key blob codec
- RekeyBlob: per-recipient {locator, wrapped} entry
- ConcordRekey: blobFor (locator = recipient pseudonym; wrapped = base64 payload
NIP-44-encrypted under the rotator<->recipient pairwise key), kind-3303 rumor
tags (scope/newepoch/prevepoch/prevcommit/chunk) and content codec, and
findNewKey (recipient computes their locator, matches, decrypts, verifies
scope+epoch) with absence == removal
Test proves remaining members recover the rotated key while a removed member
finds no matching blob, and that the locator is epoch-bound. Green on
:quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
- ConcordKeyDerivation.inviteBundleKey: derives the bundle decryption key from a
link's 16-byte unlock token via hkdf32(token, "concord/invite-key")
- InviteRelayDictionary: the v4 stock relay set + id<->url mapping
- ConcordInviteLink: encode/decode the {base}/invite/{naddr}#{fragment} link and
the [version=4][flags][relays?][token:16] fragment (stock-set flag, dictionary
ids, wss:// host and full-url relay entries), rejecting non-v4 versions; builds
the naddr (33301, link_signer, d="") and parses it back
Tests cover stock/dictionary/literal/full-url relay round-trips, wrong-version
rejection, full URL round-trip through naddr, and token-bound bundle key
derivation. Green on :quartz:jvmTest. Pinned to Concord v2 (Armada) constants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
- ChannelEntity/MetadataEntity content DTOs (CORD-02/03)
- ConcordCommunityState.fold: folds control editions + known owner into the live
community view — metadata, non-deleted channels, live roles, the owner-rooted
AuthorityResolver, and a dissolved flag from the tombstone
- Guestbook: self-signed join/leave (kind 3306, with invite attribution) and
authorized kick (kind 3309) rumor builders + parsers; off-consensus membership
motion
Tests cover metadata/channel/role folding with deleted-channel exclusion,
authority wiring, the dissolution tombstone, and join/leave/kick round-trips.
Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
CORD-03 Chat Plane vertical slice tying crypto + envelope together:
- ConcordChannelKeys: public (community_root) and private (channel_key) channel
key derivation, both via group_key with channel_id folded in so each channel
has a distinct, epoch-rotating address
- ChannelChat: channel/epoch binding tags, a kind-9 message rumor builder, and
isBoundTo validation so an event can't be replayed across channels/epochs
End-to-end test proves two members holding the same community_root independently
derive the identical public channel plane and one reads the other's message with
no key distribution, non-members can't derive the plane, cross-channel/epoch
replay is rejected, and epoch rotation rotates the address. Green on
:quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Implement CORD-04 authority resolution over a folded Control Plane:
- ControlEntities: Role and Grant content DTOs (kotlinx.serialization, lenient +
extensible) and a Banlist array parser; ConcordJson facility
- AuthorityResolver: builds roster state from the entity heads + known owner and
answers rank (lower = higher; owner = 0), effectivePermissions (union of a
member's roles), isBanned, hasPermission, and canActOn (holds the bit AND
strictly outranks the target — equal cannot act on equal; owner unremovable)
Grants are validated by an owner-rooted fixpoint: a Grant is honored only when
its signer already outranks every assigned Role and holds MANAGE_ROLES, so the
roster grows strictly outward from the owner and self-referential cycles never
bootstrap. Deleted/position-0 roles and banned members are dropped. Banlists heal
to their union.
Tests cover owner-rooted ranks/permissions, fixpoint order-independence,
unauthorized/insufficient-rank grant rejection, ban vanishing, and invalid-role
dropping. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
- ControlEdition: parses a verified kind-3308 rumor's vsk/eid/ev/ep/vac tags into
a typed edition, computes its domain-separated edition hash, and rejects
malformed editions (bad kind, missing eid/ev, unknown vsk, bad hex)
- AuthorityCitation: the vac Grant pin an actor claims rank under
- EditionFold: folds editions into each entity's current head — genesis
anchoring, intact-chain / no-downgrade advancement, deterministic lower-rumor-id
tie-break for convergence (authority-weighted tie-break layered in the resolver)
Tests cover tag parsing + hash, malformed rejection, genesis prev handling,
order-independent chain walk, downgrade/broken-chain refusal, tie-break, and the
hold-without-genesis case. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
- ConcordKinds: every Concord event kind (envelope, chat, guestbook, control,
rekey, bookkeeping), pinned to Concord v2 (Armada kinds.ts)
- ControlEntityKind: the kind-3308 `vsk` sub-kinds (metadata=0..dissolved=10)
with wire<->enum mapping
- ConcordPermissions: u64 permission bitfield with frozen bit positions, union
(effective = OR of a member's roles), and decimal-string wire codec that
preserves the high bits (no floating-point corruption)
Tests cover frozen bit positions, union, decimal round-trip incl. bit 63, blank
and garbage handling, and vsk mapping. Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Implement the Concord CORD-01 stream envelope in quartz `concord/envelope`:
the inverted NIP-59 three-layer wrap -> seal -> rumor that carries every plane's
traffic. The outer kind-1059/21059 wrap is signed by the shared stream key and
its content is NIP-44-encrypted under the plane's self-ECDH conversation key,
with an ephemeral p tag, so relays never see plaintext.
- ConcordStreamEnvelope.seal: 20014 plaintext (verbatim rumor JSON, for the
Control Plane) or 20013 encrypted seal, signed by the real author
- wrapSeal/wrap: sign+encrypt the wrap at a GroupKey plane address
- open/openOrNull: verify wrap author == stream address + wrap sig, decrypt seal,
verify seal sig, decrypt/parse rumor, enforce rumor.pubkey == seal.pubkey and
rumor.id == NIP-01 hash
- OpenedStreamEvent: verified rumor + seal kind + author
Reuses RumorAssembler, NostrSigner, NostrSignerSync and Nip44v2. Round-trip
tests cover plaintext/encrypted seals, ephemeral wraps, non-member rejection
(wrong epoch/secret), and confirm plaintext never leaks into wrap content.
Green on :quartz:jvmTest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Addresses correctness/perf issues found in the crawler + reachability audit:
- deadHosts permanent eviction (#1): an authority that accrued timeoutEvictStrikes
before its first EOSE was evicted forever — clearTimeoutStrikes only zeroed the
counter and could not un-evict, contradicting the "a host that ever produces is
never evicted" invariant. Add a producedHosts set that isDead() consults, so a
proven-productive authority is never treated as dead even if a concurrent strike
from the 24-worker fan-out raced it into deadHosts.
- Parking-disabled event loss (#2): when parking is off (no bgScope, or
parkTimeoutMs <= timeoutMs), a relay that streamed events but didn't EOSE in the
fast window had its buffer dropped without persist() and reported count 0. Drain,
persist, and return those events like the other two branches; strike only when
nothing was delivered.
- Wide-sweep over-narrowing (#4): relayListDiscoverySwept excluded an already-swept
straggler from the wide pass even though the wide net grows each round, so a 10002
hosted only on a later-learned relay was never fetched. Gate the wide pass on the
asked-relay set (wideRelaysSwept) instead: new users get the full net, older
stragglers get only newly-appeared relays, no (user, relay) pair asked twice.
- Onion detection (#10): replace loose relay.url.contains(".onion") with
RelayUrlNormalizer.isOnion() in isDead() and networkTypeOf(), fixing the
foo.onionfake.com false positive and the store/crawler disagreement.
- rtt-open=0 semantics (#9): document that the crawler's reachable records use
rtt-open purely as a liveness flag (0 = latency not probed), not a real 0 ms
measurement, and must not be published as authoritative latency data.
deadHosts is deliberately still NOT persisted to the 24h reachability cache (#8):
a timeout eviction means "too slow under our fan-out this run", not "proven
unreachable", so persisting it would blacklist slow-but-live hubs across runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
FrameDispatchStats stamped a ValueTimeMark on every relay frame and recorded a
contended atomic per frame in BasicOkHttpWebSocket — the WebSocket layer used by
the whole app, unconditionally, forever — to answer a one-time question that only
graperank --diagnose read. It served its purpose (proved the our-side dispatch
lag is ~200ms mean and the EOSE-wait is dominantly relay-side, so the crawler is
network-bound), but the ongoing per-frame Pair allocation + atomic contention on
every client's relay traffic isn't worth carrying. Revert the channel back to
Channel<String> and delete the stats holder. The diagnose-gated saturation ticker
and per-drain latency breakdown stay — they're crawler-local, off the hot path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured: relays deliver their events in ~0.6s then sit ~4.6s (86% of drain wall)
before sending EOSE — mostly relay-side (our pipeline adds only ~200ms). So instead
of waiting the full 10s fast window then parking, close a drain that has delivered
>=1 event and then gone silent for eoseIdleMs, treating it as complete ("eose-idle").
awaitTerminalOrQuiescent: the idle timer arms only AFTER the first event, so a relay
merely slow to answer still gets the full timeoutMs and is never cut prematurely; a
still-streaming relay keeps resetting the window. eose-idle paginates if the page was
capped and clears timeout strikes (it delivered), but joins notAnswered (no clean
EOSE, so its missing authors are retried elsewhere). Off by default (eoseIdleMs=0),
CLI --eose-idle-ms, so it can be A/B'd against the plain fast window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The dedicated frame-dispatch pool (ea1093ad) made dispatch lag WORSE, not better:
mean 200ms→460ms, max 3.5s→5.5s, frames>1s 43k→76k. The pool was sized cores*2
(=8 here) vs Dispatchers.IO's 64 threads, so it cut frame-processing parallelism
~8x. Lesson: the our-side lag is dominated by per-connection serial decode
throughput / thread count, NOT cross-contention with the store's IO writes — the
experiment ruled that hypothesis out. Reverting to shared IO; keep FrameDispatchStats.
EOSE-wait is confirmed dominantly relay-side (200ms our-mean vs ~5s eose-wait).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured on a GrapeRank crawl, frame decode/dispatch (per-connection consumer
coroutines) ran on the shared Dispatchers.IO — the same pool that runs the store's
blocking SQLite inserts. During event floods, frame coroutines queued behind those
inserts: mean 200ms and up to 3.5s of dispatch lag, with 43k frames waiting >1s in
our pipeline. That lag also skews the relay-idle/EOSE timing the crawler reads.
Give frame processing its own daemon thread pool (sized to a small multiple of
cores; decode is light + CPU-bound), shared across all connections. Frame delivery
stays prompt regardless of what the IO pool is doing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Adds FrameDispatchStats: the lag between a relay frame arriving on the OkHttp
reader thread and our per-connection consumer coroutine (on shared Dispatchers.IO)
pulling it off the channel — pure our-side pipeline delay, relay send-timing
excluded. BasicOkHttpWebSocket stamps arrival before enqueue and records the lag
on dequeue; the crawler resets it at start and dumps it in the --diagnose summary.
Answers whether a drain's 5s gap between the relay's last event and its EOSE is
the relay being slow to SEND eose (low dispatch-lag) or our IO pipeline backing up
so the already-arrived eose frame sits queued (high dispatch-lag).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A Nostr pubkey is x-only, exactly 32 bytes, but NPub.parse/NProfile.parse never
checked the length — they hex-encoded whatever bytes the bech32/TLV carried. A
malformed npub/nprofile that some clients encode with the full 33-byte COMPRESSED
secp256k1 key (0x02/0x03 prefix) therefore round-tripped its 66-char hex straight
into a `p`/`q` tag via the quote/mention path, and a strict relay (relay29 /
pyramid.fiatjaf.com) rejected the whole group message:
blocked: schema validation failed: tag[..]: invalid pubkey value
'02977dcf…c3402' ... pubkey should be 64-char hex
We never generate compressed keys ourselves (Nip01Crypto.pubKeyCreate strips the
prefix byte); this is purely inbound malformed input. Enforce the 32-byte length
at the decode boundary so the bad entity never becomes a mention/quote tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Answers "are we resource-bound or waiting on relays" without a profiler:
- progress ticker gains "Nw/CAPw" (drain workers busy vs drainConcurrency) and
"N rl" (rate-limit responses so far) — a rarely-full pool means the producer or
the relays are the limit, not concurrency; a climbing rl count is the external
ceiling that made concurrency 60 backfire.
- crawl-end "latency breakdown": splits each drain's wall into time-to-first-event
vs EOSE-wait-AFTER-the-relay's-last-event, and reports the % of drain wall spent
waiting for EOSE after the relay was already done, how many drains blew the fast
window and parked, and total rate-limit hits. A high EOSE-wait % is the direct
case for a shorter/adaptive fast window over more concurrency.
All gated on config.diagnose; zero cost on a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Two changes:
1. The reachability flush re-wrote the SEEDED known-dead relays with a fresh
created_at every run, refreshing their TTL without a re-probe — so a relay
marked dead once (and thereafter skipped, never re-dialed) would stay
blacklisted forever as long as crawls kept running, defeating the TTL's
re-probe. Stats.deadRelays now reports only relays actually dialed this run
(deadRelays - knownDeadRelays); seeded records keep their original timestamp
and age out on schedule so the next run re-probes them.
2. Rename GrapeRankDataCrawler -> GrapeRankCrawler (file + all references).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Wire RelayReachabilityStore into the crawler and the WoT updater so liveness is
shared across procedures and runs instead of each rediscovering dead relays.
- OperatorKeys.monitorKey(): a dedicated machine monitor identity derived from the
operator master (domain "relay-monitor:"), independent of any account — the
30166 records are published under this, not the observer key.
- Context.reachability: a RelayReachabilityStore over the shared store, signed by
the monitor key.
- Crawler: Config.knownDeadRelays seeds deadRelays before the run; Stats now
returns the final dead/live sets. GrapeRankCommand seeds from snapshot().dead
and flushes the crawl's verdicts back via reachability.record().
- Updater: Config.knownDead skips proven-dead relays from the reconcile plan — a
dead relay cannot serve its authors, so reconciling it only burns a timeout.
Live author-advertised relays are always synced.
All behind --no-reachability-cache. TTL'd (24h), so a recovered relay is retried
once its record ages out — a "skip for now", never a permanent ignore, keeping
the outbox rule that every live advertised relay is tried.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A durable, shareable relay-reachability cache backed by the EventStore as NIP-66
kind:30166 Relay Discovery events, so the crawler, the WoT updater, and future
runs share liveness knowledge instead of each rediscovering dead relays from an
in-memory set wiped at process exit.
- 30166 is addressable by its d-tag (relay URL) → one replaceable status slot per
(monitor, relay), with created_at giving a free TTL.
- Reachable → 30166 with rtt-open; dead → 30166 without (NIP-66 has no explicit
offline field; liveness is inferred from a fresh successful open). Live wins
over dead within the TTL, so third-party monitors' 30166 can be ingested.
- snapshot() loads the fresh set once (not a per-request hot-path query); record()
flushes a run's findings. A relay is only skipped for the TTL, never permanently
— consistent with the outbox rule that every advertised write relay is tried.
Reuses the existing RelayDiscoveryEvent. jvmTest covers record/reload,
live-overrides-dead, TTL expiry, and .onion→Tor network tagging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Round-8 profiling showed the crawl re-querying the same never-had-a-10002
users' outboxes every round they recirculated — ~144k slow kind:10002 drains
(p50 17.4s) against a static discovery set, dragging the round to ~18 users/s.
1. ensureRelayLists guards with `relayListDiscoverySwept`: each user's outbox
discovery runs once. The discovery relay set is static, so a second sweep of
a user still lacking a 10002 cannot find one the first missed.
2. The discovery REQ to the bounded INDEXER set co-fetches [10002, 3]: the
outbox lookup already pays the round-trip and an indexer holding a user's
10002 often holds their kind:3, so we harvest the contact list as a cheap
byproduct. The wide "every live relay" completeness sweep stays 10002-ONLY —
co-fetching kind:3 across thousands of relays downloaded the same big contact
lists repeatedly and inflated the fire-and-forget bgScope sweep the finishing
drain waits on (measured +260s at hop-3; the indexer-only co-fetch keeps
coverage flat at baseline speed).
3. harvestFromStore folds any already-stored kind:3 into the graph at Phase-A
time so Phase B never re-drains a list we hold (also speeds re-runs).
Verified same-session hop-3: pre-fix 685s / narrowed 690s / wide-co-fetch 945s,
coverage 91.74% across all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
NIP17Factory.createWraps launches all per-recipient seal builds via
mapNotNullAsync, which today runs them fully parallel. Each seal
needs nip44_encrypt + sign — for a NIP-46 (bunker) signer that means
two round-trips per recipient. A 5-recipient group send launches 10
concurrent in-flight requests against the bunker socket, and nsec.app
/ Amber / Keychat typically serialize past ~10 in-flight, so some
requests queue past the 65s timeout and silently fail.
Cap at 4 concurrent when signer is NostrSignerRemote. Local signers
(NostrSignerInternal, NostrSignerSync) bypass the semaphore and stay
fully parallel — no overhead, no behaviour change for nsec users.
The real fix is the batched nip44_get_conversation_keys NIP-46 RPC
(separate spec PR + plan) which collapses N×2 round-trips into ~2.
This commit is the interim throttle until that lands.
Three regression tests covering the NIP-17 relay-hint contract just
introduced on GiftWrapEvent.create:
- default (no hint) emits the historical two-element ["p", pubkey]
shape — guards every existing caller against a wire-format
regression.
- with-hint emits ["p", pubkey, relay-url] — the canonical NIP-17
shape with the hint on the public wrap (NOT inside the seal, which
is the encrypted envelope and would hide routing info).
- null-hint must NOT produce ["p", pubkey, ""] — that would broadcast
"this user has no canonical inbox" as a metadata leak.
Per NIP-17 §Publishing, a gift wrap (kind 1059) MAY carry the
recipient's primary DM inbox relay as a third element of the p tag.
Other clients the recipient runs (or relays acting as inbox routers)
can then locate the wrap without performing their own kind:10050
lookup — handy when the recipient is multi-device and the second
device's 10050 cache is cold.
GiftWrapEvent.create gains an optional `recipientRelayHint:
NormalizedRelayUrl?` parameter that flows into PTag.assemble (which
already accepts a relay hint). NIP17Factory.createWraps and the four
public createMessageNIP17 / createEncryptedFileNIP17 /
createReactionWithinGroup entry points gain a matching
`recipientRelayHints: (HexKey) -> NormalizedRelayUrl?` lambda so
multi-recipient sends can pass per-recipient hints in one shot.
All new parameters default to null / { null }, so every existing
caller compiles unchanged and still emits the historical
two-element ["p", recipientPubKey] shape. Callers that resolve
kind:10050 via the (forthcoming) DmInboxRelayResolver can wire the
result through to populate the hint.
While here, document the existing — but undocumented — invariant
that shared rumor created_at falls out naturally because the
rumor is signed once before the per-recipient mapNotNullAsync loop.
This is what anchors cross-recipient reaction/receipt dedupe.
RelayAuthStatus has to stay mutable — it holds LruCaches addressable from
the per-relay OkHttp dispatcher thread, and replacing the whole holder
on every mutation would be wasteful. But its mutability also makes it
useless as a StateFlow value: mutating an entry doesn't change map
identity, so distinct-until-changed downstream swallows the update and
Compose never recomposes.
Add an immutable view alongside: RelayAuthSnapshot (phase +
lastAuthSuccessAt). RelayAuthStatus.snapshot() derives it from the LRU.
RelayAuthenticator publishes a PersistentMap<NormalizedRelayUrl,
RelayAuthSnapshot> via authStateFlow on every mutation (connect,
disconnect, AUTH-submitted, AUTH-OK, AUTH-fail). PersistentMap gives
O(log32 n) updates and a fresh identity per put, so both StateFlow
equality and Compose strong-skipping work.
This is the substrate for downstream consumers — the AUTH approval
banner, the retry-queue wake on authCompleted, the indexer-fan-out gate
— none of which are wired yet. They will read authStateFlow rather than
querying RelayAuthStatus directly.
Claude
Vitor PamplonaandGitHub
davotoula
nrobi144