Replicates the PollEvent layout: all tag parsing moves to TagArray extensions
in TagArrayExt.kt with the event accessors delegating to them, builder
extensions use addUnique for single-instance tags, and construction goes
through template-returning builders instead of methods that sign internally.
- build() returns an EventTemplate<ContactCardEvent> (the signer is only used
to NIP-44 encrypt the private tags, matching TrustProviderListEvent);
create() remains as the signer.sign(build(...)) convenience and now takes
the emoji list directly
- updatePetNameAndSummary() returns an unsigned EventTemplate; callers sign
it (ContactCardsState and tests updated)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdvE4LvgkSewJXyFzyyAUY
Nicknames can now use NIP-30 custom emojis: typing : in the nickname dialog
autocompletes from the account's emoji packs, and the emoji mappings for any
shortcode used are embedded in the card's NIP-44 encrypted content next to the
petname — so even the emoji set stays private. Renderers resolve the petname's
shortcodes against the card's decrypted tags instead of the profile's metadata
tags.
- quartz: updatePetNameAndSummary replaces the private emoji tag set wholesale
and keeps it out of the public tags; round-trip test added
- commons: PetName(name, tags) holder with content equality, decryption cache
returns the merged decrypted tag list, EmojiPackState.findEmojiTags resolves
:codes: against the selected packs
- amethyst: Account embeds resolved emoji tags on save; all petname render
sites pass the card tags to the WithEmoji composables; nickname dialog gets
the : emoji autocomplete via EmojiSuggestionState
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdvE4LvgkSewJXyFzyyAUY
Adds petnames/nicknames per https://github.com/nostr-protocol/nips/pull/761,
reusing the kind:30382 contact card already used for WoT scores — a card only
acts as a nickname when signed by the account's main key. Petname and summary
are always stored in the card's NIP-44 encrypted content.
quartz:
- ContactCardEvent.updatePetNameAndSummary edits both fields in the encrypted
private tags, strips stray public copies, preserves every other tag
- TagArray.petName()/summary() parsers for decrypted tag lists + tests
commons:
- ContactCardDecryptionCache: LRU NIP-44 decrypt cache for own cards
- ContactCardsState: account-scoped access to the account's own cards,
petname flow per target user, create/update entry point
amethyst:
- Account.updateContactCardPetName publishes through the extended outbox
relays (NIP-65 write + private outbox + local + broadcast) and the login
subscription now downloads the account's own kind:30382s from its relays
- petname renders instead of the display name in usernames, profile header,
chats and @mentions (observeUserPetName)
- Edit-nickname action + dialog on the profile actions menu
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QdvE4LvgkSewJXyFzyyAUY
Clients in the wild publish structurally wrong values in profile
metadata — e.g. "nip05":{} — and the strict field serializers made
JsonMapper throw, so contactMetaData() returned null and the entire
profile (name, picture, about…) was discarded.
Generalize the BirthdayTolerantSerializer precedent: string fields now
use TolerantStringSerializer (accepts any JSON primitive, ignores
objects/arrays/null) and the bot flag uses TolerantBooleanSerializer,
so a single malformed field is skipped instead of being fatal.
Non-JSON content (e.g. "Relay initialized") remains unrecoverable and
still parses to null; a test pins that it does so without throwing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KSCGp6KgcnaepEMbxNroYe
Each PoW job now races PoWPolicy.minerWorkers(cores) = cores/2 parallel
searches (at least 1) instead of a single thread, so a post on an 8-core
phone mines ~4x faster while the other half of the cores stays free for
the UI, and the total CPU budget matches the old 2-job x 1-thread pool.
- PoWPolicy.minerWorkers(availableProcessors) is the single definition of
the per-job worker budget.
- PoWPublishQueue gains minerThreads (used by its built-in miner and
exposed for custom mine lambdas); AppModules wires maxConcurrent = 1 +
minerThreads = half the cores.
- PoWNostrSigner gains a workers param (default 1), covering reactions,
reposts and reports via Account.miningSigner; the anonymous-post paths
in the short-note and comment composers pass the same budget.
- Gift-wrap envelope mining stays single-threaded: the template
conversion is a non-suspend hook inside the synchronous NIP-59 build.
- UI time estimates (settings picker, composer chip, broadcast banner)
now benchmark at the queue's worker count via deviceHashesPerSecond(),
and PoWEstimator caches one rate per worker count.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhaF5scnAvhP9wr9R7bTWM
The miner enumerates the nonce space deterministically (the random base is
overwritten before the first hash), so naively racing N copies of the search
duplicates the exact same candidate sequence N times. PoWMiner.mine() now
races workers over disjoint slices instead: each worker's nonce carries a
distinct fixed prefix while only the bytes after it are enumerated, making the
aggregate hash rate scale with cores (~3.8x on a 4-core box).
The hot loop also switches from sha256() to sha256Into() with a reused
32-byte buffer, so hashing no longer allocates per attempt.
amy wiring:
- `pow mine` and `post --pow` mine on all cores by default; `pow mine
--threads N` overrides.
- `pow bench` measures the all-cores rate (what mining now uses, also the
basis for expected_seconds) alongside a new hashes_per_second_single_core.
- PoWEstimator benchmarks with sha256Into to match the miner, and gains a
workers overload that prices in cross-core contention.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhaF5scnAvhP9wr9R7bTWM
GiftWrapEvent.create no longer takes powDifficulty/powIsActive — it takes a
single templateConversion hook ((template, ephemeralPubKey) -> template,
default identity) that runs on the finished wrap template right before the
ephemeral key signs it. The hook receives the ephemeral pubkey because the
NIP-01 id a nonce commits to includes it and the key never leaves create().
NIP17Factory forwards the same hook through wrapSeal/createWraps and the
create*NIP17 entry points, so quartz's NIP-59/NIP-17 code no longer imports
the NIP-13 miner at all; Account builds the mining closure at the call site.
Any future pre-sign wrap adjustment flows through the same seam.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
Durability & correctness:
- Keep the job entry + disk checkpoint alive until the publish continuation
completes (was: dropped at mining-complete); failed publishes keep their
checkpoint for restart retry and surface through a failures SharedFlow
- Move draft deletion into the publish continuations in every composer so a
cancelled or process-killed mining job can't destroy the only copy of a post
- Persist gift-wrap mining: split NIP17Factory into createSeals (signer
interaction, runs inline) + wrapSeal (pure CPU, runs on the queue), new
REPLAY_WRAPS records restore pending DM wraps after process death
- Clamp synced NIP-78 difficulty (PoWPolicy.MAX_DIFFICULTY), require a sane
target in PoWMiner, bound PoWRankEvaluator against short ids
- Reaction double-tap while mining now toggles (dedupeKey + cancelByKey)
instead of publishing duplicates
- Restore checkpoints for every loaded account, not just the active one
- logOff purges the account's checkpoints and cancels its queued jobs
- Private notes: composer chip now gates on the gift-wrap kind and the
per-post override reaches sendPrivateNote
Coverage:
- Public/live chat (kinds 42 + 1311) and voice replies (1244 + kind-1 audio
replies) now route through the mining gate
Perf:
- FGS start() dedupes with a running flag; PendingIntents built once
- Banner 1 Hz clock only ticks while a job shows elapsed time
- PoWEstimator benchmark is single-flight behind a Mutex
UX / l10n:
- Post-mining failures toast with retry information
- Count strings converted to <plurals>; elapsed time via DateUtils; settings
estimate uses localized units; shared powKindLabelRes replaces three
duplicated kind→label maps; dead pow_chip_* strings removed
- CLI: pow mine lowercases the pubkey before mining (uppercase hex mined an
id that never matches the signed event) and validates via quartz Hex
New queue tests: checkpoint lifetime, failure reporting, cancelByKey toggle,
per-owner cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
Audit follow-up to the gift-wrap relay-icon fix: Marmot group chat rows
render the decrypted inner note (kind 9/7) via the standard chat feed with
RelayBadgesHorizontal, but nothing ever populated that note's relay list —
OK acceptances and relay deliveries all landed on the kind-445 envelope,
which has no link to its inner event.
- GroupEvent now implements HasInnerEvent (same @Transient @Volatile
innerEventId pattern as GiftWrapEvent/SealedRumorEvent), so
LocalCache.addRelayToNoteAndInners drills 445 -> inner for both OK
confirmations and duplicate EVENT deliveries. RouteMaker is unaffected:
it gates on the concrete wrap types before casting to HasInnerEvent.
- GroupEventHandler sets innerEventId at decrypt time and copies the
envelope's accumulated relays down to the inner note (looked up by
event.id, not the eventNote/publicNote params, which belong to the
triggering event when replayed from retryPendingFor).
- sendMarmotGroupMessage sets innerEventId on the freshly built envelope
before consuming/publishing, so acceptances for sent group messages
reach the rendered note as soon as the inner note is indexed.
Also audited the remaining chat-rendered types: NIP-04 PrivateDmEvent,
ChatMessageEvent rumors, NIP-C7 ChatEvent, ChannelMessage/Ephemeral/
LiveActivities messages all route through consumeRegularEvent (duplicates
covered by the shared helper), and NIP-37 drafts render the wrap note
itself with markAsSeen covering the version/addressable pair.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YEiq1NMK3q12KGQ2yYhPEp
Retry rounds were the dominant tail cost of deep crawls (rounds 8-10 of a
hop-8 run burned ~80 min for +6.6k lists): every 256-user batch with
attempt>0 or no-outbox users fanned the SAME ~40 backbone+fallback relays,
queueing thousands of small REQs against their 16-permit gates — pure
hot-relay head-of-line blocking.
Restructured:
- routeByOutbox now routes ONLY own-signal relays (write relays; hints when
no outbox) and flags no-outbox/retry users into a communal set.
- New Phase C communalSweep asks the shared relays ONCE per round for all
flagged users in author-chunked units — same (relay, user) coverage,
~8x fewer REQs — and records answered-empty pairs so each round shrinks.
- Attempt accounting moves to Phase C, after the round's full coverage
(own relays + shared set) has run.
- shardedSweep now also records answered-empty (user, relay) pairs, so the
communal sweep never re-asks a pair the shard pass already proved empty.
- Fixpoint recovery dedup: recoverStragglersFromAggregators skips users a
prior pass already asked the aggregators for (each repeat pass cost ~4 min
of store scans + REQs at 470k stragglers for zero new answers).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Two bugs found in a pre-merge audit:
- quartz PoolEventOutboxState: auth-required NAKs only spared the `responses`
budget, but `tries` (grown by every send/re-pump and NOT auth-aware) still
accumulated across reconnects, so a slow/flapping AUTH handshake could exhaust
Tries.isDone() and drop the event — with a spurious onEventGaveUp — before AUTH
landed. Now an auth-required NAK resets the relay's retry budget (it responded,
so it's up and just wants auth). Regression test added.
- RelayAuthPromptBus used a replay=0 SharedFlow, so a challenge that resolved to
ASK before RelayAuthPromptHost subscribed (cold start / account switch) was
dropped and the auth coroutine stalled the full timeout then DISMISSed. Add
replay so late subscribers recover pending prompts (the host already filters
resolved ones). Regression test added.
Also record the as-built design (Always/Never/Custom + toggles, venues, give-up
toast, known deny-relay-outbox limitation) in the plan doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Instead of silently dropping an event once its per-relay retry budget is
spent, PoolEventOutbox now reports it: PoolEventOutboxState.newTry returns
whether the attempt gave up on the relay, PoolEventOutbox.onSent surfaces
the dropped event, and NostrClient notifies a new (default no-op, so
non-breaking) RelayConnectionListener.onEventGaveUp(relay, event). Lets a
host surface a failed delivery rather than lose it silently; the event may
still be pending on other relays. Unit-tested via the outbox try budget.
Note: timed retry backoff (the other half of this item) is intentionally
deferred — applied in the shared syncState path it would also delay the
post-auth resend and regress the auth-required fix, so it needs
trigger-aware handling designed separately.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Add INostrClient.activeOutboxEvents(url) (backed by
PoolEventOutbox.activeOutboxEventsFor) returning the full events still
pending delivery to a relay, not just their ids like activeOutboxCache.
This lets a host explain *why* a relay is being authenticated with —
e.g. a pending kind-1059 gift wrap means we're sending a DM to its
recipient — by inspecting kind/tags. Combined with the existing
activeRequests(url) filters, it is the generic challenge context the
NIP-42 decision hook needs. Updates the INostrClient test fakes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
PoolEventOutboxState treated an `auth-required` OK-false the same as any
transient failure, recording it against the per-relay Tries budget
(responses > 2 drops the event on the next send attempt). Relays that
NAK every unauthenticated EVENT could therefore exhaust the budget and
drop the message before the AUTH handshake completed — the event was
gone by the time syncFilters re-sent it after the auth OK.
Treat `auth-required` as a deferred state instead: keep the relay in
relaysRemaining and record no failure, so the existing
syncFilters-after-auth path redelivers it. Mirrors the behavior already
present in StandaloneRelayClient. Terminal rejections (invalid/pow/
replaced/deleted) and ordinary transient errors are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EZjmYpgHP4pf79Sav5QT8a
Chat rows render the inner rumor note of a NIP-17 message, but relay
attribution only landed there through narrow windows, so accepted relays
often never showed as icons:
- Duplicate deliveries were stranded on the wrap: a gift wrap re-delivered
by a second relay hit the duplicate branch of consumeRegularEvent, which
tagged the outer wrap note only and never re-processed the event. Extract
the OK-path drilling (wrap -> seal -> rumor) into
LocalCache.addRelayToNoteAndInners and call it from both the OK
confirmation path (markAsSeen) and the duplicate EVENT path, replacing
CacheClientConnector's private copy.
- Cross-thread visibility: Note.event, Note.relays, Note.flowSet and the
innerEventId of GiftWrapEvent/SealedRumorEvent are written by decrypt/index
coroutines and read lock-free on relay socket threads; a stale read parks
an acceptance on the outer envelope permanently. Mark them @Volatile.
- Orphaned UI flows: RenderClosedRelayList/RenderAllRelayList and
createMustShowExpandButtonFlows captured note.flow().relays.stateFlow once
in remember/stateIn; MemoryTrimmingService.cleanObservers destroys the
unobserved NoteFlowSet while the lifecycle is stopped, so resumed rows
never saw another relay update. Wrap in a cold flow that re-resolves
flow() on every collection start.
- Indexing latency: sent DMs waited for the ~1s newEventBundles batcher
before the self-wrap was unwrapped and the message reached the chatroom,
parking early OKs on the wrap. broadcastPrivately and
sendNip04PrivateMessage now run the EventProcessor on the freshly
consumed note immediately; the batched re-delivery is idempotent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YEiq1NMK3q12KGQ2yYhPEp
Findings from a full NIP-13 review:
- The spec recommends updating created_at while mining. Queue jobs now
re-stamp the template to "now" when a worker picks them up (a post can
wait behind other jobs, and a job restored after process death could be
hours old); the restorer does the same. Scheduled posts are exempt —
their future created_at is intentional. Anonymous posts re-stamp before
mining against the throwaway key.
- PoWTag.assemble(nonce, null) serialized the literal string "null" as
the third tag entry; a missing commitment now omits the entry entirely.
- New tests: PoWTagTest pins the NIP-13 example tag shape and the
no-commitment round trip; a queue test asserts the created_at
re-stamp at mining start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
Implements user-facing PoW publishing (#3317) on top of the existing
quartz miner:
- quartz: PoWMiner.run gains a cooperative isActive cancellation hook
(checked every ~VALID_BYTES^2 hashes); PoWNostrSigner decorator mines
kind-scoped templates pre-signature so it composes with every signer;
GiftWrapEvent.create/NIP17Factory can mine the outer ephemeral-key
wrap (never the seal/rumor); NostrSignerWithClientTag exposes
prepareTags so mining runs over the final tag set.
- commons: PoWPublishQueue (FIFO, capped worker pool on
Dispatchers.Default, per-job cancel, in-memory only — unmined posts
are lost on process death, logged) and PoWPolicy (kind-group
categories with a hardcoded NEVER list: auth, zap requests, NWC and
bunker RPC, HTTP/Blossom auth, drafts, metadata and lists, OTS).
- amethyst: per-account synced settings (difficulty Off/16/20/24/28 or
custom, per-category checklist) in Compose Settings; Post enqueues the
template and returns immediately; reactions, reposts, reports, private
notes, DMs and long-form route through the same shouldMine gate at
their existing choke points; per-post PoW override chip in the
composer options row; "Mining proof of work… (N in queue)" phase with
per-job cancel in the broadcast banner.
Scheduled posts and anonymous posts mine against the correct key
(scheduled posts skip mining in v1); the client tag is applied to the
template before mining so signing never invalidates the nonce.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
An A/B rerun of the full-depth crawl exposed a completeness cliff: deep-hop
coverage was hostage to WHEN a contact list arrived. Lists fetched in-round
expand the frontier and get their follows crawled next round; the same lists
recovered by the terminal aggregator pass were counted but never crawled -
the rounds were already over. One weak backbone hour at round 5 shifted ~30k
lists from in-round to the terminal pass and silently amputated ~220k
hop-6/7 users (621k discovered vs 394k on otherwise-identical runs, while
total lists differed only 7.5%).
Extract the round loop into runRounds() and iterate rounds + aggregator
recovery to a fixpoint: while a recovery pass reveals new in-budget pending
users, resume rounds and recover again. Converges because each straggler
folds at most once and the hop budget bounds depth; maxRounds still
backstops. The warm pool stays up through recovery so resumed rounds start
on warm sockets, and a report-deletions top-up runs only when extra rounds
actually happened.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
CachingDns duplicated 20% of what the app's SurgeDns already did better
(stale-while-revalidate, single-flight, jittered 24-48h positive TTL,
persistence, poison filtering). Consolidate on SurgeDns and fix what the
verification pass found along the way:
- move SurgeDns + SurgeDnsStore (+ their 41 tests) to quartz jvmAndroid so
the CLI can share them; delete CachingDns
- negative TTL 10s -> 10min (class default): a dead domain burns 10-30s of
getaddrinfo per re-dial and both the relay pool and a crawl re-dial dead
hosts continuously; the relay pool's own backoff already reaches 5min
- stop the call-failure listeners from erasing negative entries they were
just written from: callFailed caused by UnknownHostException must not
invalidate (it deleted every negative entry milliseconds after creation,
silently defeating the negative TTL entirely). The invalidation remains
for its real purpose - stale positives with rotated IPs
- new SurgeDns.staleAll(): soft-expire everything WITHOUT discarding -
positives serve stale and revalidate in the background on next use,
negatives re-try on first touch. Wired to network-identity changes in
AppModules (same trigger as Tor's onNetworkChange), replacing nothing:
the full-clear invalidate() was never actually called on network change
- amy: SurgeDns wired into the CLI OkHttp client, snapshot persisted at
~/.amy/shared/dns-cache.bin across runs (best-effort load/save)
Verified: all SurgeDns/SurgeDnsStore tests pass in the new location incl.
two new staleAll tests; :amethyst compiles (main + unit tests); cli suite
green. Caveat recorded in the plan doc: behind an HTTP CONNECT proxy OkHttp
never consults the client resolver (hostname goes to the proxy), so this
layer pays off on direct-connect deployments only - which also means the
branch's A/B numbers never depended on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Every hard diagnosis this branch needed lived in blind spots of the old
output — waits that happen before its timers start and stages it never
delimited. Close them:
- permit/rate-gate wait ledger per relay (queueing for a limiter slot
happens BEFORE the drain timer starts; total + worst offender always,
top-10 queue table under --diagnose)
- [batch-walls] per round: p50/p90/max of Phase-B batch walls vs the fast
window, naming the last-resolving unit of the slowest batches - the
direct measure of pre-drain queueing
- store path split: insert wall now labeled as mutex-wait-inclusive with
an uncontended-writes estimate (solo batches), plus a read counter
(351 queries/724ms at hop-2) that makes point-query pathologies visible
- ticker gains visited/s + batches/s so retry rounds over list-less users
read as "working, zero yield" instead of "stalled" (the done-rate hid
this for hours)
- per-round limiter demotion counts; stage durations for aggregator
recovery and report deletions
- [slow-relay] lines deduped to 3 samples per authority with 3-author
samples (a hop-8 log was ~1MB of repeats; telemetry keeps full counts)
and the redundant per-drain parked line removed
Smoke-tested live on a hop-2 --diagnose crawl; all suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Cold hop-3 A/B (fresh store per leg, same observer): 48 workers beat 24
twice at identical contact-list counts (579s->559s and 474s->396s). The
old result that made 64 look 2x slower predated the multithreaded
dispatcher fix - more coroutines on one starved event-loop thread. Plan
doc gains the full A/B table (kept: background-park degating, sweep
dedup, tail overlap, dc48; rejected: 5s fast window).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Timestamped hop-3 logs split the 373s tail three ways: 222s blocking
convergence on parkedInFlight (whose late lists provably cannot fold once
pending is empty - parked filters only ever asked for in-budget users and
they are all done; the events still stream into the store), 14s of
aggregator recovery, and 137s of report-deletion fetching serialized
behind it. Convergence now breaks as soon as the frontier is empty; the
aggregator pass and report deletions run concurrently; and the aggregator
folds the lateHarvest trickle until one full park window of silence
instead of inheriting every background park's lifetime.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Straggler rounds re-offered the identical missing set to the identical
top-relay backbone each round, re-paying the rotation barriers (~20s of
Phase A per round) for ~zero new lists. Track swept users and only sweep
newcomers - but only once the backbone is at full shard width, so users
swept against round 2's proto-backbone still get a real sweep when the
top-10 exists.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
A hop-3 crawl spent ~500s of its 594s total in the finishing tail waiting
for parkedInFlight to hit zero - and the spike (1791 parked units) was the
fire-and-forget Tier-2 relay-list sweeps, whose whole point was not to
block the crawl. They only gated it because a parked unit persisted its
events at park END, so cancelling early would have lost them.
Fix in two parts: the park loop now persists incrementally (drains + stores
the queued chunk on every activity ping, plus a NonCancellable final drain),
so cancelling a park loses nothing already delivered; and drainGated grew a
background flag - Tier-2 sweep parks (and their capped-page paginations)
no longer count toward parkedInFlight and are simply cancelled at crawl
end. Round-critical parks (outbox drains, sharded sweep, aggregator
recovery) still gate convergence exactly as before. paginateIfCapped takes
(pageSize, oldest) since the incremental path no longer retains the page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Once the crawl went multithreaded, the store became the visible Phase-A
bottleneck: harvestFromStore, shardedSweep's missing-check, the Phase-B
consumer, and the aggregator straggler scan each issued one contactsOf
point query per user (~8ms each on a multi-GB store under concurrent
writers) - 100k-user folds burned minutes serially, and the aggregator
pass over 300k discovered users would have burned ~40 on its own. Add
latestContactsFor (one chunked author query per 300 users, newest
created_at per author wins) and route all four paths through it;
harvestFromStore folds chunk-by-chunk so peak memory stays one chunk.
Phase A now folds the store first and shardedSweep trusts `done` instead
of re-checking per author.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
The CLI calls crawl() from runBlocking's single-threaded event loop, so
Phase B's thousands of concurrent drain-unit coroutines (timeout timers,
channels, REQ JSON encoding, signature verifies, SQLite writes) all queued
on ONE thread: timers fired late, batch walls inflated ~5x, and the
--diagnose ticker showed 24/24 workers pinned while completing ~1 user/s
with one core pegged and three idle. It also explains why the old
64-worker A/B ran slower - more coroutines on the same thread. Hop the
whole run onto Dispatchers.IO inside the crawler so every front end gets
real parallelism; IO (not Default) because the store's blocking SQLite
calls must not starve the cores-sized pool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Most of a from-scratch crawl's wall clock was connection setup, re-paid
serially: the relay pool tears down a relay's socket ~300ms after its last
drain unsubscribes, OkHttp allowed only 256 concurrent WS handshakes, and
every dial re-ran an uncached blocking getaddrinfo (10-30s per dead domain,
once per per-user path URL of the same host). Open the connections once, in
parallel, and keep them:
- GrapeRankCrawler: the warm-pool trick (never-matching REQ that only holds
the socket) now covers the whole candidate universe instead of the top 20 -
seeded at crawl start from the reachability cache's live set (one parallel
connection storm, Config.knownLiveRelays) and refreshed each round with
newly learned outbox relays, capped by Config.preconnectCap (FD-budget
aware, --preconnect-cap / --no-preconnect).
- CachingDns (quartz jvmAndroid): 10-min positive + negative DNS cache with
in-flight per-host dedup; dead domains fail in microseconds instead of
re-burning resolver timeouts, path URLs of one host resolve once.
- cli Context: dispatcher and pre-connect caps derived from the process's
open-files limit (UnixOperatingSystemMXBean), warning when ulimit is low.
- amy graperank probe: relay census - mass-connects every relay the store
knows (kind:10002 universe deduped per authority + cached verdicts) in
waves, records live/dead with real measured rtt-open into the NIP-66
reachability cache (RelayProber + RelayReachabilityStore.recordProbed),
so the next crawl skips dead relays and waits once for the slow-but-alive.
Single-server limits (FDs, ephemeral ports, DNS, threads, conntrack) and the
design are documented in quartz/plans/2026-07-10-graperank-connect-storm.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
The Messages-list feed already filtered kind-9/1068/11/1111 content when
selecting a group's representative note, but the ChatroomEntry render fallback
still rendered ANY group-scoped note — including a kind-7 reaction lingering in
the in-memory list — as the group row, using the reaction's content and time.
Add a shared quartz helper `Event.isGroupChatContent()` and use it in both
places: the feed filter and the render fallback. A non-content group-scoped note
now falls back to the channel placeholder ("No messages yet") instead of showing
the reaction as the room's last message.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Addresses correctness/perf issues found in the crawler + reachability audit:
- deadHosts permanent eviction (#1): an authority that accrued timeoutEvictStrikes
before its first EOSE was evicted forever — clearTimeoutStrikes only zeroed the
counter and could not un-evict, contradicting the "a host that ever produces is
never evicted" invariant. Add a producedHosts set that isDead() consults, so a
proven-productive authority is never treated as dead even if a concurrent strike
from the 24-worker fan-out raced it into deadHosts.
- Parking-disabled event loss (#2): when parking is off (no bgScope, or
parkTimeoutMs <= timeoutMs), a relay that streamed events but didn't EOSE in the
fast window had its buffer dropped without persist() and reported count 0. Drain,
persist, and return those events like the other two branches; strike only when
nothing was delivered.
- Wide-sweep over-narrowing (#4): relayListDiscoverySwept excluded an already-swept
straggler from the wide pass even though the wide net grows each round, so a 10002
hosted only on a later-learned relay was never fetched. Gate the wide pass on the
asked-relay set (wideRelaysSwept) instead: new users get the full net, older
stragglers get only newly-appeared relays, no (user, relay) pair asked twice.
- Onion detection (#10): replace loose relay.url.contains(".onion") with
RelayUrlNormalizer.isOnion() in isDead() and networkTypeOf(), fixing the
foo.onionfake.com false positive and the store/crawler disagreement.
- rtt-open=0 semantics (#9): document that the crawler's reachable records use
rtt-open purely as a liveness flag (0 = latency not probed), not a real 0 ms
measurement, and must not be published as authoritative latency data.
deadHosts is deliberately still NOT persisted to the 24h reachability cache (#8):
a timeout eviction means "too slow under our fan-out this run", not "proven
unreachable", so persisting it would blacklist slow-but-live hubs across runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
FrameDispatchStats stamped a ValueTimeMark on every relay frame and recorded a
contended atomic per frame in BasicOkHttpWebSocket — the WebSocket layer used by
the whole app, unconditionally, forever — to answer a one-time question that only
graperank --diagnose read. It served its purpose (proved the our-side dispatch
lag is ~200ms mean and the EOSE-wait is dominantly relay-side, so the crawler is
network-bound), but the ongoing per-frame Pair allocation + atomic contention on
every client's relay traffic isn't worth carrying. Revert the channel back to
Channel<String> and delete the stats holder. The diagnose-gated saturation ticker
and per-drain latency breakdown stay — they're crawler-local, off the hot path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured: relays deliver their events in ~0.6s then sit ~4.6s (86% of drain wall)
before sending EOSE — mostly relay-side (our pipeline adds only ~200ms). So instead
of waiting the full 10s fast window then parking, close a drain that has delivered
>=1 event and then gone silent for eoseIdleMs, treating it as complete ("eose-idle").
awaitTerminalOrQuiescent: the idle timer arms only AFTER the first event, so a relay
merely slow to answer still gets the full timeoutMs and is never cut prematurely; a
still-streaming relay keeps resetting the window. eose-idle paginates if the page was
capped and clears timeout strikes (it delivered), but joins notAnswered (no clean
EOSE, so its missing authors are retried elsewhere). Off by default (eoseIdleMs=0),
CLI --eose-idle-ms, so it can be A/B'd against the plain fast window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The dedicated frame-dispatch pool (ea1093ad) made dispatch lag WORSE, not better:
mean 200ms→460ms, max 3.5s→5.5s, frames>1s 43k→76k. The pool was sized cores*2
(=8 here) vs Dispatchers.IO's 64 threads, so it cut frame-processing parallelism
~8x. Lesson: the our-side lag is dominated by per-connection serial decode
throughput / thread count, NOT cross-contention with the store's IO writes — the
experiment ruled that hypothesis out. Reverting to shared IO; keep FrameDispatchStats.
EOSE-wait is confirmed dominantly relay-side (200ms our-mean vs ~5s eose-wait).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Measured on a GrapeRank crawl, frame decode/dispatch (per-connection consumer
coroutines) ran on the shared Dispatchers.IO — the same pool that runs the store's
blocking SQLite inserts. During event floods, frame coroutines queued behind those
inserts: mean 200ms and up to 3.5s of dispatch lag, with 43k frames waiting >1s in
our pipeline. That lag also skews the relay-idle/EOSE timing the crawler reads.
Give frame processing its own daemon thread pool (sized to a small multiple of
cores; decode is light + CPU-bound), shared across all connections. Frame delivery
stays prompt regardless of what the IO pool is doing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Adds FrameDispatchStats: the lag between a relay frame arriving on the OkHttp
reader thread and our per-connection consumer coroutine (on shared Dispatchers.IO)
pulling it off the channel — pure our-side pipeline delay, relay send-timing
excluded. BasicOkHttpWebSocket stamps arrival before enqueue and records the lag
on dequeue; the crawler resets it at start and dumps it in the --diagnose summary.
Answers whether a drain's 5s gap between the relay's last event and its EOSE is
the relay being slow to SEND eose (low dispatch-lag) or our IO pipeline backing up
so the already-arrived eose frame sits queued (high dispatch-lag).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A Nostr pubkey is x-only, exactly 32 bytes, but NPub.parse/NProfile.parse never
checked the length — they hex-encoded whatever bytes the bech32/TLV carried. A
malformed npub/nprofile that some clients encode with the full 33-byte COMPRESSED
secp256k1 key (0x02/0x03 prefix) therefore round-tripped its 66-char hex straight
into a `p`/`q` tag via the quote/mention path, and a strict relay (relay29 /
pyramid.fiatjaf.com) rejected the whole group message:
blocked: schema validation failed: tag[..]: invalid pubkey value
'02977dcf…c3402' ... pubkey should be 64-char hex
We never generate compressed keys ourselves (Nip01Crypto.pubKeyCreate strips the
prefix byte); this is purely inbound malformed input. Enforce the 32-byte length
at the decode boundary so the bad entity never becomes a mention/quote tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Answers "are we resource-bound or waiting on relays" without a profiler:
- progress ticker gains "Nw/CAPw" (drain workers busy vs drainConcurrency) and
"N rl" (rate-limit responses so far) — a rarely-full pool means the producer or
the relays are the limit, not concurrency; a climbing rl count is the external
ceiling that made concurrency 60 backfire.
- crawl-end "latency breakdown": splits each drain's wall into time-to-first-event
vs EOSE-wait-AFTER-the-relay's-last-event, and reports the % of drain wall spent
waiting for EOSE after the relay was already done, how many drains blew the fast
window and parked, and total rate-limit hits. A high EOSE-wait % is the direct
case for a shorter/adaptive fast window over more concurrency.
All gated on config.diagnose; zero cost on a normal run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Two changes:
1. The reachability flush re-wrote the SEEDED known-dead relays with a fresh
created_at every run, refreshing their TTL without a re-probe — so a relay
marked dead once (and thereafter skipped, never re-dialed) would stay
blacklisted forever as long as crawls kept running, defeating the TTL's
re-probe. Stats.deadRelays now reports only relays actually dialed this run
(deadRelays - knownDeadRelays); seeded records keep their original timestamp
and age out on schedule so the next run re-probes them.
2. Rename GrapeRankDataCrawler -> GrapeRankCrawler (file + all references).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Wire RelayReachabilityStore into the crawler and the WoT updater so liveness is
shared across procedures and runs instead of each rediscovering dead relays.
- OperatorKeys.monitorKey(): a dedicated machine monitor identity derived from the
operator master (domain "relay-monitor:"), independent of any account — the
30166 records are published under this, not the observer key.
- Context.reachability: a RelayReachabilityStore over the shared store, signed by
the monitor key.
- Crawler: Config.knownDeadRelays seeds deadRelays before the run; Stats now
returns the final dead/live sets. GrapeRankCommand seeds from snapshot().dead
and flushes the crawl's verdicts back via reachability.record().
- Updater: Config.knownDead skips proven-dead relays from the reconcile plan — a
dead relay cannot serve its authors, so reconciling it only burns a timeout.
Live author-advertised relays are always synced.
All behind --no-reachability-cache. TTL'd (24h), so a recovered relay is retried
once its record ages out — a "skip for now", never a permanent ignore, keeping
the outbox rule that every live advertised relay is tried.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
A durable, shareable relay-reachability cache backed by the EventStore as NIP-66
kind:30166 Relay Discovery events, so the crawler, the WoT updater, and future
runs share liveness knowledge instead of each rediscovering dead relays from an
in-memory set wiped at process exit.
- 30166 is addressable by its d-tag (relay URL) → one replaceable status slot per
(monitor, relay), with created_at giving a free TTL.
- Reachable → 30166 with rtt-open; dead → 30166 without (NIP-66 has no explicit
offline field; liveness is inferred from a fresh successful open). Live wins
over dead within the TTL, so third-party monitors' 30166 can be ingested.
- snapshot() loads the fresh set once (not a per-request hot-path query); record()
flushes a run's findings. A relay is only skipped for the TTL, never permanently
— consistent with the outbox rule that every advertised write relay is tried.
Reuses the existing RelayDiscoveryEvent. jvmTest covers record/reload,
live-overrides-dead, TTL expiry, and .onion→Tor network tagging.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Round-8 profiling showed the crawl re-querying the same never-had-a-10002
users' outboxes every round they recirculated — ~144k slow kind:10002 drains
(p50 17.4s) against a static discovery set, dragging the round to ~18 users/s.
1. ensureRelayLists guards with `relayListDiscoverySwept`: each user's outbox
discovery runs once. The discovery relay set is static, so a second sweep of
a user still lacking a 10002 cannot find one the first missed.
2. The discovery REQ to the bounded INDEXER set co-fetches [10002, 3]: the
outbox lookup already pays the round-trip and an indexer holding a user's
10002 often holds their kind:3, so we harvest the contact list as a cheap
byproduct. The wide "every live relay" completeness sweep stays 10002-ONLY —
co-fetching kind:3 across thousands of relays downloaded the same big contact
lists repeatedly and inflated the fire-and-forget bgScope sweep the finishing
drain waits on (measured +260s at hop-3; the indexer-only co-fetch keeps
coverage flat at baseline speed).
3. harvestFromStore folds any already-stored kind:3 into the graph at Phase-A
time so Phase B never re-drains a list we hold (also speeds re-runs).
Verified same-session hop-3: pre-fix 685s / narrowed 690s / wide-co-fetch 945s,
coverage 91.74% across all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Claude
Vitor PamplonaandGitHub
davotoula