Commit Graph
2512 Commits
Author SHA1 Message Date
Claude a545151a34 fix: reject non-32-byte pubkeys when decoding npub/nprofile
A Nostr pubkey is x-only, exactly 32 bytes, but NPub.parse/NProfile.parse never
checked the length — they hex-encoded whatever bytes the bech32/TLV carried. A
malformed npub/nprofile that some clients encode with the full 33-byte COMPRESSED
secp256k1 key (0x02/0x03 prefix) therefore round-tripped its 66-char hex straight
into a `p`/`q` tag via the quote/mention path, and a strict relay (relay29 /
pyramid.fiatjaf.com) rejected the whole group message:

  blocked: schema validation failed: tag[..]: invalid pubkey value
  '02977dcf…c3402' ... pubkey should be 64-char hex

We never generate compressed keys ourselves (Nip01Crypto.pubKeyCreate strips the
prefix byte); this is purely inbound malformed input. Enforce the 32-byte length
at the decode boundary so the bad entity never becomes a mention/quote tag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-09 17:24:03 +00:00
Claude 0140b837b1 feat: per-type relay group discovery (follows/admins/members + topics/geo)
Relay-signed kind-39000 has no author-of-a-follow, but the people dimension
still exists: a follow may be the relay signing key, a group admin (39001), or
a member (39002). Discovery now resolves each top-nav filter into a per-relay
GroupDiscoveryConstraint instead of collapsing every filter to the same REQ.

quartz:
- GroupMetadataEvent / EditMetadataEvent: build + read #t (topics) and #g
  (geohash, mip-mapped so a coarser followed geohash still matches). Interop
  tests for parse/build round-trips.

amethyst:
- dal/RelayGroupDiscoveryFeedFilter: sealed GroupDiscoveryConstraint
  (AllGroups / ByPeople / ByHashtags / ByGeohashes / AnyOf) + toGroupConstraints()
  mapping each IFeedTopNavPerRelayFilterSet to per-relay constraints, with
  matches() covering the relay-key/admin/member people paths and topic/geo tags.
  Unit tests.
- Directory REQ narrows to 39000 #t/#g for topic/geo filters, broad directory
  otherwise (people match needs the rosters).
- ViewModel keys the feed on the constraint map and re-scans on any directory
  event (metadata OR roster) so late-arriving admins/members surface groups.
- Create/edit form gains a Discovery section (topics + geohash) threaded through
  Account.createRelayGroup/editRelayGroupMetadata and EditMetadata.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 21:42:38 +00:00
Claude 2b11dbd7d9 test(nip29): cover weird apostrophe placements + guard relay-URL possessives
Reject a single-character group id (except the default `_`) so a possessive
glued to a bare relay URL — `wss://relay.damus.io's uptime` — no longer
linkifies group "s". Real ids (relay29/Wisp/0xchat) are all longer.

Adds coverage proving only genuine ws/wss relay URLs are peeked: apostrophes
after http, nostr:, blossom:, email and bech32 tokens never become group links;
plus ws:// (insecure), second-apostrophe boundary, and multi-link cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 17:21:15 +00:00
Claude f9b24156e8 feat(nip29): linkify group invite links inline and via deep links
Recognise the de-facto `<relay>'<groupId>[?code=<code>]` NIP-29 group invite
link format used by Wisp and 0xchat, both inside rendered note content and as
an external deep link, so tapping one opens the group.

The URL detector correctly stops a host at the apostrophe (host names can't
contain `'`), so the group id is torn off before classification. Rather than
loosen the shared URL grammar — which would swallow prose possessives like
`example.com's` — group links are recovered by peeking just past each relay
URL the detector already found. This is cache-miss-only and costs nothing on
notes without a `wss://` link.

- quartz: GroupInviteLink.parse / suffixLength (+ tests)
- commons: Urls.groupLinks, UrlParser peek, RichTextParser plumbing
  (atomic span through fixMissingSpaces, new RelayGroupLinkSegment) (+ tests)
- amethyst: ClickableRelayGroupLink renderer + uriToRoute deep-link branch

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 16:15:31 +00:00
Claude e4a6e13036 fix: nostrord interop — thread titles and public group list
Two compatibility gaps found analyzing nostrord (a NIP-29 client):

- Thread titles: NIP-7D (and Amethyst) use a `title` tag, but nostrord
  writes/reads `subject`, so neither showed the other's thread titles.
  ThreadEvent.title() now reads `title` OR `subject`; we still emit only the
  spec-correct `title`.
- Joined-groups list (kind 10009): Amethyst wrote memberships as NIP-44
  private items, but both reference clients (Flotilla, nostrord) store — and
  nostrord only READS — public `["group", id, relay]` tags, so an Amethyst
  user's groups were invisible to them. follow() (and the amy CLI) now write
  public tags. NIP-29 membership is already public via the relay's kind-39002
  list, so this loses no real privacy; reads still merge any legacy private
  items so existing lists keep working.

Tests: read title from title/subject (title wins; we emit title only); public
group is a plain tag and still read through the cache; a mixed public+private
list reads as both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 14:21:42 +00:00
Claude e43c752a07 fix: keep group thread replies scoped to the group
Replying to a NIP-29 group thread went through the shared NIP-22 comment
composer, which built a plain kind-1111 comment with no group `h` tag and
broadcast it to the author's outbox. Such a reply is not group content: the
host relay rejects it and no other member — or other NIP-29 client like
Flotilla — ever sees it, and for a private/closed group it leaks to
unrelated relays.

Both fixes are tightly guarded on the replied-to event being group-scoped,
so ordinary comments are untouched:
- CommentPostViewModel inherits the group's `h` tag from the event being
  replied to (covers replies to the kind-11 root and to nested 1111
  comments — both route here).
- The reply is published only to the group's host relay (the relay the
  thread was seen on) via signAndSendPrivatelyOrBroadcast, instead of the
  outbox-computing broadcast — so it reaches the group and never leaks.

Verified: quartz test builds the reply as the composer does
(CommentEvent.replyBuilder { hTag } over a kind-11 root) and asserts it is a
1111 carrying the group `h` tag and referencing the thread.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 13:36:10 +00:00
Claude 633903b5c1 fix: relay-group audit — timeline, roster, membership, list-safety
Fixes found in a full audit of the NIP-29 relay-groups feature across
quartz/commons/amethyst/cli.

Correctness (app):
- Own group messages never appeared in the timeline until an app restart:
  the optimistic send is consumed with a null relay, so attachToRelayGroup
  bailed on the relay==null guard, and the host relay's echo (new==false)
  was skipped by the "only attach when newly consumed" gate. Attach now runs
  on every arrival, gated on the note being loaded, and the null-relay case
  attaches to the already-open channel(s) for that group id. Also avoids the
  wrong-relay phantom by only fabricating a channel from real provenance.
- Roster subscription was frozen after an in-place join/leave (state keyed
  on the stable account, never re-derived); it now invalidates on every
  liveRelayGroupList change, so a fresh join's 39002 admission is fetched.
- membershipOf demoted a 39001 admin with an empty/unknown role to MEMBER,
  hiding moderation; presence in the admins list now means at least MODERATOR.
- Members roster showed permanent truncated-hex names (one-shot
  getUserIfExists cached null); uses checkGetOrCreateUser so UsernameDisplay
  fills in when kind:0 arrives.

Protocol / data:
- GroupTag had no value equality → joined-group Sets never deduped and the
  StateFlow re-emitted on every identical re-arrival. Equality is now the
  (id, relay) pair, excluding the cosmetic name.
- create/edit emitted non-canonical ["public"]/["open"] status tags; NIP-29
  flags are presence-only, so only private/closed are emitted when set.
- Metadata/member/admin supersede guards use <= so an equal-createdAt
  duplicate isn't reprocessed (first-arrival wins); updatedMetadataAt is now
  private-set. Relay-group channels are now included in the prune loops.

CLI:
- join/leave/create updated the kind:10009 list from a network-only drain;
  a slow/empty fetch could publish a fresh list containing ONLY the new
  group, wiping the rest. Now reads the local store (source of truth) too.
- edit re-asserted both visibility axes from flag presence, so --closed on a
  private group leaked it public. It now reads current 39000 and merges,
  with --public/--open counter-flags; only the specified axis changes.
- create now tracks the new group in kind:10009 (parity with join/Android).

UI polish:
- Invite dialog no longer mints a 9009 for open groups and won't copy a code
  it never displayed. Browse "popular" list normalizes URLs before filtering.

Tests: GroupTag identity, unknown-role-admin-moderates, equal-createdAt
no-resupersede added; all quartz+commons NIP-29 suites and the amy
relaygroup harness pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-08 02:27:18 +00:00
Claude 32e8a8a7ed feat(quartz): add group()/groupSet() tag helpers for kind 10009
TagArray.groups()/groupSet() parse the NIP-51 simple-groups list `["group", id,
relay, name?]` items, mirroring ephemChat's rooms()/roomSet(). Needed for the
upcoming relay-group list state (the user's joined groups + servers).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-07 20:29:30 +00:00
Claude dc4fe7500f feat(nip29): add RelayGroupChannel model + group message datasource
First feature-layer slice for NIP-29 relay groups, cloning the ephemeral-chat
(NIP-C7) pattern and reusing the existing relay-pinned send/subscribe
primitives (RelayBasedFilter + signAndSendPrivatelyOrBroadcast) — no new
transport.

- quartz: GroupId(id, relayUrl) identifier for a relay group (host relay +
  group id), mirroring ephemChat's RoomId.
- commons: RelayGroupChannel — a metadata-backed Channel (like NIP-28's
  PublicChatChannel) keyed by GroupId, deriving name/picture/about/flags from
  the relay-signed kind 39000 event and pinning relays() to the single host.
- amethyst: filterMessagesToRelayGroup / filterMyMessagesToRelayGroup
  datasource sub-assemblers — kind 9 + poll timeline scoped by the `h` tag and
  pinned to the host relay via RelayBasedFilter, mirroring the ephemeral-chat
  sub-assemblers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-07 20:05:35 +00:00
Claude 2620ce9a50 feat(quartz): make NIP-29 event layer a verified superset of Armada
Audited Quartz's nip29RelayGroups package against every event and tag
Armada (gitlab.com/soapbox-pub/armada) creates and reads, and closed the
gaps so Quartz can round-trip all of them.

- GroupMetadataEvent (39000): fix flag parsing. `private`/`restricted`/
  `hidden`/`closed`/`livekit` are single-element presence tags, but the old
  accessors used hasTagWithContent (requires a value) AND conflated
  restricted<->closed and hidden<->private with wrong defaults (an open
  public group parsed as private+restricted). Now each flag is a plain
  presence check. Add hasLivekit() and supportedKinds(), extend GroupStatus
  with RESTRICTED/HIDDEN/LIVEKIT, and support supported_kinds in build().
- Add GroupParticipantsEvent (kind 39004, LiveKit AV presence) and register
  it in EventFactory + KindNames.
- Add TagArray.hasTagName() presence helper in nip01Core.
- Add GroupScope.kt: hTag() builder + Event.groupId()/isGroupScoped()
  readers so kind 9 (chat), 11 (thread), 7 (reaction), 1068, 1111 etc. can
  be group-scoped via the `h` tag and read back — NIP-29 reuses these
  carriers rather than defining new content kinds.
- Add Nip29ArmadaInteropTest covering parse + build for every event with
  Armada's exact tag shapes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
2026-07-07 19:45:26 +00:00
Claude e8f4c5f806 refactor(cli): align fetch default limit across paths; harden paging
Audit follow-ups before merge:

- amy fetch default limit is now the same on both paths: absent --limit → 100
  for plain AND --paginate (previously --paginate silently meant "unbounded").
  `--limit 0` is the explicit opt-in to drain everything (unbounded); negative
  is rejected. The effective limit is carried on the filter so both paths agree.

- drainAllPages sizes its SeenIds for CLI-scale fetches (initialSlotsPow2 = 12,
  ~64 KB) instead of the large-walk default (~16 MB eagerly allocated per fetch);
  it grows if an unbounded drain needs it.

- fetchAllPages clamps the inclusive advance to `min(pageMinTs, boundary)` so a
  misbehaving relay that answers with an event past the requested `until` can't
  push the cursor upward — the boundary dedup and termination rely on `until`
  never increasing. No-op for honest relays (they only return events ≤ until).

Verified live: default and --paginate both cap at 100; --limit 50 → 50; --limit 0
--paginate drains the full window (>100); paging tests + SeenIds tests still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 23:57:40 +00:00
Claude e0ebc8fad6 feat(cli): dedup drainAllPages via SeenIds; unbounded amy fetch --paginate
Two changes to the paginated fetch path:

- Cross-relay dedup before verify. drainAllPages' single consumer now runs a
  SeenIds filter: the same widely-mirrored event arrives once per relay, and the
  repeats are dropped BEFORE the expensive Schnorr verify + store instead of
  after (they were only trimmed by FetchCommand's distinctBy). An id is marked
  seen only once it verifies, so a forged copy (valid id, bad sig) delivered
  first can't suppress the genuine one from another relay. Adds SeenIds.contains
  (peek without recording) for that check-then-add.

- `amy fetch --paginate` no longer forces a --limit. With --limit N it still
  pages up to N per relay; WITHOUT --limit it drains the whole filter unbounded
  (the filter's null limit flows straight through). Plain (non-paginate) fetch
  still trims to the default 100.

Verified live: unbounded --paginate over a ~20-min nos.lol firehose window
returns 406 (all unique, 3s) vs the old 100 cap; --limit 50 caps at 50; default
caps at 100; cross-relay fetch stays count==uniq.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 23:31:09 +00:00
Claude 338c9a41af refactor(quartz): make SeenIds single-writer, move to commonMain
Drop @Synchronized from add/reset/size: SeenIds is now documented as
single-writer (not thread-safe). Callers dedup across concurrent relay
producers by funneling events into one consumer that owns the instance — the
one-consumer ingest pattern used elsewhere — which keeps a single global set,
stays lock-free, and lets resize run without coordination.

With the JVM-only @Synchronized gone the class is pure common Kotlin
(LongArray + Hex.readLong), so it moves from the jvmAndroid source set to
commonMain and is now available on every target.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 22:52:46 +00:00
Claude cb493a2291 feat(quartz): add SeenIds — a memory-lean event-id dedup filter
A run-scoped "already seen this id" filter for large, mostly-duplicate id
streams (a broad relay walk re-receiving the same event from many relays).
Keys on the first 128 bits of the id, sliced straight out of the hex with
Hex.readLong (table lookups, no parse, no allocation), in one open-addressed
LongArray — ~16 bytes/entry and the 64-char String is never retained, so tens
of millions of ids cost ~1 GB instead of a HashSet<String>'s ~6 GB. add() is
O(1) and synchronized.

Lives in the jvmAndroid source set (uses @Synchronized; a 40M-id walk is a
server-side concern). Ports the caller's implementation with the
parseUnsignedLong hot path swapped for Hex.readLong (~45-70 ns/op cheaper).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 22:38:37 +00:00
Claude 5c016fc2d4 fix(quartz): fetchAllPages must not drop events at page boundaries
fetchAllPages advanced with `until = oldest - 1` (exclusive) and no dedup. That
skips any event sharing the boundary second that didn't fit in the page — which
happens at *every* page boundary landing inside a second, not just pathological
dense ones — silently dropping events. An in-process probe with no second denser
than the relay's page cap still lost one event straddling the boundary.

Page inclusively now: `until = oldest created_at of the previous page`, and drop
the re-fetched boundary events by id. The dedup set is bounded to just the current
boundary second (`until` only decreases, so duplicates can only recur there), so
memory stays O(one second), never O(total).

A single second denser than the relay's page cap can't be drained (its tail is
unreachable — no client-side fix; raising the request limit is futile since we
already send one above the relay's cap). Once a page yields nothing new we step
strictly past that second so paging keeps progressing to older events instead of
stalling forever.

Tests: boundary-straddle retrieves all 6 (was 5); dense-second-beyond-cap steps
past without stalling and still delivers the neighbours. Verified on live relays
(strfry / nostr-rs-relay / khatru): ground-truthing each dense internal second
against the paginated set shows no gaps, incl. a 36-event second fully retrieved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 21:53:48 +00:00
Claude aa8412630e refactor(quartz): collapse fetchAllPages to a single active-filter list
The search-single-page logic left two lists with different roles: the listener
counted matches over the full `pagedFilters` (including a search filter already
dropped from paging) while the subscription only sent `remainingFilters`. That
worked — the dropped filter's count was unused and `advancesCursor` kept its
hits off the cursor — but it read as if a non-subscribed filter still mattered.

Collapse to one `activeFilters` list (index + filter) that is both what we
subscribe and what the listener iterates, so counting can't drift from what was
asked. Behavior is identical; the multi-filter and search tests still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 20:42:34 +00:00
Claude 590731f356 feat(cli): add Context.drainAllPages + shared fetchAllPagesFromPool accessory
Amy's one-shot queries all go through Context.drain, a single REQ drained to
EOSE — so a relay that caps its REQ response (strfry's per-REQ limit, ~500)
silently truncates the result with no way to page past it.

Extract the per-relay fetchAllPages fan-out that already lived privately in
EventSync into a reusable quartz accessory, fetchAllPagesFromPool: a
sliding-window pool (maxConcurrentRelays) that paginates each relay on its own
`until` cursor, tags every event with its source relay, and does not dedup
across relays. EventSync now delegates to it (its private downloadPool/
downloadFromRelay are deleted — no behavior change: perRelayFilters is already
ordered by and complete over the relay list).

Add Context.drainAllPages, the paged sibling of drain: same verify+store and
per-relay tagging, but fully draining sets larger than one REQ. Wire it into
`amy fetch` behind --paginate/--all (filter mode only), pushing the limit into
the filter so paging stays bounded. sync (NIP-77) and fetch stay separate
interfaces.

Tests: fetchAllPagesFromPool fan-out/tagging/no-cross-relay-dedup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 20:20:57 +00:00
Claude 54c8cefe69 fix(quartz): don't time-walk a search filter in fetchAllPages
NIP-50 search results are ranked by relevance, not created_at, so paging a
search filter by an `until` cursor silently degrades a top-N search into a
full time-walk of the corpus — and never terminates against a relay that
runs FTS over its whole corpus regardless of `until`.

fetchAllPages now queries a `search` filter on its first page only: it is
dropped from every later page and its hits neither advance nor drag back the
`until` cursor that co-resident non-search filters page with. onNewPage also
moves below the empty-page break so it never announces a page that isn't
fetched. Adds a test proving a search filter returns a single relay page
while a plain filter over the same capped relay still pages through the set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YEbdqCRPkszkGCoi89RMt
2026-07-06 19:58:11 +00:00
Claude e40d8df4d0 feat: add fast hex-to-Long slicing to Hex
Add Hex.toLong64/toLong128/toLong256 (plus the shared readLong helper) to
pack the first 64, 128 or 256 bits of a hex string into a single Long, two
Longs or four Longs. Big-endian, allocation-light, branch-free — 16 table
lookups and shifts per word. Useful as cheap map/set keys or bucket hashes
for 32-byte event ids and pubkeys without decoding to a ByteArray.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019CU1wR6NvQmdmNNsPe9GuN
2026-07-06 18:21:21 +00:00
davotoula 9af611faf1 fix: fetch uncached addressable thread roots; render blank PS1 blocks as empty slots 2026-07-06 00:35:58 +02:00
davotoula 836caa5cd6 Code review:
- Use the app's cached stringRes helper instead of raw stringResource,
  matching the dominant convention in ui/note/types
- Drive the icon animation from the Compose frame clock (withFrameMillis)
  instead of a delay loop, so the ticker suspends whenever the composition
  stops drawing rather than waking the main dispatcher 4x/sec from the
  back stack
- Drop the unconsumed memoryCardId/blockState/blockHash accessors; the
  tag schema stays documented in the class KDoc
- Document the frames arrays as frozen: mutating them in place would
    silently break the @Immutable skip contract; build a new instance
    to change pixels
- Replace the API-29-deprecated Bitmap.createBitmap(IntArray, ...)
    overload with createBitmap(w, h, config) + setPixels
2026-07-06 00:35:58 +02:00
davotoula c0cc2b0245 feat: render PS1 memory-card saves over nostr (kind 38192)
feat: animate the PS1 BIOS save icon on kind-38192 cards
2026-07-06 00:35:58 +02:00
Claude 5927c1837e fix(nip77): don't treat a "blocked:" NEG-ERR refusal as an over-cap overflow
A relay that refuses negentropy with a NEG-ERR whose reason merely starts
with "blocked" (e.g. "blocked: Negentropy sync is disabled" from a relay
that has NIP-77 turned off, or an auth/ban refusal) was misclassified as a
strfry `max_sync_events` overflow by `isOverflow`. Overflow triggers
created_at window-splitting, so every split re-opened, was refused again,
and the splitter fanned out breadth-first across the whole created_at range
(~2^31 windows). The call therefore never threw NegentropySyncException (so
`negentropySyncOrFetch` never took its paging fallback) and never tripped
the idle watchdog (the relay answered every NEG-OPEN promptly), so it hung
indefinitely. A second relay whose refusal string did not start with
"blocked" fell through to `Failed` -> paging and completed, which is why the
two behaved differently despite advertising the same NIPs.

Narrow `isOverflow` to genuine "result set too large" signals only; a bare
`blocked:` refusal now maps to a hard failure and fails over to paging.

Adds a regression test driving an in-process relay that refuses every
NEG-OPEN with "blocked: Negentropy sync is disabled" while still serving
plain REQ: negentropySyncOrFetch now pages and delivers every event instead
of hanging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015UF3eh76rRiwAuPg32rwiz
2026-07-05 16:08:25 +00:00
Claude 8efe8af2dc refactor(quartz): move NDJSON import/export into Quartz as store logic
The `import`/`export` engine is pure protocol/store logic — it operates only on
the `IEventStore` interface and Quartz event types (Event, OptimizedJsonMapper,
verify, Filter), with zero geode dependency — so per the sharing philosophy
("quartz = Nostr business logic, protocol, data") it belongs in Quartz, not in
the geode app. Any Quartz consumer (a relay, the `amy` CLI, a desktop
backup/restore) can now reuse it.

- move `com.vitorpamplona.geode.ImportExport` →
  `com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport` (commonMain,
  next to IEventStore); rename for a clear library-level name.
- geode keeps only the CLI glue (verb dispatch, arg parsing, file/stdin/stdout,
  the stderr summary) in Main.kt, delegating to the Quartz engine.
- move the test into quartz jvmTest, rebuilt on Quartz's own EventFactory +
  NostrSignerSync (real Schnorr signing) instead of geode fixtures.

No behavior change — `geode import`/`export` work exactly as before (verified
end-to-end previously); this is purely where the code lives.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-05 14:10:16 +00:00
Claude 5db2543cfc feat(geode): add import / export NDJSON verbs; drop the benchmark-only server
Bulk NDJSON import/export as first-class geode subcommands, mirroring
`strfry import` / `strfry export` (one JSON event per line — the interchange
format for seeding a relay, migrating between relays, or taking a backup):

  geode import [--db …] [--no-verify] [FILE…]   # files, or stdin when none
  geode export [--db …]                          # NDJSON to stdout

Both stream — memory is bounded to one batch (import) / one event (export), so
a multi-million-event corpus round-trips in roughly constant memory. `import`
verifies signatures by default (same `Event.verify()` the relay's VerifyPolicy
uses), upholding the relay's verify-by-default stance rather than trusting the
file; `--no-verify` is the trusted-input escape hatch. Verb dispatch is
backward-compatible: a bare `geode --port …` (no verb) still serves.

This makes the benchmark-only `CorpusServerMain` redundant — a corpus source is
now just `geode import` into a DB, then a normal `geode` serve — so it's
deleted, removing benchmark-only code from the production geode artifact (the
question that started this). The 1M sync-throughput plan is updated to describe
sources via `geode import` + serve.

Also fixes a native-target CI break: MergeQueryCorrectnessTest used the
deprecated `String(CharArray)` (error-level on Kotlin/Native) — switched to
`CharArray.concatToString()`.

Verified end-to-end through the packaged `geode` binary: import (file + stdin,
--no-verify), export round-trip, and verify-on rejecting bad signatures.
ImportExportTest covers the counts, duplicate handling, malformed-line
skipping, and verify accepting a freshly-signed event while rejecting bad sigs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-05 13:56:11 +00:00
Claude cece5b6e04 docs: sync comments/plans with the audit fixes
Follow-up to the audit fixes so nothing describes the pre-fix behavior:

- CorpusServerMain: drop the leftover "reuses an already loaded DB … skips
  the reload" comment above `val dbFile` — the sentinel-gated reuse it
  described is now spelled out in the block just below it.
- sync-throughput-1m plan: the up-catch-up now streams `negentropyReconcile`
  (publishing each onHaveIds batch) instead of materializing the full diff
  via negentropyReconcileIds; note the O(batch) memory win at 1M.
- follow-feed plan: the k-way merge dedups repeated authors/kinds, and its
  id-ASC tie-break is byte-exact vs the single-SQL path only when the store
  indexes id (useAndIndexIdOnOrderBy) — otherwise ties fall in rowid order.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-05 12:20:11 +00:00
Claude 2ce3e2bf5c Merge remote-tracking branch 'origin/main' into claude/benchrelay-1m-events-test-v6vtp0
# Conflicts:
#	gradle/libs.versions.toml
2026-07-05 12:13:37 +00:00
Claude 7aa6144dd1 fix(store): make k-way merge honor id tie-break and dedup repeated authors
MergeQueryExecutor's winner-picker tie-breaks equal created_at by id ASC,
but each per-stream cursor sorted by created_at DESC only, and repeated
authors/kinds opened duplicate cursors:

- id tie-break: thread the IndexingStrategy through run()/prepareStreams
  and append ", id ASC" to the per-stream ORDER BY when
  useAndIndexIdOnOrderBy is set — matching every sibling query in
  QueryBuilder. The id-indexed order comes straight off the index (no
  extra sort, lazy cursor preserved), so the merge now matches the
  single-SQL path byte-for-byte on same-second same-author events. Without
  the id index the tie stays in rowid order (a valid NIP-01 newest-N);
  documented on the class.
- dedup: streamCount/prepareStreams now operate on distinct authors and
  kinds, so a filter with a repeated pubkey can no longer open two
  identical cursors and emit each matching event twice (the single-SQL
  IN(…) path already dedups).

Adds two MergeQueryCorrectnessTest cases the suite was missing: a
within-stream same-second tie sliced by the limit, and duplicate authors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-05 04:55:19 +00:00
Claude 96674e7ce4 feat(geode): mirror strfry's two-phase model — NIP-77 sync catch-up + live REQ tail
geode's MirrorWorker mirrored `strfry router` (live REQ streaming) but had no
`strfry sync` equivalent, so backfilling a large foreign relay from empty could
not complete: a plain REQ dump of the history overruns the sink and strfry kills
the slow client at its maxPendingOutboundBytes cap (see
relayBench/plans/2026-07-04-sync-throughput-1m.md).

MirrorWorker now runs a one-shot NIP-77 "sync" catch-up per down/both upstream
before the live tail, using strfry's own vocabulary — one `[[mirror]]` entry,
one `dir` driving both phases:

- Catch-up reconciles the local set against the upstream over the
  [now - backfill_seconds, now] window and downloads only the diff via the
  existing INostrClient.negentropySyncOrFetch — client-paced (strfry can't
  overrun us) and it completes the pull. Reconcile-against-local means a warm
  restart re-fetches nothing it already holds, like `strfry sync`.
- Either mode, transparently: negentropySyncOrFetch auto-falls back to paged
  REQ for an upstream without NIP-77 — no config toggle.
- Live REQ tail unchanged; it starts at `now` when catch-up is on (history is
  the sync's job). The windows overlap at `now`; the store's unique-id
  constraint dedups the seam.

Changes:
- quartz: add a backward-compatible `localEntries` param to the public
  negentropySync / negentropySyncOrFetch (default empty = prior behavior) so the
  reconcile diffs against a caller-supplied local set.
- geode MirrorWorker: `runCatchUp()` (bounded, backpressured ingest; same
  trusted-scope re-check as the live path; failure is non-fatal). New `store` +
  `negentropyBackfill` ctor params; default off so existing live-REQ tests are
  unchanged. Main opts production in.
- Test: MirrorNegentropyCatchUpTest isolates catch-up from the live tail by
  preloading historical events a live-only sub cannot deliver, then proves the
  post-boot event still arrives (3000 catch-up + 1 live = 3001).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-05 01:33:47 +00:00
Claude bbd8b3a596 perf(nip77): stop the live index turning bulk backfill into O(n^2)
LiveNegentropyIndex kept a sorted ArrayList and paid an O(n) element shift per
incremental insert. That's cheap for near-tail live traffic (created_at ≈ now),
but a mirror/import backfill delivers historical, out-of-order events, so every
insert memmoves ~n/2 entries and the whole sync goes O(n^2) — a geode→geode 1M
mirror crawled to <300 ev/s once the index passed ~130k, versus a sustained
~20k ev/s with the index off.

When an insert lands more than REBUILD_THRESHOLD (4096) from the tail, drop the
index instead of shifting: it rebuilds in one O(n log n) scan on the next
NEG-OPEN (liveNegentropySnapshot already does this when unpopulated), and while
unpopulated newDeltaOrNull skips delta tracking, so backfill costs O(1) per
event. Near-tail live inserts keep the cheap incremental path. NIP-77
convergence and byte-exact tests still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 22:45:43 +00:00
Claude a7d549b579 test(store): guard that concurrent ingest (queue + pipeline + bg writers) is lossless
Drives the full IngestQueue pipeline — parallel verify, greedy-drain group
commit, a concurrent deferred-FTS catch-up worker taking the pool writer, and
windowed concurrent submits via the trusted (skipVerify) mirror path — over the
clean 200k corpus, asserting every Accepted regular event is queryable after.
Passes (199,612 in/accepted/stored, 0 lost), together with BatchInsertLossTest
proving geode's ingest is lossless at every in-process layer. The geode↔geode
sync event-loss therefore lives above the store+queue — in the real Ktor
WebSocket path or the benchmark harness's hand-rolled delta transfer, which the
in-process paths bypass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 21:26:33 +00:00
Claude 12e4e725c7 test(store): guard that sequential batchInsert never loses accepted events
Drives the exact store path — batchInsertEvents with geode's indexing strategy
in 64-event batches — over the clean 200k corpus and asserts every Accepted
kind-1 (regular, never replaced/deleted here) is queryable afterward. Passes
(199,612 in, 199,612 stored), which is the point: it proves the sequential
store path is lossless and narrows the geode↔geode sync event-loss to the
concurrent IngestQueue pipeline (async verify + greedy-drain batching +
concurrent WS submits), not the store itself.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 20:31:05 +00:00
davotoula 489e8ae50b Code review:
- enforce op equality by class and pin the tag-uniqueness invariant
- hoist crypto op equals/hashCode onto OpCrypto
2026-07-04 21:38:08 +02:00
davotoula 772b4ea8ed fix(quartz): repair equals/hashCode contracts in OTS ops and VerifyResult
Op instances key Timestamp.ops (MutableMap<Op, Timestamp>), so contract
violations corrupt hash-map behavior:

- OpKECCAK256 defined equals without hashCode, so equal instances hashed
  by identity — two equal keys could land in different buckets, producing
  duplicate branches or failed lookups in keccak256 timestamp trees. Add
  hashCode = TAG, mirroring OpSHA1/OpSHA256/OpRIPEMD160.
- OpBinary defined hashCode without equals — and its TAG referenced
  Op.TAG (0x00), a no-op XOR. Define the equals/hashCode pair once on
  OpBinary using tag() and drop the duplicated overrides from
  OpAppend/OpPrepend (behavior unchanged: same tag + same arg content).
- VerifyResult.equals cast without a type test (ClassCastException on
  foreign types instead of false) and hashCode force-cast the nullable
  timestamp (NPE for null-timestamp results). Convert to a data class;
  the custom toString and compareTo stay.
2026-07-04 21:37:24 +02:00
Claude 469220de77 perf(nip77): bump kmp-negentropy to v1.2.0 (faster reconcile/fingerprint internals)
v1.2.0 (on Maven Central) speeds up the library's own reconcile and fingerprint
walk on top of the v1.1.1 PrefixSumStorageVector wiring. At the 1M relayBench
slice shape (NegentropyReconcileBenchmark, converges exactly, need/have=200k):
client reconcile 264 → 178 ms, seal 424 → 320 ms, and the library's O(range)
fingerprint walk 447 → 252 ms (~1.8×). Our prefix-sum path still answers each
range fingerprint in 0.7 ms (356× the now-faster walk). All NIP-77 tests pass.

Update the reconcile-profiling plan: the fix shipped via the upstream
IStorage.fingerprint seam (v1.1.1) rather than a quartz-side fast server; record
the v1.0.2 → v1.1.1 → v1.2.0 progression.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 19:33:54 +00:00
Claude 042b6a0c76 perf(nip77): use kmp-negentropy v1.1.1 PrefixSumStorageVector for O(1) range fingerprints
kmp-negentropy v1.1.1 adds the `IStorage.fingerprint(begin, end)` seam we needed
and ships `PrefixSumStorageVector` — a drop-in IStorage that builds an additive
prefix-sum table on seal() and answers any range fingerprint in O(1) instead of
re-walking the range. Range fingerprints are the CPU-bound core of a NEG-MSG on
a large snapshot; profiling pinned them as the steady-state reconcile cost geode
lost multiples on (not serialization).

Seal a `PrefixSumStorageVector` in both `NegentropyServerSession.sealVector`
(server / relay-relay responder, also backs the `LiveNegentropyIndex` snapshot
cache) and `NegentropySession` (initiator). Byte-identical to the plain vector —
only the fingerprint path is accelerated. `NegentropyPrefixFingerprintTest` now
also asserts the library's `PrefixSumStorageVector.fingerprint` matches the plain
walk over 2000 random ranges + boundaries at 50k; the reconcile-shaped mix
measures 601× (447 ms → 0.7 ms).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 17:19:46 +00:00
Claude 9ac5106bc0 fix(store): compile the merge raw-path correctness check; validate follow-feed at 1M
The `rawQueryPathMatchesDecodedQuery` case called `store.rawQuery(filter)`, but
`EventStore` only exposes the streaming `rawQuery(filters, onEach)` — the
list-returning overload lives on the inner `SQLiteEventStore`. Point the check
at `store.store.rawQuery(filter)` so the zero-decode path is actually exercised.

Record the shipped k-way merge result in the plan doc: a fresh 1M relayBench run
has geode `follow-feed` at 18.8 ms vs strfry 17.7 ms (down from 97.7 ms, now at
parity) and 46,258 ev/s vs strfry 15,365 @8conn, both returning the same 500.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 17:19:32 +00:00
Claude 8d09671218 perf(store): k-way merge for the home-feed REQ shape
The home-feed REQ (`authors=[…] (+ kinds=[…]) [+ since/until] limit=N`,
newest-first) is one of the most common relay queries. SQLite serves it by
seeking every `(kind, pubkey)` combo and feeding *all* matching rows through a
LIMIT-bounded sorter, so it reads O(the followed set's whole matching history)
— on a cold on-disk 1M corpus that was the `follow-feed` regression (relayBench:
97.7 ms vs strfry 17.6 ms).

Add `MergeQueryExecutor`, an app-level k-way merge that opens one lazy
newest-first cursor per stream off the existing composite indexes
(`query_by_kind_pubkey_created`, or `query_by_pubkey_created` for authors-only),
merges their heads `(created_at DESC, id ASC)` and stops at the limit — reading
only O(limit + streams) rows regardless of how much history the authors have.
It reuses indexes that already exist, so write throughput and on-disk size are
untouched. Eligibility is narrow (2..2048 streams, simple filter, explicit
limit, no ids/d-tags); everything else falls through to the single-SQL plan.

Wired into both `query` and the zero-decode `rawQuery` paths (the relay REQ hot
path) and the single-element filter-list variants, so `LiveEventStore` REQs go
through it.

`MergeQueryCorrectnessTest` proves the merge returns exactly the single-SQL
top-N — vs an independent Kotlin reference and vs the SQL path — across
distinct/tied created_at, since/until windows, authors-only, fewer-than-limit,
streaming onEach, and the raw path. `FollowFeedReadBenchmark` gains a `merge`
variant: at 1.05M events it's flat ~10-12 ms across both prolific-recent and
sparse-old, where `scan` is catastrophic on sparse follows (1995 ms) and
`current` is disk-bound on prolific ones.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 16:49:13 +00:00
Claude f29196f0fc test(store): measure follow-feed read/write/size tradeoff — keep current plan
follow-feed (kinds=[1,6] × 150 authors, ORDER BY created_at DESC LIMIT 500)
was geode's 5.5× loss (97.7ms vs strfry 17.6ms). Investigated whether any
change is worth it, across read + write + size.

Read (FollowFeedReadBenchmark, in-memory, scale 5 ≈ 1.05M events):
                  prolific-recent   sparse-old
  current            5.7 ms          1.9 ms
  scan (strfry)      1.0 ms       1601.9 ms
  union            316.9 ms         20.0 ms

- scan (created_at index + early LIMIT) wins for active follows but is
  catastrophic for sparse/inactive follows AND grows with corpus size
  (234ms→1601ms from scale 1→5) — following rarely-posting accounts is
  common, so it'd be a severe regression.
- union (300 per-branch subqueries) is dominated by branch overhead.
- current is the only robust option — flat across scale, bounded by the
  followed set, never catastrophic. The 97.7ms is a worst case (the 150
  MOST prolific authors, disk-bound reading all their matching rows).

No safe SQL-level swap exists; each alternative trades geode's worst case
for a worse one on a common workload. The only universal improvement is
strfry's app-level k-way merge (O(LIMIT+streams)) — a real new executor,
not a SQL tweak.

Write & size: neutral for every candidate — all reuse existing indexes
(query_by_kind_pubkey_created / query_by_created_at_id), none adds a
CREATE INDEX, so ingest throughput and storage are untouched regardless of
choice. A new index was considered and rejected (taxes every write, helps
one shape, reverts under ANALYZE).

Decision: keep the current composite plan. Full write-up in
quartz/plans/2026-07-04-follow-feed-read-tradeoff.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 16:04:20 +00:00
Claude 3a68927829 test(store): measure why the profiles fix pins the index vs adding one
Answers 'could a new index beat the INDEXED BY pin without the pin?' for
the profiles shape (kind=0 AND pubkey IN(...) ORDER BY created_at DESC).
Measured:

- stock unhinted: scans query_by_kind_created (1.40ms) — the bug.
- pinned composite: seek + tiny sort (0.23ms) — the shipped fix.
- new (pubkey,kind,created_at) index, unhinted: STILL scans — no help.
- new (kind,pubkey,created_at ASC) index, unhinted: picks the seek (0.23ms)
  BUT that's a no-stats cost-model artifact — ANALYZE reverts it to the
  scan (1.47ms). Fragile, and a full duplicate of the DESC composite.

Root reason: ORDER BY created_at over a multi-value pubkey IN(...) needs a
sort no matter the index (no B-tree gives global created_at order across
pubkeys), and SQLite prefers the one sort-free plan — the full-kind scan.
Only the explicit pin reliably overrides that. A new index would add
write+storage cost on every event for the whole relay, help only this one
shape, and break under ANALYZE — so the free, deterministic, scoped pin is
strictly better. Diagnostic evidence for the design choice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 15:47:37 +00:00
Claude b9d7ea2574 perf(nip77): direct-build NEG-MSG wire frames (~2.5–2.8× serialization)
The server's per-round reconcile spends a large slice turning the ~1MB hex
reconcile frame into wire JSON: the generic (Jackson) serializer wraps the
hex string in a value node and scans every char for JSON escapes a
[0-9a-f] payload can never contain, then re-copies.

NegMsgMessage.toJson() now builds ["NEG-MSG","<sub>","<hex>"] directly —
no node tree, no escape scan of the hex. Fast path fires only for
escape-free printable-ASCII subIds (what the JSON encoder emits verbatim);
exotic subIds fall back to the generic serializer, so output is
byte-identical. RelaySession.send routes through message.toJson() (default
unchanged for every other message type).

Measured (toJson + UTF-8, per frame): 64KiB 2.5×, 250KiB 2.6×, 500KiB
(strfry cap) 2.8× — ~2.5ms saved per NEG-MSG, ~35ms over a 14-round
reconcile. Correctness: a subId battery asserts byte-identity with the
generic path, and GeodeVsStrfryNegentropySyncTest (real strfry) reconciles
against the fast-built frames.

Also records the ingest-latency candidate as measured-not-worth-it: the
IngestQueue pipeline overhead is only ~0.17ms p50, <10% of the ~2.4ms
receipt→queryable gap — that gap lives in the REQ-visibility path, not the
writer. Full write-up in
quartz/plans/2026-07-04-sync-serialization-and-ingest-latency.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 15:40:26 +00:00
Claude b97c74152a test(perf): add ingest-latency and NEG-MSG serialization benchmarks
Measurement harnesses for the two remaining relayBench gaps, isolating each
cost so a fix can be judged on the delta:

- IngestLatencyBenchmark: times single-event submit→onComplete through the
  group-commit IngestQueue vs a direct batchInsert. The delta is the
  pipeline's coroutine-handoff overhead — the receipt→queryable latency the
  1M run measured geode losing (4.68ms vs strfry 2.32ms).
- NegMsgSerializationBenchmark: times the NEG-MSG wire path (Hex.encode →
  MessageKSerializer JsonElement tree → UTF-8) against a direct StringBuilder
  build, asserting byte-identical output. Isolates the ~40% serialization
  slice of the server reconcile the JFR flagged.

Both compile and are self-contained; results + fixes to follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 15:23:35 +00:00
Claude ab32b21ec7 test(store): audit query plans for all relayBench scenarios
After the profiles fix, sweep every relayBench query shape with EXPLAIN
QUERY PLAN against a realistic multi-kind, tagged store to check for other
scan-that-should-seek mis-costs. Result: clean — every scenario seeks an
index (or does an index scan + LIMIT, like firehose's created_at walk).
The remaining TEMP B-TREE sorts are all over bounded result sets
(author/id count or LIMIT), not the profiles-scale pathology. The tag
queries (thread/notifications/hashtag) do an inherent subquery + sort;
hashtag is the slowest scenario but strfry is equally slow there, so it's
absolute cost, not a competitive gap.

Kept as a regression guard: asserts no scenario full-table-scans a base
table (bare SCAN without USING INDEX).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 15:10:42 +00:00
Claude 87192e7793 fix(store): pin (kind,pubkey) index for multi-author no-limit REQs
The profiles scenario — Filter(kinds=[0], authors=[50]), no limit — was
geode's ~100x loss to strfry on the 1M corpus (99.5ms vs 0.94ms).

Root cause: the REQ path always appends ORDER BY created_at DESC. For a
multi-author pubkey IN(...) filter, query_by_kind_created (kind,
created_at) satisfies that order for free by scanning an ENTIRE kind, so
SQLite prefers it over the selective query_by_kind_pubkey_created (which
would need a sort). The scan is O(all kind-0 profiles) — cheap at 2k, the
99.5ms at 1M. ANALYZE does not fix it (verified: even a reopened store
reading fresh sqlite_stat1 keeps the scan, since the ORDER BY genuinely
lets the scan skip a sort). A single author is costed right and already
seeks; only the IN-list of >1 is mis-costed.

Fix: pin INDEXED BY query_by_kind_pubkey_created for exactly that shape —
multi-author + kinds, no ids, no d-tags, no limit — keeping the ORDER BY.
SQLite seeks the authors and sorts the small result: identical rows,
identical newest-first order (zero behavior change), ~8x at 2k profiles,
growing to ~100x at 1M. Limited feeds (home/global) keep the created_at
scan + early LIMIT; single-author and d-tag queries are untouched. The
index is created unconditionally so the hint never dangles.

(Considered dropping the ORDER BY for no-limit author queries — faster and
hint-free, but it changes on-the-wire result ordering, which broke
FsParityTest's ordered-parity assertions, so it's client-visible. Rejected
in favor of the order-preserving pin.)

Adds ProfilesQueryPlanBenchmark (regression guard: asserts the live REQ
plan seeks the composite index, not the kind scan) and
plans/2026-07-04-profiles-query-plan.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 11:38:31 +00:00
davotoula b8d3791ea3 Code review:
- only parse commonName when the alt tag has the Birdstar prefix
- summary() on both Birdstar events now uses the canonical NIP-31
  tags.alt() helper instead of a raw firstTagValue("alt") lookup
- speciesReference() only returns http(s) URLs since UIs render it as
  a clickable link (rejects e.g. javascript: schemes), with a test
- Detection card: parse tags once into a single remember slot, drop
  the near-dead '?: summary' title fallback, stop rebuilding the
  italic TextStyle every recomposition
- Hoist the duplicated bird-emoji literal into a shared BIRD_PREFIX
- Trim the redundant factory test to the assertIs idiom
2026-07-04 12:42:18 +02:00
davotoula a83b8e4064 feat: support Birdstar bird detection events (kind 2473)
- fetch Birdex life lists in home and profile relay REQs
- richer Birdstar cards — common name title, Wikidata link, bird emoji
- surface Birdstar bird detections in home and profile feeds
2026-07-04 10:45:20 +02:00
Claude 9d76a6a8fa perf(store): diagnose the slow profiles query (kind-0 + authors REQ)
profiles (Filter(kinds=[0], authors=[50]), no limit) was the one query
geode lost to strfry on the 1M corpus — 99.5ms vs 0.94ms (~100x).

Root cause: the REQ path appends ORDER BY created_at DESC even with no
limit. query_by_kind_created (kind, created_at) satisfies that order for
free while scanning EVERY kind-0 profile, so SQLite prefers it over the
ideal query_by_kind_pubkey_created (which would need a sort). The scan is
O(all profiles) — cheap at 2k, the 99.5ms at 1M.

ANALYZE does not help: verified that even a reopened store reading fresh
sqlite_stat1 keeps the scan, because the ORDER BY genuinely lets the scan
avoid a sort.

Two fixes measured (both return identical rows), scoped to no-limit
kinds+authors filters:
- Fix A: drop ORDER BY when limit==null -> planner picks the composite
  index itself (~7x at 2k profiles, ~100x at 1M). Changes result order
  across authors (a NIP-01 SHOULD; clients re-sort).
- Fix B: force INDEXED BY query_by_kind_pubkey_created + keep ORDER BY
  (~same speed, newest-first preserved, at the cost of a scoped hint).

Adds ProfilesQueryPlanBenchmark (prints plans+timings, asserts row-count
equivalence) and plans/2026-07-04-profiles-query-plan.md. No production
change yet — the fix is a core QueryBuilder behavior/ordering decision.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 07:51:01 +00:00
Claude f64b2e6f1c fix(quartz): record REQ state before send in PoolRequests.syncState
On connect, syncFilters re-sends every desired REQ through
PoolRequests.syncState. It previously sent the frame and only recorded
the subscription as SENT afterward, in the post-send onSent callback. A
relay that answers faster than that callback runs — the in-process
transport used by the desktop launch-optimization tests, or any relay on
a fast path — can deliver the EOSE while the per-sub state still reads
"nothing in flight" (onConnecting cleared it, onSent hasn't recorded it).
The EOSE handler then sees empty filters, concludes it never sent a REQ,
and fires a duplicate, replaying the whole page a second time.

Pre-mark the sub as SENT under its lock before the frame leaves, mirroring
the decideCommandLocked pre-mark already used by sendToRelayIfChanged, so
a response can never race ahead of the record. The send stays
unconditional: this is a fresh-connection sync (onConnecting always
cleared the per-relay state first), so there is no in-flight REQ on the
new socket to dedupe against.

Fixes the flaky SubscribeBeforeConnectTest, which asserted a pre-connect
subscription delivers exactly its events once and intermittently saw them
doubled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W4kmtZoNUSXxwG23wD2JwP
2026-07-04 05:42:46 +00:00
Claude 8a607b08b9 perf(negentropy): profile NIP-77 reconcile, verify prefix-sum fingerprint fix
The 1M relayBench run had geode losing the negentropy phase to strfry
(initial reconcile 6066ms/27r vs 1270ms/14r; identical-set 1947ms vs
557ms). Three layered benchmarks pin where the time actually goes:

- NegentropyReconcileBenchmark (quartz): the kmp-negentropy server loop
  in isolation is ~200ms for the full 14-round exchange — the reconcile
  ALGORITHM is not the bottleneck. (An early version showed 22s/139r;
  that was a benchmark bug — index slices over randomly-sorted ids
  scatter the diff. Real relayBench slices are contiguous time ranges;
  monotonic created_at fixes it and matches strfry's round count.)
- NegentropyServerReconcileBenchmark (geode): the real in-process geode
  server over loopback is 3214ms — 15x the library loop. JFR of the
  server call-trees: ~40% hex/UTF-8/JSON serialization of the payloads,
  ~26% actual reconcile, rest allocation. The gap is the JVM
  constant-factor tax on hex-in-JSON, which strfry pays in C++, not a
  single hotspot.
- NegentropyPrefixFingerprintTest (quartz): the one algorithmic lever.
  Negentropy's fingerprint is an additive sum mod 2^256, so a prefix-sum
  table answers any range in O(1). Proven bit-for-bit identical to the
  library over 2000 random ranges, and 460x faster per call — the fix
  for the ~26% reconcile slice (dominant in the identical-set case).

Not yet wired: the library instantiates FingerprintCalculator
internally, so shipping prefix-sum needs a kmp-negentropy change (or a
quartz-side fast server). Full write-up + artifacts in
quartz/plans/2026-07-04-negentropy-reconcile-profiling.md.

Benchmarks are CI-safe (small defaults / opt-in gates); JFR via
-PnegProfile, scale via -DnegBenchN, geode server bench via
-DnegServerBench=1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EZeWww5TJnzBZKPoc6mvU
2026-07-04 05:22:20 +00:00
Vitor PamplonaandGitHub ba9db9e2cb Merge pull request #3469 from vitorpamplona/claude/relay-performance-geode-quartz-v6lbys
feat(quartz): NostrServer.ingest — local write path with per-submission verify skip
2026-07-03 22:43:49 -04:00