Compare commits

...
Author SHA1 Message Date
Vitor PamplonaandClaude Opus 5 f9bef87160 style(desktop): import Compose symbols in NotificationSettingsScreen
Follow-up to 8f8713d8 (nostr proposal 259a0bb1). CLAUDE.md forbids
fully-qualified class names inline in function bodies; the merged
proposal introduced one (androidx.compose.runtime.LaunchedEffect) and
the file already carried four more that predate it. Import them all and
reference them by simple name: LaunchedEffect, snapshotFlow,
rememberCoroutineScope, LocalWindowInfo.

Also rewrites two comments the proposal added:
- the auto-enable comment was written in the first person and described
  the author's own earlier mistake; restate it as what the code does and
  which two paths it covers.
- the "Turn on desktop notifications" comment claimed the button renders
  only when the user explicitly disabled notifications, but the guard is
  `!enabled` alone. Describe the actual condition and why it is enough.

Drops a redundant `enabled = true` on that OutlinedButton (the default).

No behaviour change. :desktopApp:compileKotlin and :commons:jvmTest green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 01:49:30 -04:00
Vitor PamplonaandGitHub a58289b62c Merge pull request #3859 from vitorpamplona/claude/yggdrasil-ipv6-compat-03mctx
Support IPv6 overlay relays (Yggdrasil) with RFC 5952 canonicalization
2026-08-05 00:36:13 -04:00
Claude 70d51cc98b fix(relay): parse the authority instead of substring-matching the url
Audit of the IPv6 work found a family of bugs in isLocalHost/isOnion, most
predating this branch, all with one root cause: the predicates ran `contains`
over the whole url rather than parsing the authority. These decide whether a
relay is exempt from Tor, and relay urls arrive from other people (NIP-65
lists, relay hints, r tags), so they are attacker-controlled input.

- A path could impersonate the host. `wss://evil.example.com/127.0.0.1`
  answered isLocalHost() == true, so any relay list could hand the app a url
  that silently dropped its own Tor routing. The IPv6 lookup added earlier on
  this branch had the same flaw via `/[fd00::1]`, and IPv6 canonicalization
  could rewrite a path outright, corrupting the url.

- `.onion:8080` never matched the `.onion/` test, so an onion relay on an
  explicit port was not treated as onion at all: never forced onto Tor, and its
  hostname went to the clearnet DNS resolver. The fully-qualified `.onion.`
  spelling missed the same way.

- Host tests were case-sensitive, but fix() asks them before the RFC 3986 pass
  folds case, so LOCALHOST:8080 and ABC.ONION:8080 were handed a wss:// scheme
  neither host can serve.

- Private IPv4 was substring-matched, which missed 10.0.0.5, 172.16.3.4 and
  127.1.2.3 — a LAN relay got wss:// and was dialed through Tor — while
  matching 192.168.evil.com and 127.0.0.1.evil.com, registrable domains that
  could therefore exempt themselves from Tor. Same for notlocalhost.example.com
  against `contains("localhost")`.

- A `://` inside a path was read as a scheme separator, so
  `relay.com/x://127.0.0.1` read its path as the authority.

Fixes: a shared hostStart/hostEnd/hostEndWithoutPort trio bounds every test to
the authority, strips :port and trailing dots and validates the scheme; private
ranges are parsed via a new Ipv4 util rather than substring-matched;
comparisons are case-insensitive per RFC 4343; NormalizedRelayUrl.isOnion()
delegates instead of keeping a second, weaker copy of the test.

No performance regression: the old form ran six full-string scans, the new one
bounds its work to the authority and rejects a DNS host from an IP parse on one
character. Ipv6.isLiteral gained a two-colon gate so the schemeless host:port
case answers without allocating the parser's buffer.

Ipv6 is now pinned by a differential test: 4000 random addresses round-trip
against java.net.InetAddress in both directions, and the canonical form is
asserted equal to OkHttp's host for the same address, so the relay identity the
app stores provably matches the host it dials.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQr8CDsznzCRUeB5tS8VYk
2026-08-05 04:22:59 +00:00
Claude 067d68b89c feat(relay): canonicalize IPv6 relay urls and support overlay meshes
Closes the four gaps the previous commit characterized for relays on an
Yggdrasil overlay, where every relay is an IPv6 literal in 0200::/7 served
over plain ws:// (no DNS, no CA-issuable certificate).

New quartz/utils/Ipv6.kt: pure-Kotlin literal parsing, RFC 5952 canonical
formatting and range classification. No java.net, so it works on every KMP
target.

- Canonicalize the bracketed host in RelayUrlNormalizer.norm(). RFC 4291 lets
  one address be spelled many ways and the RFC 3986 pass only folded hex case,
  so two spellings survived as two NormalizedRelayUrl values for one host —
  and that value keys the connection pool, the relay-list sets, the NIP-11
  cache and the per-relay stats, so the app dialed one relay twice. The
  canonical form matches what OkHttp renders when it dials; the tests assert
  that agreement differentially. Relay lists rehydrate through normalizeOrNull,
  so stored entries fold on load and no migration is needed.

- Add isOverlayNetwork() for 0200::/7 and default those relays to ws://:
  nothing can issue a certificate for the range, so wss:// could only fail its
  handshake, and the overlay already encrypts end to end.

- Teach isLocalHost() the IPv6 twins of the literals it already knew — ::1,
  fc00::/7 and fe80::/10 — so a relay on one skips TLS and Tor and stays out of
  published relay lists, as its IPv4 equivalent already did.

- Never route an overlay relay through Tor: the range is unroutable there, so
  proxying guaranteed failure rather than privacy. TorRelayEvaluation covers
  both the Android and desktop relay paths; RoleBasedHttpClientBuilder covers
  non-relay HTTP.

- Bracket a bare IPv6 literal automatically (what yggdrasilctl getSelf prints),
  but only when the whole string parses as an address, so host:port and
  addressable pointers still fall through. RelayUrlEditField now shows an error
  instead of no-opping, fixing the silent Add button for all invalid input.

Mesh relays are still published in NIP-65 and offered by the outbox model; the
plan doc explains why that is left as a maintainer's call, and records that no
live socket test was possible here (the container has no IPv6 stack).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQr8CDsznzCRUeB5tS8VYk
2026-08-04 22:28:07 +00:00
Claude 129401bdaf test(relay): characterize Yggdrasil/IPv6 relay handling
Assesses how the app fares when relays live on an Yggdrasil overlay, where
every relay is a bracketed IPv6 literal in 0200::/7 served over plain ws://
(no DNS, no CA-issuable certificate).

The happy path works: a hand-typed ws://[...]:port normalizes, survives the
RFC 3986 pass and is dialed by OkHttp; nothing in the stack is IPv4-only and
cleartext is already permitted globally.

Four gaps are pinned by the new characterization tests:

1. RelayUrlNormalizer folds hex case but not zero-compression, so two legal
   spellings of one address yield two NormalizedRelayUrl values while OkHttp
   collapses them to one host — duplicate sockets, REQs and stat entries.
2. isLocalHost() does not know 0200::/7, so a schemeless literal defaults to
   wss:// and can only fail its TLS handshake.
3. An unbracketed literal (what yggdrasilctl getSelf prints) is rejected, and
   RelayUrlEditField.submitRelay has no else branch — the Add button silently
   does nothing.
4. TorRelayEvaluation classifies mesh relays as "new", so with Tor on they are
   dialed through the SOCKS proxy, which cannot route 0200::/7.

No behavior is changed. quartz/plans/2026-08-04-yggdrasil-ipv6-relays.md records
the full assessment, the NIP-65/outbox propagation consequences of publishing a
key-derived mesh address, and what could not be verified here (the analysis
container has no IPv6 stack, so nothing below the socket was exercised).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DQr8CDsznzCRUeB5tS8VYk
2026-08-04 21:36:22 +00:00
Vitor PamplonaandClaude Opus 5 8f8713d825 Merge PR: fix(desktop): auto-enable master notif switch when OS permission already granted
Merges nostr proposal 259a0bb1 into main:
- fix(desktop): auto-enable master notif switch when OS permission already granted

Follow-up to e9475dd0, which only flipped the master notification switch on
the NotRequested -> Granted path. Adds NotificationSettings.wasExplicitlyDisabled()
(backed by a new java.util.prefs "explicitly_disabled" key) so the Settings
screen can tell "off because it defaults off on first launch" from "off
because the user turned it off", and a LaunchedEffect that auto-enables the
switch in the former case when the OS permission is Granted or NotApplicable.
Adds a "Turn on desktop notifications" recovery button for the deliberate
opt-out path.

Note: on Windows/Linux permissionState is NotApplicable from startup, so the
master switch now auto-enables the first time the user opens Notification
Settings, overriding the off-by-default first-launch state.

Verified before merge: :commons:jvmTest (4 new hermetic tests) and
:desktopApp:compileKotlin both green on top of current main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:34:02 -04:00
Vitor PamplonaandClaude Opus 5 21b02d780f Merge PR: ci: publish linux-arm64 desktop, amy, and geode release assets
Merges nostr proposal 1d98285c into main:
- ci: publish linux-arm64 desktop, amy, and geode release assets

Adds ubuntu-24.04-arm legs to the build-desktop, build-cli and build-geode
release matrices so aarch64 Linux users get .deb/.rpm/.AppImage/.flatpak/
.tar.gz for the desktop app plus amy and geode bundles. Parametrizes the
appimagetool fetch, the portable archive names and the Flatpak bundle name
by arch, computes the AppImage multiarch lib path from uname -m at launch,
and extends the desktop release-deb smoke test to arm64.

Verified before merge: the appimagetool 1.9.0 aarch64 SHA256 pin matches
the upstream release, and secp256k1-kmp-jni-jvm-linux ships a
linux-aarch64 libsecp256k1-jni.so so signing works on arm64.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:32:30 -04:00
Vitor PamplonaandGitHub d3bd7a45b9 Merge pull request #3852 from vitorpamplona/l10n_crowdin_translations
New Crowdin Translations
2026-08-04 11:56:22 -04:00
vitorpamplonaandgithub-actions[bot] a5d2d153c8 chore: sync Crowdin translations and seed translator npub placeholders 2026-08-04 15:40:56 +00:00
Vitor PamplonaandGitHub 3a702add57 Merge pull request #3857 from vitorpamplona/claude/relay-url-normalizer-mggnkh
NIP-66 relay monitoring: streaming probes, read/write checks, URL fixes
2026-08-04 11:38:07 -04:00
Claude 5ec35c7772 feat(quartz): default the read test to kind 0, limit 1
A kind-0, limit-1 REQ works everywhere: purpose relays (purplepag.es)
reject kind-less filters outright, and practically every relay stores
some profile. Verified against production — purplepag.es's read side now
measures instead of going unobserved. Pass a different kinds list to
probe a specific shelf, or null for a kind-less query on relays known to
allow one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 15:35:22 +00:00
Claude 9a4f6c6cdd feat(quartz): optional kinds on the read-test filter (production finding)
Verified the new probe surface end-to-end against production relays
(probeFlow streaming, readWriteCheck, signed 30166 templates). One
compatibility finding: purpose relays like purplepag.es reject any REQ
that names no kind ('blocked: filters must specify at least one kind'),
leaving their read side unobserved. readTestFilter/readWriteCheck now
take an optional kinds list for those; the default stays kind-less
because naming kinds also narrows the query on every other relay.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 15:26:54 +00:00
Claude c767298368 fix(quartz): audit fixes — foreign-OK confirmation bug, normalizer hot-path allocation
Audit findings across the branch, each verified with a failing test or
measurement before the fix:

- publishAndCollectResults counted an OK from a relay OUTSIDE relayList
  (same event id — a probe-wave straggler, or any republish of the same
  event to a different relay set) toward its confirmation window, ending
  the wait loop early and misreporting still-pending listed relays as
  NO_RESPONSE. The OK branch now carries the same relayList guard the
  onCannotConnect/onDisconnected branches always had. Regression test
  proves the failure without the guard. readWriteCheck additionally
  varies the probe event content per wave so wave N's confirmation window
  can never match wave N-1's event id at all.

- RelayUrlNormalizer.fix() called trimEnd('%','2','0') unconditionally,
  allocating a full string copy for ANY url merely ending in '%', '2' or
  '0' — which includes every relay port ending in zero (wss://host:3030).
  Now gated on endsWith("%20"), keeping the hot path allocation-free;
  semantics unchanged (test pins both the trim and the untouched-port
  cases).

- amy relay probe --file: unreadable file is now a clean bad_args error
  instead of a stack trace, and skipped onion urls are counted and
  reported (file_onion_skipped) instead of vanishing from the tally.

- probeFlow KDoc now states that a slow collector eats into the current
  wave's absolute deadline (answers are still recorded; silent relays get
  less listening time), not just that it delays the next wave.

Verified non-issues: androidx.collection LruCache is internally locked
(safe for CachedNip11Fetcher/normalizer concurrency); probeWave's
per-terminal emission cannot lose or double-emit verdicts (remaining-set
guard, data maps read at emission time); existing publish callers all
benefit from the OK guard rather than depending on the old behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 15:14:17 +00:00
Vitor PamplonaandGitHub ad3ce4d1a1 Merge pull request #3856 from vitorpamplona/fix/filters-changed-compares-only-first-filter
fix(relay): compare every filter in FiltersChanged, not just the first
2026-08-04 11:11:53 -04:00
Vitor PamplonaandClaude Opus 5 0d0117061a fix(relay): compare every filter in FiltersChanged, not just the first
`needsToResendRequest(List, List)` used a non-local `return` inside
`forEachIndexed`, so the loop always returned on iteration 0 and only
`filters[0]` was ever compared. A subscription whose first filter happened
to be unchanged reported "no resend needed" however much the rest had
changed, leaving the relay serving a stale filter set and the app silently
missing events. Only the size check offered any protection, so the bug was
invisible whenever the filter count stayed constant.

Replaces the loop with an indexed scan over all filters, which also drops
the lambda allocation and matches the hot-path style in this package.

Adds FiltersChangedTest. 3 of its 9 cases fail on the unfixed code — all of
them changes beyond index 0 — while the other 6 pass both before and after,
pinning the blast radius to exactly the buggy behaviour. Coverage includes
the deliberate `since`-moves-forward exemption, which must not trigger a
resend on any index.

Note for reviewers: PoolRequests.kt:490 and :528 use this inverted as a
"same as last" refusal check, so those become stricter — filter sets that
differ only beyond index 0 were previously treated as identical and will
now correctly be treated as changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 11:05:45 -04:00
Claude fd5bd994a1 feat(quartz): read+write relay checks — observed facts only, no NIP-11 claims
RelayProber.readWriteCheck(relays, signer) is the deeper check pair for
relays already proven live (warm sockets from a probe that just ran):

- READ: a real limit-1 REQ the relay must query its store for, timed
  REQ→first answer (honest rtt-read on an open socket).
- WRITE: one ephemeral RelayProbeWriteTest event signed by the monitor
  key, timed publish→OK (honest rtt-write). An OK false is a measured
  policy answer, kept with its NIP-01 machine-readable reason; only
  silence leaves the write side unobserved (writeAccepted = null).

publishAndCollectResults now stamps each OK with its elapsedMs (a
rejection is still a round trip; -1 when the relay never answered), so
any caller gets write latency for free.

toDiscoveryEventTemplate(readWrite = ...) folds the pair into the 30166
template: rtt-read/rtt-write when measured, R auth / R pow when the
write was refused with auth-required:/pow:. NIP-11-derived tags (N
supported NIPs, k kinds, T type) are deliberately NOT emitted — those
are relay self-claims, and publishing them under a monitor signature
without per-NIP compliance tests would launder claims into
measurements. Per-NIP/per-kind compliance suites can come later as
opt-in checks; open/read/write is the default surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 14:41:15 +00:00
Claude a7eec1d605 feat(quartz): NIP-66 check options — read-test filters, write-test event, cached NIP-11 fetcher
RelayProber.probe()/probeFlow() take a filters option choosing the check:
LIVENESS_FILTERS (default, impossible-id REQ — EOSE proves liveness with no
payload) or readTestFilter(limit = 1) — a REQ the relay must actually work
for, querying and streaming real events, making Verdict.rttEoseMs a genuine
read test.

RelayProbeWriteTest.build() creates the write-check event: ephemeral kind
20166 (never stored by compliant relays) carrying a NIP-40 expiration tag
60s out as belt-and-braces for relays that store unknown ephemeral kinds.
Publish it under the monitor key, time the OK for rtt-write, map rejection
prefixes to R requirement tags — an OK false still proves the write path.

Nip11Fetcher is the missing fetch seam for relay information documents,
mirroring Nip05Fetcher: the interface lives in commonMain,
OkHttpNip11Fetcher (jvmAndroid) does the Accept: application/nostr+json
GET, and CachedNip11Fetcher wraps any implementation with a TTL cache —
successes trusted for a day, failures remembered for five minutes so a
census doesn't hammer hosts that just refused.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 14:18:05 +00:00
Claude d9a58950ec feat(quartz): stream NIP-66 probe verdicts and expose them as signable 30166 templates
RelayProber.probeFlow(urls) is a cold Flow that emits each relay's Verdict
the moment the relay resolves (EOSE, CLOSED or connect failure) instead of
at the end of the whole census; only silent relays wait for their wave's
deadline. probeWave now resolves verdicts per-terminal, so the batch
probe() shares the same path.

Verdict.toDiscoveryEventTemplate() renders a verdict as an UNSIGNED
kind:30166 template (d = normalized url, n network type, rtt-open when
reachable, R auth when the probe hit a NIP-42 auth-required CLOSED) so an
external consumer signs with its own monitor key:

    prober.probeFlow(urls).map { it.toDiscoveryEventTemplate() }
        .collect { publish(signer.sign(it)) }

rtt-eose is deliberately never published as rtt-read: it is measured from
the wave start (dial + TLS + queueing + read), and aggregators rank on
rtt values. The RelayObserver/RelayMonitor path supplies honest
rtt-read/rtt-write from real traffic.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 13:57:00 +00:00
Claude ba7cc72dd2 feat(cli): amy relay probe --file to census external relay-url candidates
Feeds a file of raw candidate urls (one per line) through the same
RelayUrlNormalizer the app uses, then probes the surviving clearnet set
alongside the store's known universe. Rejected and onion counts are
reported (file_urls/file_normalized/file_rejected in the JSON output),
and results land in the NIP-66 kind:30166 reachability cache keyed by
the normalized url as d-tag, as usual.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 13:23:16 +00:00
Claude 156176f5fe fix(quartz): stop RelayUrlNormalizer from accepting urls that can never be relays
Validated against a 45k-entry corpus of relay-url hints exported from real
events (317k tag occurrences). The normalizer was converting ~30k distinct
https:// urls with paths (Mastodon/bridge actor urls from proxy tags, web
pages, images) into wss:// addresses that can never answer, wasting
connection attempts and relay-pool slots.

- http(s) → ws(s) scheme swap now only applies to bare hosts
  (host[:port] plus optional trailing slash); an http url with a path,
  query or fragment is a web resource, not a mistyped relay.
- Authority validation for all schemes: rejects empty hosts, userinfo
  (@), percent-encoding and commas in the host, and paths that start
  with // (the signature of a second pasted url, e.g. wss://https//host).
- Interior whitespace and backslashes reject the whole string (multiple
  urls or prose in one field).
- Zero-width characters (U+200B..D, U+2060, BOM) are stripped instead of
  corrupting the parse (wss://\u200Bnos.lol previously normalized to the
  scheme-less //nos.lol/).
- Schemeless candidates must look like host[:port] (single colon, numeric
  port), rejecting addressable pointers (31990:pubkey:dtag) and bare
  scheme leftovers (wss:) before the expensive RFC 3986 parse.
- Protocol-relative //host/ inputs normalize as wss:// instead of
  resolving to https://.
- normalizeOrNull now double-checks the parser output still starts with
  ws(s):// and rejects otherwise.

Corpus impact: 30,014 garbage urls (30,333 events) now rejected, 0 real
relays lost (all 15,162 kept urls normalize byte-identically), 6 broken
outputs fixed. fix() itself stays allocation-free on the happy path
(~357ns vs ~318ns per call on the garbage-heavy corpus).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9sSyh1QLJD3PZ18tVPPVK
2026-08-04 13:23:06 +00:00
Vitor PamplonaandGitHub 1a25557bdb Merge pull request #3855 from vitorpamplona/sync-accessories-from-vespa-relay
quartz/geode: sync accessories from vespa-relay, InsertOutcome.Failed contract, mirror resume coverage
2026-08-04 01:58:29 -04:00
Claude b48b87a60b Merge remote-tracking branch 'origin/main' into sync-accessories-from-vespa-relay
# Conflicts:
#	quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/backend/IngestQueue.kt
#	quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/IEventStore.kt
#	quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt
2026-08-04 05:44:17 +00:00
Claude cdbd550405 Fix audit findings across the sync accessories and their consumers
quartz:
- SQLiteEventStore: classify per-row savepoint errors — policy refusals
  (blocked:/constraint/not allowed) stay Rejected, everything else is now
  Failed, so disk-full no longer masquerades as 2M duplicate rejections
- IEventStore.batchInsert default: rethrow CancellationException and map
  unknown throws to Failed (re-offering a duplicate is idempotent;
  dropping a good event on a transient store error is not)
- IngestQueue: rethrow CancellationException instead of stamping a
  cancelled batch Failed and continuing
- HostStrikes: make the eviction verdict exactly-once under concurrency
  (deadHosts.add is the atomic gate) and re-check produced before
  publishing
- SyncCoverage: bound the identity fingerprint cache (a caller minting
  fresh Filter instances per cycle could grow it forever); legs() gains a
  floor parameter so a complete band re-opens its older span when the
  caller's window deepens; coveringWindow no longer treats a fully
  covered relay as needing the whole filter
- PagingWindowProgress: accept single-second windows (a band's re-read
  edge leg is exactly that shape)

geode:
- MirrorWorker: cap reconciledThrough at the leg's own ceiling — the
  older leg of a resumed catch-up no longer stamps the band complete
  through 'now' before the newer leg has run (silent event loss for up
  to fullResyncSeconds if that leg failed)
- MirrorWorker: run negentropy and the paged fallback by hand instead of
  negentropySyncOrFetch: drops the O(delivered-ids) dedup set from the
  mirror path, and a fallback resets the observed span so a band never
  claims interior ranges only a half-finished reconcile scattered over
- MirrorWorker: clamp a paged band's ceiling to the snapshot instant so
  one future-dated event cannot suppress the next boot's newer leg
- MirrorWorker.close(): join the workers (bounded) so the final coverage
  flush carries the last records
- Main: gate the coverage file on the store actually being persistent —
  database.file with in_memory=true (the default) persisted bands over a
  volatile store, and the next boot skipped the backfill over an empty
  database; honor --db overrides
- SyncCoverageFile: request ATOMIC_MOVE explicitly; fix the restore/dirty
  comment
- Import summary now prints the failed count; document
  mirror_sync_state_file in config.example.toml
2026-08-04 05:22:40 +00:00
Vitor PamplonaandGitHub c029b271e7 Merge pull request #3854 from vitorpamplona/claude/nip50-grammar-extractor-rejections
nip50Search: search grammar + weighted field extraction; nip01Core: shared store-semantics rules
2026-08-04 01:06:36 -04:00
Claude 4081ef1681 nip01Core: one owner rule, one supersession rule, one tag-name rule
Three semantics rules each existed as multiple independent copies:

- Event.owner() (gift-wrap recipient controls the wrap, else the
  author — NIP-09/62 authority) was derived inline in
  EventIndexesModule and again in EventStoreProjection.ownerOf.
- The NIP-01 replaceable tiebreak (newest created_at, ties to the
  lexically smallest id) lived in EventStoreProjection.supersedes,
  in SQL, and downstream.
- "Indexable tag name" was spelled `length == 1` in four places,
  which admits "5" and "#" — names the NIP-01 #x filter space
  (single a-zA-Z letters) cannot address, letting stores disagree
  about which tags filters reach. isIndexableTagName encodes the
  NIP-01 rule; converging FilterIndex and the SQLite
  IndexingStrategy on it deliberately tightens single-char
  non-letter tag names out of the index.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfwV3xgMSmfxy16ujGPxGW
2026-08-04 05:00:40 +00:00
Claude 804b850095 Audit fixes: empty exclusions, missed fallback, one vocabulary
An all-dash search token ("--") stripped to an empty exclusion that
toSearchString/stripExtensions round-tripped into a REQUIRED "-" term
reaching SQLite FTS; it is now dropped at parse. IngestQueue's
batchInsert fallback was the one site still hand-writing "insert
failed" (unprefixed) while every other path emits
RejectionReason.INSERT_FAILED. RejectionReason no longer duplicates
the NIP-01 prefixes MachineReadablePrefix already owns, and the
expiration trigger now rejects with the same words as the Kotlin
pre-check instead of its own spelling. Tests pin the "--" drop,
consecutive quoted spans, text after a closing quote, the extractor's
fallback tier, blank-content normalization, and the
unparseable-buzz-content hashtag seam; extractor KDoc now states
where the trimmed/non-empty and never-empty-Profile guarantees
actually live.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfwV3xgMSmfxy16ujGPxGW
2026-08-04 05:00:40 +00:00
Claude 42fc73f6cb nip50Search: per-kind weighted search-field extraction
SearchableEvent.indexableContent() flattens a kind's searchable text
into one blob, so a weighted full-text backend (title above summary
above body) had to re-derive the decomposition itself — and drift
every time a kind's parsing changed here. SearchFieldExtractor now
lives beside the kinds it decomposes: title-like accessors primary,
summary/description secondary, body tertiary, kind-0-shaped metadata
in profile roles, with an indexableContent() fallback so every
searchable kind, current or future, is covered.

IndexableFields is a sealed shape — Profile or Tiered — so a kind
cannot mix identity fields with content tiers, and each shape
declares its own website role (a profile's homepage; a content kind's
affiliation URLs). Multi-valued roles are carried UNJOINED, as lists:
hashtag and location tags ride raw beside the tiers (filled by the
one tiers() funnel every content branch uses, so no branch can forget
them and profile shapes never see them), and separator or weighting
choices — hashtags at summary weight, "\n" vs " ", arrays vs joined
columns — belong to the backend, not the library. Empty extractions
always normalize to None.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfwV3xgMSmfxy16ujGPxGW
2026-08-04 04:38:36 +00:00
Claude 19d8e3069d Align the sync accessories with quartz vocabulary; geode catch-up resumes
Renames from review: SyncBands -> SyncCoverage ("sync" reads negentropy-ish
in quartz, and coverage is the role — the bands are the records), and
PagingProgress moves into relay.client.paging as PagingWindowProgress,
beside RelayLoadingCursors and RelayPagingProgress, with its docs swept from
"walk" dialect to quartz's pagination vocabulary and cross-references
delineating the three: cursors are in-memory positions for demand-driven UI
paging, the window progress is fraction/ETA for a bulk pagination over a
known window, coverage is persistent intervals that license skipping work.

geode adopts both halves of the new contract. MirrorWorker counts
InsertOutcome.Failed in its own `failed` counter instead of folding it into
`rejected`, and the down catch-up gains resume memory: SyncCoverageFile
persists SyncCoverage next to the event database (admin state-file
convention, temp-file + atomic move, daemon flush), and runCatchUpDown asks
only for the legs outside the covered band. Bands are keyed on the stable
scoped filter — never the boot window, whose since/until change every start
— and clamped to the window, which only slides forward, so an old band can
never license skipping a range an earlier boot could not ask about. A clean
reconcile records completeness through its snapshot instant; a paged
fallback earns only the span it saw. For an upstream without NIP-77 this
turns the every-boot full re-download of the backfill window into a
resumed walk.

Off unless wired: MirrorWorker's coverage parameter defaults to null and
in-memory stores keep no state file, so existing tests and setups are
unchanged. Full :quartz:jvmTest and :geode:test pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Pi9YYMhdzTFxRiV2jF9R
2026-08-04 04:24:19 +00:00
Vitor PamplonaandGitHub 9231195890 Merge pull request #3853 from vitorpamplona/claude/quartz-codebase-skills-mrzz5e
docs: Add three store-implementer skills (event-store-semantics, nip85, searchable-events)
2026-08-04 00:12:36 -04:00
Claude 564cafedc4 Replace insertBisecting with a Failed outcome on the batchInsert contract
Bisecting existed to reconstruct per-event attribution after a store threw a
batch-wide exception. Better to never lose the attribution: batchInsert's
contract now requires per-row isolation, with a third outcome telling whose
fault a miss was. Rejected is the EVENT's fault (duplicate, expired, invalid,
blocked) and is final; Failed is the STORE's fault (schema drift, a failed
feed, a resource error) — the event was good, it is lost unless re-offered,
and a rising Failed count means the store is broken rather than that
upstreams send junk. Throwing is reserved for failures with no per-event
answer (engine unreachable, transaction never started), readable as "nothing
in this batch was written".

Consumers updated: RelaySession maps Failed to OK false with NIP-01's
"error:" prefix; IngestQueue converts a thrown batch and a missing outcome to
Failed instead of Rejected; NdjsonImportExport counts failed apart from
rejected; geode's MirrorWorker logs store failures at warn instead of
folding them into debug-level rejections. BisectingInsert and its test are
removed — with attribution guaranteed by the contract, retry-by-splitting
has nothing left to do.

Full :quartz:jvmTest passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Pi9YYMhdzTFxRiV2jF9R
2026-08-04 03:53:39 +00:00
Claude e54a546d10 SearchQuery: parse quoted phrases and -word exclusions
NIP-50 search strings in the wild carry Google-style syntax the
extension tokenizer could not see: "exact phrase" requirements,
-"phrase" and -word exclusions. SearchQuery now lifts quoted spans
BEFORE the extension pass — the order is load-bearing: the extension
pass is quote-blind, so a span ending in an extension-shaped token
would lose its closing quote, and lifting first also lets quotes
protect extension-shaped tokens ("include:spam" is a phrase, not an
extension). toSearchString() and stripExtensions() reassemble the
full grammar; existing parses are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfwV3xgMSmfxy16ujGPxGW
2026-08-04 03:48:04 +00:00
Claude b46063713e Centralize the insert-rejection vocabulary in RejectionReason
Every store spelled its rejection reasons inline — the expired-event
string was duplicated four times across ObservableEventStore and
SQLiteEventStore. RejectionReason now carries the NIP-01 OK
machine-readable prefixes plus the standard store reasons, so
InsertOutcome.Rejected tallies and OK-frame building see one
vocabulary no matter which store produced the outcome.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MfwV3xgMSmfxy16ujGPxGW
2026-08-04 03:48:04 +00:00
Claude d54e54b48a Add battle-tested sync accessories from vespa-relay's mirror
Four pieces extracted from a production Nostr mirror (NosFabrica/vespa-relay),
generalized to quartz's multiplatform primitives, with their test suites:

- nip01Core.store.insertBisecting: batchInsert fails as a unit, so one bad
  event costs its whole batch (999 good events per bad one at a 1000-event
  batch). Bisecting isolates the offender in ~2*log2(n) extra writes, and a
  fixed write budget keeps a store-wide failure (full disk, dead engine) from
  turning one failed write into ~2n.

- accessories.SyncBands: resume memory for fetchAllPages. Remembers the
  created_at band covered per (relay, filter) and asks only for the legs
  outside it, with inclusive edges so a page boundary cannot strand a run of
  same-second events. A finished negentropy reconcile records completeness
  through its start instant; a periodic full re-walk keeps stale claims from
  narrowing forever. Persistence is the caller's, via export/restore and an
  onChange hook.

- accessories.PagingProgress: progress for paged walks measured on the time
  axis, the only axis whose end is known in advance - count-based percentages
  degenerate to downloaded/downloaded = 100%. Needs no COUNT support.

- nip66RelayMonitor.reachability.HostStrikes: per-authority strike counting
  for outbox-scale fan-outs, where a filtering relay mints one url per user
  and per-url counters never converge. Ever-delivered overrides eviction in
  both race orders, and eviction surfaces exactly once for publishing.
  reachability.Unreachability (jvmAndroid): which failures may be published
  as a signed NIP-66 unreachable record - connection-level only, so a relay
  that answered the handshake and hung up mid-page is never libelled, and a
  caller's own bug is never the relay's fault.

57 tests pass on the JVM target; the common code uses quartz's ConcurrentMap,
ConcurrentSet and TimeUtils only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4Pi9YYMhdzTFxRiV2jF9R
2026-08-04 03:33:48 +00:00
Claude 2b2e47256b docs: add nip85-trusted-assertions and searchable-events skills
Completes the store-implementer skill set requested by the vespa-eventstore
consumer: the NIP-85 trust-assertion model (kind map, tag vocabulary, value
semantics, authorization conventions) and the NIP-50 indexing surface (the
SearchableEvent contract plus an exhaustive kind -> indexableContent table
external search engines can diff at version bumps).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADBfjWhsXyHZb7ea2eeBhJ
2026-08-04 03:04:17 +00:00
Claude 7b29f57526 docs: add event-store-semantics skill (IEventStore/SQLite store contract)
Documents the store's observable behavior as named rules (STORE-F/W/D/C/S/N)
so external IEventStore implementations can review pin bumps and annotate
divergences against a stated contract instead of reverse-engineering
QueryBuilder. Requested by the vespa-eventstore consumer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADBfjWhsXyHZb7ea2eeBhJ
2026-08-04 02:55:14 +00:00
mstrofnone a110ce0a30 ci: publish linux-arm64 desktop, amy, and geode release assets
Amethyst v1.13.1 (and every prior release) shipped only linux-x64 desktop
binaries — .deb, .rpm, .AppImage, .flatpak, .tar.gz. Same for the amy
CLI and geode relay. Users on aarch64 hardware (Pinebook, Ampere
Altra, Raspberry Pi 4/5, AWS Graviton, arm64 servers, arm64 Chromebooks
running crostini, etc.) can't install any of them.

This teaches the release matrix about arm64:

- Add `ubuntu-24.04-arm` legs to build-desktop, build-cli, and
  build-geode. This is a standard, free public-repo GitHub-hosted
  runner (4 CPU / 16 GB / 14 GB SSD / arm64) since early 2025. No
  cross-compilation: jpackage / jlink / Compose Multiplatform 1.11
  all produce host-native artifacts.

- Fetch the matching `appimagetool-<arch>.AppImage` from the same
  1.9.0 release with an arch-specific SHA256 pin. `APPIMAGETOOL_URL`
  becomes `APPIMAGETOOL_VERSION` + per-arch SHA256 env vars.

- Parametrize the portable tarball/zip filename by `matrix.arch`
  (`amethyst-desktop-<ver>-linux-arm64.tar.gz` is now produced).

- Parametrize the Flatpak bundle filename and rewrite the manifest's
  `GST_PLUGIN_SYSTEM_PATH` from `x86_64-linux-gnu` to
  `aarch64-linux-gnu` on the arm64 leg. The Flathub-submission manifest
  (`desktopApp/packaging/flatpak/flathub/`) still gates on
  `only-arches: x86_64` — flipping that to include aarch64 is a
  follow-up once a Flathub aarch64 build has been validated end-to-end.

- Make the `createReleaseAppImage` gradle task pick its host arch from
  `System.getProperty("os.arch")` (amd64/x86_64 → `x86_64`, aarch64/
  arm64 → `aarch64`). Same task, same command, drives both legs.

- Fix `desktopApp/packaging/appimage/AppRun` to compute the multiarch
  library path from `uname -m` at launch time instead of hard-coding
  `x86_64-linux-gnu`. One script works in both AppImages on the target
  machine.

- Extend the desktop smoke test to run the release .deb build + launch
  probe on `ubuntu-24.04-arm` too, so arch-specific ProGuard/jlink
  breakage (missing native lib, arch-specific reflection root) is
  caught at PR time.

- Update BUILDING.md and scripts/asset-name.sh docs with the new
  arm64 asset names.

Follow-up assets published for the next tag push (v1.13.2+):

- amethyst-desktop-<ver>-linux-arm64.{deb,rpm,AppImage,flatpak,tar.gz}
- amy-<ver>-linux-arm64.{deb,rpm,tar.gz}
- geode-<ver>-linux-arm64.{deb,rpm,tar.gz}

Verification (local, before submitting):

- `python3 -c 'import yaml; yaml.safe_load(open(".github/workflows/create-release.yml"))'` — parses clean
- `bash -n scripts/asset-name.sh desktopApp/packaging/appimage/AppRun` — parses clean
- `actionlint` — reports only pre-existing shellcheck style hints; no new errors
- Confirmed `linuxdeploy-aarch64.AppImage` and
  `appimagetool-aarch64.AppImage` exist under the same pinned release
  tags used for x86_64; SHA256 recorded from a fresh download.

Not addressed (out of scope for this PR):

- Homebrew / winget bump workflows (`bump-homebrew*.yml`,
  `bump-winget.yml`) — those consume the assets by name; the new arm64
  filenames don't change any x86_64 name they already reference.
- Android arm64 continues to ship as before (already had it).
2026-08-04 10:51:16 +10:00
Vitor PamplonaandGitHub ba78eb599e Merge pull request #3851 from vitorpamplona/fix/relay-subscription-lock-convoy
fix(relay): stripe the subscription-state lock per relay to end an ANR convoy
2026-08-03 19:38:06 -04:00
Vitor PamplonaandClaude Opus 5 5896ae5eff fix(relay): stripe the subscription-state lock per relay to end an ANR convoy
A production ANR on a Pixel 8 (Amethyst 1.13.1, anr_2026-08-03-12-55-26-256)
showed the app burning 596% CPU — 6 of 9 cores — with the main thread stuck in
WaitingForGcToComplete. 37 of 52 runnable DefaultDispatcher workers sat at ONE
program point inside PoolRequests.onIncomingMessage and 12 more at one point in
syncState, all state=R, while the single thread actually holding the lock was
itself parked in GC.

Root cause: RequestSubscriptionState.withLock was a raw busy-wait
(`while (lock.exchange(true)) { while (lock.load()) {} }`) with no yield or
backoff, and being `inline` it disappeared into its callers' frames. The lock is
per subId, but one subId spans every relay it runs on — 191 live sockets on that
device — so dozens of relay-dispatch threads piled onto a single AtomicBoolean.
Spinning is only correct when the holder cannot be descheduled; on Android it
always can.

The fix stripes the lock per (subId, relay) rather than making waiting cheaper.
All 11 withLock bodies in PoolRequests are already scoped to exactly one relay,
and every field of RequestSubscriptionState is keyed by relay, so the sharing was
purely an artifact of mutableMapOf not being thread-safe. State moves into a
ConcurrentMap<T, RelayState>; locks live in a fixed 32-entry stripe array that is
never mutated, so lock identity stays stable — if locks lived inside the map
values, a thread holding one while another dropped and re-created that entry
would leave both inside the critical section excluding nothing.

A suspending Mutex was measured and rejected: it needs 262 method overrides and
110 call sites to become suspend, and ran at 0.35-0.63x the current throughput.

Measured (LockDesignComparisonBenchmark, 191 relays / 64 dispatcher threads):
  striped vs per-sub lock  1.5-2.8x throughput, bystander p50 halved
On device (SM-T220, playBenchmark, same account, n=3 per design):
  DefaultDispatcher CPU  -35% mean / -30% median vs the spin lock,
  with non-overlapping ranges; GC -18%
Plus 10 min of driven UI stress (feed, profiles, chat, notifications,
communities): no ANRs, no crashes, thread pools stable.

Also here:
- PlatformLock: new expect/actual parking lock (ReentrantLock on jvmAndroid,
  NSRecursiveLock on Apple, spin only on linuxX64 which is a CI target). quartz
  cannot use commons' equivalent KmpLock because commons depends on quartz.
- LiveNegentropyIndex had the identical busy-wait with a full list SORT inside
  the critical section; switched to PlatformLock.
- ConcurrentMap.remove (+ tests), with a caution that a removable value must not
  own a lock callers acquire.
- SpinLockConvoyBenchmark: regression guard asserting contended waiters PARK
  rather than spin (fails-before / passes-after). Pure benchmarks are gated
  behind -PprodRelayBench=1, so CI cost is 0.3s rather than 51.5s.

Analysis and measurements: quartz/plans/2026-08-03-poolrequests-lock-contention.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 19:19:56 -04:00
Vitor PamplonaandGitHub fdc68e801a Merge pull request #3849 from vitorpamplona/l10n_crowdin_translations
New Crowdin Translations
2026-08-03 00:40:58 -04:00
vitorpamplonaandgithub-actions[bot] 30742ab352 chore: sync Crowdin translations and seed translator npub placeholders 2026-08-03 04:35:55 +00:00
Vitor PamplonaandGitHub 1622bd7109 Merge pull request #3850 from vitorpamplona/claude/fetchallpages-timeoutms-d30cl1
Standardize timeout semantics to idle windows across all accessory APIs
2026-08-03 00:33:06 -04:00
mandClaude ebd9a163bc fix(desktop): auto-enable master notif switch when OS permission already granted
Follow-up to e9475dd079. That commit fixed the case where the user
clicked the "Enable OS notifications" button on a fresh install
(permission NotRequested \u2192 Granted) but the master toggle stayed off.
It missed the two closely-related cases the user was still hitting on
v1.13.1:

1. Permission was already granted from a previous session or install
   (e.g. an earlier v1.13.0 build, or the user allowed it via
   System Settings \u2192 Notifications directly). In this state,
   permissionState == Granted, so the "Enable OS notifications"
   button never renders \u2014 the button label promised the whole
   handshake but the code path that flipped the master switch only
   ran under NotRequested.
2. On Windows/Linux `permissionState` defaults to `NotApplicable`
   from the moment the app starts. The master switch is off by
   default (first-launch UX choice) and nothing ever flips it, so
   the auto-dispatcher stayed muted forever unless the user found
   the switch manually.

Fix:

- Add `NotificationSettings.wasExplicitlyDisabled()` so the Settings
  screen can distinguish "master switch is off because it defaults
  off on first launch" (auto-enable is fine) from "master switch is
  off because the user turned it off" (leave alone). Backed by a
  new java.util.prefs key `explicitly_disabled` that flips true on
  `setEnabled(false)` and gets cleared on `setEnabled(true)`.
- In `NotificationSettingsScreen`, a `LaunchedEffect(permissionState,
  enabled)` observes when the OS permission is Granted OR NotApplicable
  and the master switch is off. If the user has never explicitly
  turned it off, it auto-flips on \u2014 matching the "Enable OS
  notifications" contract for the paths the previous fix missed.
- Also render a "Turn on desktop notifications" button in the
  Granted branch when the user has explicitly turned notifications
  off. That's the recovery path for users who deliberately opted
  out and later want to opt back in without hunting for the
  master switch two rows away.

Behaviour on the fresh-install macOS path (permission NotRequested)
is unchanged \u2014 that path still runs the `requestPermission()`
flow inside the button's onClick, and the auto-enable happens via
the same LaunchedEffect once permissionState flips to Granted.

Tests (jvmTest, hermetic \u2014 UUID-scoped prefs nodes so tests never
share state or pollute real user prefs):

  PreferencesNotificationSettingsExplicitDisableTest:
    - fresh install defaults to not-explicitly-disabled
    - turning off marks explicitly disabled
    - turning on clears the explicit-disable flag
    - flag persists across new instances on the same prefs node

\ud83e\udd16 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 16:32:11 +10:00
96 changed files with 9357 additions and 351 deletions
+27
View File
@@ -85,3 +85,30 @@ skills verified clean):
`ParseReturn.entity` (the `Nip19Parser.Return.*` sealed class never existed);
Event Store section corrected from "Android only" to commonMain/all platforms
with the real `store.sqlite.EventStore` import and suspend generic `query<T>`.
## Phase 4 (2026-08): Store-implementer skills (external consumer request)
Three skills added at the request of an external Quartz consumer
(vespa-eventstore — a server-side `IEventStore` on Vespa that asserts result
parity against the SQLite store in CI). All three document the
**store/relay-implementer's perspective**, which `quartz-integration` and
`nostr-expert` (client-side) did not cover. Requirements doc: the skill-requests
file reviewed 2026-08-04; the requester's items #4 (storage-lifecycle-nips) was
folded into `event-store-semantics` per their own recommendation, and #5
(relay-server/geode policies) was declined as not currently needed.
- **`event-store-semantics/`** — the `IEventStore`/SQLite-store behavioral
contract as named rules (STORE-Fxx/Wxx/Dxx/Cxx/Sxx/Nxx) with a semantics
changelog for pin-bump review. Written from `QueryBuilder`,
`MergeQueryExecutor`, the seven `*Module.kt` files, and `IEventStore` KDoc.
- **`nip85-trusted-assertions/`** — the NIP-85 model (10040/30382/30383/30384/
30385), full tag vocabulary with value semantics, authorization conventions,
worked JSON examples, stability notes.
- **`searchable-events/`** — the `SearchableEvent` contract + maintenance
mandate, with `references/searchable-kinds.md` holding the exhaustive
kind → class → `indexableContent()` table (126 classes / 129 kinds) that
external search engines diff at version bumps.
Follow-ups suggested but not implemented: a shared JSON test-vector corpus for
filter semantics (testFixtures both the SQLite tests and external parity suites
could run), and a snapshot test pinning the searchable-kind set.
@@ -0,0 +1,325 @@
---
name: event-store-semantics
description: The authoritative behavioral contract of Quartz's event stores — `IEventStore` and its reference SQLite implementation (`nip01Core/store/sqlite/`). Use when implementing or asserting parity with a Quartz event store (external engines like Vespa, the filesystem store, geode), answering filter-semantics questions (since/until inclusivity, tag OR/AND, multi-filter limits, ordering tiebreaks), or working on the write-path rules for replaceable/addressable supersession, NIP-09 deletions, NIP-40 expiration, NIP-62 vanish, NIP-45 counts, or NIP-50 search inside the store. Every behavior has a named rule id (STORE-Fxx/Wxx/Dxx/Sxx/Cxx) so downstream implementations can annotate divergences precisely.
---
# Event Store Semantics — the `IEventStore` / SQLite-store contract
The SQLite `EventStore` (`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/`)
is the de-facto **reference implementation** of what a Quartz event store must do. Other
implementations — the in-repo filesystem store (`nip01Core/store/fs/`, held to parity by
`quartz/src/jvmTest/.../store/fs/FsParityTest.kt`) and external engines (e.g. a Vespa-backed
store) — reimplement its *observable behavior* and assert parity in CI. This skill states that
behavior as **named, numbered decisions** so a parity divergence becomes a lookup, not an
archaeology session through `QueryBuilder`/`MergeQueryExecutor`.
Every rule below was verified against the code as of this skill's last update. When you change
store behavior, **update the rule here in the same PR** and add a line to the
[Semantics changelog](#semantics-changelog) — downstream implementations pin Quartz by commit and
review pin bumps against this file.
## Key files
| Concern | File |
|---|---|
| Public contract (KDoc is normative) | `nip01Core/store/IEventStore.kt` |
| High-level store (owns pool + planner) | `sqlite/EventStore.kt`, `sqlite/SQLiteEventStore.kt` |
| Filter → SQL, ordering, limits, counts | `sqlite/QueryBuilder.kt` |
| k-way merge fast path (feed shapes) | `sqlite/MergeQueryExecutor.kt` |
| Schema, tag hashing, immutability | `sqlite/EventIndexesModule.kt`, `sqlite/TagNameValueHasher.kt`, `sqlite/SeedModule.kt` |
| Replaceable / addressable supersession | `sqlite/ReplaceableModule.kt`, `sqlite/AddressableModule.kt` |
| NIP-09 / NIP-40 / NIP-62 / ephemeral | `sqlite/DeletionRequestModule.kt`, `sqlite/ExpirationModule.kt`, `sqlite/RightToVanishModule.kt`, `sqlite/EphemeralModule.kt` |
| NIP-50 FTS | `sqlite/FullTextSearchModule.kt` (see also the `searchable-events` skill) |
| Index/feature toggles | `sqlite/IndexingStrategy.kt` (client default) and geode's `RelayIndexingStrategy.kt` (relay preset) |
| Operational README | `sqlite/README.md` (concurrency, pragmas, maintenance) |
Executable spec: the test suites in
`quartz/src/commonTest/.../store/sqlite/` (`BasicTest`, `ReplaceableTest`, `AddressableTest`,
`DeletionTest`, `ExpirationTest`, `RightToVanishTest`, `SearchTest`, `SearchRelevanceOrderTest`,
`MergeQueryCorrectnessTest`, `TagMergeCorrectnessTest`, `QueryAssemblerTest`,
`SnapshotIdsForNegentropyTest`, `FilterMatcherTest`, …). If a rule here ever contradicts a test,
the test wins — and this file has a bug to fix.
## Kind classes (used throughout)
- **Replaceable**: kind `0`, kind `3`, and `10000 ≤ kind < 20000`.
- **Ephemeral**: `20000 ≤ kind < 30000`.
- **Addressable**: `30000 ≤ kind < 40000`.
- Everything else is a regular event.
---
## Filter matching (STORE-F)
**STORE-F01 — `since`/`until` are both inclusive.** `since` compiles to
`created_at >= ?`, `until` to `created_at <= ?` (`QueryBuilder` uses
`greaterThanOrEquals`/`lessThanOrEquals` everywhere). An event with
`created_at == since == until` matches.
**STORE-F02 — `ids` and `authors` are exact-match only.** They compile to `=`/`IN` against the
full 64-char hex columns. **NIP-01 prefix matching is NOT supported** anywhere in the store.
(`Filter`'s constructor logs an error for non-64-char ids/authors but still sends them; they
simply never match.)
**STORE-F03 — tag filter combination.** Within one tag name, values are **OR**
(`tag_hash IN (…)`). Across different tag names in the same filter, conditions are **AND**
(each extra name becomes another `event_tags` self-join). `tagsAll` (NIP-91 `&x` syntax) demands
**every listed value** be present on the event — one join + equality per value — and composes by
AND with any plain `tags` in the same filter.
**STORE-F04 — only single-letter tag names are indexed (by default).**
`DefaultIndexingStrategy.shouldIndex` indexes a tag iff `tag.size >= 2 && tag[0].length == 1`.
A filter on a multi-letter tag name (`#title`, `#alt`) matches **nothing** in the SQLite store.
Deployments can widen `shouldIndex`, but the stock contract is single-letter-only.
**STORE-F05 — `d` is special-cased out of the tag index.** `#d` values are matched against the
`event_headers.d_tag` column, not `event_tags` (`Filter.toFilterWithDTags()`). Consequences:
`#d` works on addressable events (which populate `d_tag`); when all `kinds` are addressable the
query adds `kind >= 30000 AND kind < 40000` to pin the addressable index. **Only use `#d` via
plain `tags`.** A `#d` under `tagsAll` is handled inconsistently: on the simple (no other
tags/search) path it degrades to OR semantics (`toFilterWithDTags` folds it into `dTags`), and
when `tags["d"]` is also present it is dropped entirely; on the tag-join path it is ignored.
(An event has one d-tag, so AND-across-values could never match anyway.)
**STORE-F06 — tag and author matching in the tag path is hash-based.** `event_tags` stores a
64-bit MurmurHash3 of `(tag name, value)` keyed by a per-database random seed (`SeedModule`,
`TagNameValueHasher`); the p/e/a-owner columns are hashes too. There is **no post-verification**
of hash matches, so a hash collision would return a false positive. Probability is negligible in
practice but nonzero — a parity harness comparing against an exact-match engine should know this
is the one place the reference can (theoretically) over-match.
**STORE-F07 — multiple filters are a union with dedup; `limit` is per-filter.** Each filter
becomes its own row-id subquery with its **own** `ORDER BY … LIMIT`; branches are combined with
SQL `UNION` (dedup by row). There is **no global limit** — a 3-filter query with limits
10/20/30 can return up to 60 events, presented in one merged `created_at DESC` ordering. NIP-45
counts and negentropy snapshots dedup the same way (`SELECT DISTINCT` / `UNION`).
**STORE-F08 — result ordering.** Non-search queries order `created_at DESC`. The `id ASC`
tiebreak on equal `created_at` is applied **only when
`IndexingStrategy.useAndIndexIdOnOrderBy = true`** — which is `false` in the client default
**and** in geode's relay preset. So by default, same-second ordering is unspecified (SQLite
returns them in storage order). Any newest-N is valid; a parity suite must not assert
same-`created_at` order unless it configures the flag. One extra caveat with the flag ON: the
`MergeQueryExecutor` tag-stream path still yields same-second ties in rowid order (its cursors
run off `event_tags`, which has no id column) — a valid newest-N that may differ byte-for-byte
from the single-SQL ordering.
**STORE-F09 — the merge fast path returns the same *set*.** Single-filter queries of the shape
"authors (+kinds) + limit" or "one `#x` IN-list (+kinds) + limit" (≤2048 streams) route through
`MergeQueryExecutor`, a k-way newest-first merge over per-(kind,author) / per-(tag-value,kind)
index cursors with dedup by id on the tag shape. This is an optimization, not a semantics
change — `MergeQueryCorrectnessTest`/`TagMergeCorrectnessTest` assert set-equality with the
single-SQL plan (ordering caveat per STORE-F08).
**STORE-F10 — empty filter.** `query(Filter())` / `count(Filter())` match **everything**
(`Filter.isEmpty()` → the "everything" query). `delete(Filter())` is deliberately asymmetric:
it deletes **nothing** and returns 0, so a stray empty filter can't wipe the store (documented
on `QueryBuilder.delete`).
**STORE-F11 — empty lists (`kinds = emptyList()` etc.) are a client error with inconsistent
handling; don't rely on either outcome.** On the single-filter simple path an empty list
renders as `1 = 0` → matches nothing. But `Filter.isEmpty()` treats empty lists the same as
`null`, so on the multi-filter union path such a filter contributes no subquery — and a list of
*only* empty-list filters degrades to the match-everything query. Known quirk; treat
empty-list filters as invalid input rather than replicating this shape.
**STORE-F12 — `limit` edge cases.** `limit = 0` compiles to `LIMIT 0` → zero rows.
`limit = null` means unbounded. Negative limits are not defended against (don't send them).
**STORE-F13 — the in-memory matcher is a separate (simpler) implementation.**
`Filter.match(event)` (`FilterMatcher`) is used for live-stream matching, not storage queries;
it checks ids/authors/kinds/tags/tagsAll/since/until but not `search` or `limit`. Parity work
targets the SQL semantics above, not `FilterMatcher`.
---
## Write path (STORE-W)
Inserts run every module in one transaction: header+tags → NIP-09 side effects → expiration
row → FTS row → vanish side effects. A trigger `RAISE(ABORT, …)` rejects the whole row with the
messages quoted below (they surface as the NIP-01 `OK false` reason).
**STORE-W01 — replaceable supersession.** Unique index on `(kind, pubkey)` for replaceable
kinds. A `BEFORE INSERT` trigger deletes any stored version that is *older* — meaning
`created_at` smaller, **or equal `created_at` with lexicographically larger id** (NIP-01
lowest-id-wins). Inserting a version that is *not* newer under that ordering leaves the stored
row in place and fails the unique index → rejected (`UNIQUE constraint failed`). Net contract:
exactly one version stored; newest wins; ties broken by lowest id; older re-inserts blocked.
**STORE-W02 — addressable supersession.** Same as W01 with unique index
`(kind, pubkey, d_tag)` over `30000 ≤ kind < 40000`. Nuance: `d_tag` is populated from the
*parsed* event class (`AddressableEvent.dTag()`); an addressable-range kind whose class doesn't
parse as `AddressableEvent` stores `d_tag NULL`, and SQLite treats NULLs as distinct in unique
indexes — such events don't supersede each other. An event with no `d` tag parses as `dTag() = ""`
(empty string), which *does* dedupe normally.
**STORE-W03 — ephemeral events are never stored but are acked as accepted.**
`insert()` returns silently and `batchInsert` reports `Accepted` for `20000 ≤ kind < 30000`
without writing (the live relay stream still broadcasts them). A DB-level backstop trigger
(`blocked: cannot store ephemeral events`) rejects any that sneak past the app-level check.
**STORE-W04 — expired events are rejected at insert.** App-level check
(`event.isExpired()`) plus a trigger on the expiration-row insert
(`blocked: this event is expired` when `expiration <= unixepoch()`). Single-event `insert`
**throws**; `batchInsert` returns `Rejected`.
**STORE-W05 — expiry is enforced at insert and by sweep, NOT at query time.** Events with a
future `expiration` store a row in `event_expirations`. Nothing filters them out of queries
after the timestamp passes: **a query between expiry and the next `deleteExpiredEvents()` sweep
returns the expired event.** Operators run the sweep periodically (README recommends ~15 min).
Re-inserting an already-expired event after the sweep is rejected per W04.
**STORE-W06 — GiftWrap ownership is the recipient.** For kind 1059 the store computes
`pubkey_owner_hash` from the `p`-tag recipient (falling back to the random signer key if
absent). All owner-scoped machinery — NIP-09 re-insert blocking, NIP-62 vanish deletion and
blocking — operates on that owner hash, so **a user's deletions/vanish remove giftwraps
addressed to them**, even though the wrap's `pubkey` is a one-time key. (Consequently GiftWraps
are also excluded from `authorsMissingOutbox()`.)
**STORE-W07 — immutability.** `event_headers`/`event_tags` rows are never updated
(`BEFORE UPDATE` triggers abort). All supersession is delete + insert; `event_tags`,
`event_expirations`, `event_vanish`, and the FTS row follow the header by
`ON DELETE CASCADE` / trigger.
**STORE-W08 — batch insert.** One outer transaction, one SAVEPOINT per row: a bad row rolls
back alone and reports `Rejected(reason)`; the rest commit. If the **outer commit** fails, every
entry is treated as `Rejected` (the `IEventStore.batchInsert` contract). Outcomes are returned
in input order; OK frames pair by event id, not order.
---
## Deletion lifecycle — NIP-09 / NIP-62 (STORE-D)
**STORE-D01 — delete by id.** A kind-5's `e` tags delete stored events with those ids **whose
owner is the kind-5's author** (`pubkey_owner_hash` match — recipient for giftwraps per W06).
The id path has **no timestamp condition**: it deletes the target regardless of the relative
`created_at` values.
**STORE-D02 — delete by address.** A kind-5's `a` tags delete events at that
`(kind, pubkey, d_tag)` coordinate with `created_at <= deletion.created_at`**inclusive**; a
version newer than the deletion survives. Only coordinates whose pubkey equals the kind-5's
author are honored. Replaceable coordinates (`kind:pubkey:` with no d-tag) get the same
`created_at <=` treatment against `(kind, pubkey)`.
**STORE-D03 — cross-author kind-5s are stored but inert.** A deletion naming someone else's
events is inserted like any regular event (it may be useful to other relays/clients) but its
delete pass removes zero rows and creates no blocking.
**STORE-D04 — re-insert blocking.** A `BEFORE INSERT` trigger rejects
(`blocked: a deletion event exists`) any event whose id (`e`-hash) **or** address (`a`-hash) is
named by a stored kind-5 from the same owner with `deletion.created_at >= event.created_at`.
Note the asymmetry with D01: a *backdated* id-deletion (older `created_at` than its target)
still deletes on arrival, but would not block a later re-insert.
**STORE-D05 — a kind-5 CAN delete another kind-5, and doing so un-blocks its targets.**
Nothing excludes kind 5 from the id path (D01). Deleting a deletion removes its tombstone rows
from `event_tags`, so events it had deleted become re-insertable. **Status: known quirk, not a
considered decision.** NIP-09 leaves it open; at least one external implementation
(vespa-eventstore) deliberately diverges by treating deletion-of-a-deletion as a no-op, which is
the safer reading (tombstones shouldn't be revocable). If you change this, update this rule and
the changelog — parity suites key off it.
**STORE-D06 — NIP-62 vanish is relay-scoped.** A kind-62 only cascades when
`shouldVanishFrom(relay)` — its `relay` tags name this store's `relay` URL or `ALL_RELAYS`.
(A store constructed with `relay = null` matches only `ALL_RELAYS` requests.) Out-of-scope
vanish events are stored as regular events with no side effects.
**STORE-D07 — vanish scope and horizon.** An in-scope vanish deletes every event whose
**owner** (W06) is the vanishing pubkey with `created_at < vanish.created_at` (strict — the
vanish event itself survives), and blocks inserts of owned events with
`created_at <= vanish.created_at` (`blocked: a request to vanish event exists`; note blocking is
inclusive where deletion is strict). Newer vanish requests supersede older ones per pubkey
(unique on `pubkey_hash`).
**STORE-D08 — manual deletes.** `delete(id)` removes one row unconditionally (no blocking
created). `delete(filter)` deletes matching rows honoring per-filter limits, with the F10
empty-filter no-op guard. Neither creates re-insert blocking — only stored kind-5/kind-62
events do that.
---
## NIP-45 count (STORE-C)
**STORE-C01 — count = size of the deduped match set, honoring per-filter limits.** Single
filter: `COUNT(*)` over that filter's row-id subquery (including its `LIMIT`, so
`count(Filter(kinds=…, limit=10))` is at most 10). Multiple filters: branches are `UNION`ed
(dedup) **before** counting — an event matching several filters counts once. FTS-off + search
term → 0 (F-series search rules apply).
---
## NIP-50 search inside the store (STORE-S)
The indexing surface (which kinds are searchable, what text they contribute) is the
`searchable-events` skill; these rules are the store's query-side contract.
**STORE-S01 — extension stripping at the store boundary.** Every filter-accepting method runs
`strippingSearchExtensions()`: NIP-50 `key:value` tokens (`include:spam`, `domain:…`, …) are
removed before FTS. Unsupported extensions are **ignored, never matched as literal text and
never match-nothing** — an extensions-only search collapses to an unconstrained query. Stores
that *do* implement extensions receive the raw string through the relay layer and parse it with
`nip50Search.SearchQuery.parse` (see the `IEventStore` KDoc).
**STORE-S02 — relevance ordering.** Search results order by FTS5 `bm25` rank (best match
first), with `created_at DESC` only as tiebreak; the `LIMIT` keeps the most *relevant* N, not
the newest N. A multi-filter REQ is relevance-ordered only when **every** filter carries a
search term (best/min rank per event across branches); mixing search and non-search filters
falls back to `created_at DESC`.
**STORE-S03 — search combines by AND with the structural parts** (ids/authors/kinds/tags/
since/until) of the same filter — an FTS `MATCH` join on top of the normal conditions.
**STORE-S04 — search grammar is SQLite FTS5 `MATCH`.** The raw (post-strip) string is passed to
FTS5, so implicit-AND terms, `"phrase queries"`, `OR`, and `prefix*` follow FTS5 semantics.
Tokenization details live in `FullTextSearchModule` (see `searchable-events`).
**STORE-S05 — FTS off.** With `IndexingStrategy.indexFullTextSearch = false`: a filter with a
non-empty search term matches **nothing** (query/count/delete alike); an empty-string search
imposes no constraint. Everything else is unchanged.
**STORE-S06 — deferred FTS.** Relays may set `deferFullTextSearchIndexing = true` (geode does):
tokenization moves off the insert path to a watermark-driven catch-up
(`needsFtsCatchUp`/`ftsCatchUp`), and search queries drain the backlog first — so NIP-50
results are exactly as fresh as the synchronous path.
---
## Negentropy / NIP-77 (STORE-N)
**STORE-N01 —** `snapshotIdsForNegentropy(filters)` returns `(created_at, id)` pairs under the
**same filter semantics as `query`** (per-filter limits included, multi-filter dedup), order
unspecified (negentropy re-sorts). `maxEntries` returns up to `maxEntries + 1` as an overflow
sentinel. `liveNegentropySnapshot` serves full-corpus NEG-OPENs from an in-memory index when
`maintainLiveNegentropyIndex` is on; the delta plumbing in `SQLiteEventStore` keeps it exact
across replaceable displacement, kind-5s, and vanish (invalidate-and-rebuild for the
non-itemizable cases).
---
## Configuration presets
- **Client default** (`DefaultIndexingStrategy()`): FTS on (synchronous), optional indexes off,
`useAndIndexIdOnOrderBy` off, no live negentropy index.
- **Relay preset** (geode's `relayIndexingStrategy()`): adds created_at-alone, pubkey-alone and
tag+kind+pubkey indexes, defers FTS, maintains the live negentropy index — still leaves
`useAndIndexIdOnOrderBy` off.
- Flag-gated indexes are runtime config, not schema: flipping one on an existing DB builds the
index on next open (`ensureOptionalIndexes`), no migration.
## For parity implementers
- Treat the rule ids above as the vocabulary for divergence notes
(e.g. "diverges from STORE-D05: we no-op deletion-of-a-deletion").
- The commonTest suites are the executable spec; `FsParityTest` shows the in-repo pattern for
holding a second engine to it.
- Remember F06 (hash-based tag matching) and F08 (unordered same-second ties by default) when
diffing results byte-for-byte — both are places where a "divergence" may be the reference's
own slack, not your bug.
## Semantics changelog
Add one line per behavior change, newest first: `YYYY-MM-DD <short sha> <rule id> — what changed`.
- 2026-08-04 (baseline) — rules F01F13, W01W08, D01D08, C01, S01S06, N01 written from the
code at the time this skill was introduced. Changes before this date are not itemized;
archaeology starts at `git log` on `nip01Core/store/`.
@@ -0,0 +1,232 @@
---
name: nip85-trusted-assertions
description: The NIP-85 trusted-assertions model in Quartz (`nip85TrustedAssertions/`) — kind 10040 trust-provider lists, kind 30382 contact cards / user assertions, 30383 event assertions, 30384 addressable assertions, 30385 external-id assertions. Use when building or parsing these events, working with the typed tags (RankTag, HopsTag, FollowerCountTag, ServiceProviderTag/ServiceType, …), wiring a consumer that resolves a 10040 provider entry to the 30382s it signs, ranking on assertion values, or touching the GrapeRank publisher, contact-card nicknames, or the trust projection of an external store.
---
# NIP-85 Trusted Assertions — the Quartz model
Package: `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/`.
NIP-85 is still an evolving spec; **this package is the operative definition** of what
Amethyst-family software writes and reads. This skill states the model (who signs what about
whom), the exact kind/d-tag/tag vocabulary, and what consumers may — and may not — assume.
## The model in one paragraph
An **assertion is signed by the asserting party** (a trust provider service, or the user
themself) **about a subject named in the d-tag**. All assertion kinds are addressable, so
"latest card by provider P about subject S" is just the addressable coordinate
`(kind, P, S)` and supersession is standard NIP-01 latest-wins. Discovery is the observer's
**kind 10040 list**: each entry says *"for metric M on kind K, I trust provider P — fetch
their assertions at relay R"*. Quartz enforces none of this cryptographically beyond normal
event signatures; the 10040→assertion link is **consumer-side convention** (see
"Authorization" below).
## Kind map
| Kind | Class | Kind class | d-tag = the subject | Content |
|---|---|---|---|---|
| 10040 | `list/TrustProviderListEvent` | replaceable | *(none — always `""`)* | NIP-44 private provider entries (optional) |
| 30382 | `users/ContactCardEvent` | addressable | **target user's pubkey** (hex) | NIP-44 private tags (petname/summary/emoji) |
| 30383 | `events/EventAssertionEvent` | addressable | **target event id** (hex) | `""` |
| 30384 | `addressables/AddressableAssertionEvent` | addressable | **target coordinate** `kind:pubkey:dtag` | `""` |
| 30385 | `externalIds/ExternalIdAssertionEvent` | addressable | **external identifier** (e.g. `isbn:978-0-13-468599-1`) | `""` |
Addresses: `ContactCardEvent.createAddress(owner, target)``Address(30382, owner, target)`
(owner = signer, target = subject). `TrustProviderListEvent.createAddress(pubKey)` uses
`FIXED_D_TAG = ""`. `AssertionEventTest.eventKindsAreCorrect` pins all five numbers.
`ContactCardEvent` is also a `SearchableEvent` — it indexes only the **public** petname/summary
tags plus topics; the encrypted card content is intentionally never indexed.
## The 10040 provider entry (`ServiceProviderTag` / `ServiceType`)
There is **no fixed tag name**: `tag[0]` *is* the service string.
```json
["30382:rank", "<provider pubkey, 64 hex>", "wss://nip85.brainstorm.world"]
```
- `ServiceType(kind, type)` parses/renders `"<kind>:<type>"` — kind must be an int, the first
`:` splits, colons in the remainder stay in `type`. `ServiceType.isOfKind` is the
allocation-free prefix check.
- `ServiceProviderTag.parse` requires ≥3 elements, non-empty service, 64-char pubkey
(length-only check), and a **normalizable relay URL** (`RelayUrlNormalizer.normalizeOrNull`) —
entries failing any check are silently dropped, which is what keeps foreign tags like
`["client","nostria"]` out (regression-tested in `ServiceTypeParserTest`).
- Entries may be **public** (tag array) or **private** (NIP-44 content); `create`/`add` take
`isPrivate`. `remove` always needs decryption and strips from both sides by parsed-value
equality.
- `object ProviderTypes` (`list/tags/ServiceType.kt`) enumerates the *known* service types —
`30382:rank`, `30382:followers`, `30382:first_created_at`, per-metric `30383:*`/`30384:*`/
`30385:*`, etc. It is an **open vocabulary**: real 10040s in the wild (see the fiatjaf →
brainstorm fixture in `commonTest/.../nip85TrustedAssertions/ServiceParser.kt`) carry types
Quartz doesn't enumerate (`30382:personalizedGrapeRank_influence`, `30382:hops`,
`30382:verifiedFollowersCount`, …). Parse any `kind:type`; special-case only what you rank on.
## Authorization — what a consumer may assume
- **A 30382 (or 30383/…) is meaningful to an observer only if its author is listed in the
observer's 10040 for a matching service type.** Quartz does not enforce this; the consuming
code does. The in-repo pattern is `commons/.../model/nip85TrustedAssertions/UserCardsCache.kt`:
`rankFlow(trustProviderList)` picks the received card whose **author pubkey equals the
provider entry's pubkey** and reads `rank()` from it. Assertions from unlisted signers are
simply ignored for trust purposes (they may still be stored; dropping them — as an external
store's orphan sweep does — is a legitimate storage policy, not a protocol rule).
- What an entry authorizes is scoped by its `ServiceType`: `30382:rank` authorizes that
provider's user-rank cards, nothing else. Amethyst models this as one provider slot per
metric (`liveUserRankProvider`, `liveUserFollowerCount` in
`amethyst/.../model/trustedAssertions/TrustProviderListState.kt`).
- **Multi-provider combination is unprescribed.** When two listed providers assert different
ranks, there is no spec'd merge; Amethyst avoids the question by selecting one provider per
metric slot. Consumers choose their own policy — document it.
- The relay URL in the entry is a **fetch hint, and it is honored**:
`amethyst/.../UserCardsSubAssembler.kt` subscribes for cards at the provider's declared relay
(`kinds=[30382], authors=[provider], #d=[targets]`).
### The dual use of kind 30382
The same kind serves two roles, distinguished **by author**:
1. **Provider WoT cards** — signed by a trust provider; public metric tags (`rank`,
`followers`, `hops`, …); this is what 10040 discovery points at.
2. **The account's own contact cards (nicknames, NIP-81-style)** — signed by the account,
one per target user. The petname, summary, and their NIP-30 emoji mappings **always live in
the NIP-44 encrypted content, never in public tags** (`ContactCardEvent.build`/
`updatePetNameAndSummary` strip stray public copies; asserted by `ContactCardPetNameTest`).
`commons/.../ContactCardsState.kt` keys everything on `author == account` and ignores
provider cards.
## Tag vocabulary and value semantics
All tag classes share one shape: `TAG_NAME` + `parse(tag)` (null on wrong name/empty/non-numeric
value — a bad tag is *dropped*, never an error) + `assemble(value)``[name, value.toString()]`.
**A missing tag means "unknown" (`null` accessor), never zero.** There is deliberately no range
validation (rank isn't clamped, hours aren't checked against 023, counts may be negative) —
consumers must defend.
**On 30382** (`users/tags/`, accessors on `ContactCardEvent` and as `TagArray` extensions in
`users/TagArrayExt.kt` so they also work on decrypted private arrays):
| Tag name | Accessor | Type | Semantics |
|---|---|---|---|
| `rank` | `rank()` | Int | Provider-relative score; higher is better. GrapeRank publishes `round(score × 100)` (so 0100 in practice), but nothing enforces a scale — treat it as comparable only *within one provider*. |
| `followers` | `followerCount()` | Int | Follower count as the provider computes it (cumulative, provider-defined). |
| `hops` | `hops()` | Int | Shortest follow-path length **from the observer the provider computed for** to the subject (1 = directly followed). Mirrors Brainstorm GrapeRank's `hops`. The only tag with KDoc. |
| `first_created_at` | `firstCreatedAt()` | Long | Unix seconds of subject's earliest known event. |
| `post_cnt` / `reply_cnt` / `reactions_cnt` | `postCount()` etc. | Int | Activity counts. |
| `zap_amt_recd` / `zap_amt_sent` | `zapAmountReceived()`/`…Sent()` | Long | Sats. |
| `zap_cnt_recd` / `zap_cnt_sent` | `zapCountReceived()`/`…Sent()` | Int | Counts. |
| `zap_avg_amt_day_recd` / `zap_avg_amt_day_sent` | `zapAvgAmountDay…()` | Long | Sats/day averages. |
| `reports_cnt_recd` / `reports_cnt_sent` | `reportsCount…()` | Int | NIP-56 report counts. |
| `t` (repeatable) | `topics()` | List\<String> | Subject's topics/interests. |
| `active_hours_start` / `active_hours_end` | `activeHours…()` | Int | Hour-of-day; **no timezone is specified in code** — treat as provider-defined (UTC in practice) and unclamped. |
| `petname` / `summary` | `petName()`/`summary()` | String | Nickname fields — conventionally private (see dual use above). |
**On 30383/30384** (`tags/`, shared): `rank`, `comment_cnt`, `quote_cnt`, `repost_cnt`,
`reaction_cnt`, `zap_cnt` (Int) and `zap_amount` (Long, sats).
**On 30385**: only `rank`, `comment_cnt`, `reaction_cnt`.
## Building and parsing (use the typed helpers, not raw `arrayOf`)
```kotlin
// Provider list: declare a rank provider (this is what `amy graperank register` does)
val tag = ServiceProviderTag(ProviderTypes.rank, providerPubkeyHex, relayUrl)
val list = TrustProviderListEvent.create(tag, isPrivate = false, signer)
// or append to an existing one:
val updated = TrustProviderListEvent.add(existing, tag, isPrivate = false, signer)
val providers: List<ServiceProviderTag> = updated.serviceProviders() // public
val private = updated.privateTags(signer)?.serviceProviders() // private side
// Provider-style contact card (public metrics) — the GrapeRankPublisher pattern:
val card = ContactCardEvent.create(
targetUser = subjectPubkey,
signer = providerSigner,
publicInitializer = {
rank(87)
followers(1234)
hops(2)
},
)
card.aboutUser() // d-tag → subject pubkey
card.rank() // 87
// Event assertion: unsigned template only (30383/84/85 have build(), no create())
val template = EventAssertionEvent.build(targetEventId) {
rank(12)
reactionCount(40)
zapAmount(2100)
}
val signed = signer.sign(template)
```
## Worked end-to-end example
Observer `O` trusts provider `P` for user ranks (kind 10040, replaceable, by `O`):
```json
{ "kind": 10040, "pubkey": "<O>",
"tags": [
["30382:rank", "<P>", "wss://nip85.brainstorm.world"],
["30382:followers", "<P>", "wss://nip85.brainstorm.world"]
],
"content": "" }
```
Provider `P` asserts about subject `S` (kind 30382, addressable at `30382:<P>:<S>`):
```json
{ "kind": 30382, "pubkey": "<P>",
"tags": [
["d", "<S>"],
["rank", "87"], ["followers", "1234"], ["hops", "2"]
],
"content": "" }
```
`P` asserts about an event `E` (kind 30383, addressable at `30383:<P>:<E>`):
```json
{ "kind": 30383, "pubkey": "<P>",
"tags": [["d", "<E>"], ["rank", "12"], ["reaction_cnt", "40"], ["zap_amount", "2100"]],
"content": "" }
```
Consumption chain: read `O`'s 10040 → entry matching `ServiceType(30382, "rank")` → subscribe
`{kinds:[30382], authors:["<P>"], "#d":["<S>", …]}` at the hinted relay → newest card per
address wins → `rank()`.
Literal fixtures: `quartz/src/commonTest/.../nip85TrustedAssertions/ServiceParser.kt` (a real
10040 — fiatjaf's, pointing at the Brainstorm provider) and `AssertionEventTest.kt` (all four
assertion kinds with every tag populated).
## Freshness / supersession
Assertions are addressable: **latest per `(kind, author, d-tag)` wins**; there is no expiry tag
convention and **no prescribed refresh cadence** — staleness policy is the consumer's.
Writers should avoid churn: `GrapeRankPublisher` re-signs a card only when
`(rank, followers, hops)` actually changed, and retracts with a NIP-09 kind-5 carrying the
card's `a`-tag (`30382:<provider>:<target>`).
## Stability notes (as of 2026-08)
- **Settled** (shipped consumers on both ends): the kind map; `ServiceProviderTag` entry shape;
`rank`/`followers`/`hops` on 30382; petname/summary-in-encrypted-content; 10040 relay-hint
consumption.
- **Written but lightly consumed** (parse, but gate ranking features carefully): the activity/
zap/report count tags, `active_hours_*` (no timezone semantics), 30383/30384/30385 (builders +
tests exist; no in-repo publisher yet).
- **Known warts**: `ServiceProviderTag.assemble(id: ServiceProviderTag)` infers `Array<Any>`
dead code, don't use it; `SummaryTag.assemble(ip:)`/`ActiveHours*Tag.assemble(count:)` params
are misnamed; the tests live under `commonTest/.../experimental/nip85TrustedAssertions/`
(stale path); `TrustProviderListEvent` extends the addressable base, so a stray on-wire `d`
tag is reflected by `dTag()` even though the convention is `""`.
## Where it's consumed (reading list)
- **Publisher**: `quartz/.../experimental/graperank/GrapeRankPublisher.kt` (canonical 30382
writer), `cli/.../graperank/` (`amy graperank register|unregister|providers|publish`).
- **Client model**: `commons/.../model/nip85TrustedAssertions/` (`ContactCardsState`,
`UserCardsCache`, `ContactCardDecryptionCache`, `TrustProviderListDecryptionCache`),
`amethyst/.../model/trustedAssertions/TrustProviderListState.kt`.
- **Relay plumbing**: `commons/.../relayClient/assemblers/ContactCardFilters.kt`,
`amethyst/.../reqCommand/user/watchers/UserCardsSubAssembler.kt`.
+117
View File
@@ -0,0 +1,117 @@
---
name: searchable-events
description: The NIP-50 indexing surface of Quartz — the `SearchableEvent` interface, which event kinds are searchable, exactly what text each kind's `indexableContent()` contributes, how the SQLite/filesystem stores consume it, and the NIP-50 `SearchQuery` extension grammar plus `SearchRelayListEvent` (kind 10007). Use when making a kind searchable, changing what a kind indexes, diffing the searchable set at a Quartz version bump (external search engines mirror this table), debugging why an event is or isn't found by search, or working with search extensions (`include:spam`, `domain:`, …).
---
# Searchable Events — the NIP-50 indexing surface
## The contract
`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip50Search/SearchableEvent.kt`:
```kotlin
interface SearchableEvent {
fun indexableContent(): String
}
```
One method; marker and extractor in one. An event kind is searchable **iff** its event class
implements this interface **and** the class is wired into `EventFactory` (the stores probe
searchability by kind through `EventFactory.create` — an unwired implementor is invisible).
Rules every implementation follows (keep them when adding one):
- **Plain text out.** Return the human-meaningful fields joined with `"\n"` (a handful of
metadata-ish kinds use `" "`); no markup stripping is performed — markdown/asciidoc content
goes in raw, JSON-content kinds (kind 0 metadata, marketplace stalls, channel info) **parse
first and join the extracted fields**, never the raw JSON.
- **Never throw, never null.** There is no defensive wrapper at any call site; a throw aborts
the insert transaction. Parsed-JSON implementations use `?.let { … } ?: ""`.
- **Only public data.** Encrypted content stays out (e.g. kind 30382 contact cards index only
the public petname/summary/topics, never the NIP-44 payload).
- Typical shapes: `content` alone (~33 kinds); `listOfNotNull(title(), content)`;
`listOfNotNull(title(), summary(), content)`; lists index `title() + description()`.
## The full kind table
**`references/searchable-kinds.md`** in this skill holds the authoritative table — every
implementor with its kind number, class, and the exact `indexableContent()` expression
(126 concrete classes / 129 kind values as of 2026-08). Diff that file at a version bump to
answer "did the searchable set or any kind's indexed text change?".
Notables that surprise people:
- **Kind 9735 (zap receipt) indexes the embedded zap request's content**
(`zapRequest?.content.orEmpty()`) — receipts are searchable by the zapper's comment.
- **Kind 0 / 31990** index many profile fields space-joined (name, about, nip05, lud16,
website, picture URL, …).
- **Kind 30063 is claimed twice** (`ReleaseArtifactSetEvent` in nip51Lists and the experimental
`SoftwareReleaseEvent`); `EventFactory` resolves 30063 to `ReleaseArtifactSetEvent`, so
`title()\ndescription()` is what actually gets indexed — `SoftwareReleaseEvent.indexableContent()`
is dead on the store path.
- Poll kinds (1068, 6969) append each option label on its own line.
## MANDATORY maintenance when you touch this surface
Adding `SearchableEvent` to a kind, removing it, or changing any `indexableContent()` body:
1. **Update `references/searchable-kinds.md`** in the same PR (external search engines — e.g.
the Vespa-backed store's `SearchExtractors` — mirror this table at pin bumps; a silent
change ships them stale search results).
2. **Remember existing databases don't reindex themselves.** Old rows keep their old (or
missing) FTS text until `IEventStore.reindexFullTextSearch()` runs — the KDoc on that method
is the contract. App-side, schedule the resumable overload after shipping such a change.
3. New implementors must be **registered in `EventFactory`** or the reindex scan and kind
pre-filter (`FullTextSearchModule.isSearchableKind`) will never see them.
Eligibility policy: a kind becomes searchable when it carries human-authored, human-meaningful
text (titles, bodies, names, descriptions). Pure-machine kinds (reactions, follow lists, zaps
minus their comment, relay lists) stay out to keep the index small.
## How the stores consume it
**SQLite** (`nip01Core/store/sqlite/FullTextSearchModule.kt`):
`CREATE VIRTUAL TABLE event_fts USING fts5(content, content='', contentless_delete=1)`
contentless, `rowid` = `event_headers.row_id`, an `AFTER DELETE` trigger keeps it in sync. On
insert (when FTS is on and not deferred): `if (event is SearchableEvent)` → bind
`event.indexableContent()` — the only method ever called. Tokenization is entirely SQLite's
default FTS5 `unicode61`; queries are always a bound `event_fts MATCH ?` (never concatenated),
ordered by bm25 `rank` then `created_at DESC`. Query-side semantics (relevance ordering,
extension stripping, FTS-off behavior, deferred catch-up) are rules STORE-S01…S06 in the
`event-store-semantics` skill.
**Filesystem store** (`jvmMain/.../store/fs/FsIndexer.kt` + `FsSearchTokenizer.kt`): tokenizes
`indexableContent()` itself, approximating `unicode61` (split on non-letter/digit, lowercase);
the same tokenizer runs on queries so drift cancels.
## NIP-50 client side
**`SearchQuery`** (`nip50Search/SearchQuery.kt`) — typed parse of the `search` filter string
into `terms` + `extensions`. A whitespace token is an extension iff it looks like
`lowercasekey:value` (the value not starting with `//`, so URLs stay free text); duplicate keys
keep the last; unknown extensions are preserved (`extension(key)`). Typed accessors:
`includeSpam`, `domain`, `language`, `sentiment`, `nsfw`. `stripExtensions()` /
`Filter.strippingSearchExtensions()` is the bridge the built-in stores use — unsupported
extensions are **ignored** (NIP-50), so an extensions-only search collapses to an unconstrained
query, never match-nothing. A server-side store that implements its own extensions
(`observer:`, `sort:rank`, …) receives the raw string (see the `IEventStore` KDoc) and should
parse with `SearchQuery.parse` so its syntax stays compatible with what clients send.
**`SearchRelayListEvent`** — **kind 10007**, the user's search-relay list (NIP-51-style, public
tags + NIP-44 private tags; *not* a `SearchableEvent` itself). Client consumption:
`commons/.../actions/SearchActions.kt`, bootstrap defaults in
`commons/.../account/AccountBootstrapEvents.kt`.
Don't confuse it with `commons/.../commons/search/SearchQuery.kt` — an app-level local-feed
query model (authors/kinds/hashtags/or-terms), unrelated to the NIP-50 wire string.
## Tests (executable spec)
- `commonTest/.../nip50Search/SearchQueryTest.kt` — the extension grammar, token by token.
- `commonTest/.../store/sqlite/SearchTest.kt` — per-kind indexing (kind 0 profile fields,
40/41 channel JSON, 31924/30617), extension-token ignoring, reindex/resumable-reindex,
FTS cleanup on replaceable rotation.
- `commonTest/.../store/sqlite/SearchRelevanceOrderTest.kt` — bm25-before-recency ordering,
limit-after-score, multi-filter rank union.
- `commonTest/.../store/sqlite/NoFullTextSearchTest.kt` — FTS-off contract.
- `jvmTest/.../store/fs/FsSearchTest.kt` — tokenizer parity for the filesystem store.
@@ -0,0 +1,166 @@
# Searchable kinds — the authoritative implementor table
Every concrete `SearchableEvent` implementor in Quartz, with the exact `indexableContent()`
expression. **Update this file in the same PR as any change to the searchable set or to an
`indexableContent()` body** (see SKILL.md). Verified against the code 2026-08-04.
Counts: 126 concrete classes covering 129 kind values (`GitStatusEvent` spans 4 kinds;
kind 30063 has a collision — see the footnote). File paths are under
`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/`.
Separator legend: **NL** = `joinToString("\n")`, **SP** = `joinToString(" ")`.
| Kind | Class | Package | `indexableContent()` |
|---|---|---|---|
| 0 | MetadataEvent | nip01Core/metadata | `contactMetaData()?.let { listOfNotNull(it.name, it.displayName, it.about, it.nip05, it.lud06, it.lud16, it.website, it.picture, it.banner).joinToString(" ") } ?: ""` (SP) |
| 1 | TextNoteEvent | nip10Notes | `listOfNotNull(subject(), content)` NL |
| 9 | ChatEvent | nipC7Chats | `content` |
| 11 | ThreadEvent | nip7DThreads | `listOfNotNull(title(), content)` NL |
| 14 | ChatMessageEvent | nip17Dm/messages | `content` |
| 20 | PictureEvent | nip68Picture | `listOfNotNull(title(), content)` NL |
| 21 | VideoNormalEvent | nip71Video | inherited `RegularVideoEvent`: `listOfNotNull(title(), content)` NL |
| 22 | VideoShortEvent | nip71Video | inherited `RegularVideoEvent`: `listOfNotNull(title(), content)` NL |
| 24 | PublicMessageEvent | nipA4PublicMessages | `content` |
| 40 | ChannelCreateEvent | nip28PublicChat/admin | `channelInfo().let { listOfNotNull(it.name, it.about, it.picture).joinToString(" ") }` (SP) |
| 41 | ChannelMetadataEvent | nip28PublicChat/admin | same as kind 40 (SP) |
| 42 | ChannelMessageEvent | nip28PublicChat/message | `content` |
| 54 | PodcastEpisodeEvent | nipF4Podcasts/episode | `listOfNotNull(title(), description(), content)` NL |
| 1010 | TextNoteModificationEvent | experimental/edits | `listOfNotNull(content, summary())` NL (content first) |
| 1063 | FileHeaderEvent | nip94FileMetadata | `listOfNotNull(summary(), content)` NL |
| 1065 | FileStorageHeaderEvent | experimental/nip95/header | `listOfNotNull(summary())` NL |
| 1068 | PollEvent | nip88Polls/poll | `buildString { append(content); options().forEach { append('\n').append(it.label) } }` |
| 1111 | CommentEvent | nip22Comments | `(listOf(content) + tags.hashtags())` NL |
| 1163 | ProfileGalleryEntryEvent | experimental/profileGallery | `listOfNotNull(summary())` NL |
| 1301 | WorkoutRecordEvent | experimental/fitness/workout | `listOfNotNull(title(), content)` NL |
| 1311 | LiveActivitiesChatMessageEvent | nip53LiveActivities/chat | `(listOf(content) + tags.hashtags())` NL |
| 1312 | LiveActivitiesRaidEvent | nip53LiveActivities/raid | `content` |
| 1313 | LiveActivitiesClipEvent | nip53LiveActivities/clip | `listOfNotNull(title(), content)` NL |
| 1315 | RoadEventReportEvent | experimental/roadstr/report | `content` |
| 1337 | CodeSnippetEvent | nipC0CodeSnippets | `listOfNotNull(snippetName(), snippetDescription(), content)` NL |
| 1617 | GitPatchEvent | nip34Git/patch | `content` |
| 1618 | GitPullRequestEvent | nip34Git/pr | `listOfNotNull(subject(), content)` NL |
| 1621 | GitIssueEvent | nip34Git/issue | `listOfNotNull(subject(), content)` NL |
| 1622 | GitReplyEvent | nip34Git/reply | `content` |
| 16301633 | GitStatusEvent | nip34Git/status | `content` (open/applied/closed/draft) |
| 1808 | AudioHeaderEvent | experimental/audio/header | `content` |
| 1985 | LabelEvent | nip32Labeling | `(listOf(content) + labels().map { it.label }).filter { it.isNotEmpty() }` NL |
| 2003 | TorrentEvent | nip35Torrents | `listOfNotNull(title(), content)` NL |
| 2004 | TorrentCommentEvent | nip35Torrents | `content` |
| 2473 | BirdDetectionEvent | experimental/birdstar | `listOfNotNull(summary(), speciesName())` NL |
| 3302 | ConcordChatEditEvent | concord/cord03Channels | `content` |
| 5050 | NIP90TextGenerationRequestEvent | nip90Dvms/textGeneration | `inputs().filter { it.type == "prompt" \|\| it.type == "text" }.joinToString(" ") { it.value }` (SP) |
| 5100 | NIP90ImageGenerationRequestEvent | nip90Dvms/imageGeneration | `listOfNotNull(prompt(), negativePrompt()).joinToString(" ")` (SP) |
| 5129 | NappletSnapshotEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
| 5250 | NIP90TextToSpeechRequestEvent | nip90Dvms/textToSpeech | `text() ?: ""` |
| 5302 | NIP90ContentSearchRequestEvent | nip90Dvms/contentSearch | `searchQuery() ?: ""` |
| 5303 | NIP90PeopleSearchRequestEvent | nip90Dvms/peopleSearch | `searchQuery() ?: ""` |
| 6969 | ZapPollEvent | experimental/zapPolls | `buildString { append(content); pollOptionsArray().forEach { append('\n').append(it.descriptor) } }` |
| 8333 | OnchainZapEvent | nipBCOnchainZaps/zap | `content` |
| 9002 | EditMetadataEvent | nip29RelayGroups/moderation | `(listOfNotNull(name(), about()) + hashtags())` NL |
| 9041 | GoalEvent | nip75ZapGoals | `listOfNotNull(summary(), content)` NL |
| 9321 | NutzapEvent | nip61Nutzaps/nutzap | `content` |
| 9734 | LnZapRequestEvent | nip57Zaps | `content` |
| 9735 | LnZapEvent | nip57Zaps | `zapRequest?.content.orEmpty()` — indexes the **embedded 9734's** content |
| 9736 | Bolt12ZapEvent | nipB1Bolt12Zaps/zap | `content` |
| 9737 | Bolt12ZapIntentEvent | nipB1Bolt12Zaps/intent | `content` |
| 9802 | HighlightEvent | nip84Highlights | `listOfNotNull(comment(), context(), content)` NL |
| 10003 | BookmarkListEvent | nip51Lists/bookmarkList | `listOfNotNull(title())` NL |
| 10100 | AgentProfileEvent | buzz/agentProfiles | `profileOrNull()?.let { listOfNotNull(it.name, it.displayName).joinToString("\n") } ?: ""` |
| 10154 | PodcastMetadataEvent | nipF4Podcasts/metadata | `listOfNotNull(title(), description())` NL |
| 11871 | AttestorProficiencyEvent | experimental/attestations/proficiency | `listOfNotNull(description())` NL |
| 12473 | BirdexEvent | experimental/birdstar | `(listOfNotNull(summary()) + speciesNames())` NL |
| 15128 | RootSiteEvent | nip5aStaticWebsites | `listOfNotNull(title(), description())` NL |
| 15129 | RootNappletEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
| 30000 | PeopleListEvent | nip51Lists/peopleList | `listOfNotNull(titleOrName(), description())` NL |
| 30001 | OldBookmarkListEvent | nip51Lists/bookmarkList | `listOfNotNull(title())` NL |
| 30002 | RelaySetEvent | nip51Lists/relaySets | `listOfNotNull(title(), description())` NL |
| 30003 | LabeledBookmarkListEvent | nip51Lists/labeledBookmarkList | `listOfNotNull(titleOrName(), description())` NL |
| 30004 | ArticleCurationSetEvent | nip51Lists/articleCurationSet | `listOfNotNull(title(), description())` NL |
| 30005 | VideoCurationSetEvent | nip51Lists/videoCurationSet | `listOfNotNull(title(), description())` NL |
| 30006 | PictureCurationSetEvent | nip51Lists/pictureCurationSet | `listOfNotNull(title(), description())` NL |
| 30009 | BadgeDefinitionEvent | nip58Badges/definition | `listOfNotNull(name(), description(), content)` NL |
| 30015 | InterestSetEvent | nip51Lists/interestSet | `(listOfNotNull(title(), description()) + publicHashtags())` NL |
| 30017 | StallEvent | nip15Marketplace/stall | `stallData()?.let { listOfNotNull(it.name, it.description).joinToString("\n") } ?: ""` |
| 30018 | ProductEvent | nip15Marketplace/product | `productData()?.let { (listOfNotNull(it.name, it.description) + categories()).joinToString("\n") } ?: ""` |
| 30019 | MarketplaceEvent | nip15Marketplace/marketplace | `marketplaceData()?.let { listOfNotNull(it.name, it.about).joinToString("\n") } ?: ""` |
| 30020 | AuctionEvent | nip15Marketplace/auction | `auctionData()?.let { (listOfNotNull(it.name, it.description) + tags.hashtags()).joinToString("\n") } ?: ""` |
| 30023 | LongTextNoteEvent | nip23LongContent | `listOfNotNull(title(), summary(), content)` NL |
| 30030 | EmojiPackEvent | nip30CustomEmoji/pack | `listOfNotNull(titleOrName(), description(), content)` NL |
| 30054 | Podcasting20EpisodeEvent | nipXXPodcasting20/episode | `(listOfNotNull(title(), description(), content) + topics())` NL |
| 30055 | Podcasting20TrailerEvent | nipXXPodcasting20/trailer | `listOfNotNull(title(), content)` NL |
| 30063 | ReleaseArtifactSetEvent † | nip51Lists/releaseArtifactSet | `listOfNotNull(title(), description())` NL |
| 30175 | PersonaEvent | buzz/apPersonas | `personaOrNull()?.let { listOfNotNull(it.displayName, it.systemPrompt).joinToString("\n") } ?: ""` |
| 30176 | TeamEvent | buzz/teams | `teamOrNull()?.let { listOfNotNull(it.name, it.description, it.instructions).joinToString("\n") } ?: ""` |
| 30177 | ManagedAgentEvent | buzz/managedAgents | `agentOrNull()?.let { listOfNotNull(it.name, it.systemPrompt).joinToString("\n") } ?: ""` |
| 30267 | AppCurationSetEvent | nip51Lists/appCurationSet | `listOfNotNull(title(), description())` NL |
| 30296 | InteractiveStoryPrologueEvent | experimental/interactiveStories | inherited base: `listOfNotNull(title(), summary(), content)` NL |
| 30297 | InteractiveStorySceneEvent | experimental/interactiveStories | inherited base: `listOfNotNull(title(), summary(), content)` NL |
| 30311 | LiveActivitiesEvent | nip53LiveActivities/streaming | `listOfNotNull(title(), summary(), content)` NL |
| 30312 | MeetingSpaceEvent | nip53LiveActivities/meetingSpaces | `listOfNotNull(room(), summary(), content)` NL |
| 30313 | MeetingRoomEvent | nip53LiveActivities/meetingSpaces | `listOfNotNull(title(), summary())` NL |
| 30315 | StatusEvent | nip38UserStatus | `content` |
| 30382 | ContactCardEvent | nip85TrustedAssertions/users | `(listOfNotNull(petName(), summary()) + topics())` NL — public tags only, never the NIP-44 content |
| 30402 | ClassifiedsEvent | nip99Classifieds | `listOfNotNull(title(), summary(), content)` NL |
| 30617 | GitRepositoryEvent | nip34Git/repository | `listOfNotNull(name(), description(), content)` NL |
| 30620 | WorkflowDefEvent | buzz/workflow | `listOfNotNull(name(), content)` NL |
| 30817 | NipTextEvent | experimental/nipsOnNostr | `listOfNotNull(title(), content)` NL |
| 30818 | WikiNoteEvent | nip54Wiki | `listOfNotNull(title(), summary(), content)` NL |
| 31337 | AudioTrackEvent | experimental/audio/track | `listOfNotNull(subject())` NL |
| 31871 | AttestationEvent | experimental/attestations/attestation | `content` |
| 31872 | AttestationRequestEvent | experimental/attestations/request | `content` |
| 31873 | AttestorRecommendationEvent | experimental/attestations/recommendation | `listOfNotNull(description())` NL |
| 31890 | FeedDefinitionEvent | feedDefinition | `title().orEmpty()` |
| 31922 | CalendarDateSlotEvent | nip52Calendar/appt/day | `listOfNotNull(title(), summary(), content)` NL |
| 31923 | CalendarTimeSlotEvent | nip52Calendar/appt/time | `listOfNotNull(title(), summary(), content)` NL |
| 31924 | CalendarEvent | nip52Calendar/calendar | `listOfNotNull(title(), content)` NL |
| 31925 | CalendarRSVPEvent | nip52Calendar/rsvp | `content` |
| 31990 | AppDefinitionEvent | nip89AppHandlers/definition | `appMetaData()?.let { listOfNotNull(it.name, it.username, it.displayName, it.about, it.nip05, it.lud06, it.lud16, it.website, it.picture, it.banner, it.image).joinToString(" ") } ?: ""` (SP) |
| 32267 | SoftwareApplicationEvent | experimental/nip82SoftwareApps/application | `listOfNotNull(name(), summary(), content)` NL |
| 33401 | ExerciseTemplateEvent | experimental/fitness/workout | `listOfNotNull(title(), content)` NL |
| 33863 | FundraiserEvent | experimental/agora | `listOfNotNull(title(), content)` NL |
| 34139 | MusicPlaylistEvent | experimental/music/playlist | `listOfNotNull(title(), description(), content)` NL |
| 34235 | VideoHorizontalEvent | nip71Video | inherited `AddressableVideoEvent`: `listOfNotNull(title(), content)` NL |
| 34236 | VideoVerticalEvent | nip71Video | inherited `AddressableVideoEvent`: `listOfNotNull(title(), content)` NL |
| 34550 | CommunityDefinitionEvent | nip72ModCommunities/definition | `listOfNotNull(name(), description(), rules(), content)` NL |
| 35128 | NamedSiteEvent | nip5aStaticWebsites | `listOfNotNull(title(), description())` NL |
| 35129 | NamedNappletEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
| 36787 | MusicTrackEvent | experimental/music/track | `listOfNotNull(title(), artist(), album(), content)` NL |
| 38000 | MintRecommendationEvent | nip87Ecash/recommendation | `content` |
| 38192 | Ps1SaveEvent | experimental/ps1saves | `listOfNotNull(summary(), saveTitle(), region(), filename())` NL |
| 38383 | P2POrderEvent | nip69P2pOrderEvents | `(listOfNotNull(makerName(), currency()) + paymentMethods().orEmpty()).joinToString(" ")` (SP) |
| 39000 | GroupMetadataEvent | nip29RelayGroups/metadata | `listOfNotNull(name(), about())` NL |
| 39089 | FollowListEvent | nip51Lists/followList | `listOfNotNull(title(), description())` NL |
| 39092 | MediaStarterPackEvent | nip51Lists/mediaStarterPack | `listOfNotNull(title(), description())` NL |
| 39701 | WebBookmarkEvent | nipB0WebBookmarks | `listOfNotNull(title(), description())` NL |
| 40002 | StreamMessageV2Event | buzz/stream | `content` |
| 40100 | CanvasEvent | buzz/stream | `content` |
| 45001 | ForumPostEvent | buzz/forum | `content` |
| 45003 | ForumCommentEvent | buzz/forum | `content` |
| 48106 | HuddleGuidelinesEvent | buzz/huddles | `content` |
**Kind 30063 collision:** `experimental/nip82SoftwareApps/release/SoftwareReleaseEvent` also
declares `KIND = 30063` and implements `SearchableEvent` (`content`), but `EventFactory` maps
30063 to `ReleaseArtifactSetEvent`, so on every store path kind 30063 indexes
`title()\ndescription()`. If the factory mapping ever changes, this table changes with it.
## Abstract bases (no kind of their own)
| Base class | Body | Concrete kinds |
|---|---|---|
| `InteractiveStoryBaseEvent` | `listOfNotNull(title(), summary(), content)` NL | 30296, 30297 |
| `AddressableVideoEvent` | `listOfNotNull(title(), content)` NL | 34235, 34236 |
| `RegularVideoEvent` | `listOfNotNull(title(), content)` NL | 21, 22 |
## How to regenerate / verify this table
```bash
# All implementor files:
grep -rln "override fun indexableContent" quartz/src/commonMain
# For each, pair the KIND constant with the indexableContent() body.
# Searchability on the store path additionally requires EventFactory registration:
grep -n "<ClassName>" quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/EventFactory.kt
```
A CI-diffable snapshot test (assert the set of kinds whose `EventFactory` product implements
`SearchableEvent` against a checked-in list) would make this table impossible to go stale —
suggested follow-up, not yet implemented.
+50 -18
View File
@@ -26,8 +26,12 @@ env:
# bundle deps — that fights jpackage's self-contained JRE (libjvm.so has
# $ORIGIN RPATH so ldd can't resolve it standalone). appimagetool only
# embeds the AppDir as-is, which is what we actually want.
APPIMAGETOOL_URL: https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-x86_64.AppImage
APPIMAGETOOL_SHA256: 46fdd785094c7f6e545b61afcfb0f3d98d8eab243f644b4b17698c01d06083d1
#
# Both arch binaries come from the same appimagetool release so their SHA256
# values move in lockstep on version bumps.
APPIMAGETOOL_VERSION: '1.9.0'
APPIMAGETOOL_SHA256_X86_64: 46fdd785094c7f6e545b61afcfb0f3d98d8eab243f644b4b17698c01d06083d1
APPIMAGETOOL_SHA256_AARCH64: 04f45ea45b5aa07bb2b071aed9dbf7a5185d3953b11b47358c1311f11ea94a96
jobs:
# ---------------------------------------------------------------------------
@@ -38,11 +42,17 @@ jobs:
strategy:
fail-fast: false
matrix:
# Linux legs run on x64 and arm64 GitHub-hosted runners (the
# ubuntu-24.04-arm label is a standard free public-repo runner as of
# early 2025). jpackage / jlink / Compose Multiplatform 1.11 all
# produce host-native artifacts — no cross-compilation needed.
include:
- { os: macos-14, arch: arm64, family: macos, tasks: "packageReleaseDmg" }
- { os: windows-latest, arch: x64, family: windows, tasks: "packageReleaseMsi createReleaseDistributable" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
- { os: ubuntu-latest, arch: x64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
- { os: macos-14, arch: arm64, family: macos, tasks: "packageReleaseDmg" }
- { os: windows-latest, arch: x64, family: windows, tasks: "packageReleaseMsi createReleaseDistributable" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
- { os: ubuntu-latest, arch: x64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
- { os: ubuntu-24.04-arm, arch: arm64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
runs-on: ${{ matrix.os }}
timeout-minutes: 60 # linux-portable leg also downloads the freedesktop runtime + builds the Flatpak bundle
defaults:
@@ -93,13 +103,21 @@ jobs:
set -euo pipefail
# appimagetool 1.9.0 validates the .desktop file via desktop-file-validate.
sudo apt-get update && sudo apt-get install -y desktop-file-utils
curl -fsSL --retry 3 "$APPIMAGETOOL_URL" -o desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
actual=$(sha256sum desktopApp/packaging/appimage/appimagetool-x86_64.AppImage | awk '{print $1}')
if [[ "$actual" != "$APPIMAGETOOL_SHA256" ]]; then
echo "::error::appimagetool SHA256 mismatch. Expected $APPIMAGETOOL_SHA256, got $actual"
# Map runner arch → upstream AppImage suffix (x86_64 / aarch64).
case "${{ matrix.arch }}" in
x64) TOOL_ARCH=x86_64 ; EXPECTED_SHA="$APPIMAGETOOL_SHA256_X86_64" ;;
arm64) TOOL_ARCH=aarch64; EXPECTED_SHA="$APPIMAGETOOL_SHA256_AARCH64" ;;
*) echo "::error::unsupported arch for AppImage: ${{ matrix.arch }}"; exit 1 ;;
esac
URL="https://github.com/AppImage/appimagetool/releases/download/${APPIMAGETOOL_VERSION}/appimagetool-${TOOL_ARCH}.AppImage"
DEST="desktopApp/packaging/appimage/appimagetool-${TOOL_ARCH}.AppImage"
curl -fsSL --retry 3 "$URL" -o "$DEST"
actual=$(sha256sum "$DEST" | awk '{print $1}')
if [[ "$actual" != "$EXPECTED_SHA" ]]; then
echo "::error::appimagetool SHA256 mismatch for $TOOL_ARCH. Expected $EXPECTED_SHA, got $actual"
exit 1
fi
chmod +x desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
chmod +x "$DEST"
# Flatpak tooling + the freedesktop runtime/sdk the manifest pins
# (runtime-version is greped from the manifest so this never drifts).
@@ -208,12 +226,13 @@ jobs:
run: |
set -euo pipefail
VER="${{ steps.ver.outputs.version }}"
ARCH="${{ matrix.arch }}"
APP="desktopApp/build/compose/binaries/main-release/app"
mkdir -p desktopApp/build/portable
if [[ "${{ matrix.family }}" == "windows" ]]; then
( cd "$APP" && 7z a -tzip "../../../../portable/amethyst-desktop-${VER}-windows-x64.zip" Amethyst/ )
( cd "$APP" && 7z a -tzip "../../../../portable/amethyst-desktop-${VER}-windows-${ARCH}.zip" Amethyst/ )
else
( cd "$APP" && tar czf "../../../../portable/amethyst-desktop-${VER}-linux-x64.tar.gz" Amethyst/ )
( cd "$APP" && tar czf "../../../../portable/amethyst-desktop-${VER}-linux-${ARCH}.tar.gz" Amethyst/ )
fi
# Flatpak bundle: wraps the same createReleaseDistributable tree the
@@ -230,6 +249,17 @@ jobs:
PKG="desktopApp/packaging/flatpak"
APP_ID="com.vitorpamplona.amethyst.Desktop"
OUT="desktopApp/build/flatpak"
# AppImage-style arch names for the bundle filename.
case "${{ matrix.arch }}" in
x64) BUNDLE_ARCH=x86_64 ; GST_TRIPLET=x86_64-linux-gnu ;;
arm64) BUNDLE_ARCH=aarch64 ; GST_TRIPLET=aarch64-linux-gnu ;;
*) echo "::error::unsupported arch for Flatpak: ${{ matrix.arch }}"; exit 1 ;;
esac
# Rewrite the arch-specific GStreamer plugin path in the manifest
# (checked-in default is x86_64-linux-gnu). Idempotent — the sed only
# matches the original triplet.
sed -i "s|/usr/lib/x86_64-linux-gnu/gstreamer-1.0|/usr/lib/${GST_TRIPLET}/gstreamer-1.0|g" \
"${PKG}/${APP_ID}.yml"
# Inject the AppStream <release> entry for this build (the checked-in
# metainfo deliberately carries none — CI is the source of truth).
sed -i "s|<releases>|<releases>\n <release version=\"${VER}\" date=\"$(date -u +%F)\" />|" \
@@ -241,7 +271,7 @@ jobs:
"${OUT}/build-dir" \
"${PKG}/${APP_ID}.yml"
flatpak build-bundle "${OUT}/repo" \
"${OUT}/Amethyst-${VER}-x86_64.flatpak" \
"${OUT}/Amethyst-${VER}-${BUNDLE_ARCH}.flatpak" \
"$APP_ID" \
--runtime-repo=https://dl.flathub.org/repo/flathub.flatpakrepo
ls -la "$OUT"
@@ -325,8 +355,9 @@ jobs:
fail-fast: false
matrix:
include:
- { os: macos-14, arch: arm64, family: macos, tasks: "amyImage" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
- { os: macos-14, arch: arm64, family: macos, tasks: "amyImage" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
runs-on: ${{ matrix.os }}
timeout-minutes: 45 # macOS leg also codesigns + notarizes the jlink image
defaults:
@@ -574,8 +605,9 @@ jobs:
fail-fast: false
matrix:
include:
- { os: macos-14, arch: arm64, family: macos, tasks: "geodeImage" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
- { os: macos-14, arch: arm64, family: macos, tasks: "geodeImage" }
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
runs-on: ${{ matrix.os }}
timeout-minutes: 45 # macOS leg also codesigns + notarizes the jlink image
defaults:
+11 -2
View File
@@ -49,9 +49,17 @@ jobs:
# package, installs it, and verifies the process stays alive for 10s.
# Catches ProGuard stripping (JNI, reflection), missing jlink modules
# (java.management, java.prefs), and native lib bundling issues.
#
# Runs on both x64 and arm64 hosted runners so release-time arm64 breakage
# (e.g. ProGuard rules missing an arch-specific reflection root) is caught
# at PR time instead of on the tag build.
# -------------------------------------------------------------------------
release-deb-launch:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-24.04-arm]
runs-on: ${{ matrix.os }}
timeout-minutes: 45
steps:
- name: Checkout code
@@ -139,5 +147,6 @@ jobs:
if: always()
uses: actions/upload-artifact@v7
with:
name: Release DEB (smoke-tested)
# Artifact names must be unique across a run — disambiguate per arch.
name: Release DEB (smoke-tested, ${{ matrix.os }})
path: desktopApp/build/compose/binaries/main-release/deb/*.deb
+8 -5
View File
@@ -57,9 +57,12 @@ Install appimagetool locally (CI fetches its own — SHA-verified):
# Debian/Ubuntu — appimagetool calls desktop-file-validate on the .desktop entry
sudo apt-get install -y desktop-file-utils
curl -fsSL -o desktopApp/packaging/appimage/appimagetool-x86_64.AppImage \
https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-x86_64.AppImage
chmod +x desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
# createReleaseAppImage picks appimagetool-<arch>.AppImage matching the JVM's
# os.arch — fetch the one for your host (x86_64 on Intel/AMD, aarch64 on ARM).
ARCH="$(uname -m)"
curl -fsSL -o "desktopApp/packaging/appimage/appimagetool-${ARCH}.AppImage" \
"https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-${ARCH}.AppImage"
chmod +x "desktopApp/packaging/appimage/appimagetool-${ARCH}.AppImage"
```
---
@@ -110,8 +113,8 @@ are **not** required to build Amethyst from the committed sources.
| Windows MSI | `./gradlew :desktopApp:packageReleaseMsi` | `desktopApp/build/compose/binaries/main-release/msi/Amethyst-*.msi` |
| Linux `.deb` | `./gradlew :desktopApp:packageReleaseDeb` | `desktopApp/build/compose/binaries/main-release/deb/amethyst_*.deb` |
| Linux `.rpm` | `./gradlew :desktopApp:packageReleaseRpm` | `desktopApp/build/compose/binaries/main-release/rpm/amethyst-*.rpm` |
| Linux AppImage | `./gradlew :desktopApp:createReleaseAppImage` | `desktopApp/build/appimage/Amethyst-*-x86_64.AppImage` |
| Linux Flatpak | `flatpak-builder` over `createReleaseDistributable` output — see [`desktopApp/packaging/flatpak/README.md`](desktopApp/packaging/flatpak/README.md) | `desktopApp/build/flatpak/Amethyst-*-x86_64.flatpak` (CI) |
| Linux AppImage | `./gradlew :desktopApp:createReleaseAppImage` | `desktopApp/build/appimage/Amethyst-*-<arch>.AppImage` (x86_64 or aarch64, from host) |
| Linux Flatpak | `flatpak-builder` over `createReleaseDistributable` output — see [`desktopApp/packaging/flatpak/README.md`](desktopApp/packaging/flatpak/README.md) | `desktopApp/build/flatpak/Amethyst-*-<arch>.flatpak` (CI; x86_64 or aarch64) |
| Windows `.zip` portable | See below (inline `7z`) | — |
| Linux `.tar.gz` portable | See below (inline `tar`) | — |
@@ -67,7 +67,9 @@ class RoleBasedHttpClientBuilder(
normalizedUrl: String,
final: Boolean,
): Boolean =
if (RelayUrlNormalizer.isLocalHost(normalizedUrl)) {
if (RelayUrlNormalizer.isLocalHost(normalizedUrl) || RelayUrlNormalizer.isOverlayNetwork(normalizedUrl)) {
// Overlay-mesh hosts (0200::/7) are reachable only through the local mesh
// interface — Tor cannot route the range, so proxying only breaks the fetch.
false
} else if (RelayUrlNormalizer.isOnion(normalizedUrl)) {
true
@@ -113,7 +115,7 @@ class RoleBasedHttpClientBuilder(
isOnionRelaysActive: Boolean,
final: Boolean,
): Boolean =
if (RelayUrlNormalizer.isLocalHost(normalizedUrl)) {
if (RelayUrlNormalizer.isLocalHost(normalizedUrl) || RelayUrlNormalizer.isOverlayNetwork(normalizedUrl)) {
false
} else if (RelayUrlNormalizer.isOnion(normalizedUrl)) {
isOnionRelaysActive
@@ -170,6 +170,7 @@ fun RelayUrlEditField(
nav: INav,
) {
var url by remember { mutableStateOf("") }
var isInvalid by remember { mutableStateOf(false) }
fun submitRelay() {
if (url.isNotBlank()) {
@@ -177,7 +178,13 @@ fun RelayUrlEditField(
if (relay != null) {
onNewRelay(relay)
url = ""
isInvalid = false
relaySuggestions.reset()
} else {
// Without this the Add button is a silent no-op, which reads as a broken button.
// Bare IPv6 literals are the common way to land here: an overlay-mesh address
// pasted straight out of `yggdrasilctl getSelf` needs brackets to carry a port.
isInvalid = true
}
}
}
@@ -189,8 +196,23 @@ fun RelayUrlEditField(
value = url,
onValueChange = {
url = it
isInvalid = false
relaySuggestions.processInput(it)
},
isError = isInvalid,
// Null, not an empty lambda: a non-null slot reserves its line height even when it
// draws nothing, which would pad the field permanently for every user.
supportingText =
if (isInvalid) {
{
Text(
text = stringRes(R.string.relay_url_not_valid),
color = MaterialTheme.colorScheme.error,
)
}
} else {
null
},
placeholder = {
Text(
text = "server.com",
@@ -332,7 +332,7 @@
<string name="concord_leave_title">क्या समुदाय छोडें।</string>
<string name="concord_leave_message">क्या %1$s छोडें। इसे हटाया जाएगा इस लेखा की सूची से तथा आपके यन्त्रों पर समचरणीकरण रुक जाएगा। समुदाय को सूचित नहीं किया जाएगा। तथा आपको उसके सदस्य कार्यसूची से हटाया नहीं जाएगा। सन्देश जिनका आप अरहस्यीकरण नहीं कर सकेंगे सम्भाव्यतः पुनःप्राप्तव्य नहीं होंगे। तथा आप केवल नए आमन्त्रण के साथ लौट सकेंगे।</string>
<string name="concord_leave_owner_warning">आपने इस समुदाय को बनाया। छोड जाने से यह मिटेगा नहीं। किसी अन्य के हाथ सौंपा नहीं जाएगा। परन्तु स्वत्वधारी कुंचिका जो आपकी सूची में हैं वह हटाया जाएगा। आप आगे से इसका प्रबन्धन नहीं कर पाएँगे।</string>
<string name="concord_edit_relays_desc">जहाँ इस समुदाय के रहस्यीकृत पत्रों का प्रकाशन तथा पठन किया जाता है।</string>
<string name="concord_edit_relays_desc">जहाँ इस समुदाय के रहस्यीकृत समतलों का प्रकाशन तथा पठन किया जाता है।</string>
<string name="concord_dissolved_read_only">इस समुदाय को विघटित किया गया है तथा अब पठनेवशक्य है। आप इसका इतिहास पढ सकते हैं परन्तु कोई नए सन्देश नहीं भेज सकते।</string>
<string name="concord_typing_one">%1$s टंकण मध्य…</string>
<string name="concord_typing_two">%1$s तथा %2$s टंकण मध्य…</string>
@@ -1233,7 +1233,7 @@
<string name="nest_participant_unmute">मौन हटाएँ</string>
<string name="nest_force_mute_note">सम्भाव्यतः उन ग्राहकों द्वारा उपेक्षित जो आज्ञा का सम्मान नहीं करते।</string>
<string name="nest_confirm_kick_title">क्या शाला से निष्कासित करें।</string>
<string name="nest_confirm_kick_body">%1$s को ध्वनि तल से हटाए जाएँगे तथा सहभागी सूची से भी। वे पुनः जुड सकते हैं यदि वे शाला योजक प्राप्त कर लें।</string>
<string name="nest_confirm_kick_body">%1$s को ध्वनि समतल से हटाए जाएँगे तथा सहभागी सूची से भी। वे पुनः जुड सकते हैं यदि वे शाला योजक प्राप्त कर लें।</string>
<string name="nest_confirm_kick_confirm">पदप्रहार</string>
<string name="nest_confirm_force_mute_title">क्या वक्ता को मौन करें।</string>
<string name="nest_confirm_force_mute_body">%1$s के ग्राहक को अपना ध्वनिग्राहक मौन करने का अनुरोध करता है। कुछ ग्राहक इस आदेश की उपेक्षा कर सकते हैं।</string>
@@ -1942,14 +1942,95 @@
</plurals>
<!-- Expanded-only breakdown of the always-on notification. Counts overlap: one relay commonly
serves several jobs at once, so these deliberately sum to more than the relay count. -->
<plurals name="relay_purpose_line">
<item quantity="one">%1$s \u00b7 %2$d पुनःप्रसारक</item>
<item quantity="other">%1$s \u00b7 %2$d पुनःप्रसारक</item>
</plurals>
<string name="relay_purpose_browsing">जालभ्रमण</string>
<string name="relay_purpose_media">ध्वनिचित्राभिलेख</string>
<string name="relay_purpose_tags">विषयसूचक</string>
<string name="relay_purpose_topics">विषय सूची</string>
<string name="relay_purpose_thread">वार्तालाप</string>
<string name="relay_purpose_search">खोज</string>
<string name="relay_purpose_referenced">लुप्त घटनाओं को ढूँढें</string>
<string name="relay_purpose_engagement">घटना अवलोकन</string>
<string name="relay_explain_referenced">घटनाओं को विभेदक अनुसार ले आता है जिसका उल्लेख आपके पटल पर अमुक करता है पर जिसकी प्राप्ती अभी नहीं हुई। एक उद्धरण अथवा एक प्रत्युत्तर का पूर्वपत्र अथवा एक सूत्र का मूल।</string>
<string name="relay_explain_engagement">घटनाओं का अवलोकन करता है जो वर्तमान में प्रदर्शित हो रहे हैं नए प्रत्युत्तर प्रतिक्रियाएँ उद्धरण ज्साप तथा वृत्तान्तों के लिए जिससे गिनतियों का नवीकरण होता है जब आप पढ रहे हैं।</string>
<string name="relay_purpose_add_ons">संलग्न</string>
<string name="relay_purpose_relay_info">पुनःप्रसारक जानकारी</string>
<string name="relay_purpose_other">अन्य</string>
<!-- Activity labels, used where the app's own noun for the data type already means something
else to the user: "Outbox Relays" is their own relay list in settings, and "Profile" is
their profile screen. Naming the job avoids an active misreading. -->
<string name="relay_purpose_relay_list_finder">पुनःप्रसारक सूची खोजकर्ता</string>
<string name="relay_purpose_observing_profiles">परिचय अवलोकन</string>
<string name="relay_purpose_your_account">लेखा जानकारी</string>
<string name="relay_purpose_home_feed">मुख्य सूचनावली</string>
<string name="relay_purpose_relay_groups">पुनःप्रसारक समूह</string>
<plurals name="active_subs_groups">
<item quantity="one">%1$d समूह</item>
<item quantity="other">%1$d समूह</item>
</plurals>
<string name="relay_purpose_ephemeral_chats">अस्थायी चर्चाएँ</string>
<string name="relay_purpose_geohash_chats">स्थानीय चर्चाएँ</string>
<string name="relay_purpose_live_chat">वर्तमानप्रवाह चर्चा</string>
<string name="relay_explain_relay_groups">निप॰२९ समूह जिनसे आप जुडे हैं। प्रत्येक समूह एक जालावास पुनःप्रसारक में रहता है। इसलिए क्रमक प्रत्येक पुनःप्रसारक से संयोजन करता है जो आपके किसी समूह का जालावास है।</string>
<string name="relay_explain_ephemeral_chats">चर्चाशालाएँ जो कोई इतिहास नहीं रखते। सन्देश केवल तब तक रहते हैं जब तक आप संयोजित हैं। इसलिए ये ग्राहकता बनाए रखते हैं कुछ भी प्राप्त होने के लिए।</string>
<string name="relay_explain_geohash_chats">स्थान आधारित शालाएँ उन क्षेत्रों के लिए जिनका आप अनुगमन करते हैं। पृष्ट उन पुनःप्रसारको से जो इनके जालावास हैं।</string>
<string name="relay_explain_live_chat">चर्चा तथा ज्साप उद्देश्य जो वर्तमानप्रवाहों से संलग्न हैं जिन्हें आप खोले हुए हैं अथवा अनुगमन करते हैं।</string>
<string name="relay_purpose_dm_inbox">सीधासन्देश आगतपेटिका</string>
<string name="relay_purpose_your_wallet">धनकोष</string>
<string name="relay_purpose_nutzap_inbox">नटज्साप आगतपेटिका</string>
<string name="relay_purpose_mint_directory">टकसाल निर्देशिका</string>
<string name="relay_purpose_nwc">धनकोष संयोजन</string>
<string name="relay_purpose_community_chats">समुदाय चर्चाएँ</string>
<string name="relay_purpose_community_feeds">समुदाय सूचनावलियाँ</string>
<!-- How each subscription actually works, shown on the Active Subscriptions screen.
Describe the real strategy, not the intent — these are read by people trying to explain
a relay count they think is too high. -->
<string name="relay_explain_notifications">आपके आगतपेटिका पुनःप्रसारक तथा कुछ अल्पमात्रा परिभ्रमणवर्ती दृष्टान्त पुनःप्रसारक जिनपर आपके अनुचरित पत्र प्रकाशित करते हैं। यदि कोई उल्लेख अन्यत्र भेजा गया।</string>
<string name="relay_explain_direct_messages">आपके सीधासन्देश पुनःप्रसारक। जहाँ उपहारकोषयुक्त सन्देश भेजे जाते हैं।</string>
<string name="relay_explain_public_chats">मुख्य पुनःप्रसारक प्रत्येक चर्चा का जिन्हें आप खोल रखे हैं अथवा जिनसे आप जुड चुके हैं।</string>
<string name="relay_explain_community_chats">पुनःप्रसारक जिनपर प्रत्येक समुदाय अपने समतलों को प्रकाशित करते हैं।</string>
<string name="relay_explain_encrypted_groups">समूह सन्देश तथा कुंचिकापेटलियाँ। प्रत्येक समूह के पुनःप्रसारकों पर।</string>
<string name="relay_explain_live_rooms">शाला के पुनःप्रसारक। जब वह खुला हो।</string>
<string name="relay_explain_account_data">आपके अपने परिचय तथा स्थापना विकल्प तथा पाण्डुलिपियाँ। आपके मुख्य पुनःप्रसारकों पर।</string>
<string name="relay_explain_profiles">वर्तमानतः पटल पर लोगों के परिचय।</string>
<string name="relay_explain_relay_lists">खोजता है किन पुनःप्रसारकों पर प्रत्येक व्यक्ति प्रकाशन करता है। जिससे कि उनके पत्र सम्यक स्थल से प्राप्प हो।</string>
<string name="relay_explain_follows">अनुचरण सूचियाँ। आपकी सूचनावली तथा आपका विश्वासजाल का निर्माण के लिए उपयुक्त।</string>
<string name="relay_explain_moderation">वृत्तान्त जो आपके अनुचरितों ने लिखा वर्तमानतः आपके पटल पर दिखनेवाले परिचयों के विषय में। पृष्ट प्रत्येक पुनःप्रसारक से जिनपर वे पत्र प्रकाशन करते हैं।</string>
<string name="relay_purpose_reports_from_follows">अनुचरित से वृत्तान्त</string>
<string name="relay_explain_wallet">आपके अपने धनकोष घटनाएँ। पुनःपठित उन पुनःप्रसारकों से जिनपर आपने उनके प्रकाशन किए।</string>
<string name="relay_explain_nutzap_inbox">सुनता है आपके नटज्साप पुनःप्रसारकों पर तथा आपके आगतपेटिका तथा सीधासन्देश पुनःप्रसारकों पर। जिससे कि कोई भी भुगतान छूट ना जाए।</string>
<string name="relay_explain_mint_directory">पुनःप्रसारकों का वीक्षण करता है यह देखने के लिए कि कौनसे टकसाल हैं तथा लोग किनकी अनुशम्सा करते हैं।</string>
<string name="relay_explain_nwc">आपके संयोजित धनकोष से सूचनाएँ।</string>
<string name="active_subs_title">सक्रिय पुनःप्रसारक ग्राहकताएँ</string>
<!-- Two countable nouns, so two plurals composed at the call site rather than one string with
two %d in it: filter and relay decline independently in Slavic/Baltic/Semitic languages. -->
<plurals name="active_subs_filters">
<item quantity="one">%1$d छलनी</item>
<item quantity="other">%1$d छलनियाँ</item>
</plurals>
<plurals name="active_subs_relays">
<item quantity="one">%1$d पुनःप्रसारक</item>
<item quantity="other">%1$d पुनःप्रसारक</item>
</plurals>
<plurals name="active_subs_untagged">
<item quantity="one">%1$d छलनी आरोपित नहीं अब तक</item>
<item quantity="other">%1$d छलनियाँ आरोपित नहीं अब तक</item>
</plurals>
<string name="active_subs_pair">%1$s \u00b7 %2$s</string>
<string name="active_subs_unattributed">किसी लेखा प्रति आरोपित नहीं</string>
<string name="active_subs_no_entity">सभी</string>
<string name="active_subs_scope_global">सभी</string>
<string name="active_subs_scope_follows">आपके द्वारा अनुचरित लोग</string>
<string name="active_subs_scope_authors">चयनित लोगों की सूची</string>
<string name="active_subs_scope_muted">मौनकृत लोग</string>
<string name="active_subs_scope_all_communities">आपके समुदाय</string>
<string name="active_subs_scope_algo">एक प्रिय कलनविधि सूचनावली</string>
<string name="active_subs_share">%1$dप्रतिशतप्रतिशत सब में से</string>
<string name="active_subs_search_keywords">ग्राहकताएँ छलनियाँ पुनःप्रसारक अनुरोध अनु॰ संयोजन क्यों निदानतन्त्र</string>
<string name="relay_explain_home">आपके अनुचरितों के पत्र। पठित उन पुनःप्रसारकों से जिनपर उनमें से प्रत्येक प्रकाशन करते हैं।</string>
<string name="always_on_notif_connecting">आगतपेटिका पुनःप्रसारकों के साथ संयोजन किया जा रहा है \u2026</string>
<string name="always_on_notif_setting_title">सदैव सक्रिय सूचना सेवा</string>
<string name="always_on_notif_setting_description">अनवरत संयोजन बनाए रखता है आपके आगतपेटिका पुनःप्रसारकों के साथ तत्काल सूचना वितरण के लिए। एक स्थायी सूचना दिखाता है। विद्युत्कोष का अधिक उपयोग करता है पर निश्चित करता है कि आप कभी भी सन्देश नहीं खोएँगे।</string>
File diff suppressed because it is too large Load Diff
+1
View File
@@ -214,6 +214,7 @@
<string name="connection_success_rate_description">Percentage of successful connections to the relay</string>
<string name="search_and_add_a_user">Search and add user</string>
<string name="add_a_relay">Add a Relay</string>
<string name="relay_url_not_valid">Not a valid relay address. Use a host name, or an IP address in brackets (for example [201:d0e:9ba5:8bbc::1]:8080).</string>
<string name="my_name">My @tag name</string>
<string name="display_name">Display Name</string>
<string name="my_display_name">My display name</string>
@@ -28,6 +28,7 @@ import com.vitorpamplona.quartz.marmot.mip00KeyPackages.KeyPackageRelayListEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrlOrNull
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
import com.vitorpamplona.quartz.nip11RelayInfo.Nip11RelayInformation
@@ -50,6 +51,7 @@ import com.vitorpamplona.quartz.nip65RelayList.tags.AdvertisedRelayType
import com.vitorpamplona.quartz.nip66RelayMonitor.reachability.RelayProber
import okhttp3.OkHttpClient
import okhttp3.Request
import java.io.File
/**
* `amy relay …` — manage every relay list this account maintains, mirroring
@@ -112,10 +114,12 @@ object RelayCommands {
| relay info URL fetch + print a relay's NIP-11 info document (stateless)
| relay probe [--timeout SECS] relay census: mass-connect every relay the store
| [--concurrency N] knows and record live/dead + measured rtt-open
| into the reachability cache (NIP-66 kind:30166),
| [--file PATH] into the reachability cache (NIP-66 kind:30166),
| so reachability-aware commands (graperank crawl/
| refresh) skip dead relays and wait once
| (--timeout: per wave, default 15s)
| (--timeout: per wave, default 15s; --file: also
| probe candidate urls, one per line, each run
| through the relay url normalizer first)
""".trimMargin()
// ------------------------------------------------------------------
@@ -337,12 +341,41 @@ object RelayCommands {
// Relays dialed at once; --relay-concurrency accepted as the alias the
// graperank verbs spell it with.
val waveSize = args.intFlag("concurrency", args.intFlag("relay-concurrency", Context.defaultPreconnectCap))
// Optional external candidate list: one raw url per line, run through the
// same RelayUrlNormalizer the app uses, so a probe doubles as a census of
// how a corpus of relay hints normalizes (rejects are counted, not dialed).
val fromFile = args.flag("file")
args.rejectUnknown()
var fileRaw = 0
var fileRejected = 0
var fileOnion = 0
val fileRelays = HashSet<NormalizedRelayUrl>()
if (fromFile != null) {
val candidates = File(fromFile)
if (!candidates.canRead()) return Output.error("bad_args", "cannot read --file $fromFile")
candidates.forEachLine { line ->
if (line.isBlank()) return@forEachLine
fileRaw++
val normalized = line.normalizeRelayUrlOrNull()
if (normalized == null) {
fileRejected++
} else if (RelayUrlNormalizer.isOnion(normalized.url)) {
fileOnion++
} else {
fileRelays.add(normalized)
}
}
System.err.println(
"[relay-probe] $fromFile: $fileRaw urls → ${fileRelays.size} unique clearnet relays " +
"($fileRejected rejected by the normalizer, $fileOnion onion skipped)",
)
}
Context.openOrAnonymous(dataDir).use { ctx ->
ctx.prepare()
val cached = ctx.reachability.snapshot()
val universe = RelayProber.knownRelayUniverse(ctx.store) + cached.live + cached.dead
val universe = RelayProber.knownRelayUniverse(ctx.store) + cached.live + cached.dead + fileRelays
if (universe.isEmpty()) {
Output.emit(
linkedMapOf<String, Any?>(
@@ -381,6 +414,10 @@ object RelayCommands {
Output.emit(
linkedMapOf<String, Any?>(
"probed" to result.verdicts.size,
"file_urls" to (if (fromFile != null) fileRaw else null),
"file_normalized" to (if (fromFile != null) fileRelays.size else null),
"file_rejected" to (if (fromFile != null) fileRejected else null),
"file_onion_skipped" to (if (fromFile != null) fileOnion else null),
"reachable" to result.reachable.size,
"dead" to result.dead.size,
"closed_by_policy" to authWalled,
@@ -1,17 +1,95 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<!-- Login & Auth -->
<string name="login_title">Üdvözöljük az Amethystben</string>
<string name="login_subtitle">Jelentkezzen be Nostr a-fiókjába</string>
<string name="login_subtitle_desktop">Asztali Nostr-kliens</string>
<string name="login_card_title">Jelentkezzen be Nostr-a kulcsával</string>
<string name="login_card_subtitle">nsec a teljes hozzáféréshez, bunker:// a távoli aláíróhoz, vagy npub a csak olvasható módhoz</string>
<string name="login_with_key">Bejelentkezés kulccsal</string>
<string name="login_button">Bejelentkezés</string>
<string name="login_generate_new">Új kulcs előállítása</string>
<string name="login_generate_button">Új előállítása</string>
<string name="login_key_hint">Adja meg a privát kulcsát (nsec) vagy a nyilvános kulcsát (npub)</string>
<string name="login_key_label">nsec, bunker:// vagy npub</string>
<string name="login_key_placeholder">nsec1… / bunker://… / npub1…</string>
<string name="login_show_key">Kulcs megjelenítése</string>
<string name="login_hide_key">Kulcs elrejtése</string>
<!-- New Key Warning -->
<string name="new_key_warning_title">FONTOS: Mentse el a kulcsait!</string>
<string name="new_key_warning_message">A titkos kulcsa (nsec) az EGYETLEN módja annak, hogy hozzáférjen a fiókjához. Ha elveszíti, akkor a fiókja végleg elvész. Mentse el egy biztonságos helyre!</string>
<string name="new_key_public_label">Nyilvános kulcs (megosztható):</string>
<string name="new_key_secret_label">Titkos kulcs (SOHA ne ossza meg!):</string>
<string name="new_key_continue_button">Elmentettem a kulcsaimat, folytatás</string>
<!-- Common Actions -->
<string name="action_copy">Másolás</string>
<string name="action_paste">Beillesztés</string>
<string name="action_cancel">Mégse</string>
<string name="action_ok">OK</string>
<string name="action_save">Mentés</string>
<string name="action_delete">Törlés</string>
<string name="action_share">Megosztás</string>
<!-- Errors -->
<string name="error_invalid_key">Érvénytelen kulcsformátum. Ellenőrizze, és próbálja újra.</string>
<string name="error_network">Hálózati hiba. Ellenőrizze a kapcsolatot.</string>
<string name="error_generic">Hiba történt. Próbálja újra.</string>
<!-- Loading & Empty States -->
<string name="action_refresh">Frissítés</string>
<string name="action_try_again">Próbálja újra</string>
<string name="feed_empty">A hírfolyam üres</string>
<string name="error_loading_feed">Hiba történt a hírfolyam betöltésekor: %s</string>
<!-- Placeholder Screens -->
<string name="screen_search_title">Keresés</string>
<string name="screen_search_description">Keressen felhasználókat, bejegyzéseket és kulcsszavakat.</string>
<string name="screen_messages_title">Üzenetek</string>
<string name="screen_messages_description">Az Ön titkosított közvetlen üzenetei itt fognak megjelenni.</string>
<string name="screen_notifications_title">Értesítések</string>
<string name="screen_notifications_description">Az említések, válaszok és reakciók itt fognak megjelenni.</string>
<!-- Accessibility -->
<string name="accessibility_user_avatar">Felhasználó profilképe</string>
<string name="accessibility_navigate">Navigáció</string>
<!-- Relay history paging (shared feed markers + status card) -->
<string name="chats_history_loading_label">Betöltés:</string>
<string name="chats_history_fully_loaded_label">Teljesen betöltve:</string>
<string name="chats_history_fully_loaded">(teljesen betöltve)</string>
<string name="chats_history_by_relay">Előzmények átjátszónként</string>
<string name="chats_history_stalled_retry">Újra megpróbálja, amint újra megnyitja ezt a képernyőt</string>
<string name="chats_history_older">%1$s korábbi üzenet</string>
<string name="chats_history_all_caught_up">Naprakész</string>
<string name="chats_history_reached_start">Elérte a(z) %1$s üzeneteinek elejét</string>
<string name="chats_history_subtitle">%1$s · %2$s · betöltve ekkortól: %3$s</string>
<string name="chats_history_subtitle_no_date">%1$s · %2$s</string>
<string name="chats_history_waiting">várakozás erre: %1$s</string>
<string name="chats_history_incomplete">Néhány átjátszó nem válaszolt</string>
<string name="chats_history_incomplete_sub">%1$s nem érhető el · koppintson a részletekért</string>
<string name="chats_history_relays_title">%1$s · előzmények átjátszónként</string>
<string name="chats_history_relay_since">ekkortól: %1$s</string>
<string name="action_dismiss">Eltüntetés</string>
<plurals name="chats_history_relays">
<item quantity="one">%1$d átjátszó</item>
<item quantity="other">%1$d relé</item>
</plurals>
<!-- Notes & Replies -->
<string name="replying_to">válasz neki: </string>
<!-- Static sites (NIP-5A) & napplets (NIP-5D) feed card -->
<string name="nsite_title">nOldal: %1$s</string>
<string name="napplet_card_title">nKisalkalmazás: %1$s</string>
<string name="napplet_card_kind">nKisalkalmazás</string>
<string name="nsite_website_kind">nOldal</string>
<string name="napplet_card_permissions">Amihez hozzáférhet</string>
<string name="nsite_root_site">Gyökéroldal</string>
<string name="nsite_source">Forrás:</string>
<string name="nsite_servers">Kiszolgálók:</string>
<string name="nsite_open">Megnyitás</string>
<!-- Custom emoji suggestions (NIP-30) -->
<string name="use_direct_url">Közvetlen webcím használata</string>
<!-- Nicknames (NIP-85 contact cards) -->
<string name="nickname_dialog_title">Becenév</string>
<string name="nickname_dialog_explainer">Ez jelenik meg Önnek ezen felhasználó neve helyett az alkalmazásban bárhol. Titkosítva tárolódik el a kapcsolatkártyájára: csak Ön olvashatja. Írjon be kettőspontot (:) az egyéni emodzsik használatához.</string>
<string name="nickname_label">Becenév</string>
<string name="nickname_summary_label">Privát megjegyzés erről a felhasználóról</string>
<string name="nickname_save">Mentés</string>
<string name="nickname_cancel">Mégse</string>
<string name="git_status_open">Nyitva</string>
<string name="git_status_merged">Beolvasztva</string>
<string name="git_status_closed">Lezárva</string>
@@ -53,6 +131,7 @@
<string name="road_event_traffic_jam">Forgalmi dugó</string>
<string name="road_event_unknown">Útesemény</string>
<string name="podcast_value_zap_split_hint">Az erre küldött Zapek megoszlanak a következők között:</string>
<string name="podcast_value_split_percent">%1$d%%</string>
<string name="podcast_value_for_value">Értéket az értékért</string>
<string name="relay_monitor_rtt_open">Megnyitás</string>
<string name="relay_monitor_rtt_read">Olvasás </string>
@@ -61,6 +140,7 @@
<string name="relay_monitor_relay_type">Típus</string>
<string name="relay_monitor_requirements">Követelmények</string>
<string name="relay_monitor_supported_nips">Támogatott NIP-ek</string>
<string name="relay_monitor_ms">%1$d ms</string>
<string name="relay_discovery_accepted_kinds">Elfogadott típusok</string>
<string name="relay_discovery_geohash">Helyszín</string>
<string name="calendar_rsvp_going">Ott leszek</string>
@@ -81,6 +81,16 @@ interface NotificationSettings {
fun setEnabled(v: Boolean)
/**
* True iff the user has taken an explicit action to disable
* notifications (i.e. flipped the master switch OFF at some point).
* Used by the Settings screen to distinguish "master switch is off
* because it defaults to off on first launch" from "master switch
* is off because the user asked for it to be off". Only the former
* gets auto-enabled when the OS permission check passes.
*/
fun wasExplicitlyDisabled(): Boolean
fun setKindToggle(
kind: NotifKind,
v: Boolean,
@@ -23,6 +23,7 @@ package com.vitorpamplona.amethyst.commons.tor
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isLocalHost
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOnion
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOverlayNetwork
class TorRelayEvaluation(
val torSettings: TorRelaySettings,
@@ -36,6 +37,11 @@ class TorRelayEvaluation(
} else {
if (relay.isLocalHost()) {
false
} else if (relay.isOverlayNetwork()) {
// An overlay-mesh relay (0200::/7, e.g. Yggdrasil) is reachable only through the
// local mesh interface: Tor cannot route the range at all, so proxying it would
// guarantee failure rather than privacy. The overlay already encrypts end to end.
false
} else if (relay.isOnion()) {
// .onion is only reachable over Tor regardless of any other classification.
torSettings.onionRelaysViaTor
@@ -0,0 +1,70 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.tor
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* An overlay-mesh relay (`0200::/7`, e.g. Yggdrasil) must never be dialed through the Tor SOCKS
* proxy: Tor cannot route the range, so proxying guarantees failure rather than privacy. The
* overlay already encrypts end to end and authenticates the peer by its key-derived address.
*/
class YggdrasilTorRoutingTest {
private val yggdrasilRelay = NormalizedRelayUrl("ws://[201:d0e:9ba5:8bbc::1]:8080/")
private val yggdrasilSubnetRelay = NormalizedRelayUrl("ws://[300:1b5d:d0e9:ba58::1]:4848/")
private val lanRelay = NormalizedRelayUrl("ws://192.168.1.100:8080/")
private val ulaRelay = NormalizedRelayUrl("ws://[fd12:3456::1]:8080/")
private val clearnetIpv6Relay = NormalizedRelayUrl("wss://[2001:db8::1]:8080/")
private fun evaluation(newViaTor: Boolean) =
TorRelayEvaluation(
torSettings =
TorRelaySettings(
torType = TorType.INTERNAL,
onionRelaysViaTor = true,
dmRelaysViaTor = true,
newRelaysViaTor = newViaTor,
trustedRelaysViaTor = false,
moneyOperationsViaTor = false,
),
trustedRelayList = emptySet(),
dmRelayList = emptySet(),
)
@Test
fun overlayRelaysAreNeverTorifiedEvenWhenNewRelaysViaTorIsOn() {
val eval = evaluation(newViaTor = true)
assertFalse(eval.useTor(yggdrasilRelay), "0200::/8 node address must not be proxied")
assertFalse(eval.useTor(yggdrasilSubnetRelay), "0300::/8 subnet address must not be proxied")
assertFalse(eval.useTor(lanRelay), "LAN relay stays off Tor")
assertFalse(eval.useTor(ulaRelay), "IPv6 unique local address stays off Tor")
}
@Test
fun clearnetIpv6RelaysStillFollowTheTorSetting() {
// The overlay exemption must not leak into ordinary IPv6 relays.
assertTrue(evaluation(newViaTor = true).useTor(clearnetIpv6Relay))
assertFalse(evaluation(newViaTor = false).useTor(clearnetIpv6Relay))
}
}
@@ -53,8 +53,20 @@ class PreferencesNotificationSettings(
override fun setEnabled(v: Boolean) {
_enabled.value = v
prefs.putBoolean(KEY_ENABLED, v)
// Track explicit user intent so the Settings screen can auto-enable
// on next visit for users who never touched the switch, while
// respecting users who deliberately turned it off. Only false
// → "explicit disable"; going from off to on clears the flag so
// subsequent auto-enable heuristics work normally.
if (v) {
prefs.remove(KEY_EXPLICITLY_DISABLED)
} else {
prefs.putBoolean(KEY_EXPLICITLY_DISABLED, true)
}
}
override fun wasExplicitlyDisabled(): Boolean = prefs.getBoolean(KEY_EXPLICITLY_DISABLED, false)
override fun setKindToggle(
kind: NotifKind,
v: Boolean,
@@ -92,6 +104,7 @@ class PreferencesNotificationSettings(
companion object {
const val NODE = "com/vitorpamplona/amethyst/notifications"
private const val KEY_ENABLED = "enabled"
private const val KEY_EXPLICITLY_DISABLED = "explicitly_disabled"
private const val KEY_DND_UNTIL = "dnd_until"
private const val KEY_PREVIEW = "preview_in_toast"
@@ -0,0 +1,86 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.moderation.notifications
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
import java.util.UUID
import java.util.prefs.Preferences
/**
* Pins the semantics of the new [NotificationSettings.wasExplicitlyDisabled]
* flag added to unblock the "Enable OS notifications button doesn't work on
* desktop" bug's second failure mode.
*
* The Settings screen auto-enables the master notifications switch when it
* detects that the OS permission is fine and the switch is off. That's the
* common path for users who click the "Enable OS notifications" button and
* expect it to fully take effect (button label promise). But it MUST NOT
* override users who deliberately turned notifications off. The
* [wasExplicitlyDisabled] flag is how we distinguish the two.
*/
class PreferencesNotificationSettingsExplicitDisableTest {
private fun freshNode(): Preferences {
// Use a UUID-scoped node so tests never share state and never
// pollute the real user prefs on the machine running CI/dev builds.
return Preferences.userRoot().node("amethyst-test-" + UUID.randomUUID())
}
@Test
fun `fresh install defaults to not-explicitly-disabled`() {
val settings = PreferencesNotificationSettings(freshNode())
assertFalse(
"First launch must not look like a deliberate opt-out; otherwise auto-enable stays off forever",
settings.wasExplicitlyDisabled(),
)
}
@Test
fun `turning off marks explicitly disabled`() {
val prefs = freshNode()
val settings = PreferencesNotificationSettings(prefs)
settings.setEnabled(false)
assertTrue(settings.wasExplicitlyDisabled())
}
@Test
fun `turning on clears the explicit-disable flag`() {
val prefs = freshNode()
val settings = PreferencesNotificationSettings(prefs)
settings.setEnabled(false)
assertTrue(settings.wasExplicitlyDisabled())
settings.setEnabled(true)
assertFalse(
"Toggling back on must clear the flag so subsequent OFF->auto-enable cycles work",
settings.wasExplicitlyDisabled(),
)
}
@Test
fun `flag persists across new instances on the same prefs node`() {
val prefs = freshNode()
PreferencesNotificationSettings(prefs).setEnabled(false)
// Second instance opens the same node \u2014 flag must survive process restart.
val reopened = PreferencesNotificationSettings(prefs)
assertTrue(reopened.wasExplicitlyDisabled())
}
}
+16 -4
View File
@@ -246,18 +246,30 @@ compose.desktop {
// - amethyst.png 512x512 icon
//
// appimagetool binary is fetched by CI (SHA-verified) into
// desktopApp/packaging/appimage/ as appimagetool-x86_64.AppImage.
// desktopApp/packaging/appimage/ as appimagetool-<arch>.AppImage.
// The arch is selected at task-execution time from the host JVM's os.arch, so
// the same task builds the correct AppImage on both x86_64 and aarch64 hosts.
// BUILDING.md documents local-dev fetch.
val createReleaseAppImage by tasks.registering(Exec::class) {
group = "compose desktop"
description = "Package createReleaseDistributable output into a Linux AppImage via appimagetool."
dependsOn("createReleaseDistributable")
// AppImage's ARCH env accepts the Linux kernel arch names: x86_64 / aarch64
// / armhf / i686. jpackage produces host-native binaries, so mirror the
// host JVM arch. Do not read the property inside doFirst — it needs to be
// resolved at configuration time so outputs.file() below is stable.
val hostArch = when (val a = System.getProperty("os.arch").lowercase()) {
"amd64", "x86_64" -> "x86_64"
"aarch64", "arm64" -> "aarch64"
else -> a
}
val distDir = layout.buildDirectory.dir("compose/binaries/main-release/app/Amethyst")
val appDir = layout.buildDirectory.dir("appimage/Amethyst.AppDir")
val outFile = layout.buildDirectory.file("appimage/Amethyst-$appVersion-x86_64.AppImage")
val outFile = layout.buildDirectory.file("appimage/Amethyst-$appVersion-$hostArch.AppImage")
val toolRoot = layout.projectDirectory.dir("packaging/appimage")
val appimagetool = toolRoot.file("appimagetool-x86_64.AppImage")
val appimagetool = toolRoot.file("appimagetool-$hostArch.AppImage")
inputs.dir(distDir)
inputs.dir(toolRoot)
@@ -292,7 +304,7 @@ val createReleaseAppImage by tasks.registering(Exec::class) {
appDir.get().asFile.absolutePath,
outFile.get().asFile.absolutePath,
)
environment("ARCH", "x86_64")
environment("ARCH", hostArch)
// Bypass FUSE requirement on CI runners (ubuntu-latest lacks libfuse.so.2).
// AppImage standard env var: extracts + runs without mounting.
environment("APPIMAGE_EXTRACT_AND_RUN", "1")
+4 -1
View File
@@ -7,7 +7,10 @@
# (Equivalent packages on Fedora/Arch.)
set -eu
HERE="$(dirname "$(readlink -f "${0}")")"
export LD_LIBRARY_PATH="${HERE}/usr/lib:${HERE}/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}"
# Multiarch lib path is set by the host, not baked at build time — same
# AppRun works in both x86_64 and aarch64 AppImages.
GNU_TRIPLET="$(uname -m)-linux-gnu"
export LD_LIBRARY_PATH="${HERE}/usr/lib:${HERE}/usr/lib/${GNU_TRIPLET}:${LD_LIBRARY_PATH:-}"
export PATH="${HERE}/usr/bin:${PATH}"
export APPDIR="${HERE}"
exec "${HERE}/usr/bin/Amethyst" "$@"
+6 -2
View File
@@ -28,8 +28,12 @@ used two ways:
manifest (archive source pinned to the release tarball URL + sha256, with
`x-checker-data` so Flathub's update bot bumps it), its own metainfo
(carries the permanent `<releases>` history Flathub requires), desktop
entry, icon, and `flathub.json` (`only-arches: x86_64` — we publish no
aarch64 tarball, and jpackage can't cross-compile one)
entry, icon, and `flathub.json` — currently gated to `only-arches:
x86_64` so the Flathub build machinery never tries the aarch64 tarball
before we've validated it end-to-end on Flathub's aarch64 builders. GitHub
releases already ship aarch64 flatpak bundles (built from the same source
tree on `ubuntu-24.04-arm`); flipping `only-arches` to include `aarch64`
is the follow-up once we've smoke-tested a Flathub aarch64 build.
## Local build
@@ -39,13 +39,17 @@ import androidx.compose.material3.Switch
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.runtime.snapshotFlow
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalWindowInfo
import androidx.compose.ui.unit.dp
import com.vitorpamplona.amethyst.commons.moderation.notifications.HostOs
import com.vitorpamplona.amethyst.commons.moderation.notifications.NotifKind
@@ -112,7 +116,7 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
dispatcher?.nativeAvailable?.collectAsState()
?: remember { mutableStateOf(false) }
)
val coroutineScope = androidx.compose.runtime.rememberCoroutineScope()
val coroutineScope = rememberCoroutineScope()
var testStatus by remember { mutableStateOf<String?>(null) }
var requestingPermission by remember { mutableStateOf(false) }
var sendingTest by remember { mutableStateOf(false) }
@@ -120,18 +124,36 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
// Re-sync permission state whenever this screen enters composition
// and whenever the window regains focus — user may have toggled
// Amethyst in System Settings → Notifications while we were open.
val windowInfo = androidx.compose.ui.platform.LocalWindowInfo.current
androidx.compose.runtime.LaunchedEffect(dispatcher) {
val windowInfo = LocalWindowInfo.current
LaunchedEffect(dispatcher) {
dispatcher?.refreshPermission()
}
androidx.compose.runtime.LaunchedEffect(dispatcher, windowInfo) {
androidx.compose.runtime
.snapshotFlow { windowInfo.isWindowFocused }
LaunchedEffect(dispatcher, windowInfo) {
snapshotFlow { windowInfo.isWindowFocused }
.collect { focused ->
if (focused) dispatcher?.refreshPermission()
}
}
// Covers the two paths the earlier fix (e9475dd0) missed: the OS
// permission was already granted in a prior session (an older
// build asked, or the user allowed Amethyst in System Settings
// directly), and Windows/Linux, where permissionState is
// NotApplicable from startup. On both, permissionState is not
// NotRequested, so the "Enable OS notifications" button never
// renders, yet the master switch is still OFF from first-launch
// defaults — leaving no affordance that both explains the problem
// and fixes it. Auto-enable when the permission is fine, the
// master switch is off, and the user has never explicitly turned
// it off; [NotificationSettings.wasExplicitlyDisabled] is what
// keeps a deliberate opt-out from being overruled.
LaunchedEffect(permissionState, enabled) {
val allowed = permissionState == PermissionState.Granted || permissionState == PermissionState.NotApplicable
if (allowed && !enabled && !settings.wasExplicitlyDisabled()) {
settings.setEnabled(true)
}
}
PlatformStatusCard(
host = host,
nativeAvailable = nativeAvailable,
@@ -219,6 +241,20 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
}
}
PermissionState.Granted, PermissionState.NotApplicable -> {
// Recovery affordance for the deliberate-opt-out path.
// The LaunchedEffect above already re-enables the
// switch for anyone who never touched it, so in
// practice the only state that still reaches here with
// `enabled == false` is an explicit opt-out. Guarding
// on `!enabled` alone (rather than also calling
// wasExplicitlyDisabled) keeps this a pure Compose
// state read and leaves the button visible for the one
// frame before the effect runs.
if (!enabled) {
OutlinedButton(
onClick = { settings.setEnabled(true) },
) { Text("Turn on desktop notifications") }
}
OutlinedButton(
onClick = {
if (sendingTest) return@OutlinedButton
+13 -10
View File
@@ -84,27 +84,34 @@
"Dutch"
]
},
{
"user": "summoner001",
"languages": [
"Hungarian"
]
},
{
"user": "maxblake2015",
"languages": [
"Polish"
]
},
{
"user": "rajs19420616",
"languages": [
"Hindi"
]
},
{
"user": "vitorpamplona",
"languages": [
"Czech",
"German",
"Polish",
"Portuguese, Brazilian",
"Swedish"
]
},
{
"user": "rajs19420616",
"languages": [
"Hindi"
]
},
{
"user": "greenart7c3",
"languages": []
@@ -265,10 +272,6 @@
"user": "D4rkFIow",
"languages": []
},
{
"user": "summoner001",
"languages": []
},
{
"user": "fiddleway",
"languages": []
+9
View File
@@ -112,6 +112,15 @@ require_auth = false
# the future. Enforced by RejectFutureEventsPolicy.
# reject_future_seconds = 1800
# Path for the JSON file that remembers what the [[mirror]] catch-up
# has already synced (per upstream, per scope), so a restart resumes
# instead of re-downloading each upstream's whole backfill window.
# Defaults to "<database file>.sync-coverage.json" next to the event
# store; only written when the store itself is file-backed (an
# in-memory store keeps no resume state — saved coverage would
# describe events that no longer exist).
# mirror_sync_state_file = "/var/lib/geode/events.db.sync-coverage.json"
[authorization]
# Allow / deny lists. Allow is a permissive ceiling; deny still
# removes specific entries inside it. Enforced by Pubkey/KindAllowDenyPolicy.
@@ -28,6 +28,7 @@ import com.vitorpamplona.geode.config.StaticConfig
import com.vitorpamplona.geode.mirror.MirrorDirection
import com.vitorpamplona.geode.mirror.MirrorUpstream
import com.vitorpamplona.geode.mirror.MirrorWorker
import com.vitorpamplona.geode.mirror.SyncCoverageFile
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
@@ -44,6 +45,7 @@ import com.vitorpamplona.quartz.nip01Core.store.IEventStore
import com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
@@ -223,7 +225,8 @@ private fun runImport(args: Array<String>) {
}
System.err.println(
"geode import: read=${stats.read} imported=${stats.imported} " +
"rejected=${stats.rejected} invalid-sig=${stats.invalid} malformed=${stats.malformed} " +
"rejected=${stats.rejected} failed=${stats.failed} " +
"invalid-sig=${stats.invalid} malformed=${stats.malformed} " +
"${ctx.dbFile ?: "(in-memory — not persisted; pass --db)"}",
)
} finally {
@@ -411,6 +414,27 @@ private fun serve(args: Array<String>) {
require(upstreams.none { it.url.displayUrl() == advertisedIdentity }) {
"[[mirror]] must not list this relay's own URL ($advertisedUrl)"
}
// Resume state for the mirror catch-up, following the admin state-file
// convention: next to the event database unless configured. Keyed to the
// store's ACTUAL persistence, not to `database.file` being set: a
// volatile store with a persistent coverage file would claim, on the
// next boot, that an empty database already holds the backfill window —
// and the mirror would never fetch it.
val sqliteBackend =
config.database.backend
.trim()
.lowercase() in StoreFactory.SQLITE_BACKEND_KEYWORDS
val persistentLocation =
a.opt("--db") ?: config.database.file?.takeUnless { sqliteBackend && config.database.in_memory }
val syncCoverage =
if (upstreams.isEmpty() || persistentLocation == null) {
if (upstreams.isNotEmpty() && config.options.mirror_sync_state_file != null) {
Log.w("Main") { "mirror_sync_state_file ignored: the event store is in-memory, so saved coverage would outlive the events it describes" }
}
null
} else {
SyncCoverageFile(File(config.options.mirror_sync_state_file ?: "$persistentLocation.sync-coverage.json"))
}
val mirror =
if (upstreams.isEmpty()) {
null
@@ -425,6 +449,9 @@ private fun serve(args: Array<String>) {
// for the historical window, then live REQ tail. Auto-falls back
// to paged REQ for upstreams without NIP-77.
negentropyBackfill = true,
// Without NIP-77 the catch-up is a paged re-download; the
// coverage bands remember what previous boots already walked.
coverage = syncCoverage?.coverage,
).also { it.start() }
}
@@ -462,6 +489,8 @@ private fun serve(args: Array<String>) {
// queue and store beneath them shut down.
runCatching { maintenanceScope.cancel() }
runCatching { mirror?.close() }
// After the mirror, so the final flush carries the last bands.
runCatching { syncCoverage?.close() }
runCatching { server.stop() }
runCatching { relay.close() }
},
@@ -185,6 +185,16 @@ data class StaticConfig(
* that don't offer NIP-50 at all (strfry, for example).
*/
val full_text_search: Boolean = true,
/**
* Where the mirror catch-up's resume state lives: the per-upstream
* `created_at` coverage bands (quartz's `SyncCoverage`). Without it
* every restart re-syncs each upstream's whole backfill window —
* a full re-download for an upstream without NIP-77. Defaults to
* `<database file>.sync-coverage.json` when the store is
* file-backed; an in-memory store keeps no resume state (bands
* only pay off across restarts).
*/
val mirror_sync_state_file: String? = null,
)
/**
@@ -23,8 +23,11 @@ package com.vitorpamplona.geode.mirror
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
import com.vitorpamplona.quartz.nip01Core.relay.client.NostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.NegentropySyncException
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.SyncCoverage
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPages
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropyReconcile
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySyncOrFetch
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySync
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.EventMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.ReqCmd
@@ -40,12 +43,15 @@ import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.channels.trySendBlocking
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeoutOrNull
import okhttp3.OkHttpClient
import java.time.Duration
import java.util.concurrent.atomic.AtomicLong
@@ -154,11 +160,18 @@ class MirrorWorker(
* historical window before the live REQ tail takes over. The geode binary
* turns this on (see `Main`); it defaults **off** so the many existing
* MirrorWorker tests keep exercising the pure live-REQ path unchanged.
* When on, `negentropySyncOrFetch` automatically falls back to paged REQ
* against an upstream that doesn't speak NIP-77 — so "either mode" is
* transparent and needs no separate toggle.
* When on, the catch-up automatically falls back to paged REQ against an
* upstream that doesn't speak NIP-77 — so "either mode" is transparent
* and needs no separate toggle.
*/
private val negentropyBackfill: Boolean = false,
/**
* Resume memory for the down catch-up, shared across upstreams and — via
* [SyncCoverageFile] — across restarts. Null keeps the old behavior:
* every boot re-syncs the whole backfill window, which for an upstream
* without NIP-77 is a full re-download.
*/
private val coverage: SyncCoverage? = null,
) : AutoCloseable {
private val scope = CoroutineScope(Dispatchers.IO + SupervisorJob())
@@ -201,6 +214,14 @@ class MirrorWorker(
/** Events the store rejected — mostly duplicate replays after a reconnect. */
val rejected = AtomicLong(0)
/**
* Good events the store could not write ([IEventStore.InsertOutcome.Failed]).
* The store's fault, not the event's, and nothing re-offers them — counted
* apart from [rejected] so a schema drift reads as store damage rather
* than as upstreams sending junk.
*/
val failed = AtomicLong(0)
/**
* Deliveries dropped before ever reaching the store: events outside
* the [MirrorUpstream.filter] scope (an upstream answering outside
@@ -300,6 +321,13 @@ class MirrorWorker(
rejected.incrementAndGet()
Log.d("MirrorWorker") { "rejected ${msg.event.id}: ${outcome.reason}" }
}
is IEventStore.InsertOutcome.Failed -> {
// The store's error, not the event's: the event
// was good and nothing will re-offer it. Louder
// than a rejection on purpose.
failed.incrementAndGet()
Log.w("MirrorWorker") { "store failed ${msg.event.id}: ${outcome.reason}" }
}
}
}
} catch (e: CancellationException) {
@@ -399,9 +427,9 @@ class MirrorWorker(
* **client-paced** so a fast upstream can't overrun the sink: a plain REQ
* backfill of a large set dies here (strfry kills a slow REQ client once its
* unsent-outbound buffer crosses `maxPendingOutboundBytes`), which is exactly
* why this uses negentropy. [INostrClient.negentropySyncOrFetch] falls back
* to paged REQ automatically when the upstream doesn't speak NIP-77, so the
* mirror is compatible with either kind of upstream with no config.
* why this uses negentropy. When the upstream doesn't speak NIP-77 the
* catch-up falls back to paged REQ, so the mirror is compatible with
* either kind of upstream with no config.
*
* A failure here is non-fatal: the live subscription keeps the mirror current
* and the reconnect watermark narrows any residual gap.
@@ -412,16 +440,26 @@ class MirrorWorker(
initialSince: Long,
until: Long,
) {
val catchUpFilter = scopedBase.copy(since = initialSince, until = until)
// Reconcile against what we already hold in this window → download only
// the diff (like `strfry sync`). No store wired → empty local set → the
// whole window is downloaded and the store's unique-id constraint dedups.
val localEntries = store?.snapshotIdsForNegentropy(listOf(catchUpFilter)) ?: emptyList()
// Resume memory: coverage is keyed on the STABLE scoped filter, never
// on the boot window — whose since/until change every start and would
// never match a stored band. The window itself rides along as the
// floor argument, so a band recorded against a shallower window
// re-opens the older span when the operator deepens the backfill.
val legs =
coverage
?.legs(up.url, scopedBase, initialSince)
?.mapNotNull { clampToWindow(it, initialSince, until) }
?: listOf(scopedBase.copy(since = initialSince, until = until))
if (legs.isEmpty()) {
Log.i("MirrorWorker") { "catch-up from ${up.url.url}: window already covered - nothing outside the synced band" }
return
}
// Bounded hand-off → one ingest consumer. `onEvent` can't suspend, so it
// blocks here when the sink falls behind; because negentropySyncOrFetch's
// own delivery pipeline is bounded, that backpressure reaches all the way
// to the upstream — no unbounded buffering (unlike the live-tail path).
// blocks here when the sink falls behind; because negentropySync's own
// delivery pipeline is bounded (and a paged REQ is paced by its pages),
// that backpressure reaches all the way to the upstream — no unbounded
// buffering (unlike the live-tail path).
val handoff = Channel<Event>(capacity = CATCHUP_HANDOFF)
val consumer =
scope.launch {
@@ -431,6 +469,10 @@ class MirrorWorker(
when (outcome) {
IEventStore.InsertOutcome.Accepted -> accepted.incrementAndGet()
is IEventStore.InsertOutcome.Rejected -> rejected.incrementAndGet()
is IEventStore.InsertOutcome.Failed -> {
failed.incrementAndGet()
Log.w("MirrorWorker") { "store failed ${event.id}: ${outcome.reason}" }
}
}
}
} catch (e: CancellationException) {
@@ -443,24 +485,99 @@ class MirrorWorker(
}
try {
val result =
client.negentropySyncOrFetch(
relay = up.url,
filter = catchUpFilter,
localEntries = localEntries,
onEvent = { event ->
// Same containment as the live path: even a trusted
// upstream may only inject events inside the declared scope.
if (up.filter == null || up.filter.match(event)) {
handoff.trySendBlocking(event)
} else {
filtered.incrementAndGet()
var downloaded = 0
var paged = false
for (leg in legs) {
// Reconcile against what we already hold in this leg → download
// only the diff (like `strfry sync`). No store wired → empty
// local set → the whole leg is downloaded and the store's
// unique-id constraint dedups.
val localEntries = store?.snapshotIdsForNegentropy(listOf(leg)) ?: emptyList()
// Coverage is stamped from when the local ids were read — that
// is the state the relay is being compared against.
val syncStartedAt = TimeUtils.now()
var seenMin: Long? = null
var seenMax: Long? = null
fun observe(event: Event) {
// Same containment as the live path: even a trusted
// upstream may only inject events inside the declared scope.
if (up.filter == null || up.filter.match(event)) {
// Only plausible stamps widen a band — one
// misdated event must not discard the rest.
if (SyncCoverage.isPlausible(event.createdAt)) {
seenMin = minOf(seenMin ?: event.createdAt, event.createdAt)
seenMax = maxOf(seenMax ?: event.createdAt, event.createdAt)
}
},
)
handoff.trySendBlocking(event)
} else {
filtered.incrementAndGet()
}
}
// The two phases run by hand rather than through
// negentropySyncOrFetch, for two reasons. The combinator keeps
// every delivered id for cross-phase dedup — a multi-million-
// event catch-up cannot afford that heap, and the store's
// unique-id constraint dedups anyway. And the fallback must
// reset the observed span: a half-finished reconcile delivers
// events scattered across the whole leg, and a band built from
// that scatter would claim interior ranges nobody walked.
val legPaged =
try {
downloaded +=
client
.negentropySync(
relay = up.url,
filter = leg,
localEntries = localEntries,
onEvent = ::observe,
).downloaded
false
} catch (e: NegentropySyncException) {
seenMin = null
seenMax = null
// The watchdog matches negentropySync's default rather
// than fetchAllPages' shorter one: a paged catch-up
// sits behind the same slow upstreams.
downloaded += client.fetchAllPages(up.url, listOf(leg), idleTimeoutMs = 120_000L) { observe(it) }
true
}
paged = paged || legPaged
// Recorded per leg, so a failure between legs keeps the ground
// the first one gained — and no more: a reconcile compared only
// its own leg, so completeness reaches the leg's ceiling, never
// "now" while a later leg is still pending. A paged fallback
// earns only the span it actually saw, capped at the snapshot
// instant so one future-dated event cannot lift the band's
// ceiling past what was asked.
if (legPaged) {
coverage?.record(
up.url,
scopedBase,
seenMin,
seenMax?.coerceAtMost(syncStartedAt),
paged = true,
)
} else {
val legFloor = leg.since ?: initialSince
coverage?.record(
up.url,
scopedBase,
// The compared range starts at the leg's floor whether
// or not anything was observed there — that is what a
// clean reconcile proves.
observedMin = minOf(seenMin ?: legFloor, legFloor),
observedMax = null,
paged = false,
reconciledThrough = minOf(leg.until ?: syncStartedAt, syncStartedAt),
)
}
}
Log.i("MirrorWorker") {
val how = if (result.pagedFallback) "paged REQ (upstream has no NIP-77)" else "negentropy"
"catch-up from ${up.url.url}: ${result.downloaded} events via $how"
val how = if (paged) "paged REQ (upstream has no NIP-77)" else "negentropy"
val resumed = if (coverage != null && legs.size > 1) " [resumed: ${legs.size} legs outside the synced band]" else ""
"catch-up from ${up.url.url}: $downloaded events via $how$resumed"
}
} catch (e: CancellationException) {
throw e
@@ -671,11 +788,21 @@ class MirrorWorker(
runCatching { client.close() }
inbound.close()
scope.cancel()
// Wait (bounded) for the workers to land: a coverage.record racing
// past the state file's final flush would be recorded and lost.
// Bounded because a worker parked in a blocking hand-off does not
// feel the cancel, and shutdown must not hang on it.
runBlocking {
withTimeoutOrNull(CLOSE_JOIN_MS) { scope.coroutineContext[Job]?.join() }
}
okhttp?.dispatcher?.executorService?.shutdown()
okhttp?.connectionPool?.evictAll()
}
private companion object {
/** How long [close] waits for the worker coroutines to land. */
const val CLOSE_JOIN_MS = 5_000L
/** Matches the Android app's relay-pool WebSocket ping interval. */
const val PING_INTERVAL_SECS = 120L
@@ -701,7 +828,7 @@ class MirrorWorker(
/**
* Depth of the catch-up hand-off between the negentropy download and the
* ingest consumer. Small: negentropySyncOrFetch is already internally
* ingest consumer. Small: the negentropy download is already internally
* backpressured, so this only smooths the seam — the bounded IngestQueue
* behind `server.ingest` is the real limiter.
*/
@@ -720,3 +847,20 @@ class MirrorWorker(
const val UP_SYNC_SETTLE_MS = 1_500L
}
}
/**
* [leg] intersected with the boot window `[since, until]`, or null when the
* band already covers everything this window could ask. Coverage legs come
* off the stable scoped filter and are unbounded on one side; the catch-up
* only ever asks inside its own window.
*/
internal fun clampToWindow(
leg: Filter,
since: Long,
until: Long,
): Filter? {
val newSince = maxOf(leg.since ?: since, since)
val newUntil = minOf(leg.until ?: until, until)
if (newSince > newUntil) return null
return leg.copy(since = newSince, until = newUntil)
}
@@ -0,0 +1,158 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.geode.mirror
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.SyncCoverage
import com.vitorpamplona.quartz.utils.Log
import kotlinx.serialization.json.Json
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.boolean
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.jsonObject
import kotlinx.serialization.json.jsonPrimitive
import kotlinx.serialization.json.long
import kotlinx.serialization.json.put
import java.io.File
import java.nio.file.AtomicMoveNotSupportedException
import java.nio.file.Files
import java.nio.file.StandardCopyOption
/**
* File persistence for [SyncCoverage], so the mirror's catch-up resumes
* across restarts instead of re-syncing each upstream's whole backfill
* window — which, for an upstream without NIP-77, is a full re-download
* every boot.
*
* Same shape as the admin state file: JSON next to the event database,
* written via a temp file and an atomic move so a reader never sees a half
* map. A daemon timer flushes changed state so progress survives a hard
* kill; [close] flushes the rest. A corrupt file starts fresh — the cost of
* losing it is one re-sync, the cost of refusing to start is the relay.
*/
class SyncCoverageFile(
private val file: File,
flushSeconds: Long = DEFAULT_FLUSH_SECONDS,
) : AutoCloseable {
@Volatile private var dirty = false
val coverage = SyncCoverage(onChange = { dirty = true })
private val flusher: Thread
init {
load()
// restore() bypasses onChange, but stay defensive: reopening a file
// must never count as a change, or every boot rewrites it.
dirty = false
flusher =
Thread {
while (!Thread.currentThread().isInterrupted) {
try {
Thread.sleep(flushSeconds * 1000)
} catch (_: InterruptedException) {
return@Thread
}
flush()
}
}.apply {
isDaemon = true
name = "mirror-sync-coverage-flush"
start()
}
}
/** Write the map if anything changed since the last write. */
@Synchronized
fun flush() {
if (!dirty) return
dirty = false
save()
}
override fun close() {
flusher.interrupt()
flush()
}
private fun load() {
if (!file.isFile) return
runCatching {
val root = Json.parseToJsonElement(file.readText()).jsonObject
coverage.restore(
root.mapValues { (_, v) ->
val o = v.jsonObject
SyncCoverage.Band(
o.getValue("min").jsonPrimitive.long,
o.getValue("max").jsonPrimitive.long,
o["complete"]?.jsonPrimitive?.boolean ?: false,
o["fullAt"]?.jsonPrimitive?.long ?: 0L,
)
},
)
}.onFailure {
Log.w("SyncCoverageFile") { "could not read ${file.path} (${it.message}); starting fresh" }
}
}
@Synchronized
private fun save() {
runCatching {
val doc =
buildJsonObject {
coverage.export().forEach { (key, band) ->
put(
key,
buildJsonObject {
put("min", band.minCreatedAt)
put("max", band.maxCreatedAt)
put("complete", band.complete)
put("fullAt", band.fullAt)
},
)
}
}
file.parentFile?.mkdirs()
val tmp = File(file.parentFile ?: File("."), "${file.name}.tmp")
tmp.writeText(json.encodeToString(JsonObject.serializer(), doc))
// ATOMIC_MOVE requested explicitly: without it the JVM may
// legally fall back to copy+delete, and a reader could see a
// half map. Same-directory rename, so support is the norm; a
// filesystem that truly can't gets the plain move (and the
// corrupt-file recovery absorbs the residual risk).
try {
Files.move(tmp.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE)
} catch (_: AtomicMoveNotSupportedException) {
Files.move(tmp.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING)
}
}.onFailure {
Log.w("SyncCoverageFile") { "could not write ${file.path}: ${it.message}" }
}
}
companion object {
// Pretty-printed: this file is read by a human debugging why an
// upstream re-synced.
private val json = Json { prettyPrint = true }
// Often enough that a kill costs little, rare enough to be free.
private const val DEFAULT_FLUSH_SECONDS = 30L
}
}
@@ -0,0 +1,130 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.geode.mirror
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import java.io.File
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNull
import kotlin.test.assertTrue
/**
* The catch-up's resume memory across restarts. Without it every boot
* re-syncs each upstream's whole backfill window — a full re-download for an
* upstream without NIP-77. These pin the restart round-trip and the window
* clamping that keys bands on the stable filter rather than the sliding
* boot window.
*/
class SyncCoverageFileTest {
private val relay = RelayUrlNormalizer.normalize("wss://relay.example")
private val profiles = Filter(kinds = listOf(0))
private fun tempFile(): File {
val f = File.createTempFile("sync-coverage", ".json")
f.delete()
return f
}
@Test
fun `bands survive a restart`() {
val f = tempFile()
SyncCoverageFile(f).use {
it.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
}
// A fresh instance, as a restart would build.
SyncCoverageFile(f).use { reopened ->
val band = reopened.coverage.band(relay, profiles)!!
assertEquals(1_700_001_000L, band.minCreatedAt)
assertEquals(1_700_002_000L, band.maxCreatedAt)
assertFalse(band.complete)
}
}
@Test
fun `a complete band survives with its completeness`() {
val f = tempFile()
SyncCoverageFile(f).use {
it.coverage.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_005_000L)
}
SyncCoverageFile(f).use { reopened ->
assertTrue(reopened.coverage.band(relay, profiles)!!.complete)
}
}
@Test
fun `a corrupt file starts fresh instead of refusing to start`() {
val f = tempFile()
f.writeText("{ not json")
SyncCoverageFile(f).use {
assertNull(it.coverage.band(relay, profiles))
}
}
@Test
fun `recording does not write but closing does`() {
val f = tempFile()
val store = SyncCoverageFile(f)
store.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
assertFalse(f.isFile, "a record marks dirty; only a flush writes")
store.close()
assertTrue(f.isFile, "close flushes")
}
@Test
fun `reopening without new records does not rewrite the file`() {
val f = tempFile()
SyncCoverageFile(f).use {
it.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
}
val written = f.lastModified()
SyncCoverageFile(f).close()
assertEquals(written, f.lastModified(), "restoring bands must not mark the store dirty")
}
// ---- the window clamp --------------------------------------------------
@Test
fun `an unbanded filter clamps to exactly the boot window`() {
val leg = clampToWindow(profiles, since = 1_000L, until = 2_000L)!!
assertEquals(1_000L, leg.since)
assertEquals(2_000L, leg.until)
}
@Test
fun `a leg outside the window is dropped rather than inverted`() {
// The band covers past the window's floor: the older leg would ask
// [since..band.min] with since above until — a range nothing can be in.
val olderLeg = profiles.copy(until = 500L)
assertNull(clampToWindow(olderLeg, since = 1_000L, until = 2_000L))
}
@Test
fun `a leg inside the window keeps its own tighter bound`() {
val newerLeg = profiles.copy(since = 1_500L)
val clamped = clampToWindow(newerLeg, since = 1_000L, until = 2_000L)!!
assertEquals(1_500L, clamped.since, "the band's ceiling wins over the window floor")
assertEquals(2_000L, clamped.until)
}
}
@@ -0,0 +1,221 @@
# PoolRequests subscription-state lock: what a suspending Mutex would cost
**Date:** 2026-08-03
**Status:** analysis + measurements; recommendation is NOT to use `Mutex`
**Harness:** `quartz/src/jvmTest/.../prodbench/LockDesignComparisonBenchmark.kt`
(`./gradlew :quartz:jvmTest --tests "*.LockDesignComparisonBenchmark"`)
**Context:** follows the Pixel 8 ANR (`anr_2026-08-03-12-55-26-256`) that replaced
`RequestSubscriptionState`'s busy-wait with a parking `PlatformLock`.
## The question
The parking lock removed the CPU burn, but it still **blocks** rather than suspends.
Relay consumers run as coroutines on `Dispatchers.IO` (limitedParallelism 64) and
production has ~191 live relays, so a blocked waiter occupies one of 64 dispatcher
threads. kotlinx's scheduler treats IO tasks as blocking and grows the pool when they
block — which is why the on-device fix moved CPU but left thread count flat (+7%).
Would `kotlinx.coroutines.sync.Mutex` (a waiter *suspends*, freeing its thread) be
better?
## What Mutex actually costs
The lock sits at the bottom of a call chain that is **entirely non-suspend**:
```
OkHttpWebSocket: scope.launch { for (m in incomingMessages) out.onMessage(m) } <- coroutine
WebSocketListener.onMessage (non-suspend)
BasicRelayClient.MyWebsocketListener.onMessage
RelayPool.onIncomingMessage
NostrClient.onIncomingMessage (RelayConnectionListener)
PoolRequests.onIncomingMessage
RequestSubscriptionState.withLock <- the lock
SubscriptionListener.onEvent (non-suspend, fans out to the app)
```
`Mutex.lock()` is `suspend`, so every frame above it must become `suspend`. Measured
blast radius:
| interface / API | count |
|---|---|
| `override fun onEvent(` | 59 |
| `override fun onEose(` | 52 |
| `override fun onIncomingMessage(` | 35 |
| `override fun onCannotConnect(` | 32 |
| `override fun onClosed(` | 25 |
| `override fun onDisconnected(` | 17 |
| `override fun onConnected(` | 16 |
| `override fun onSent(` | 14 |
| `override fun onConnecting(` | 11 |
| others (`onSubscriptionStarted`, …) | 1 |
| **total overrides to convert** | **262** |
| `.subscribe(` / `.unsubscribe(` call sites | **110** |
Worse than the count: the *entry points* are not all coroutines. `INostrClient` is
non-suspend by design — `subscribe`, `unsubscribe`, `publish`, `syncFilters`,
`connect` — and is called from ViewModels, filter assemblers and Compose effects.
`PoolRequests.addOrUpdate` / `remove` / `sendToRelayIfChanged` reach the lock from
those paths. Making them suspend pushes coroutine scoping into every call site that
today just calls `subscribe(...)` synchronously.
## The cheaper alternative: stripe the lock per relay
**Enabling invariant (verified by reading all 11 `withLock` bodies):** *every*
critical section in `PoolRequests` is scoped to exactly one relay. Each one takes
`url` / `relay` / `relay.url` as its key:
| line | body | key |
|---|---|---|
| 204 | `state.connecting(url)` | url |
| 218 | `state.onOpenReq(relay, cmd.filters)` | relay |
| 228 | `state.onSubscriptionClosed(relay)` | relay |
| 249 | `onNewEvent` / `currentState` / `lastKnownFilterStates` | relay.url |
| 267 | `onEose` + `decideCommandLocked` | relay.url |
| 296 | `onClosed` + `recordRefusalIfStructural` + `decideCommandLocked` | relay.url |
| 325 | `state.disconnected(url)` | url |
| 354 | `isStructurallyRefused` + `onOpenReq` | relay |
| 382 | `lastKnownFilterStates(url)` | url |
| 403 | `decideCommandLocked(state, subId, relay)` | relay |
Every field in `RequestSubscriptionState` is a `Map<T, …>` keyed by relay
(`subStates`, `filterStates`, `lastKnownFilterStates`, `refusedFilters`,
`refusalCounts`). The single cross-relay accessor, `currentFilters()` (no-arg,
returns the whole map), has **zero usages** — dead code, delete it.
So the lock is only shared across relays as an artifact of `mutableMapOf` not being
thread-safe. One lock per `(subId, relay)` is semantically equivalent and drops
contention from ~191 threads to ~12 (that relay's consumer, plus the occasional app
thread in `sendToRelayIfChanged`).
Blast radius: `RequestSubscriptionState` + `PoolRequests` only. Used by 4 production
files (`PoolRequests`, `RelayActiveRequestStates`, `RelayReqRefusals`) and 2 tests.
**No public API change.**
## Measurements
191 relay coroutines on a 64-thread limited-parallelism dispatcher, 3s windows. Each
iteration yields, modelling the real per-message suspension point of
`for (message in incomingMessages)` — without it the tight loops monopolise the
dispatcher and the harness measures itself, not the lock.
`bystander` = a task that never touches the lock, on the same dispatcher. Its latency
answers "is the lock stealing dispatcher threads from unrelated work?"
| subs | design | ops/s | bystander p50 | p99 |
|---|---|---|---|---|
| 1 | PER_SUB_BLOCKING (today) | 548,608 | 196.5µs | 770.9µs |
| 1 | PER_SUB_MUTEX | 190,672 | **1.9µs** | 30.4µs |
| 1 | **STRIPED** | **1,543,037** | 88.8µs | 237.6µs |
| 4 | PER_SUB_BLOCKING | 833,090 | 136.1µs | 507.6µs |
| 4 | PER_SUB_MUTEX | 523,440 | **3.1µs** | 26.0µs |
| 4 | **STRIPED** | **1,499,757** | 92.3µs | 337.5µs |
| 16 | PER_SUB_BLOCKING | 1,198,581 | 94.3µs | 444.3µs |
| 16 | PER_SUB_MUTEX | 749,487 | **5.0µs** | 16.5µs |
| 16 | **STRIPED** | **1,789,509** | 85.7µs | 219.8µs |
Reading:
- **STRIPED gives the most throughput** — 2.8× / 1.8× / 1.5× over today — and roughly
halves bystander p50. It removes the contention rather than tolerating it.
- **MUTEX is the slowest of the three** (0.35× / 0.63× / 0.63× of today): per-acquisition
suspend/resume is not free at these rates.
- **MUTEX does win bystander latency decisively** (1.95.0µs vs 85196µs, and it collected
626k vs 24k samples). But read that honestly — part of the win is that its coroutines
spend their time *suspended waiting for the mutex instead of doing work*. Better
thread-yielding is partly a symptom of lower throughput, not purely a win.
## On-device result (SM-T220, playBenchmark, same account, n=3 per design)
Cold-start burst, 75s window, exact per-thread CPU accounting from `/proc/<tid>/stat`.
Warmups matched (~220-255 established sockets, ~200 relay reader threads each time).
| design | DefaultDispatcher CPU (ms/75s) mean / median / range | GC ms | threads |
|---|---|---|---|
| spin lock (pre-fix) | 136,743 / 117,750 / 112,230180,250 | 33,777 | 475 |
| parking, one lock per sub | 103,780 / 99,160 / 86,120130,680 | 30,235 | 510 |
| **striped, per (sub, relay)** | **88,953 / 82,160 / 72,860111,840** | **27,687** | 489 |
- **Striped vs spin: 35% mean, 30% median CPU, 18% GC — and the ranges do not
overlap** (striped max 111,840 < spin min 112,230). That separation is what the
parking-only change could not show; its range overlapped the baseline heavily.
- **Striped vs parking: 14% mean / 17% median**, ranges still overlap — directional,
not conclusive at n=3.
- **Thread count is unchanged across all three** (475 / 510 / 489). Expected: the lock
is blocking, and kotlinx marks IO tasks blocking and grows the pool. Striping reduces
how *often* threads block, not the fact that they can.
- Not a false win from broken subscriptions: the feed rendered live notes 3-15 min old
with avatars and reaction counts, on 223 sockets / 200 relay reader threads.
## Recommendation
**Do the striping; do not convert to `Mutex`.**
Striping attacks the cause — the lock is contended only because it is shared across
relays that touch disjoint keys. Once contention is ~0, the blocking-vs-suspending
question is moot: *a lock that is never contended never blocks a thread*. It also
happens to be the fastest option and is contained to two files, versus 262 overrides
and 110 call sites for a design that measures slower.
`Mutex` only becomes the right answer if a future critical section must genuinely span
relays (or do I/O), which none does today.
## The lock must not live inside a removable entry
The obvious shape — `ConcurrentMap<T, PerRelayState>` where `PerRelayState` owns both
the fields *and* its lock — is **wrong as soon as entries can be removed**.
`connecting()` / `disconnected()` genuinely mean "forget this relay's wire state", so
they want removal, and then:
```
T1: getOrPut(R) -> stateA ; stateA.lock.lock() // in a critical section
T2: disconnected(R) -> remove(R) // stateA is now orphaned
T3: getOrPut(R) -> stateB (NEW lock) ; stateB.lock.lock() // acquires immediately
-> T1 and T3 are both "in" relay R's critical section, excluding nothing.
```
Two ways out:
- **(A) never remove; null the fields.** Lock identity is stable, but entries
accumulate for every relay a sub has *ever* seen. Bounded but monotonic — and slow
monotonic growth in a long-lived process is precisely the class of bug this whole
investigation was about.
- **(B) stable stripe locks + removable state.** A fixed `Array(N) { PlatformLock() }`
indexed by `relay.hashCode()`, never mutated, so lock identity can't change; the
`ConcurrentMap<T, RelayState>` entries are then free to be added and removed with
exact `connecting`/`disconnected` semantics and no growth.
**(B) is the recommendation.** With N = 32 and ~191 relays, ~6 relays share a stripe —
still a ~32x contention reduction versus today's single lock per sub, with none of the
lock-lifetime hazard. `ConcurrentMap.remove` (added 2026-08-03, with tests in
`ConcurrentCollectionsTest`) is what makes (B) possible.
## Implementation sketch
1. Delete the dead `currentFilters()` (no-arg).
2. In `RequestSubscriptionState`, replace the five relay-keyed maps with a single
`ConcurrentMap<T, RelayState>` holding the five fields — **no lock inside**.
3. Add a fixed `private val stripes = Array(32) { PlatformLock() }` and
`withLock(reference)` = `stripes[reference.hashCode().absoluteValue % 32].withLock { }`.
The array is never mutated, so lock identity is stable for the object's life.
4. `connecting()` / `disconnected()` become `map.remove(reference)` — exact current
semantics, no residue.
5. Update the 11 `withLock` call sites in `PoolRequests` to pass the relay.
`decideCommandLocked`'s "MUST hold the lock" contract becomes "MUST hold *that
relay's stripe*" — tighten the kdoc.
6. Extend `PoolRequestsRefusalTest` with concurrent multi-relay access on one subId,
and add a striped variant to `LockDesignComparisonBenchmark` to confirm the
measured win survives stripe collisions.
## Risks
- **Stripe collisions serialize unrelated relays.** With 32 stripes and 191 relays this
is ~6-way sharing; it is a contention *reduction*, not elimination. If a future
profile shows it mattering, raise N — it is a one-line change with no semantic effect.
- **Two relays on one stripe must never be locked simultaneously by one thread** — that
would self-deadlock (`PlatformLock` is reentrant on JVM/Apple, so same-thread
re-entry is survivable, but the invariant should be stated). `PoolRequests` already
locks one relay at a time.
- The state machine's atomicity comments are load-bearing; the per-relay scoping must
be re-verified against any new `withLock` body added between now and the change.
- Striping is NOT a fix for a critical section that does I/O or spans relays. If one is
ever added, this analysis must be redone.
@@ -0,0 +1,154 @@
# Amethyst over Yggdrasil (IPv6 overlay)
Status: **fixed** — the four gaps found in the original assessment are closed. The last
section records what was deliberately left alone.
## What Yggdrasil looks like to the app
Yggdrasil is an encrypted end-to-end mesh. Every node gets an IPv6 address derived from its
public key inside `0200::/7` (nodes in `0200::/8`, subnets in `0300::/8`). Consequences:
- **No DNS.** A relay on the mesh is addressed as an IPv6 literal, always.
- **No certificates.** No CA issues for `0200::/7`, so relays run plain `ws://`. Not a
downgrade — the overlay already encrypts end to end and authenticates the peer by an
address derived from its public key.
- **On Android it is a `VpnService`**, so the app's default network becomes the VPN network.
- `0200::/7` is deprecated NSAP space, so nothing else routes there. An address in the range
is reachable *only* through a running mesh interface — which is what makes it safe to key
behavior off the prefix.
## What was wrong, and what fixed it
### 1. One relay, two identities
`RelayUrlNormalizer` folded hex case but not zero-compression, so
`[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]` stayed two distinct
`NormalizedRelayUrl`s for one host — while OkHttp collapsed both to the same host when
dialing. Since that value keys the connection pool, the relay-list sets, the NIP-11 cache and
the per-relay stat maps, the app opened two sockets to one relay and counted it twice.
**Fix:** new `Ipv6` util (`quartz/utils/Ipv6.kt`) — pure-Kotlin parse, RFC 5952 canonical
format and range classification, no `java.net`, so it works on every KMP target.
`RelayUrlNormalizer.norm()` now canonicalizes the bracketed host. The canonical form is
byte-for-byte what OkHttp renders, so the key the app stores is the host it actually dials —
asserted differentially against OkHttp in `YggdrasilCompatCharacterizationTest`.
Affects every IPv6 relay, not just mesh ones; it only bit Yggdrasil users because on the mesh
a literal is the *only* way to name a relay. No migration needed: relay lists are rehydrated
from event tags through `normalizeOrNull`, so stored entries fold on load.
### 2. Schemeless entry defaulted to `wss://`
`isLocalHost()` knew `127.0.0.1` / `localhost` / `//umbrel:` / `192.168.` / `.local`, so a
mesh address fell through to the clearnet default and produced a `wss://` url whose TLS
handshake could never succeed.
**Fix:** new `RelayUrlNormalizer.isOverlayNetwork()` recognizes `0200::/7` and joins
`isOnion` / `isLocalHost` in choosing `ws://`. Clearnet IPv6 (`2001:db8::1`) still gets
`wss://`.
`isLocalHost()` separately grew the IPv6 twins of the literals it already knew — `::1`
(loopback), `fc00::/7` (unique local, the 192.168. analogue) and `fe80::/10` (link-local).
Those are the same question every caller is asking, so a relay on one now correctly skips TLS
and Tor and stays out of published relay lists.
### 3. Unbracketed literal silently rejected
`yggdrasilctl getSelf` prints the address unbracketed — exactly what gets pasted into "add a
relay". Normalization returned null (correctly: it is ambiguous with a scheme) and
`RelayUrlEditField.submitRelay` had no else branch, so the Add button did nothing at all.
**Fix, two halves:**
- `fix()` brackets a bare literal automatically, but only when the whole string parses as an
IPv6 address — so `31990:hex:dtag` (addressable pointer), `abcd:1234` (host:port) and
`relay.example.com:8080` still fall through untouched.
- The edit field now sets `isError` and shows `relay_url_not_valid` instead of no-opping.
That fixes the dead button for *all* invalid input, not just IPv6.
### 4. Tor routing broke mesh relays
`TorRelayEvaluation` classified mesh relays as "new", so with Tor on and the default "new
relays via Tor" they were dialed through the SOCKS proxy — which cannot route `0200::/7`.
Guaranteed failure, not privacy.
**Fix:** `useTor()` returns false for `isOverlayNetwork()`, checked right after the localhost
branch. Both the Android and desktop relay paths delegate here (`TorRelayState`,
`DesktopHttpClient`), so one change covers both. `RoleBasedHttpClientBuilder` got the same
treatment for non-relay HTTP (images, previews, NIP-05, money ops). Clearnet IPv6 relays keep
following the Tor setting — asserted in `YggdrasilTorRoutingTest`.
## Audit round: bugs found in the host predicates
Auditing the change above turned up a family of bugs in `isLocalHost` / `isOnion` that predate
it. All shared one root cause — the predicates ran `contains` over the **whole url** instead of
parsing the authority — and all are now anchored, parsed and covered by
`RelayUrlAuthorityAnchoringTest`.
These predicates decide whether a relay is exempt from Tor, and relay urls arrive from other
people (NIP-65 lists, relay hints, `r` tags), so they are attacker-controlled input.
| # | Bug | Effect |
|---|---|---|
| 1 | A path could impersonate the host: `wss://evil.example.com/127.0.0.1` answered `isLocalHost() == true` | Any relay list could hand the app a url that silently dropped its own Tor routing |
| 2 | `.onion:8080` never matched the `.onion/` test | An onion relay on an explicit port was not treated as onion — never forced onto Tor, hostname sent to the clearnet DNS resolver |
| 3 | `.onion.` / `localhost.` (RFC 1034 fully-qualified form) matched nothing | Same leak as #2, via a different spelling |
| 4 | Host tests were case-sensitive, but `fix()` runs *before* the RFC 3986 pass folds case | `LOCALHOST:8080` and `ABC.ONION:8080` were given a `wss://` scheme neither host can serve |
| 5 | Private IPv4 was substring-matched | `10.0.0.5`, `172.16.3.4`, `127.1.2.3` were not local (LAN relay got `wss://` and Tor), while `192.168.evil.com` — a registrable domain — was |
| 6 | `contains("localhost")` matched `notlocalhost.example.com` | Same Tor exemption as #1, via a registrable domain |
| 7 | A `://` inside a path was read as a scheme separator | `relay.com/x://127.0.0.1` read its path as the authority |
Two bugs were introduced by this branch and caught in the same pass: the IPv6 host lookup had
the #1 flaw (`wss://evil.example.com/[fd00::1]` read as localhost), and IPv6 canonicalization
could rewrite a **path** (`/x[0:0:0:0:0:0:0:1]y``/x[::1]y`), corrupting the url.
Fixes: a shared `hostStart` / `hostEnd` / `hostEndWithoutPort` trio bounds every test to the
authority, strips `:port` and trailing dots, and validates the scheme; private ranges are
parsed via the new `Ipv4` util and `Ipv6` rather than substring-matched; comparisons are
case-insensitive per RFC 4343; `NormalizedRelayUrl.isOnion()` now delegates instead of keeping
a second, weaker copy of the test.
Performance: no regression, likely a small win. The old form ran six full-string `contains`
scans; the new one bounds its work to the authority and rejects a DNS host from an IP parse on
a single character. `Ipv6.isLiteral` gained a two-colon gate so the schemeless `host:port` case
answers without allocating the parser's 16-byte buffer.
`Ipv6` itself is pinned by `Ipv6DifferentialTest`: 4000 random addresses round-trip against
`java.net.InetAddress` in both directions, and the canonical form is asserted equal to
OkHttp's host for the same address — so the relay identity the app stores provably matches the
host it dials.
## Deliberately not changed
**Mesh relays are still published and recommended.** `AdvertisedRelayInfoTag` (NIP-65) and
`RelayListRecommendationProcessor.filterValidRelays` only exclude localhost, so a mesh relay
in your relay list is still published to public relays and offered to other users via the
outbox model. Two consequences worth a maintainer's decision:
- Peers not on the mesh dial `[201:…]` and burn reconnect attempts on an unreachable host.
- Your Yggdrasil address — a stable, key-derived node identifier — becomes public.
Onion relays already have precedent for both readings: they *are* published, but
`filterValidRelays` gates them behind `hasOnionConnection`. The equivalent for overlay relays
would be a `hasMeshConnection` gate. That is a product call about whether mesh relays are
meant to be discoverable, so it is flagged rather than decided here.
## Not verified here
- **No live socket test.** The analysis container has no IPv6 stack at all (`AF_INET6`
`EAFNOSUPPORT`), so everything is verified below the socket: normalization, OkHttp URL/host
agreement, and the Tor routing decision. An on-device run against a real mesh relay is
still needed to confirm the happy path end to end.
- **Android VPN interaction untested.** `ConnectivityFlow` uses
`registerDefaultNetworkCallback`, so it follows the app into the VPN network. Whether
`isMeteredOrMobileData()` reads correctly through Yggdrasil's `VpnService` depends on
whether that app declares underlying networks — worth checking on device before assuming
data-saving mode behaves.
- Media loading (Coil) and NIP-05 resolution against mesh hosts were not exercised.
## Tests
- `quartz/…/utils/Ipv6Test.kt` — parser, RFC 5952 formatting, range classification.
- `quartz/…/relay/YggdrasilCompatCharacterizationTest.kt` — normalization end to end, plus
the differential assertions that our identity matches the host OkHttp dials.
- `commons/…/tor/YggdrasilTorRoutingTest.kt` — overlay relays never Torified, clearnet IPv6
still follows the setting.
@@ -0,0 +1,36 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils.concurrent
import platform.Foundation.NSRecursiveLock
// NSRecursiveLock parks contended waiters in the kernel, matching the
// ReentrantLock semantics of the jvmAndroid actual (and the same choice
// commons' KmpLock made for iOS). This must NOT be a spin lock: quartz's
// relay client runs here too, and spinning is what produced the Android
// ANR documented on the expect declaration.
actual class PlatformLock {
private val delegate = NSRecursiveLock()
actual fun lock() = delegate.lock()
actual fun unlock() = delegate.unlock()
}
@@ -25,12 +25,13 @@ import com.vitorpamplona.quartz.nip01Core.core.AddressableEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.core.isReplaceable
import com.vitorpamplona.quartz.nip01Core.core.supersedes
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.store.ObservableEventStore
import com.vitorpamplona.quartz.nip01Core.store.ObservableEventStore.StoreChange
import com.vitorpamplona.quartz.nip01Core.store.owner
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
import com.vitorpamplona.quartz.nip40Expiration.isExpirationBefore
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
import com.vitorpamplona.quartz.nip62RequestToVanish.RequestToVanishEvent
import com.vitorpamplona.quartz.utils.SortedList
import com.vitorpamplona.quartz.utils.TimeUtils
@@ -342,19 +343,14 @@ class EventStoreProjection<T : Event>(
private fun supersedes(
new: Event,
existing: Event,
): Boolean =
when {
new.createdAt > existing.createdAt -> true
new.createdAt < existing.createdAt -> false
else -> new.id < existing.id
}
): Boolean = new.supersedes(existing)
/**
* Owner pubkey for ownership checks (NIP-09 author match,
* NIP-62 vanish target). For GiftWrap the owner is the p-tag
* recipient; for everything else it's `event.pubKey`.
*/
private fun ownerOf(event: Event): HexKey = (event as? GiftWrapEvent)?.recipientPubKey() ?: event.pubKey
private fun ownerOf(event: Event): HexKey = event.owner()
/**
* created_at DESC, id ASC. Sort keys are frozen at slot
@@ -0,0 +1,35 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.core
/**
* The NIP-01 replaceable/addressable supersession rule: the winner of an
* address is the highest `created_at`, with ties broken by the LEXICALLY
* SMALLEST id. True when THIS event beats [existing] — equal events (same id)
* do not supersede themselves. One rule, shared by every store; the SQLite
* triggers encode the same comparison in SQL.
*/
fun Event.supersedes(existing: Event): Boolean =
when {
createdAt > existing.createdAt -> true
createdAt < existing.createdAt -> false
else -> id < existing.id
}
@@ -34,6 +34,7 @@ import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.channels.Channel.Factory.UNLIMITED
import kotlinx.coroutines.coroutineScope
import kotlinx.coroutines.withTimeoutOrNull
import kotlin.time.TimeSource
/**
* One relay's verdict on a published event: [accepted] plus the reason the
@@ -45,6 +46,12 @@ import kotlinx.coroutines.withTimeoutOrNull
class PublishResult(
val accepted: Boolean,
val message: String,
/**
* Milliseconds from the publish to this relay's OK (true or false — a rejection
* is still a measured round trip), or -1 when the relay never answered with an
* OK. On an already-open socket this is an honest NIP-66 `rtt-write`.
*/
val elapsedMs: Long = -1,
) {
/**
* True when this failure came from the transport (never connected,
@@ -102,6 +109,7 @@ suspend fun INostrClient.publishAndCollectResults(
timeoutInSeconds: Long = 15,
): Map<NormalizedRelayUrl, PublishResult> {
val resultChannel = Channel<DetailedResult>(UNLIMITED)
val mark = TimeSource.Monotonic.markNow()
Log.d("publishAndConfirm") { "Waiting for ${relayList.size} responses" }
@@ -133,8 +141,13 @@ suspend fun INostrClient.publishAndCollectResults(
when (msg) {
is OkMessage -> {
if (msg.eventId == event.id) {
resultChannel.trySend(DetailedResult(relay.url, msg.success, msg.message))
// The relayList guard matters, not just the id: the same event may
// have been published to OTHER relays by an earlier call (probe
// waves, republish), and counting their late OKs here would inflate
// receivedResults and end the wait loop before every listed relay
// answered — misreporting the missing ones as NO_RESPONSE.
if (msg.eventId == event.id && relay.url in relayList) {
resultChannel.trySend(DetailedResult(relay.url, msg.success, msg.message, mark.elapsedNow().inWholeMilliseconds))
Log.d("publishAndConfirm") { "onSendResponse Received response for ${msg.eventId} from relay ${relay.url} message ${msg.message} success ${msg.success}" }
}
}
@@ -160,7 +173,7 @@ suspend fun INostrClient.publishAndCollectResults(
val currentResult = receivedResults[result.relay]
// do not override a successful result.
if (currentResult == null || !currentResult.accepted) {
receivedResults[result.relay] = PublishResult(result.success, result.message)
receivedResults[result.relay] = PublishResult(result.success, result.message, result.elapsedMs)
}
}
}
@@ -191,4 +204,5 @@ private class DetailedResult(
val relay: NormalizedRelayUrl,
val success: Boolean,
val message: String,
val elapsedMs: Long = -1,
)
@@ -0,0 +1,313 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.accessories
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.TimeUtils
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
/**
* How much of a filter's history has already been pulled from one relay, so a
* restart does not pull it again.
*
* A negentropy relay needs none of this — reconciliation downloads only the
* diff. Most relays lack NIP-77, and a paged fetch ([fetchAllPages]) has no
* memory: it re-downloads everything it walked last time, every restart,
* forever. So for those, remember the band of `created_at` covered per
* (relay, filter), and the next run asks only for the two legs outside it:
*
* stored band: |<-------- covered -------->|
* next fetch: <------| |------>
*
* Keyed by the WHOLE filter deliberately: any edit to a filter is a new key
* with no band, so the next run starts over — the safe direction to be wrong
* in, and the intended way to force a re-walk.
*
* A band does not guarantee completeness (a truncating relay, an event
* back-dated into a walked span). The trade is deliberate: re-reading a
* corpus on every restart is a certain daily cost, while both holes are
* occasional and self-heal on the next filter change or full re-walk.
*
* Persistence is the caller's: [export] the map on a schedule and [restore]
* it at startup. [onChange] fires whenever a band changes, so a persistence
* layer can mark itself dirty without polling.
*
* Not to be confused with the `relay.client.paging` package: its
* `RelayLoadingCursors` are in-memory POSITIONS for demand-driven UI paging
* within one session, while these are persistent INTERVALS — a claim about
* coverage that outlives the process and licenses skipping work.
*/
class SyncCoverage(
// How long a band may narrow work before the whole filter is walked
// again. Everything a band claims is a claim about the past; this is how
// long to trust it without re-testing.
private val fullResyncSeconds: Long = DEFAULT_FULL_RESYNC_SECONDS,
private val now: () -> Long = { TimeUtils.now() },
private val onChange: () -> Unit = {},
) {
/**
* What is already covered for one (relay, filter) pair.
*
* [complete] is the difference between "we walked this span" (a paged
* fetch) and "we are in sync below this point" (a finished negentropy
* reconcile, which compared the whole range). Only a complete band may
* skip its older leg.
*
* [fullAt] is when the last pass that started from nothing finished — the
* clock for the periodic re-walk.
*/
data class Band(
val minCreatedAt: Long,
val maxCreatedAt: Long,
val complete: Boolean = false,
val fullAt: Long = 0,
)
private val bands = ConcurrentMap<String, Band>()
// filter -> its canonical json. Filter.toJson() runs to tens of thousands
// of characters for author-scoped filters, and a fan-out keys once per
// relay per cycle over the SAME handful of filter instances. Filter
// compares by identity, so this map is an identity cache — and an
// identity cache retains every distinct instance it is handed. A caller
// that rebuilds its filter each cycle would grow it forever, so past
// MAX_FINGERPRINTS new instances key correctly but are not cached.
private val fingerprints = ConcurrentMap<Filter, String>()
/**
* The filters to actually run now, given what is already covered: the
* whole filter when nothing is recorded (or the band went stale),
* otherwise the legs outside the band, clamped to the filter's own
* `since`/`until`.
*
* The legs are INCLUSIVE of the band's edges (`until = min`, not
* `min - 1`): a page boundary can split a run of events sharing one
* `created_at`, and excluding the edge would strand the rest of that
* second in no leg at all. The cost is re-reading one second's worth of
* events per leg, which a store rejects as duplicates.
*/
fun legs(
url: NormalizedRelayUrl,
filter: Filter,
floor: Long? = null,
): List<Filter> {
val band = bands[key(url, filter)] ?: return listOf(filter)
// Time for another full pass: relays gain old events, and without
// this the band's claim is never re-tested.
if (isStale(band)) return listOf(filter)
val legs = mutableListOf<Filter>()
// Older: up to and including the band's floor, but not past the
// filter's (or, when the filter has no `since`, the caller's
// [floor] — a sync window the filter itself must not carry, or it
// would change the band's key every run). A complete band compared
// its whole range already, but only down to the floor it ran
// against: a caller now reaching deeper — a raised backfill window
// — re-opens the span below the band.
val since = filter.since ?: floor
val wantsOlder =
if (band.complete) {
since != null && since < band.minCreatedAt
} else {
since == null || band.minCreatedAt >= since
}
if (wantsOlder) {
legs.add(filter.copy(until = minOf(band.minCreatedAt, filter.until ?: Long.MAX_VALUE)))
}
// Newer: from the band's ceiling on, but not past the filter's.
if (filter.until == null || band.maxCreatedAt <= filter.until) {
legs.add(filter.copy(since = maxOf(band.maxCreatedAt, filter.since ?: Long.MIN_VALUE)))
}
return legs
}
/**
* Widen the band for (url, filter) to include what a completed fetch saw.
*
* [paged] gates the mechanism: a negentropy sync needs no band, and
* recording one would only risk narrowing a future reconciliation.
* Nothing is recorded for a fetch that saw no events — an empty result
* says nothing about what the relay holds.
*
* [reconciledThrough] is the strong case: a FINISHED reconcile compared
* the filter's whole range, so the caller is in sync up to the instant
* the sync STARTED — recorded against that instant rather than the newest
* event seen, because "the relay had nothing newer" and "we never asked"
* must not look alike.
*/
fun record(
url: NormalizedRelayUrl,
filter: Filter,
observedMin: Long?,
observedMax: Long?,
paged: Boolean,
reconciledThrough: Long? = null,
) {
if (reconciledThrough != null) {
put(url, filter, observedMin ?: reconciledThrough, reconciledThrough, complete = true)
return
}
if (!paged) return
// Guarded even though callers should filter with [isPlausible] per
// event: a 1970 floor or a far-future ceiling would make the band
// claim the whole timeline, and the leg outside it would ask for a
// range nothing can be in, forever.
if (observedMin == null || observedMax == null) return
if (!isPlausible(observedMin, now()) || !isPlausible(observedMax, now())) return
put(url, filter, observedMin, observedMax, complete = false)
}
/**
* Widen (or reset) the band. A pass that ran because the previous band
* had gone stale REPLACES it: it re-walked the whole filter, so its own
* span is the complete picture and [Band.fullAt] restarts from here.
*/
private fun put(
url: NormalizedRelayUrl,
filter: Filter,
min: Long,
max: Long,
complete: Boolean,
) {
val fresh = Band(min, max, complete, now())
bands.merge(key(url, filter), fresh) { old, new ->
if (isStale(old)) {
new
} else {
Band(
minOf(old.minCreatedAt, new.minCreatedAt),
maxOf(old.maxCreatedAt, new.maxCreatedAt),
old.complete || new.complete,
old.fullAt,
)
}
}
onChange()
}
private fun isStale(band: Band): Boolean = now() - band.fullAt >= fullResyncSeconds
/**
* The narrowest single filter that still covers what every one of [urls]
* needs — the window a shared negentropy snapshot has to be taken over.
*
* In steady state every relay carries a complete band and this collapses
* to `since = the oldest of their ceilings` — the difference between
* snapshotting an id set of millions and one of a few thousand. One relay
* that has never synced puts it back to the full filter, correctly: that
* relay genuinely needs everything.
*/
fun coveringWindow(
urls: List<NormalizedRelayUrl>,
filter: Filter,
): Filter {
if (urls.isEmpty()) return filter
var since = Long.MAX_VALUE
for (url in urls) {
val legs = legs(url, filter)
// Nothing outside its band: this relay asks nothing of the
// snapshot at all — the best case must not widen the window.
if (legs.isEmpty()) continue
// More than one leg means an older gap this relay still wants, so
// the snapshot cannot start above the filter's own floor.
val only = legs.singleOrNull() ?: return filter
val legSince = only.since ?: return filter
since = minOf(since, legSince)
}
// Every relay fully covered: any window would do; the unnarrowed
// filter is merely safe, and callers usually skip the sync entirely.
return if (since == Long.MAX_VALUE) filter else filter.copy(since = since)
}
/** What is currently covered, for logging and tests. */
fun band(
url: NormalizedRelayUrl,
filter: Filter,
): Band? = bands[key(url, filter)]
fun size(): Int = bands.size()
/** A point-in-time copy of every band, for a persistence layer to write out. */
fun export(): Map<String, Band> = bands.snapshot()
/** Load previously [export]ed bands, e.g. at startup. */
fun restore(entries: Map<String, Band>) {
for ((key, band) in entries) bands[key] = band
}
/**
* The identity of one (relay, filter) pair. [Filter.toJson] is the
* protocol's own canonical form, so two filters that mean the same thing
* key the same way and any edit keys differently — exactly the "config
* changed, start over" rule.
*/
private fun key(
url: NormalizedRelayUrl,
filter: Filter,
): String {
val fingerprint =
fingerprints[filter]
?: filter.toJson().also {
// Bounded: stable callers hit the cache at any size a real
// config produces; a caller minting fresh instances just
// pays the toJson each time instead of growing the heap.
if (fingerprints.size() < MAX_FINGERPRINTS) fingerprints[filter] = it
}
return "${url.url} $fingerprint"
}
companion object {
// More filter instances than any deliberate configuration holds; only
// a caller rebuilding filters per cycle ever reaches it.
private const val MAX_FINGERPRINTS = 1_000
/**
* A week. Long enough that the narrow path is the normal one, short
* enough that anything a band is wrong about is wrong for days, not
* forever.
*/
const val DEFAULT_FULL_RESYNC_SECONDS = 7L * 24 * 60 * 60
/**
* 2020-01-01. Below this a `created_at` is a bug, not a date — the
* protocol did not exist. Also the natural floor for measuring a
* paged walk's progress when a filter names no `since`.
*/
const val PLAUSIBLE_FLOOR = 1_577_836_800L
// Clock skew a relay may legitimately be ahead by. Past this, a
// created_at is the author's fiction rather than a time.
private const val FUTURE_SKEW_SECONDS = 86_400L
/**
* Whether a `created_at` can be believed as evidence of coverage.
* Filter with this per EVENT, not over a leg's aggregate: one
* misdated event among hundreds of thousands would otherwise discard
* the whole relay's band.
*/
fun isPlausible(
createdAt: Long,
now: Long = TimeUtils.now(),
): Boolean = createdAt in PLAUSIBLE_FLOOR..(now + FUTURE_SKEW_SECONDS)
}
}
@@ -0,0 +1,141 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
import com.vitorpamplona.quartz.utils.TimeUtils
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
import kotlin.concurrent.Volatile
/**
* How far a bulk pagination has got, measured on the time axis — the only
* axis whose end is known in advance.
*
* A paged fetch (`fetchAllPages`) has no event denominator: how many events
* exist is exactly what it is finding out, so every count-based percentage
* degenerates to `downloaded/downloaded = 100%`. The time axis has both ends
* before the first request — the filter's `until` (or now) down to its
* `since` (or the accessories' `SyncCoverage.PLAUSIBLE_FLOOR`) — with each
* page's new `until` cursor reporting the exact position between them. It
* needs no COUNT support.
*
* The estimate assumes events are spread evenly over time, which they are
* not — so it errs pessimistic on the tail, and is a bound, not a promise.
*
* One instance can serve many concurrent paginations: keys are
* `"group|name"`, and the group prefix scopes [fraction], [reached] and
* [etaMs] so two groups never report each other's numbers.
*
* Its siblings in this package track different things: [RelayLoadingCursors]
* is demand-driven `until`+`limit` paging for one scope (a feed pulling
* older pages on demand, no window), and [RelayPagingProgress] is the
* per-relay display state derived from it. This class is for a BULK
* pagination over a known `[since, until]` window, where "how far through
* the window, and when will it finish" is the question.
*/
class PagingWindowProgress(
private val nowMillis: () -> Long = { TimeUtils.nowMillis() },
) {
private class Window(
val top: Long,
val bottom: Long,
val startedMs: Long,
@Volatile var current: Long,
)
private val windows = ConcurrentMap<String, Window>()
/**
* Begin a pagination over `[bottom, top]` seconds. An inverted window is
* not one; a single-second window (`top == bottom`) is — coverage legs
* that re-read a band's edge second are exactly that shape.
*/
fun begin(
key: String,
top: Long,
bottom: Long,
) {
if (top >= bottom) windows[key] = Window(top, bottom, nowMillis(), top)
}
/**
* The pagination reached [until]; monotonic, so a page that jumps back
* cannot un-advance it. The check-then-set is unsynchronized on purpose:
* one pagination is one coroutine, and a display racing a mark can only
* ever read a value one page stale.
*/
fun mark(
key: String,
until: Long,
) {
windows[key]?.let {
// Clamped to the window's own floor: relays serve events stamped
// 0, and one of those would drag the position to the epoch. Below
// the floor means the pagination is done, not time travel.
val reached = until.coerceAtLeast(it.bottom)
if (reached < it.current) it.current = reached
}
}
fun finish(key: String) {
windows.remove(key)
}
/**
* Fraction complete, averaged over every pagination still going in
* [group] (or all of them when null) — averaged rather than summed
* because each covers its own span, so "half of them done and half at
* zero" is 50%.
*/
fun fraction(group: String? = null): Double? {
val live = live(group)
if (live.isEmpty()) return null
return live.sumOf { w ->
val span = (w.top - w.bottom).coerceAtLeast(1)
((w.top - w.current).toDouble() / span).coerceIn(0.0, 1.0)
} / live.size
}
private fun live(group: String?): List<Window> =
if (group == null) {
windows.snapshot().values.toList()
} else {
windows
.snapshot()
.entries
.filter { it.key.startsWith("$group|") }
.map { it.value }
}
/** The oldest second [group] has reached, or null when nothing is paging. */
fun reached(group: String? = null): Long? = live(group).minOfOrNull { it.current }
/** Milliseconds left at the rate achieved so far, or null before it means anything. */
fun etaMs(group: String? = null): Long? {
val f = fraction(group) ?: return null
// Under a few percent the extrapolation is dominated by connect time
// and produces numbers worse than saying nothing.
if (f < 0.02) return null
val oldestStart = live(group).minOfOrNull { it.startedMs } ?: return null
val elapsed = nowMillis() - oldestStart
if (elapsed < 5_000) return null
return ((elapsed / f) - elapsed).toLong()
}
}
@@ -29,10 +29,14 @@ object FiltersChanged {
): Boolean {
if (oldFilters.size != newFilters.size) return true
oldFilters.forEachIndexed { index, oldFilter ->
val newFilter = newFilters.getOrNull(index) ?: return true
return needsToResendRequest(oldFilter, newFilter)
// Every filter must be compared. This used to be a forEachIndexed whose body
// `return`ed on the first iteration — a non-local return from this function — so
// only filters[0] was ever checked and a subscription whose first filter happened
// to be unchanged reported "no resend needed" however much the rest had changed,
// leaving the relay serving a stale filter set.
// Indexing is safe: the sizes were just proven equal.
for (i in oldFilters.indices) {
if (needsToResendRequest(oldFilters[i], newFilters[i])) return true
}
return false
}
@@ -84,25 +84,26 @@ class PoolRequests(
/*
* Locking model: every compound access to a subscription's state machine
* ([RequestSubscriptionState]) — including the check-then-send decision in
* [decideCommandLocked] — runs inside THAT subscription's own lock
* ([RequestSubscriptionState.withLock]).
* [decideCommandLocked] — runs inside the stripe for THAT (subscription, relay)
* pair ([RequestSubscriptionState.withLock], which takes the relay).
*
* A single subscription can span many relays, and each relay's
* socket-reader thread delivers messages into this class concurrently
* while the app thread adds/removes subscriptions — so the plain maps
* inside [RequestSubscriptionState] are written from several threads at
* once. That is both a memory hazard (concurrent map mutation) and a
* logic hazard: two threads must never both observe "no REQ in flight"
* and both send a REQ for the same sub id.
* A single subscription can span many relays, and each relay's socket-reader
* thread delivers messages into this class concurrently while the app thread
* adds/removes subscriptions. Two threads must never both observe "no REQ in
* flight" and both send a REQ for the same (sub, relay).
*
* The lock is per subscription, not global, because different subIds
* share no wire state: EVENT frames for different subs coming from
* different relay consumer threads must not serialize on each other (a
* global lock here measured negative scaling under 4 concurrent relay
* feeders). Listener callbacks and the actual socket sends are ALWAYS
* performed outside the lock — they re-enter this class through
* [onSent], so holding the lock across them would self-deadlock, and the
* lock is non-reentrant. Never hold two subscriptions' locks at once.
* The lock is striped PER RELAY rather than per subscription: every critical
* section below touches only one relay's slice of the state, so EVENT frames
* arriving for the same subId from different relays no longer serialize on each
* other. With ~191 relays that previously made a single per-sub lock contended
* ~191 threads deep — the production ANR in
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`.
*
* Two rules this file must keep:
* - Never hold two relays' stripes (or two subscriptions') at once. Every loop
* below locks exactly one relay at a time.
* - Listener callbacks and socket sends are ALWAYS performed outside the lock —
* they re-enter this class through [onSent].
*/
/**
@@ -201,7 +202,7 @@ class PoolRequests(
fun onConnecting(url: NormalizedRelayUrl) {
// Change states to connecting. One sub's lock at a time.
relayState.forEach { subId, state ->
state.withLock { state.connecting(url) }
state.withLock(url) { state.connecting(url) }
}
}
@@ -215,7 +216,7 @@ class PoolRequests(
when (cmd) {
is ReqCmd -> {
subState(cmd.subId).let { state ->
state.withLock { state.onOpenReq(relay, cmd.filters) }
state.withLock(relay) { state.onOpenReq(relay, cmd.filters) }
}
desiredSubListeners.get(cmd.subId)?.onSubscriptionStarted(
relay = relay.url,
@@ -225,7 +226,7 @@ class PoolRequests(
is CloseCmd -> {
subState(cmd.subId).let { state ->
state.withLock { state.onSubscriptionClosed(relay) }
state.withLock(relay) { state.onSubscriptionClosed(relay) }
}
desiredSubListeners.get(cmd.subId)?.onSubscriptionClosed(
relay = relay.url,
@@ -246,7 +247,7 @@ class PoolRequests(
var isLive = false
var forFilters: List<Filter>? = null
relayState.get(msg.subId)?.let { state ->
state.withLock {
state.withLock(relay.url) {
state.onNewEvent(relay.url)
isLive = state.currentState(relay.url) == ReqSubStatus.LIVE
forFilters = state.lastKnownFilterStates(relay.url)
@@ -264,7 +265,7 @@ class PoolRequests(
var forFilters: List<Filter>? = null
val cmd =
relayState.get(msg.subId)?.let { state ->
state.withLock {
state.withLock(relay.url) {
state.onEose(relay.url)
forFilters = state.lastKnownFilterStates(relay.url)
// Decide (and pre-mark) the resend while still holding the
@@ -293,7 +294,7 @@ class PoolRequests(
var forFilters: List<Filter>? = null
val cmd =
relayState.get(msg.subId)?.let { state ->
state.withLock {
state.withLock(relay.url) {
state.onClosed(relay.url)
forFilters = state.lastKnownFilterStates(relay.url)
recordRefusalIfStructural(state, relay.url, msg.message, forFilters)
@@ -322,7 +323,7 @@ class PoolRequests(
*/
fun onDisconnected(url: NormalizedRelayUrl) {
relayState.forEach { subId, state ->
state.withLock { state.disconnected(url) }
state.withLock(url) { state.disconnected(url) }
}
}
@@ -351,7 +352,7 @@ class PoolRequests(
if (!filters.isNullOrEmpty()) {
val send =
subState(subId).let { state ->
state.withLock {
state.withLock(relay) {
if (isStructurallyRefused(state, relay, filters)) {
false
} else {
@@ -379,7 +380,7 @@ class PoolRequests(
// These are all my subs.. need to figure out which relays have them
val subs = desiredSubs.get(subId)
if (subs != null && url in subs.keys) {
toNotify.add(subId to state.withLock { state.lastKnownFilterStates(url) })
toNotify.add(subId to state.withLock(url) { state.lastKnownFilterStates(url) })
}
}
@@ -400,7 +401,7 @@ class PoolRequests(
val state = subState(subId)
relaysToUpdate.forEach { relay ->
// Decide + pre-mark atomically under the sub's lock, then send outside it.
val cmd = state.withLock { decideCommandLocked(state, subId, relay) }
val cmd = state.withLock(relay) { decideCommandLocked(state, subId, relay) }
if (cmd != null) {
sync(relay, cmd)
}
@@ -21,84 +21,116 @@
package com.vitorpamplona.quartz.nip01Core.relay.client.reqs
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import kotlin.concurrent.atomics.AtomicBoolean
import kotlin.concurrent.atomics.ExperimentalAtomicApi
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
import com.vitorpamplona.quartz.utils.concurrent.PlatformLock
/**
* Manages the State of Subscriptions by logging states as the
* subscription progresses.
*
* Thread-safety: the plain maps below are only touched inside [withLock].
* The lock lives HERE — one per subscription — instead of a single global
* lock in PoolRequests, because every mutation is scoped to one subId:
* relays delivering EVENTs for *different* subscriptions have no shared
* state and must not serialize on each other. (A single global spin lock
* measured *negative* scaling: 4 relay consumer threads pushed less
* aggregate throughput through it than 1 — see
* quartz/plans/2026-07-02-nostrclient-receiver-perf.md.)
* **Thread-safety: the lock is striped per reference (per relay), not per
* subscription.** Every operation below is scoped to a single [reference] — all state
* lives in [RelayState] objects held in a [ConcurrentMap] keyed by it — so two relays
* delivering EVENTs for the SAME subscription touch disjoint state and no longer
* serialize on each other.
*
* That mattered in production: one subId spans every relay it is subscribed on, so a
* single per-subscription lock was contended ~191 threads deep on the Pixel 8 that
* produced `anr_2026-08-03-12-55-26-256` (it was a *spin* lock then, which burned 6 of
* 9 cores — see [PlatformLock]). Striping removes the contention instead of making
* waiting cheaper: measured 1.5-2.8x the throughput of one lock per sub in
* `quartz/src/jvmTest/.../prodbench/LockDesignComparisonBenchmark.kt`. Full analysis,
* including why a suspending `Mutex` was rejected, is in
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`.
*
* The stripe array is allocated once and NEVER mutated, so a stripe's identity is
* stable for this object's whole life. That is load-bearing: if locks lived inside the
* per-relay values, a thread holding one while another thread dropped and re-created
* that entry would leave both "inside" the critical section excluding nothing.
*
* Callers MUST NOT hold two references' stripes at once ([PoolRequests] locks one relay
* at a time, including inside its all-subs iterations), and MUST keep socket sends and
* listener callbacks outside the critical section — they re-enter this class through
* `onSent`, and the point of a lock is to be held briefly.
*/
@OptIn(ExperimentalAtomicApi::class)
class RequestSubscriptionState<T> {
class RequestSubscriptionState<T : Any> {
/**
* Tiny non-reentrant spin lock (same primitive as BasicRelayClient's
* connecting mutex). Critical sections are a handful of map operations,
* never I/O — callers MUST NOT re-enter and MUST NOT hold two
* subscriptions' locks at once (PoolRequests locks one sub at a time,
* including inside its all-subs iterations).
* One reference's (relay's) slice of this subscription's state. Plain `var`s: every
* field is written and read under that reference's stripe by [PoolRequests].
*
* `@PublishedApi internal` only because [withLock] is inline (this sits
* on the per-EVENT hot path; inlining avoids a closure allocation per
* message) — treat it as private.
* Note `RelayActiveRequestStates` uses this class WITHOUT locking. There the fields
* may be read stale — but the backing [ConcurrentMap] can no longer be structurally
* corrupted the way the plain `HashMap`s this replaced could.
*/
private class RelayState {
/** Null == no REQ state on this relay (fresh, or wiped by connecting/disconnected). */
var status: ReqSubStatus? = null
/** Filters of the REQ currently believed to be in flight. */
var filters: List<Filter>? = null
/**
* Survives connect/disconnect so that if new events still arrive we can link
* them with the filters the relay was processing.
*/
var lastKnownFilters: List<Filter>? = null
/**
* Refused-filter memory. Unlike [status]/[filters] — per-connection wire state
* wiped by [connecting]/[disconnected] — this SURVIVES reconnects on purpose: a
* relay that structurally refuses a filter (a search-only relay CLOSING a plain
* kinds REQ, a relay that "does not accept REQs", "too many filters", …) refuses
* it again on every new socket, so [PoolRequests.syncState] replaying it each
* reconnect is pure waste. [refusalCount] accumulates repeated refusals of the
* same shape so a one-off (transient) close isn't mistaken for a structural one.
* Cleared on a successful REQ ([onEose]/[onNewEvent]) or when the caller observes
* the desired filter meaningfully changed.
*/
var refusedFilters: List<Filter>? = null
var refusalCount: Int = 0
}
private val states = ConcurrentMap<T, RelayState>()
/**
* Fixed stripe array — allocated once, never mutated, so lock identity is stable.
* [STRIPE_COUNT] stripes over ~191 relays is roughly 6-way sharing: a ~32x
* contention reduction versus one lock per subscription. References colliding on a
* stripe merely serialize; correctness never depends on N.
*/
@PublishedApi
internal val lock = AtomicBoolean(false)
internal val stripes = Array(STRIPE_COUNT) { PlatformLock() }
inline fun <R> withLock(block: () -> R): R {
while (lock.exchange(true)) {
// Test-and-test-and-set: spin-read until it looks free (cheaper
// on the cache line than hammering exchange), then retry above.
while (lock.load()) { }
}
@PublishedApi
internal fun stripeFor(reference: T): PlatformLock = stripes[(reference.hashCode() and 0x7FFFFFFF) % STRIPE_COUNT]
/**
* Runs [block] holding [reference]'s stripe. Inline so the per-EVENT hot path
* allocates no closure.
*/
inline fun <R> withLock(
reference: T,
block: () -> R,
): R {
val lock = stripeFor(reference)
lock.lock()
try {
return block()
} finally {
lock.store(false)
lock.unlock()
}
}
// Logs the state of each channel to:
// 1. inform when an event is received as live
// 2. to block REQs being sent before finished (receiving an EOSE or Closed)
//
// If 2 happens, the relay might send multiple EOSEs in sequence
// for the same sub and we won't know which REQ was it for.
private val subStates = mutableMapOf<T, ReqSubStatus>()
private val filterStates = mutableMapOf<T, List<Filter>>()
/** Read-only lookup — never creates an entry. */
private fun peek(reference: T): RelayState? = states[reference]
/**
* This cache is used to make sure we know what the relay was processing
* before a close or disconnect so that if new events still arrive
* we can link them with the appropriate filters.
*/
private val lastKnownFilterStates = mutableMapOf<T, List<Filter>>()
/** Write lookup — creates the entry on first use. */
private fun mutable(reference: T): RelayState = states.getOrPut(reference) { RelayState() }
/**
* Refused-filter memory. Unlike [subStates]/[filterStates] above — per-connection
* wire state wiped by [connecting]/[disconnected] — this SURVIVES reconnects on
* purpose: a relay that structurally refuses a filter (a search-only relay CLOSING
* a plain kinds REQ, a relay that "does not accept REQs", "too many filters", …)
* refuses it again on every new socket, so [PoolRequests.syncState] replaying it
* each reconnect is pure waste. [refusalCounts] accumulates repeated refusals of
* the same shape so a one-off (transient) close isn't mistaken for a structural
* one. Cleared on a successful REQ ([onEose]/[onNewEvent]) or when the caller
* observes the desired filter meaningfully changed.
*/
private val refusedFilters = mutableMapOf<T, List<Filter>>()
private val refusalCounts = mutableMapOf<T, Int>()
fun refusedFilters(reference: T) = peek(reference)?.refusedFilters
fun refusedFilters(reference: T) = refusedFilters[reference]
fun refusalCount(reference: T) = refusalCounts[reference] ?: 0
fun refusalCount(reference: T) = peek(reference)?.refusalCount ?: 0
/**
* Records that [reference] refused [filters]. [sameAsLastRefusal] must be true when
@@ -111,73 +143,91 @@ class RequestSubscriptionState<T> {
filters: List<Filter>,
sameAsLastRefusal: Boolean,
) {
val state = mutable(reference)
if (sameAsLastRefusal) {
refusalCounts[reference] = refusalCount(reference) + 1
state.refusalCount += 1
} else {
refusedFilters[reference] = filters
refusalCounts[reference] = 1
state.refusedFilters = filters
state.refusalCount = 1
}
}
fun clearRefusal(reference: T) {
refusedFilters.remove(reference)
refusalCounts.remove(reference)
peek(reference)?.let {
it.refusedFilters = null
it.refusalCount = 0
}
}
fun currentFilters() = filterStates
fun currentFilters(reference: T) = peek(reference)?.filters
fun currentFilters(reference: T) = filterStates[reference]
fun lastKnownFilterStates(reference: T) = peek(reference)?.lastKnownFilters
fun lastKnownFilterStates(reference: T) = lastKnownFilterStates[reference]
fun currentState(reference: T) = subStates[reference]
fun currentState(reference: T) = peek(reference)?.status
fun onNewEvent(reference: T) {
val state = mutable(reference)
// The relay is serving this REQ (it matched an event), so any past refusal
// no longer applies — let it be tried freely again.
clearRefusal(reference)
if (subStates[reference] == ReqSubStatus.SENT) {
subStates[reference] = ReqSubStatus.QUERYING_PAST
state.refusedFilters = null
state.refusalCount = 0
if (state.status == ReqSubStatus.SENT) {
state.status = ReqSubStatus.QUERYING_PAST
}
}
fun onEose(reference: T) {
val state = mutable(reference)
// Reaching EOSE means the relay accepted and finished the REQ; clear any refusal.
clearRefusal(reference)
subStates[reference] = ReqSubStatus.LIVE
state.refusedFilters = null
state.refusalCount = 0
state.status = ReqSubStatus.LIVE
}
fun onClosed(reference: T) {
subStates[reference] = ReqSubStatus.CLOSED
// Closed messages are usually relays refusing to process a REQ
// This message keeps the state of filterStates intact to
// avoid sending the same filter, and getting immediately closed,
// over and over again.
// filterStates.remove(reference)
// Closed messages are usually relays refusing to process a REQ. This keeps
// [RelayState.filters] intact to avoid sending the same filter, and getting
// immediately closed, over and over again.
mutable(reference).status = ReqSubStatus.CLOSED
}
fun onOpenReq(
reference: T,
filters: List<Filter>,
) {
subStates[reference] = ReqSubStatus.SENT
filterStates[reference] = filters
lastKnownFilterStates[reference] = filters
val state = mutable(reference)
state.status = ReqSubStatus.SENT
state.filters = filters
state.lastKnownFilters = filters
}
fun onSubscriptionClosed(reference: T) {
subStates[reference] = ReqSubStatus.CLOSED
filterStates.remove(reference)
val state = mutable(reference)
state.status = ReqSubStatus.CLOSED
state.filters = null
}
fun connecting(reference: T) {
subStates.remove(reference)
filterStates.remove(reference)
// Wipes per-connection wire state only; lastKnownFilters and the refusal memory
// deliberately survive (see [RelayState]).
peek(reference)?.let {
it.status = null
it.filters = null
}
}
fun disconnected(reference: T) {
subStates.remove(reference)
filterStates.remove(reference)
peek(reference)?.let {
it.status = null
it.filters = null
}
}
companion object {
/**
* Comfortably above the IO dispatcher's thread count (64 / 2) so collisions stay
* rare even when many workers are inside this class at once.
*/
const val STRIPE_COUNT = 32
}
}
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.relay.filters
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.tags.isIndexableTagName
import kotlinx.collections.immutable.PersistentMap
import kotlinx.collections.immutable.PersistentSet
import kotlinx.collections.immutable.persistentHashMapOf
@@ -238,7 +239,7 @@ class FilterIndex<S : Any> {
s.kinds[event.kind]?.let { result.addAll(it) }
if (s.tags.isNotEmpty()) {
for (tag in event.tags) {
if (tag.size >= 2 && tag[0].length == 1) {
if (tag.size >= 2 && isIndexableTagName(tag[0])) {
s.tags[tag[0]]?.get(tag[1])?.let { result.addAll(it) }
}
}
@@ -348,14 +349,14 @@ class FilterIndex<S : Any> {
if (!filter.tags.isNullOrEmpty()) {
val first =
filter.tags.entries.firstOrNull {
it.key.length == 1 && it.value.isNotEmpty()
isIndexableTagName(it.key) && it.value.isNotEmpty()
}
if (first != null) return first.value.map { TagKey(first.key, it) }
}
if (!filter.tagsAll.isNullOrEmpty()) {
val first =
filter.tagsAll.entries.firstOrNull {
it.key.length == 1 && it.value.isNotEmpty()
isIndexableTagName(it.key) && it.value.isNotEmpty()
}
if (first != null) return first.value.map { TagKey(first.key, it) }
}
@@ -44,6 +44,11 @@ fun NormalizedRelayUrl.toHttp() =
"https://$url"
}
fun NormalizedRelayUrl.isOnion() = url.contains(".onion/")
// Delegates rather than re-implementing `contains(".onion/")`: that copy missed
// `wss://host.onion:8080/`, so an onion relay on an explicit port was never forced onto Tor.
fun NormalizedRelayUrl.isOnion() = RelayUrlNormalizer.isOnion(this.url)
fun NormalizedRelayUrl.isLocalHost() = RelayUrlNormalizer.isLocalHost(this.url)
/** True for a relay inside an encrypted IPv6 overlay mesh. See [RelayUrlNormalizer.isOverlayNetwork]. */
fun NormalizedRelayUrl.isOverlayNetwork() = RelayUrlNormalizer.isOverlayNetwork(this.url)
@@ -21,6 +21,8 @@
package com.vitorpamplona.quartz.nip01Core.relay.normalizer
import androidx.collection.LruCache
import com.vitorpamplona.quartz.utils.Ipv4
import com.vitorpamplona.quartz.utils.Ipv6
import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.Rfc3986
import kotlinx.coroutines.CancellationException
@@ -38,15 +40,183 @@ val normalizedUrls = LruCache<String, NormalizationResult>(5000)
class RelayUrlNormalizer {
companion object {
fun isLocalHost(url: String) =
url.contains("127.0.0.1") ||
url.contains("localhost") ||
url.contains("//umbrel:") ||
url.contains("192.168.") ||
url.contains(".local:") ||
url.contains(".local/")
/**
* Every host test below is anchored to the **authority** (`host[:port]`), never to the
* whole url. A plain `contains` reads the path and query too, so
* `wss://evil.example.com/127.0.0.1` used to answer true here — and since [isLocalHost]
* is what exempts a relay from Tor, any relay list could hand the app a url that quietly
* dropped its own Tor routing. Relay urls arrive from other people (NIP-65 lists, relay
* hints, `r` tags), so they are attacker-controlled input and have to be parsed as such.
*
* Anchoring is also what makes `host:port` work: `.onion:8080` never matched the old
* `.onion/` test, so an onion relay on an explicit port was not recognized as onion at
* all and its hostname went to the clearnet DNS resolver.
*/
fun isLocalHost(url: String): Boolean {
val start = hostStart(url)
val end = hostEnd(url, start)
if (end <= start) return false
val hostEnd = hostEndWithoutPort(url, start, end)
return isPrivateIpv4(url, start, hostEnd) ||
// RFC 6761: `localhost` and anything under it — the same rule SurgeDns applies
// when deciding whether a loopback answer is legitimate. A substring test would
// also match `notlocalhost.example.com`, which is registrable.
regionEquals(url, "localhost", start, hostEnd) ||
regionEndsWith(url, ".localhost", start, hostEnd) ||
regionEquals(url, "umbrel", start, hostEnd) ||
regionEndsWith(url, ".local", start, hostEnd) ||
isPrivateIpv6(url, start, end)
}
fun isOnion(url: String) = url.endsWith(".onion") || url.contains(".onion/")
/**
* The IPv4 ranges that are never a public relay: `127.0.0.0/8` loopback, the RFC 1918
* private blocks, `169.254.0.0/16` link-local and `0.0.0.0/8`.
*
* Parsed rather than substring-matched, which was wrong both ways: `contains("192.168.")`
* missed `10.0.0.5` and `172.16.3.4` — so a LAN relay was given `wss://` and dialed
* through Tor — while matching `192.168.evil.com`, a registrable domain that could
* therefore exempt itself from Tor.
*/
private fun isPrivateIpv4(
url: String,
start: Int,
end: Int,
): Boolean {
val bytes = Ipv4.parse(url, start, end) ?: return false
return Ipv4.isLoopback(bytes) ||
Ipv4.isPrivate(bytes) ||
Ipv4.isLinkLocal(bytes) ||
Ipv4.isUnspecified(bytes)
}
fun isOnion(url: String): Boolean {
val start = hostStart(url)
val end = hostEnd(url, start)
if (end <= start) return false
return regionEndsWith(url, ".onion", start, hostEndWithoutPort(url, start, end))
}
/**
* True for a relay inside an encrypted IPv6 overlay mesh — today `0200::/7`, the range
* Yggdrasil derives node addresses and subnets from.
*
* Unlike [isLocalHost] this is not a private address: it is reachable from anywhere on
* the mesh. But it is unreachable *off* the mesh, which has two consequences the relay
* stack has to honour — it can never be dialed through a SOCKS/Tor proxy, and it can
* never present a CA-issued certificate, so it speaks plain `ws://`. Both are safe:
* the overlay already encrypts end to end and authenticates the peer by its address,
* which is derived from the peer's public key.
*/
fun isOverlayNetwork(url: String): Boolean {
val start = hostStart(url)
val bytes = ipv6HostOf(url, start, hostEnd(url, start)) ?: return false
return Ipv6.isOverlayMesh(bytes)
}
/**
* The IPv6 twins of the literals in [isLocalHost]: `::1` (127.0.0.1), `fc00::/7` unique
* local addresses (192.168.0.0/16) and `fe80::/10` link-local. All three name a host that
* only exists on this machine or this LAN, which is what every caller of [isLocalHost]
* means by the question — so a relay on one must not be Torified, must not need TLS,
* and must not be advertised to the network.
*/
private fun isPrivateIpv6(
url: String,
start: Int,
end: Int,
): Boolean {
val bytes = ipv6HostOf(url, start, end) ?: return false
return Ipv6.isLoopback(bytes) || Ipv6.isUniqueLocal(bytes) || Ipv6.isLinkLocal(bytes)
}
/**
* Parses the authority of [url] as a bracketed IPv6 literal, dropping any `%zone` suffix.
* Returns null — on a single char comparison — for the overwhelmingly common case of a
* url with a DNS host.
*/
private fun ipv6HostOf(
url: String,
start: Int,
end: Int,
): ByteArray? {
if (start >= end || url[start] != '[') return null
val close = url.indexOf(']', start + 1)
if (close < 0 || close >= end || close <= start + 1) return null
val zone = url.indexOf('%', start + 1)
val addressEnd = if (zone in (start + 1) until close) zone else close
return Ipv6.parse(url.substring(start + 1, addressEnd))
}
/**
* Index of the first char of the authority: past `://`, or 0 for a schemeless host.
*
* The `://` only counts when what precedes it is a real RFC 3986 scheme
* (`ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )`). Otherwise `relay.com/x://127.0.0.1`
* would have its *path* read as the authority and answer true to [isLocalHost].
*/
private fun hostStart(url: String): Int {
val scheme = url.indexOf("://")
if (scheme <= 0 || !url[0].isLetter()) return 0
for (i in 1 until scheme) {
val c = url[i]
if (!c.isLetterOrDigit() && c != '+' && c != '-' && c != '.') return 0
}
return scheme + 3
}
/** Index just past the authority — the first `/`, `?` or `#`, or the end of [url]. */
private fun hostEnd(
url: String,
start: Int,
): Int {
var i = start
while (i < url.length) {
val c = url[i]
if (c == '/' || c == '?' || c == '#') return i
i++
}
return i
}
/**
* [end] trimmed back past a `:port` and any trailing dots, so the host tests see the
* name alone. RFC 1034's fully-qualified form ends in a dot (`abc.onion.`), and missing
* that spelling on [isOnion] would send a `.onion` name to the clearnet DNS resolver.
*/
private fun hostEndWithoutPort(
url: String,
start: Int,
end: Int,
): Int {
var stop =
if (url[start] == '[') {
// In an IPv6 authority only the `:` after the `]` can start a port.
val close = url.indexOf(']', start + 1)
if (close in start until end) close + 1 else end
} else {
val colon = url.lastIndexOf(':', end - 1)
if (colon >= start) colon else end
}
while (stop > start && url[stop - 1] == '.') stop--
return stop
}
// Host names are case-insensitive (RFC 4343), and `fix()` asks these questions *before*
// the RFC 3986 pass folds the case — so a case-sensitive test gave `LOCALHOST:8080` and
// `ABC.ONION:8080` a `wss://` scheme neither host can ever serve.
private fun regionEndsWith(
url: String,
suffix: String,
start: Int,
end: Int,
): Boolean = end - start >= suffix.length && url.regionMatches(end - suffix.length, suffix, 0, suffix.length, ignoreCase = true)
private fun regionEquals(
url: String,
name: String,
start: Int,
end: Int,
): Boolean = end - start == name.length && url.regionMatches(start, name, 0, name.length, ignoreCase = true)
fun isRelaySchemePrefix(url: String) = url.length > 6 && url[0] == 'w' && url[1] == 's'
@@ -82,19 +252,135 @@ class RelayUrlNormalizer {
return false
}
private fun norm(url: String) = NormalizedRelayUrl(Rfc3986.normalize(url))
private fun norm(url: String) = NormalizedRelayUrl(canonicalizeIpv6Host(Rfc3986.normalize(url)))
/**
* Rewrites a bracketed IPv6 host into its RFC 5952 canonical form.
*
* RFC 4291 lets one address be spelled many ways, and the RFC 3986 pass only folds hex
* case — so `[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]`
* survive as two different [NormalizedRelayUrl]s for one host. That value keys the
* connection pool, the relay-list sets, the NIP-11 cache and the per-relay stats, so the
* app would dial the same relay twice and count it twice. OkHttp canonicalizes to this
* exact form when it dials, so folding here makes the stored key the host on the wire.
*
* Returns [url] itself — no allocation — when there is no literal or it is already
* canonical, which is every url with a DNS host.
*/
private fun canonicalizeIpv6Host(url: String): String {
// Anchored to the authority: a `[...]` in a path or query is data, and rewriting it
// would silently corrupt the url.
val open = hostStart(url)
if (open >= url.length || url[open] != '[') return url
val close = url.indexOf(']', open + 1)
if (close <= open + 1 || close >= hostEnd(url, open)) return url
val inner = url.substring(open + 1, close)
val canonical = Ipv6.canonicalizeOrNull(inner) ?: return url
if (canonical == inner) return url
return url.substring(0, open + 1) + canonical + url.substring(close)
}
private fun isInvisible(c: Char) = c == '\u200B' || c == '\u200C' || c == '\u200D' || c == '\u2060' || c == '\uFEFF'
/**
* Scans the authority (host[:port]) that starts at [start] and ends at the first
* `/`, `?` or `#`. Returns the end index, or -1 when the authority is empty or
* contains characters that never appear in a real relay host (`@` userinfo,
* percent-encoding, commas).
*/
private fun authorityEnd(
url: String,
start: Int,
): Int {
if (start >= url.length) return -1
var i = start
if (url[i] == '[') {
// IPv6 literal: defer validation to the RFC 3986 parser
while (i < url.length && url[i] != '/' && url[i] != '?' && url[i] != '#') i++
return i
}
while (i < url.length) {
val c = url[i]
if (c == '/' || c == '?' || c == '#') break
if (c == '@' || c == '%' || c == ',') return -1
i++
}
return if (i == start) -1 else i
}
/**
* Accepts a ws/wss url whose host starts at [hostStart] if the authority is sane
* and the path does not start with `//` (the signature of a second URL or a broken
* `https//` pasted after the scheme, e.g. `wss://https//nostr.watch/relay/x`).
*/
private fun fixWs(
url: String,
hostStart: Int,
): String? {
val end = authorityEnd(url, hostStart)
if (end < 0) return null
if (end + 1 < url.length && url[end] == '/' && url[end + 1] == '/') return null
return url
}
/**
* Converts an http(s) url to ws(s) only when it is a bare host — nothing after
* `host[:port]` but an optional trailing `/`. An http url with a path, query or
* fragment (Mastodon actor urls from bridge `proxy` tags, web pages, images) is
* a web resource, not a relay: converting it creates a wss:// url that can never
* answer and only wastes connection attempts.
*/
private fun fixHttp(
url: String,
hostStart: Int,
newScheme: String,
): String? {
val end = authorityEnd(url, hostStart)
if (end < 0) return null
val bareHost = end == url.length || (end == url.length - 1 && url[end] == '/')
if (!bareHost) return null
return "$newScheme${url.substring(hostStart)}"
}
/**
* Validates a schemeless candidate: the part before the first `/` must look like
* `host` or `host:port` — letters, digits, `.`, `-`, `_`, plus at most one `:`
* followed by digits only. Rejects addressable-event pointers (`31990:hex:dtag`),
* bare scheme leftovers (`wss:`) and anything else that would otherwise be blindly
* prefixed with `wss://`.
*/
private fun isBareHostAndPath(url: String): Boolean {
if (url[0] == '[') return true // IPv6 literal: defer to the RFC 3986 parser
var i = 0
var portStart = -1
while (i < url.length) {
val c = url[i]
if (c == '/') break
if (c == ':') {
if (portStart >= 0) return false
portStart = i + 1
} else if (portStart >= 0) {
if (c < '0' || c > '9') return false
} else if (!c.isLetterOrDigit() && c != '.' && c != '-' && c != '_') {
return false
}
i++
}
if (i == 0) return false
if (portStart >= 0 && portStart == i) return false
return true
}
@OptIn(ExperimentalContracts::class)
fun fix(rawUrl: String): String? {
if (rawUrl.length < 4) return null
if (rawUrl.contains("%00")) return null
// Trim trailing %20 (percent-encoded spaces from malformed event data)
val url =
rawUrl.trimEnd('%', '2', '0').let { trimmed ->
// Only accept if we actually removed a trailing %20 pattern
if (trimmed.length < rawUrl.length && rawUrl.endsWith("%20")) trimmed else rawUrl
}
// Trim trailing %20 (percent-encoded spaces from malformed event data).
// The endsWith gate keeps the hot path allocation-free: trimEnd would
// copy the string for ANY url merely ending in '%', '2' or '0' — which
// includes every port ending in zero ("wss://host:3030").
val url = if (rawUrl.endsWith("%20")) rawUrl.trimEnd('%', '2', '0') else rawUrl
if (url.length < 4) return null
// Reject URLs with %20 in the middle — these are garbage
@@ -109,17 +395,33 @@ class RelayUrlNormalizer {
}
}
val trimmed =
var trimmed =
if (url[0].isWhitespace() || url[url.length - 1].isWhitespace()) {
url.trim()
} else {
url
}
// Single pass: interior whitespace means multiple urls or prose in one field,
// backslashes never appear in a real relay url; both are garbage. Invisible
// characters (zero-width spaces, BOM) are copy-paste artifacts — strip them.
var hasInvisible = false
for (c in trimmed) {
if (c == '\\') return null
if (c.isWhitespace()) return null
if (isInvisible(c)) hasInvisible = true
}
if (hasInvisible) {
trimmed = buildString(trimmed.length) { for (c in trimmed) if (!isInvisible(c)) append(c) }
if (trimmed.length < 4) return null
}
// fast for good wss:// urls
if (isRelaySchemePrefix(trimmed)) {
if (isRelaySchemePrefixSecure(trimmed) || isRelaySchemePrefixInsecure(trimmed)) {
return trimmed
if (isRelaySchemePrefixSecure(trimmed)) {
return fixWs(trimmed, 6)
} else if (isRelaySchemePrefixInsecure(trimmed)) {
return fixWs(trimmed, 5)
}
}
@@ -127,31 +429,31 @@ class RelayUrlNormalizer {
if (isHttpPrefix(trimmed)) {
if (isHttpSSuffix(trimmed)) {
// https://
return "wss://${trimmed.drop(8)}"
return fixHttp(trimmed, 8, "wss://")
} else if (isHttpSuffix(trimmed)) {
// http://
return "ws://${trimmed.drop(7)}"
return fixHttp(trimmed, 7, "ws://")
}
}
// fast for good ww:// urls
if (trimmed.startsWith("ww://")) {
return "wss://${trimmed.drop(5)}"
return fixWs("wss://${trimmed.drop(5)}", 6)
}
// fast for good ww:// urls
if (trimmed.startsWith("was://")) {
return "wss://${trimmed.drop(6)}"
return fixWs("wss://${trimmed.drop(6)}", 6)
}
// fast for good ww:// urls
if (trimmed.startsWith("Wws://")) {
return "wss://${trimmed.drop(6)}"
return fixWs("wss://${trimmed.drop(6)}", 6)
}
// fast for good ww:// urls
if (trimmed.startsWith("Wss://")) {
return "wss://${trimmed.drop(6)}"
return fixWs("wss://${trimmed.drop(6)}", 6)
}
if (trimmed.contains("://")) {
@@ -160,10 +462,33 @@ class RelayUrlNormalizer {
return null
}
return if (isOnion(trimmed) || isLocalHost(trimmed)) {
"ws://$trimmed"
// protocol-relative urls (`//host/`) are just missing the scheme
val protocolRelative = if (trimmed.startsWith("//")) trimmed.drop(2) else trimmed
if (protocolRelative.length < 4) return null
// A bare IPv6 literal is missing its brackets, not malformed. This is the shape a
// user actually has in hand — `yggdrasilctl getSelf` prints the address unbracketed
// — and without the brackets `isBareHostAndPath` rejects it below as a host with too
// many colons. Only a string that parses as a whole address is bracketed, so an
// addressable-event pointer (`31990:hex:dtag`) or a `host:port` still falls through.
val bare =
if (protocolRelative[0] != '[' && Ipv6.isLiteral(protocolRelative)) {
"[$protocolRelative]"
} else {
protocolRelative
}
if (!isBareHostAndPath(bare)) {
Log.d("RelayUrlNormalizer") { "Rejected $url" }
return null
}
// Overlay and localhost relays cannot hold a certificate, so wss:// could only ever
// fail its handshake. Both carry their own encryption, so ws:// is not a downgrade.
return if (isOnion(bare) || isLocalHost(bare) || isOverlayNetwork(bare)) {
"ws://$bare"
} else {
"wss://$trimmed"
"wss://$bare"
}
}
@@ -186,6 +511,13 @@ class RelayUrlNormalizer {
val fixed = fix(url)
if (fixed != null) {
val normalized = norm(fixed)
// the RFC 3986 parser can drop or replace the scheme on odd inputs;
// anything that is not ws(s):// at this point cannot be connected to.
if (!isRelayUrl(normalized.url)) {
Log.d("NormalizedRelayUrl") { "Rejected $url" }
normalizedUrls.put(url, NormalizationResult.Error)
return null
}
normalizedUrls.put(url, NormalizationResult.Success(normalized))
normalized
} else {
@@ -214,6 +214,13 @@ class RelaySession(
is IEventStore.InsertOutcome.Rejected -> {
send(OkMessage(cmd.event.id, false, outcome.reason))
}
is IEventStore.InsertOutcome.Failed -> {
// The store's error, not the event's — NIP-01's
// machine-readable prefix for that is "error:".
val reason = outcome.reason
send(OkMessage(cmd.event.id, false, if (reason.startsWith("error:")) reason else "error: $reason"))
}
}
}
} catch (_: ClosedSendChannelException) {
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.relay.server.backend
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
@@ -36,6 +37,7 @@ import kotlin.concurrent.atomics.AtomicBoolean
import kotlin.concurrent.atomics.AtomicInt
import kotlin.concurrent.atomics.ExperimentalAtomicApi
import kotlin.coroutines.CoroutineContext
import kotlin.coroutines.cancellation.CancellationException
/**
* Group-commit writer for incoming EVENT publishes.
@@ -268,8 +270,8 @@ class IngestQueue(
* Run the SQLite transaction for the verified subset of [batch]
* and stitch outcomes back to a per-batch-index array. Failed
* verifies pre-mark `Rejected` and skip the insert. A whole-batch
* commit failure converts every persisted entry to `Rejected`
* with the throw message.
* commit failure converts every persisted entry to `Failed` with
* the throw message — the events were good; the store was not.
*/
private suspend fun runInsertStage(
batch: List<Submission>,
@@ -290,10 +292,15 @@ class IngestQueue(
} else {
try {
store.batchInsert(toInsert)
} catch (e: CancellationException) {
// Shutdown, not a store failure: rethrow so the loop
// stops instead of stamping the batch Failed and
// carrying on while cancelled.
throw e
} catch (e: Throwable) {
Log.w("IngestQueue") { "batchInsert failed for ${toInsert.size} events: ${e.message}" }
val reason = e.message ?: e::class.simpleName ?: "insert failed"
List(toInsert.size) { IEventStore.InsertOutcome.Rejected(reason) }
val reason = e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED
List(toInsert.size) { IEventStore.InsertOutcome.Failed(reason) }
}
}
@@ -349,7 +356,7 @@ class IngestQueue(
* inserts), so the message is informational, not user-facing.
*/
private val missingOutcome =
IEventStore.InsertOutcome.Rejected("internal error: missing outcome")
IEventStore.InsertOutcome.Failed("internal error: missing outcome")
/**
* Cap per batch. Sized to keep per-batch latency low (each
@@ -180,6 +180,10 @@ class LiveEventStore(
is IEventStore.InsertOutcome.Rejected -> {
done.completeExceptionally(IllegalStateException(outcome.reason))
}
is IEventStore.InsertOutcome.Failed -> {
done.completeExceptionally(IllegalStateException(outcome.reason))
}
}
}
done.await()
@@ -0,0 +1,35 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
/**
* The pubkey that CONTROLS this event for ownership checks — NIP-09 deletion
* authority and NIP-62 vanish targeting. A gift wrap (kind 1059) is signed by
* a random one-time key, so control belongs to its p-tag RECIPIENT (the
* recipient deletes the wraps addressed to them); every other event is
* controlled by its author. One rule, shared by every store — do not re-derive
* it inline.
*/
fun Event.owner(): HexKey = (this as? GiftWrapEvent)?.recipientPubKey() ?: pubKey
@@ -27,6 +27,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
import kotlin.coroutines.cancellation.CancellationException
/**
* Storage contract for Nostr events: insert, filter-query, count, delete,
@@ -92,37 +93,64 @@ interface IEventStore : AutoCloseable {
/**
* Per-row outcome from [batchInsert]. The OK frame on the wire is
* built from this — `Accepted` becomes `OK true`, `Rejected.reason`
* becomes the false reason. NIP-01 says OK pairs to its EVENT by
* built from this — `Accepted` becomes `OK true`, the other two
* become the false reason. NIP-01 says OK pairs to its EVENT by
* id, not by order, so callers may dispatch outcomes in any order.
*/
sealed class InsertOutcome {
data object Accepted : InsertOutcome()
/**
* The EVENT's fault: policy said no — a duplicate, an expired
* or invalid event, a blocked author. Final: re-offering the
* same event yields the same answer, so dropping it is correct.
*/
data class Rejected(
val reason: String,
) : InsertOutcome()
/**
* The STORE's fault: the event was acceptable but could not be
* written — schema drift, a failed feed, a resource error. The
* event is lost unless the caller re-offers it, and nothing
* else will. Callers should count these apart from [Rejected]:
* a rising [Rejected] is usually the protocol working
* (duplicates on a wide fan-out), while a rising [Failed]
* means the store is losing good events.
*/
data class Failed(
val reason: String,
) : InsertOutcome()
}
/**
* Bulk insert in a single transaction with per-row error isolation.
* Returns one outcome per input event in the same order.
* Bulk insert with per-row attribution. Returns one outcome per
* input event in the same order.
*
* Implementations must isolate per-row failures so one bad event
* doesn't roll back the others (SQLite uses SAVEPOINTs). If the
* outer commit itself fails, every entry in the returned list is
* `Rejected` with the commit-failure reason.
* Implementations must isolate per-row problems so one bad event
* never costs the batch: a policy refusal is [InsertOutcome.Rejected],
* a store-side write error is [InsertOutcome.Failed] (SQLite uses
* SAVEPOINTs for the isolation). Throwing is reserved for failures
* with no per-event answer — the engine unreachable, the transaction
* never started — and a caller may read a throw as "nothing in this
* batch was written".
*
* Default impl runs each insert in its own transaction — correct
* but loses the group-commit win. SQLite overrides this.
* but loses the group-commit win — and cannot classify a throw from
* [insert], so it reports `Failed`: re-offering a duplicate is
* idempotent, while dropping a good event on a transient store
* error is not. Implementations that can tell a refusal from a
* write error should override and say which.
*/
suspend fun batchInsert(events: List<Event>): List<InsertOutcome> =
events.map { event ->
try {
insert(event)
InsertOutcome.Accepted
} catch (e: CancellationException) {
throw e
} catch (e: Throwable) {
InsertOutcome.Rejected(e.message ?: e::class.simpleName ?: "insert failed")
InsertOutcome.Failed(e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED)
}
}
@@ -52,15 +52,17 @@ object NdjsonImportExport {
val imported: Long,
/** Events the store rejected — overwhelmingly duplicates (unique-id). */
val rejected: Long,
/** Good events the store could not write — a store-side error, not the event's. */
val failed: Long,
/** Events dropped for a bad signature (only when verifying). */
val invalid: Long,
/** Lines that didn't parse as a NIP-01 event. */
val malformed: Long,
) {
operator fun plus(o: ImportStats) = ImportStats(read + o.read, imported + o.imported, rejected + o.rejected, invalid + o.invalid, malformed + o.malformed)
operator fun plus(o: ImportStats) = ImportStats(read + o.read, imported + o.imported, rejected + o.rejected, failed + o.failed, invalid + o.invalid, malformed + o.malformed)
companion object {
val ZERO = ImportStats(0, 0, 0, 0, 0)
val ZERO = ImportStats(0, 0, 0, 0, 0, 0)
}
}
@@ -81,6 +83,7 @@ object NdjsonImportExport {
var read = 0L
var imported = 0L
var rejected = 0L
var failed = 0L
var invalid = 0L
var malformed = 0L
val batch = ArrayList<Event>(batchSize)
@@ -91,6 +94,7 @@ object NdjsonImportExport {
when (outcome) {
IEventStore.InsertOutcome.Accepted -> imported++
is IEventStore.InsertOutcome.Rejected -> rejected++
is IEventStore.InsertOutcome.Failed -> failed++
}
}
batch.clear()
@@ -112,7 +116,7 @@ object NdjsonImportExport {
if (batch.size >= batchSize) flush()
}
flush()
return ImportStats(read, imported, rejected, invalid, malformed)
return ImportStats(read, imported, rejected, failed, invalid, malformed)
}
/**
@@ -103,7 +103,7 @@ class ObservableEventStore(
// order. Already-expired ephemerals are dropped (matching
// [insert]). Accepted events are emitted on [_changes] only
// after the inner batch returns, so a commit failure that
// converts everything to Rejected suppresses the emits.
// converts everything to Failed suppresses the emits.
if (events.isEmpty()) return emptyList()
val outcomes = arrayOfNulls<IEventStore.InsertOutcome>(events.size)
@@ -114,7 +114,7 @@ class ObservableEventStore(
if (event.kind.isEphemeral()) {
outcomes[i] =
if (event.isExpired()) {
IEventStore.InsertOutcome.Rejected("blocked: Cannot insert an expired event")
IEventStore.InsertOutcome.Rejected(RejectionReason.EXPIRED)
} else {
IEventStore.InsertOutcome.Accepted
}
@@ -0,0 +1,54 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store
/**
* The machine-readable insert-rejection vocabulary, shared by every
* [IEventStore] implementation so the same condition always rejects with the
* same words — a caller tallying [IEventStore.InsertOutcome.Rejected] reasons,
* or a relay building `OK false` frames, must never see two stores spell
* "duplicate" differently.
*
* NIP-01: an `OK false` message SHOULD begin with a single-word
* machine-readable prefix followed by `:`. The full-sentence constants here
* are the standard reasons the built-in stores emit. `replaced:` is not in
* NIP-01 but is the de-facto prefix (strfry and others) for a replaceable
* event that lost to a stored newer version — distinct from `duplicate:`
* (the exact event is already held).
*/
object RejectionReason {
/**
* The NIP-01 prefix vocabulary itself lives in
* [com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.MachineReadablePrefix]
* — use its `format`/`parse` instead of hand-writing prefixes. The one
* prefix that enum lacks is the de-facto (not in NIP-01) word for a stale
* version of a replaceable/addressable event:
*/
const val PREFIX_REPLACED = "replaced:"
// The standard store reasons.
const val DUPLICATE = "duplicate: already have this event"
const val EXPIRED = "blocked: Cannot insert an expired event"
const val DELETED = "blocked: a deletion event exists"
const val VANISHED = "blocked: a request to vanish event exists"
const val REPLACED = "replaced: a newer version exists"
const val INSERT_FAILED = "error: insert failed"
}
@@ -25,7 +25,7 @@ import com.vitorpamplona.quartz.nip01Core.core.AddressSerializer
import com.vitorpamplona.quartz.nip01Core.core.AddressableEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
import com.vitorpamplona.quartz.nip01Core.store.owner
class EventIndexesModule(
val hasher: (db: SQLiteConnection) -> TagNameValueHasher,
@@ -206,12 +206,8 @@ class EventIndexesModule(
val kindLong = event.kind.toLong()
val pubkeyHash = hasher.hash(event.pubKey)
val eventOwnerHash =
if (event is GiftWrapEvent) {
event.recipientPubKey()?.let { hasher.hash(it) } ?: pubkeyHash
} else {
pubkeyHash
}
val ownerKey = event.owner()
val eventOwnerHash = if (ownerKey == event.pubKey) pubkeyHash else hasher.hash(ownerKey)
val eTagHash = hasher.hashETag(event.id)
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.store.sqlite
import androidx.sqlite.SQLiteConnection
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
import com.vitorpamplona.quartz.nip40Expiration.expiration
class ExpirationModule : IModule {
@@ -44,7 +45,7 @@ class ExpirationModule : IModule {
FOR EACH ROW
BEGIN
-- Check for existing newer record
SELECT RAISE(ABORT, 'blocked: this event is expired')
SELECT RAISE(ABORT, '${RejectionReason.EXPIRED}')
WHERE NEW.expiration <= unixepoch();
END;
""".trimIndent(),
@@ -21,6 +21,7 @@
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import com.vitorpamplona.quartz.nip01Core.core.Tag
import com.vitorpamplona.quartz.nip01Core.tags.isIndexableTagName
interface IndexingStrategy {
/**
@@ -165,5 +166,5 @@ class DefaultIndexingStrategy(
override fun shouldIndex(
kind: Int,
tag: Tag,
) = tag.size >= 2 && tag[0].length == 1
) = tag.size >= 2 && isIndexableTagName(tag[0])
}
@@ -37,6 +37,7 @@ import com.vitorpamplona.quartz.nip01Core.store.FtsReindexProgress
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
import com.vitorpamplona.quartz.nip01Core.store.RawEvent
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
import com.vitorpamplona.quartz.nip40Expiration.isExpired
import com.vitorpamplona.quartz.nip50Search.strippingSearchExtensions
@@ -433,7 +434,7 @@ class SQLiteEventStore(
}
suspend fun insertEvent(event: Event) {
if (event.isExpired()) throw SQLiteException("blocked: Cannot insert an expired event")
if (event.isExpired()) throw SQLiteException(RejectionReason.EXPIRED)
if (event.kind.isEphemeral()) return
pool.useWriter { db ->
@@ -463,8 +464,9 @@ class SQLiteEventStore(
* stream still surfaces them; persistence is intentionally a
* no-op per NIP-01.
*
* Outer-commit failure throws; the caller treats every entry as
* `Rejected` (this is what the IEventStore contract documents).
* Outer-commit failure throws; per the IEventStore contract a
* throw means "nothing in this batch was written", and the
* IngestQueue converts it to per-event `Failed`.
*/
suspend fun batchInsertEvents(events: List<Event>): List<IEventStore.InsertOutcome> {
if (events.isEmpty()) return emptyList()
@@ -489,7 +491,7 @@ class SQLiteEventStore(
delta: LiveIndexDelta?,
): IEventStore.InsertOutcome {
if (event.isExpired()) {
return IEventStore.InsertOutcome.Rejected("blocked: Cannot insert an expired event")
return IEventStore.InsertOutcome.Rejected(RejectionReason.EXPIRED)
}
if (event.kind.isEphemeral()) return IEventStore.InsertOutcome.Accepted
@@ -509,7 +511,31 @@ class SQLiteEventStore(
// ROLLBACK shouldn't mask the original cause.
runCatching { db.execSQL("ROLLBACK TRANSACTION TO SAVEPOINT $sp") }
runCatching { db.execSQL("RELEASE SAVEPOINT $sp") }
IEventStore.InsertOutcome.Rejected(e.message ?: e::class.simpleName ?: "insert failed")
classifyRowError(e)
}
}
/**
* Which side failed decides whether the caller may drop the event.
* Policy refusals are recognizable — every schema trigger RAISEs with
* a `blocked:` prefix, the immutability guards say "not allowed", and
* a duplicate id is a constraint violation. Anything else (disk full,
* I/O error, schema drift) is the store failing to write an acceptable
* event: `Failed`, so a rising count is loud instead of blending into
* the duplicate tally.
*/
private fun classifyRowError(e: Throwable): IEventStore.InsertOutcome {
val message = e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED
val refusal =
message.contains("blocked:") ||
message.contains("duplicate:") ||
message.contains(RejectionReason.PREFIX_REPLACED) ||
message.contains("not allowed") ||
message.contains("constraint", ignoreCase = true)
return if (refusal) {
IEventStore.InsertOutcome.Rejected(message)
} else {
IEventStore.InsertOutcome.Failed(message)
}
}
@@ -518,7 +544,7 @@ class SQLiteEventStore(
private val delta: LiveIndexDelta?,
) : IEventStore.ITransaction {
override fun insert(event: Event) {
if (event.isExpired()) throw SQLiteException("blocked: Cannot insert an expired event")
if (event.isExpired()) throw SQLiteException(RejectionReason.EXPIRED)
if (event.kind.isEphemeral()) return
innerInsertEvent(event, db, delta)
@@ -0,0 +1,30 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.tags
/**
* Whether [name] is a tag name the NIP-01 `#x` filter space can address: a
* single ASCII LETTER (`a`-`z` / `A`-`Z`), per NIP-01's "single-letter
* (a-zA-Z) English-alphabet letters". Deliberately stricter than
* `length == 1`: a `"5"` or `"#"` tag name is not filterable and indexing
* it would let stores disagree about which tags `#x` filters reach.
*/
fun isIndexableTagName(name: String): Boolean = name.length == 1 && (name[0] in 'a'..'z' || name[0] in 'A'..'Z')
@@ -0,0 +1,97 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip11RelayInfo
import androidx.collection.LruCache
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlinx.coroutines.CancellationException
/**
* TTL cache around any [Nip11Fetcher]. NIP-11 documents change rarely, so a
* successful fetch is served from memory for [ttlSeconds]; a FAILED fetch is
* also remembered — for the shorter [errorTtlSeconds] — so a mass census does
* not hammer a host that just refused, while still retrying it soon.
*
* Concurrent first fetches of the same relay are not deduplicated: both hit the
* network and the second result wins the cache slot. That is harmless (the
* document is idempotent) and keeps this class lock-free.
*/
class CachedNip11Fetcher(
private val delegate: Nip11Fetcher,
private val ttlSeconds: Long = DEFAULT_TTL_SECONDS,
private val errorTtlSeconds: Long = DEFAULT_ERROR_TTL_SECONDS,
maxEntries: Int = 1000,
private val now: () -> Long = { TimeUtils.now() },
) : Nip11Fetcher {
private sealed interface Cached {
val at: Long
}
private class Hit(
val info: Nip11RelayInformation,
override val at: Long,
) : Cached
private class Miss(
val message: String?,
override val at: Long,
) : Cached
private val cache = LruCache<NormalizedRelayUrl, Cached>(maxEntries)
/** The cached document if present and fresh; null otherwise. Never touches the network. */
fun cachedOrNull(relay: NormalizedRelayUrl): Nip11RelayInformation? {
val hit = cache[relay] as? Hit ?: return null
return if (now() - hit.at < ttlSeconds) hit.info else null
}
/** Drops the cache entry (success or failure) so the next [fetch] is fresh. */
fun invalidate(relay: NormalizedRelayUrl) {
cache.remove(relay)
}
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation {
when (val cached = cache[relay]) {
is Hit -> if (now() - cached.at < ttlSeconds) return cached.info
is Miss ->
if (now() - cached.at < errorTtlSeconds) {
throw Nip11FetchException(cached.message ?: "cached NIP-11 failure for ${relay.url}")
}
null -> {}
}
return try {
delegate.fetch(relay).also { cache.put(relay, Hit(it, now())) }
} catch (e: Exception) {
if (e is CancellationException) throw e
cache.put(relay, Miss(e.message, now()))
throw e
}
}
companion object {
/** Documents are near-static: trust a success for a day. */
const val DEFAULT_TTL_SECONDS = 24L * 60 * 60
/** Failures are often transient: retry after five minutes. */
const val DEFAULT_ERROR_TTL_SECONDS = 5L * 60
}
}
@@ -0,0 +1,44 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip11RelayInfo
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
/**
* Fetches and parses a relay's NIP-11 information document (the
* `application/nostr+json` answer on the relay's https url). Mirrors the
* [com.vitorpamplona.quartz.nip05DnsIdentifiers.Nip05Fetcher] seam: the HTTP
* transport lives in a platform implementation (OkHttpNip11Fetcher on
* JVM/Android), so common code — probes, monitors, the CLI — depends only on
* this interface. Wrap any implementation in [CachedNip11Fetcher] to add a TTL
* cache.
*
* Throws [Nip11FetchException] (or a transport exception) when the document is
* unavailable or unparseable.
*/
interface Nip11Fetcher {
suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation
}
/** The relay answered, but not with a usable NIP-11 document (bad status, not JSON). */
class Nip11FetchException(
message: String,
) : Exception(message)
@@ -0,0 +1,94 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip50Search
/**
* An event's searchable text decomposed by ROLE, for full-text backends that
* weight fields instead of indexing one concatenated blob
* ([SearchableEvent.indexableContent]'s flat form). Produced per kind by
* [SearchFieldExtractor].
*
* A kind is either PROFILE-shaped ([Profile] — kind-0-style identity: kind 0
* itself, app handlers) or CONTENT-shaped ([Tiered] — title above summary
* above body). The shape is part of the value, so a consumer discriminates on
* the type instead of keeping its own list of profile kinds. Both shapes
* carry a website role — a profile's homepage ([Profile.website]), or a
* content kind's affiliation URLs ([Tiered.websites]: repo, trackers,
* bookmarked page) — declared on each shape rather than the interface, so
* [None] answers no question that doesn't apply to it.
*
* Multi-valued roles are carried UNJOINED, as lists: which separator to use —
* or whether to index the values separately — is the backend's decision, and
* once values are pre-joined a backend can't unmix them. Values produced by
* [SearchFieldExtractor] are trimmed and non-empty; the types themselves do
* not enforce it.
*/
sealed interface IndexableFields {
fun isEmpty(): Boolean
/** No searchable text — the extraction result for non-searchable kinds. */
data object None : IndexableFields {
override fun isEmpty(): Boolean = true
}
/**
* Kind-0-shaped identity, each field in its own role. [SearchFieldExtractor]
* never RETURNS an empty Profile — extract() normalizes every empty shape
* to [None] — so an all-null value only exists mid-extraction or when
* hand-built. Note the equality trap for hand-built values: an empty
* shape isEmpty() but is not equal to [None].
*/
data class Profile(
val name: String? = null,
val displayName: String? = null,
val about: String? = null,
val nip05: String? = null,
val lud16: String? = null,
val website: String? = null,
) : IndexableFields {
override fun isEmpty(): Boolean = this == EMPTY
private companion object {
val EMPTY = Profile()
}
}
/**
* Content decomposed by priority tier: [primary] (title-like values),
* [secondary] (summary/description-like values), [text] (the body — the
* one inherently single-valued role), plus the raw [hashtags] and
* [locations] tag values.
*/
data class Tiered(
val primary: List<String> = emptyList(),
val secondary: List<String> = emptyList(),
val text: String? = null,
val hashtags: List<String> = emptyList(),
val locations: List<String> = emptyList(),
val websites: List<String> = emptyList(),
) : IndexableFields {
override fun isEmpty(): Boolean = this == EMPTY
private companion object {
val EMPTY = Tiered()
}
}
}
@@ -0,0 +1,486 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip50Search
import com.vitorpamplona.quartz.buzz.agentProfiles.AgentProfileEvent
import com.vitorpamplona.quartz.buzz.apPersonas.PersonaEvent
import com.vitorpamplona.quartz.buzz.managedAgents.ManagedAgentEvent
import com.vitorpamplona.quartz.buzz.teams.TeamEvent
import com.vitorpamplona.quartz.buzz.workflow.WorkflowDefEvent
import com.vitorpamplona.quartz.experimental.agora.FundraiserEvent
import com.vitorpamplona.quartz.experimental.audio.track.AudioTrackEvent
import com.vitorpamplona.quartz.experimental.fitness.workout.ExerciseTemplateEvent
import com.vitorpamplona.quartz.experimental.fitness.workout.WorkoutRecordEvent
import com.vitorpamplona.quartz.experimental.interactiveStories.InteractiveStoryBaseEvent
import com.vitorpamplona.quartz.experimental.music.playlist.MusicPlaylistEvent
import com.vitorpamplona.quartz.experimental.music.track.MusicTrackEvent
import com.vitorpamplona.quartz.experimental.nip82SoftwareApps.application.SoftwareApplicationEvent
import com.vitorpamplona.quartz.experimental.nipsOnNostr.NipTextEvent
import com.vitorpamplona.quartz.feedDefinition.FeedDefinitionEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
import com.vitorpamplona.quartz.nip01Core.tags.hashtags.hashtags
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.nip14Subject.subject
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelCreateEvent
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelMetadataEvent
import com.vitorpamplona.quartz.nip29RelayGroups.metadata.GroupMetadataEvent
import com.vitorpamplona.quartz.nip30CustomEmoji.pack.EmojiPackEvent
import com.vitorpamplona.quartz.nip34Git.issue.GitIssueEvent
import com.vitorpamplona.quartz.nip34Git.pr.GitPullRequestEvent
import com.vitorpamplona.quartz.nip34Git.repository.GitRepositoryEvent
import com.vitorpamplona.quartz.nip35Torrents.TorrentEvent
import com.vitorpamplona.quartz.nip51Lists.appCurationSet.AppCurationSetEvent
import com.vitorpamplona.quartz.nip51Lists.articleCurationSet.ArticleCurationSetEvent
import com.vitorpamplona.quartz.nip51Lists.bookmarkList.BookmarkListEvent
import com.vitorpamplona.quartz.nip51Lists.bookmarkList.OldBookmarkListEvent
import com.vitorpamplona.quartz.nip51Lists.followList.FollowListEvent
import com.vitorpamplona.quartz.nip51Lists.interestSet.InterestSetEvent
import com.vitorpamplona.quartz.nip51Lists.labeledBookmarkList.LabeledBookmarkListEvent
import com.vitorpamplona.quartz.nip51Lists.mediaStarterPack.MediaStarterPackEvent
import com.vitorpamplona.quartz.nip51Lists.peopleList.PeopleListEvent
import com.vitorpamplona.quartz.nip51Lists.pictureCurationSet.PictureCurationSetEvent
import com.vitorpamplona.quartz.nip51Lists.relaySets.RelaySetEvent
import com.vitorpamplona.quartz.nip51Lists.releaseArtifactSet.ReleaseArtifactSetEvent
import com.vitorpamplona.quartz.nip51Lists.videoCurationSet.VideoCurationSetEvent
import com.vitorpamplona.quartz.nip52Calendar.appt.day.CalendarDateSlotEvent
import com.vitorpamplona.quartz.nip52Calendar.appt.time.CalendarTimeSlotEvent
import com.vitorpamplona.quartz.nip52Calendar.calendar.CalendarEvent
import com.vitorpamplona.quartz.nip53LiveActivities.clip.LiveActivitiesClipEvent
import com.vitorpamplona.quartz.nip53LiveActivities.meetingSpaces.MeetingRoomEvent
import com.vitorpamplona.quartz.nip53LiveActivities.meetingSpaces.MeetingSpaceEvent
import com.vitorpamplona.quartz.nip53LiveActivities.streaming.LiveActivitiesEvent
import com.vitorpamplona.quartz.nip54Wiki.WikiNoteEvent
import com.vitorpamplona.quartz.nip58Badges.definition.BadgeDefinitionEvent
import com.vitorpamplona.quartz.nip5aStaticWebsites.NamedSiteEvent
import com.vitorpamplona.quartz.nip5aStaticWebsites.RootSiteEvent
import com.vitorpamplona.quartz.nip5dNapplets.NamedNappletEvent
import com.vitorpamplona.quartz.nip5dNapplets.NappletSnapshotEvent
import com.vitorpamplona.quartz.nip5dNapplets.RootNappletEvent
import com.vitorpamplona.quartz.nip68Picture.PictureEvent
import com.vitorpamplona.quartz.nip71Video.AddressableVideoEvent
import com.vitorpamplona.quartz.nip71Video.RegularVideoEvent
import com.vitorpamplona.quartz.nip72ModCommunities.definition.CommunityDefinitionEvent
import com.vitorpamplona.quartz.nip75ZapGoals.GoalEvent
import com.vitorpamplona.quartz.nip7DThreads.ThreadEvent
import com.vitorpamplona.quartz.nip84Highlights.HighlightEvent
import com.vitorpamplona.quartz.nip89AppHandlers.definition.AppDefinitionEvent
import com.vitorpamplona.quartz.nip94FileMetadata.FileHeaderEvent
import com.vitorpamplona.quartz.nip99Classifieds.ClassifiedsEvent
import com.vitorpamplona.quartz.nipB0WebBookmarks.WebBookmarkEvent
import com.vitorpamplona.quartz.nipC0CodeSnippets.CodeSnippetEvent
import com.vitorpamplona.quartz.nipF4Podcasts.episode.PodcastEpisodeEvent
import com.vitorpamplona.quartz.nipF4Podcasts.metadata.PodcastMetadataEvent
/**
* Decomposes every [SearchableEvent] into [IndexableFields] by priority tier:
* title-like accessors primary, summary/description secondary, body tertiary,
* with hashtag and location tags carried raw beside the tiers. Each explicit
* branch splits exactly the accessors that kind's `indexableContent()`
* concatenates — keep the two in sync when a kind's parsing changes. Kinds
* without an explicit branch fall back to the [SearchableEvent] branch (whole
* `indexableContent()` in the tertiary tier), so EVERY searchable kind,
* current or future, is extracted.
*
* A kind may also fill the profile roles when it carries that shape: kind
* 31990 goes through the kind-0 fields wholesale, and any kind with a
* homepage/site URL fills [IndexableFields.websites]. Hashtags and `location`
* tags are filled SYSTEMICALLY by the [tiers] funnel every content branch
* uses, so recall never depends on a branch remembering them.
*
* Non-searchable kinds return [IndexableFields.None]. The extraction is
* derived data baked into the build: stores should re-derive after upgrades
* (see [com.vitorpamplona.quartz.nip01Core.store.IEventStore.reindexFullTextSearch]).
*/
object SearchFieldExtractor {
/** Empty extractions always come back as [IndexableFields.None], whatever shape produced them. */
fun extract(event: Event): IndexableFields = base(event).let { if (it.isEmpty()) IndexableFields.None else it }
private fun base(event: Event): IndexableFields =
when (event) {
// kind 0 -> the profile fields, each in its own role.
is MetadataEvent -> {
val md = event.contactMetaData()
if (md == null) {
IndexableFields.Profile()
} else {
IndexableFields.Profile(
name = clean(md.name),
displayName = clean(md.displayName),
about = clean(md.about),
nip05 = clean(md.nip05),
lud16 = clean(md.lud16),
website = clean(md.website),
)
}
}
is LongTextNoteEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is WikiNoteEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is ClassifiedsEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is GitRepositoryEvent -> {
tiers(event, listOf(event.name()), listOf(event.description()), event.content, websites = event.webs())
}
is GitIssueEvent -> {
tiers(event, event.subject(), null, event.content)
}
is GitPullRequestEvent -> {
tiers(event, event.subject(), null, event.content)
}
is CommunityDefinitionEvent -> {
tiers(event, listOf(event.name()), listOf(event.description(), event.rules()), event.content)
}
is EmojiPackEvent -> {
tiers(event, event.titleOrName(), event.description(), event.content)
}
is ChannelCreateEvent -> {
event.channelInfo().let { tiers(event, it.name, it.about, null) }
}
is ChannelMetadataEvent -> {
event.channelInfo().let { tiers(event, it.name, it.about, null) }
}
is PictureEvent -> {
tiers(event, event.title(), null, event.content)
}
is RegularVideoEvent -> {
tiers(event, event.title(), null, event.content)
}
is AddressableVideoEvent -> {
tiers(event, event.title(), null, event.content)
}
// Torrents are searched by FILE NAME above all — index the file
// list into the secondary tier, trackers as the affiliation URL.
is TorrentEvent -> {
tiers(event, listOf(event.title()), event.files().map { it.fileName }, event.content, websites = event.trackers())
}
is ThreadEvent -> {
tiers(event, event.title(), null, event.content)
}
is FundraiserEvent -> {
tiers(event, event.title(), null, event.content)
}
is NipTextEvent -> {
tiers(event, event.title(), null, event.content)
}
is ExerciseTemplateEvent -> {
tiers(event, event.title(), null, event.content)
}
is WorkoutRecordEvent -> {
tiers(event, event.title(), null, event.content)
}
is CalendarEvent -> {
tiers(event, event.title(), null, event.content)
}
is LiveActivitiesClipEvent -> {
tiers(event, event.title(), null, event.content)
}
is CalendarDateSlotEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is CalendarTimeSlotEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is LiveActivitiesEvent -> {
tiers(event, event.title(), event.summary(), event.content, website = event.streaming())
}
is InteractiveStoryBaseEvent -> {
tiers(event, event.title(), event.summary(), event.content)
}
is MeetingSpaceEvent -> {
tiers(event, event.room(), event.summary(), event.content)
}
is MeetingRoomEvent -> {
tiers(event, event.title(), event.summary(), null)
}
// Code snippets are searched by language/runtime as much as name —
// fold those keywords into the secondary tier, repo as affiliation.
is CodeSnippetEvent -> {
tiers(
event,
listOf(event.snippetName()),
listOf(event.snippetDescription(), event.language(), event.extension(), event.runtime()),
event.content,
websites = listOf(event.repo()),
)
}
is BadgeDefinitionEvent -> {
tiers(event, event.name(), event.description(), event.content)
}
is MusicPlaylistEvent -> {
tiers(event, event.title(), event.description(), event.content)
}
is MusicTrackEvent -> {
tiers(event, listOf(event.title()), listOf(event.artist(), event.album()), event.content)
}
is SoftwareApplicationEvent -> {
tiers(event, listOf(event.name()), listOf(event.summary()), event.content, websites = listOf(event.url(), event.repository()))
}
is PodcastEpisodeEvent -> {
tiers(event, event.title(), event.description(), event.content)
}
is PodcastMetadataEvent -> {
tiers(event, listOf(event.title()), listOf(event.description()), null, websites = event.websites())
}
is GroupMetadataEvent -> {
tiers(event, event.name(), event.about(), null)
}
is InterestSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is FollowListEvent -> {
tiers(event, event.title(), event.description(), null)
}
is MediaStarterPackEvent -> {
tiers(event, event.title(), event.description(), null)
}
is PictureCurationSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is ArticleCurationSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is VideoCurationSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is ReleaseArtifactSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is AppCurationSetEvent -> {
tiers(event, event.title(), event.description(), null)
}
is RelaySetEvent -> {
tiers(event, event.title(), event.description(), null)
}
// A web bookmark IS its URL — route it to the affiliation website
// field so the bookmark is findable by its domain.
is WebBookmarkEvent -> {
tiers(event, event.title(), event.description(), null, website = event.url())
}
is NamedSiteEvent -> {
tiers(event, event.title(), event.description(), null)
}
is RootSiteEvent -> {
tiers(event, event.title(), event.description(), null)
}
is RootNappletEvent -> {
tiers(event, event.title(), event.description(), null)
}
is NappletSnapshotEvent -> {
tiers(event, event.title(), event.description(), null)
}
is NamedNappletEvent -> {
tiers(event, event.title(), event.description(), null)
}
is FeedDefinitionEvent -> {
tiers(event, event.title(), null, null)
}
is LabeledBookmarkListEvent -> {
tiers(event, event.titleOrName(), event.description(), null)
}
is PeopleListEvent -> {
tiers(event, event.titleOrName(), event.description(), null)
}
is BookmarkListEvent -> {
tiers(event, event.title(), null, null)
}
is OldBookmarkListEvent -> {
tiers(event, event.title(), null, null)
}
is GoalEvent -> {
tiers(event, null, event.summary(), event.content)
}
is HighlightEvent -> {
tiers(event, emptyList(), listOf(event.comment(), event.context()), event.content)
}
is FileHeaderEvent -> {
tiers(event, null, event.summary(), event.content)
}
is AudioTrackEvent -> {
tiers(event, event.subject(), null, null)
}
// Buzz agent/workspace kinds carry their metadata as JSON in `content`;
// split each decoded object the way its indexableContent() concatenates it.
is AgentProfileEvent -> {
event.profileOrNull()?.let { tiers(event, listOf(it.name, it.displayName), emptyList(), null) } ?: tiers(event, null, null, null)
}
is PersonaEvent -> {
event.personaOrNull()?.let { tiers(event, it.displayName, null, it.systemPrompt) } ?: tiers(event, null, null, null)
}
is ManagedAgentEvent -> {
event.agentOrNull()?.let { tiers(event, it.name, null, it.systemPrompt) } ?: tiers(event, null, null, null)
}
is TeamEvent -> {
event.teamOrNull()?.let { tiers(event, it.name, it.description, it.instructions) } ?: tiers(event, null, null, null)
}
is WorkflowDefEvent -> {
tiers(event, event.name(), null, event.content)
}
// kind 31990 — the app handler's metadata IS a UserMetadata clone,
// so route it through the kind-0 profile fields: an app's
// @-handle and site get the same treatment a person's do.
is AppDefinitionEvent -> {
val md = event.appMetaData()
if (md == null) {
IndexableFields.Profile()
} else {
IndexableFields.Profile(
// Per NIP-24 the deprecated `username` folds into `name`.
name = clean(md.name ?: md.username),
displayName = clean(md.displayName),
about = clean(md.about),
nip05 = clean(md.nip05),
lud16 = clean(md.lud16),
website = clean(md.website),
)
}
}
// kind 1 LAST among the explicit branches, defensively: a future
// kind extending the text-note base must hit its own branch first.
is TextNoteEvent -> {
tiers(event, event.subject(), null, event.content)
}
// Everything else Quartz can search, current or future: the whole
// indexableContent lands in the tertiary tier.
is SearchableEvent -> {
tiers(event, null, null, event.indexableContent())
}
else -> {
IndexableFields.None
}
}
/** Single-value convenience over the list funnel — most kinds carry one title, one summary, one body. */
private fun tiers(
event: Event,
primary: String?,
secondary: String?,
text: String?,
website: String? = null,
) = tiers(event, listOf(primary), listOf(secondary), text, listOf(website))
/**
* The one funnel every content branch uses — [IndexableFields.Tiered.hashtags]
* and [IndexableFields.Tiered.locations] are filled here, so no branch can
* forget them. Values stay UNJOINED: separator choices belong to the backend.
*/
private fun tiers(
event: Event,
primary: List<String?>,
secondary: List<String?>,
text: String?,
websites: List<String?> = emptyList(),
) = IndexableFields.Tiered(
primary = cleanAll(primary),
secondary = cleanAll(secondary),
text = clean(text),
hashtags = cleanAll(event.tags.hashtags()),
locations = locationValues(event),
websites = cleanAll(websites),
)
/** Trim and drop empties at the single funnel every derived string passes through. */
private fun clean(s: String?): String? = s?.trim()?.ifEmpty { null }
private fun cleanAll(parts: List<String?>): List<String> = parts.mapNotNull { clean(it) }
/**
* Every `location` tag value, on ANY kind. Deliberately a raw scan, not a
* typed accessor: Quartz's LocationTag classes are per-NIP (calendar,
* picture, classifieds) and only those kinds expose locations(), while
* this funnel must also catch location tags on kinds whose class doesn't
* model them.
*/
private fun locationValues(event: Event): List<String> = event.tags.mapNotNull { tag -> if (tag.getOrNull(0) != "location") null else clean(tag.getOrNull(1)) }
}
@@ -28,43 +28,70 @@ import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
* NIP-50 defines the [com.vitorpamplona.quartz.nip01Core.relay.filters.Filter.search]
* field as "a string describing a query in a human-readable form", optionally
* carrying `key:value` extension tokens such as `domain:example.com` or
* `language:en`. This class splits that raw string into the free-text [terms]
* and the recognized [extensions], giving relays (and search redirectors) a
* typed view of the query instead of forcing each one to re-parse the string.
* `language:en`. This class splits that raw string into the free-text [terms],
* the Google-style term syntax ([phrases], [notPhrases], [notTerms]), and the
* recognized [extensions], giving relays (and search redirectors) a typed view
* of the query instead of forcing each one to re-parse the string.
*
* Example:
* ```
* val q = SearchQuery.parse("best nostr apps domain:example.com language:en")
* q.terms // "best nostr apps"
* val q = SearchQuery.parse("best \"nostr apps\" -spam domain:example.com")
* q.terms // "best"
* q.phrases // ["nostr apps"]
* q.notTerms // ["spam"]
* q.domain // "example.com"
* q.language // "en"
* ```
*
* ## Tokenization
* ## Parse order (load-bearing)
*
* The string is split on whitespace. A token is treated as an extension when:
* - it contains a `:`,
* - the part before the `:` is a non-empty run of lowercase ASCII letters
* (`a``z`), and
* - the part after the `:` is non-empty and does not start with `//` (so URLs
* like `https://example.com` stay in [terms]).
* Quoted spans are lifted off the RAW string FIRST, then the residual is
* tokenized for extensions, then `-word` exclusions split off. The order
* matters twice over: the extension pass is quote-blind, so a span ending in
* an extension-shaped token (`"pizza sort:rank" -spam`) would otherwise lose
* its closing quote and the unclosed quote would swallow the rest of the
* query; and lifting first lets quotes protect extension-shaped tokens —
* `"include:spam"` is the phrase [include, spam], not an extension.
*
* Everything else is free text. Per NIP-50 unknown extensions are kept (so they
* can be forwarded to a backend) — relays "SHOULD ignore extensions they don't
* support", which this models by simply not having a typed accessor for them;
* they remain readable through [extensions] / [extension].
* ## Term syntax
*
* Extension keys are matched case-sensitively against the lowercase forms
* documented by NIP-50. Duplicate keys keep the last occurrence.
* - A `"quoted span"` is an exact-phrase requirement ([phrases]); `-"…"` is a
* phrase exclusion ([notPhrases]). A quote opens a span only at a token
* boundary — mid-token quotes stay ordinary characters. An unclosed span
* runs to the end of the string. Empty spans are dropped, but a positive
* phrase keeps content a text index may not hold ("⚡"): it is an
* unsatisfiable requirement the backend turns into provably-no-match —
* dropping it here would silently flip that into match-all.
* - A leading `-` on a 2+ character token makes it an exclusion ([notTerms],
* all leading dashes stripped); a lone `-` stays an ordinary term. There is
* no `-extension` syntax: extension keys are strictly `a``z`, so
* `-include:spam` fails the key test and becomes the excluded literal.
*
* ## Extension tokenization
*
* The residual is split on whitespace. A token is treated as an extension when
* it contains a `:`, the part before the `:` is a non-empty run of lowercase
* ASCII letters (`a``z`), and the part after is non-empty and does not start
* with `//` (so URLs like `https://example.com` stay in [terms]). Per NIP-50
* unknown extensions are kept (readable through [extensions] / [extension]) so
* they can be forwarded to a backend; relays "SHOULD ignore extensions they
* don't support". Extension keys are matched case-sensitively against the
* lowercase forms documented by NIP-50. Duplicate keys keep the last
* occurrence.
*/
class SearchQuery(
/** The human-readable search terms with all extension tokens removed. */
/** The loose human-readable search terms: extensions, phrases, and exclusions all removed. */
val terms: String,
/**
* All recognized `key:value` extension tokens, in the order they appeared.
* Known keys: [INCLUDE], [DOMAIN], [LANGUAGE], [SENTIMENT], [NSFW].
*/
val extensions: Map<String, String>,
/** Exact-phrase requirements (`"nostr apps"`), quotes removed, in order. */
val phrases: List<String> = emptyList(),
/** Exact-phrase exclusions (`-"nostr apps"`), quotes removed, in order. */
val notPhrases: List<String> = emptyList(),
/** Single-word exclusions (`-spam`), dashes removed, in order. */
val notTerms: List<String> = emptyList(),
) {
/** `true` when the query carries the `include:spam` token (NIP-50: disable spam filtering). */
val includeSpam: Boolean
@@ -93,20 +120,41 @@ class SearchQuery(
val nsfwIncluded: Boolean
get() = nsfw ?: true
/**
* Whether the query REQUIRES any text — loose terms or phrases. Exclusions
* alone don't count: an exclusions-only query is plain recall minus the
* excluded words, not a ranked text search.
*/
val hasText: Boolean
get() = terms.isNotEmpty() || phrases.isNotEmpty()
/** Returns the raw value of an arbitrary extension key (including unknown ones), or null. */
fun extension(key: String): String? = extensions[key]
/** Returns true when there are no free-text terms (the query is extensions-only or empty). */
/** Returns true when there are no loose free-text terms. Phrases don't count — see [hasText] for "any required text". */
fun isTermsEmpty(): Boolean = terms.isEmpty()
/**
* Re-assembles a canonical NIP-50 search string: the free-text [terms]
* followed by each `key:value` extension. Useful for a redirector that
* normalizes the incoming query before forwarding it to a backend.
* Re-assembles a canonical NIP-50 search string: the free-text [terms],
* then each `"phrase"`, `-exclusion`, `-"phrase exclusion"`, and
* `key:value` extension. Canonical, not order-preserving. Useful for a
* redirector that normalizes the incoming query before forwarding it.
*/
fun toSearchString(): String =
buildString {
append(terms)
for (phrase in phrases) {
if (isNotEmpty()) append(' ')
append('"').append(phrase).append('"')
}
for (word in notTerms) {
if (isNotEmpty()) append(' ')
append('-').append(word)
}
for (phrase in notPhrases) {
if (isNotEmpty()) append(' ')
append("-\"").append(phrase).append('"')
}
for ((key, value) in extensions) {
if (isNotEmpty()) append(' ')
append(key).append(':').append(value)
@@ -134,20 +182,24 @@ class SearchQuery(
private val WHITESPACE = Regex("\\s+")
/** Empty query — no terms and no extensions. */
/** Empty query — no terms, no syntax, no extensions. */
val EMPTY = SearchQuery("", emptyMap())
/**
* Parses a raw NIP-50 [search] string into a [SearchQuery]. A null or
* blank input yields [EMPTY].
* blank input yields [EMPTY]. See the class KDoc for the grammar and
* why the quote pass runs before the extension pass.
*/
fun parse(search: String?): SearchQuery {
if (search.isNullOrBlank()) return EMPTY
val quoted = liftQuotedSpans(search)
val extensions = LinkedHashMap<String, String>()
val terms = StringBuilder()
val notTerms = ArrayList<String>()
for (token in search.trim().split(WHITESPACE)) {
for (token in quoted.residual.trim().split(WHITESPACE)) {
if (token.isEmpty()) continue
val colon = token.indexOf(':')
if (colon > 0 && colon < token.length - 1) {
val key = token.substring(0, colon)
@@ -157,18 +209,63 @@ class SearchQuery(
continue
}
}
if (token.length > 1 && token[0] == '-') {
// All-dash tokens ("--") strip to nothing: an
// exclude-nothing token is dropped, not surfaced — an
// empty exclusion would round-trip into a required "-".
token.trimStart('-').takeIf { it.isNotEmpty() }?.let { notTerms += it }
continue
}
if (terms.isNotEmpty()) terms.append(' ')
terms.append(token)
}
return SearchQuery(terms.toString(), extensions)
return SearchQuery(terms.toString(), extensions, quoted.phrases, quoted.notPhrases, notTerms)
}
private fun isExtensionKey(key: String): Boolean = key.isNotEmpty() && key.all { it in 'a'..'z' }
/** The quoted spans lifted off the raw text, plus the residual for the extension and `-word` passes. */
private class QuotedSpans(
val phrases: List<String>,
val notPhrases: List<String>,
val residual: String,
)
/** Stage one, over the RAW string: lift every `"…"` / `-"…"` span. See the class KDoc for the rules. */
private fun liftQuotedSpans(text: String): QuotedSpans {
val phrases = ArrayList<String>()
val notPhrases = ArrayList<String>()
val residual = StringBuilder()
var i = 0
var boundary = true
while (i < text.length) {
val c = text[i]
val neg = c == '-' && i + 1 < text.length && text[i + 1] == '"'
if (boundary && (c == '"' || neg)) {
val start = i + if (neg) 2 else 1
val close = text.indexOf('"', start)
val end = if (close < 0) text.length else close
val span = text.substring(start, end).trim()
i = if (close < 0) text.length else close + 1
if (span.isNotEmpty()) {
if (neg) notPhrases += span else phrases += span
}
// The lifted span's place stays a token boundary for what follows.
residual.append(' ')
} else {
residual.append(c)
boundary = c.isWhitespace()
i++
}
}
return QuotedSpans(phrases, notPhrases, residual.toString())
}
/**
* Returns [search] with every `key:value` extension token removed,
* leaving only the free-text terms (tokenized as in [parse]).
* leaving the free-text query (terms, phrases, and exclusions,
* re-assembled as in [toSearchString]).
*
* Backends that hand the search string to an engine with its own
* query syntax — e.g. SQLite FTS, where `:` is column-filter
@@ -184,7 +281,8 @@ class SearchQuery(
fun stripExtensions(search: String?): String? {
if (search.isNullOrBlank()) return search
val parsed = parse(search)
return if (parsed.extensions.isEmpty()) search else parsed.terms
if (parsed.extensions.isEmpty()) return search
return SearchQuery(parsed.terms, emptyMap(), parsed.phrases, parsed.notPhrases, parsed.notTerms).toSearchString()
}
}
}
@@ -0,0 +1,139 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentSet
/**
* Which relays are worth dialling, within one fan-out cycle. A discovered
* relay list is five figures of urls and most of them are corpses; without
* this, every cycle re-dials all of them and the working relays queue behind
* hosts that stopped existing years ago.
*
* Failures are counted per AUTHORITY (`host[:port]`), not per url: the outbox
* model mints one url per user for a filtering relay, so a per-url counter
* never reaches a threshold on any single one. The authority is host-only and
* does NOT fold a subdomain into its parent — those are different servers.
*
* [produced] overrides a strike race: a host that has ever delivered is never
* treated as dead for the rest of the cycle, whichever order the two events
* land in. Cycle-local; nothing persists — see [RelayReachabilityStore] for
* the part that survives a restart.
*/
class HostStrikes(
private val strikeLimit: Int = DEFAULT_STRIKE_LIMIT,
// Relays a previous run proved unreachable, and still within their TTL.
private val knownDead: Set<NormalizedRelayUrl> = emptySet(),
) {
private val strikes = ConcurrentMap<String, Int>()
private val deadHosts = ConcurrentSet<String>()
private val producedHosts = ConcurrentSet<String>()
private val delivered = ConcurrentSet<NormalizedRelayUrl>()
private val failed = ConcurrentSet<NormalizedRelayUrl>()
/** Relays this cycle actually got something from — worth remembering as live. */
val reachable: Set<NormalizedRelayUrl> get() = delivered.snapshot()
/** Relays this cycle could not reach at all. A relay that later delivered is not in it. */
val unreachable: Set<NormalizedRelayUrl> get() = failed.snapshot() - delivered.snapshot()
/**
* Skip this relay? True when a previous run proved it dead (and no
* [produced] since), or when its whole authority has been struck out here.
*/
fun isDead(url: NormalizedRelayUrl): Boolean {
val authority = authorityOf(url.url)
if (authority in producedHosts) return false
return url in knownDead || authority in deadHosts
}
/**
* This relay connected but delivered nothing before giving up. Count it
* against its authority and, at [strikeLimit], stop dialling the host.
* Returns the eviction — for the caller to publish — exactly when this
* strike is the one that took the host down: that is the only point where
* the evidence exists, because every sibling url is skipped without being
* dialled from here on.
*/
fun strike(url: NormalizedRelayUrl): Evicted? {
failed.add(url)
if (strikeLimit <= 0) return null
val authority = authorityOf(url.url)
if (authority in producedHosts || authority in deadHosts) return null
if (strikes.merge(authority, 1) { old, new -> old + new } < strikeLimit) return null
// Concurrent strikers can cross the threshold together; add() is the
// atomic exactly-once gate on who publishes. Re-check produced after
// winning it: a delivery that landed while this strike was in flight
// outranks the verdict, and a verdict for a host that just answered
// would be a false public record.
if (!deadHosts.add(authority)) return null
if (authority in producedHosts) return null
return Evicted(authority, strikeLimit)
}
/** An authority struck out, and the evidence for it. */
class Evicted(
val authority: String,
val strikes: Int,
)
/** This relay delivered. Its authority is alive, whatever else happened. */
fun produced(url: NormalizedRelayUrl) {
delivered.add(url)
producedHosts.add(authorityOf(url.url))
}
/** For a cycle's closing line: how many hosts were dropped. */
fun evictedHosts(): Int = deadHosts.size()
fun summary(total: Int): String =
"${reachable.size} live, ${unreachable.size} unreachable, " +
"${deadHosts.size()} host(s) struck out, ${knownDead.size} skipped as known-dead of $total"
companion object {
/**
* Three, because a single timeout is ordinary — a busy relay that
* never answered one REQ is not a dead one — while three separate
* urls on the same host all going silent is a server, not a
* coincidence.
*/
const val DEFAULT_STRIKE_LIMIT = 3
/**
* `host[:port]` — everything between the scheme and the first path
* slash. The port is part of it: two ports on one machine are two
* relays.
*/
fun authorityOf(url: String): String {
val afterScheme =
when {
url.startsWith("wss://") -> url.substring(6)
url.startsWith("ws://") -> url.substring(5)
else -> url
}
val slash = afterScheme.indexOf('/')
return if (slash >= 0) afterScheme.substring(0, slash) else afterScheme
}
}
}
@@ -0,0 +1,59 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
import com.vitorpamplona.quartz.nip01Core.signers.eventTemplate
import com.vitorpamplona.quartz.nip40Expiration.ExpirationTag
import com.vitorpamplona.quartz.utils.TimeUtils
/**
* The event a NIP-66 monitor publishes to measure a relay's WRITE path: publish
* one of these (signed with the monitor key), time the `OK`, and read the
* rejection prefix when refused (`auth-required:`/`restricted:`/`pow:` map to
* the discovery record's `R` requirement tags; a timed acceptance is `rtt-write`).
*
* The kind is EPHEMERAL (2000029999 per NIP-01), so a compliant relay serves it
* to current subscribers and never stores it — the probe leaves nothing behind.
* [KIND] 20166 is this library's convention (30166 discovery minus the
* addressable range), not something NIP-66 standardizes; any ephemeral kind
* works. Belt-and-braces, the template also carries a NIP-40 `expiration` tag
* [EXPIRATION_SECONDS] out, so a relay that stores unknown ephemeral kinds
* anyway purges it promptly.
*
* A rejection is still a MEASUREMENT: an `OK false` proves the write path works
* and documents the relay's policy. Only silence is a failed write test.
*/
object RelayProbeWriteTest {
const val KIND = 20166
/** Storage-window ceiling for non-compliant relays that store ephemeral events. */
const val EXPIRATION_SECONDS = 60L
fun build(
content: String = "NIP-66 write probe",
createdAt: Long = TimeUtils.now(),
): EventTemplate<Event> =
eventTemplate(KIND, content, createdAt) {
add(ExpirationTag.assemble(createdAt + EXPIRATION_SECONDS))
}
}
@@ -22,6 +22,8 @@ package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.PublishResult
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.publishAndCollectResults
import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient
@@ -29,10 +31,20 @@ import com.vitorpamplona.quartz.nip01Core.relay.client.single.newSubId
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.RelayDiscoveryEvent
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.networkType
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.requirement
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.rtt
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.tags.RttType
import com.vitorpamplona.quartz.utils.TimeUtils
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.withTimeoutOrNull
import kotlin.time.TimeSource
@@ -74,6 +86,19 @@ class RelayProber(
val error: String?,
)
/**
* One relay's read+write check outcome (see [readWriteCheck]). Latencies are
* -1 when unobserved; [writeAccepted] is null when the relay never answered
* the write with an OK (transport failure or silence).
*/
class ReadWriteVerdict(
val relay: NormalizedRelayUrl,
val rttReadMs: Long,
val rttWriteMs: Long,
val writeAccepted: Boolean?,
val writeMessage: String?,
)
class Result(
val verdicts: List<Verdict>,
val elapsedMs: Long,
@@ -91,18 +116,24 @@ class RelayProber(
/**
* Probe every relay in [relays], [waveSize] at a time, giving each wave up to
* [timeoutMs] to reach terminals. Returns one [Verdict] per input relay.
*
* [filters] is the REQ each relay is asked to answer. The default
* [LIVENESS_FILTERS] matches nothing, so an EOSE proves liveness without
* streaming a payload; pass [readTestFilter] to make [Verdict.rttEoseMs] a
* real read test instead (the relay must query and stream an actual event).
*/
suspend fun probe(
relays: Collection<NormalizedRelayUrl>,
timeoutMs: Long = 15_000,
waveSize: Int = 1000,
filters: List<Filter> = LIVENESS_FILTERS,
): Result {
val mark = TimeSource.Monotonic.markNow()
val all = ArrayList<Verdict>(relays.size)
val distinct = relays.toSet()
var done = 0
for (wave in distinct.chunked(waveSize.coerceAtLeast(1))) {
all += probeWave(wave, timeoutMs)
probeWave(wave, timeoutMs, filters) { all += it }
done += wave.size
if (distinct.size > wave.size) {
val liveSoFar = all.count { it.reachable }
@@ -112,10 +143,94 @@ class RelayProber(
return Result(all, mark.elapsedNow().inWholeMilliseconds)
}
/**
* Streaming variant of [probe]: a cold [Flow] that emits each relay's [Verdict]
* the moment that relay resolves — an answering relay's verdict arrives as soon
* as its EOSE/CLOSED/connect-failure lands, not when the whole census ends. Only
* relays that stay silent wait for their wave's [timeoutMs] deadline.
*
* [filters] picks the check, as in [probe]: [LIVENESS_FILTERS] (default) or
* [readTestFilter].
*
* Probing starts when the flow is collected, and emission is sequential — a slow
* collector delays the next wave AND eats into the current wave's [timeoutMs]
* window (the deadline is absolute; answers keep being recorded while the
* collector runs, but silent relays get less listening time). Keep per-verdict
* work light, or buffer, when precise deadlines matter. Pair each verdict with
* [toDiscoveryEventTemplate] to turn the stream into signable NIP-66 kind:30166
* records for another process to sign and publish.
*/
fun probeFlow(
relays: Collection<NormalizedRelayUrl>,
timeoutMs: Long = 15_000,
waveSize: Int = 1000,
filters: List<Filter> = LIVENESS_FILTERS,
): Flow<Verdict> =
flow {
for (wave in relays.toSet().chunked(waveSize.coerceAtLeast(1))) {
probeWave(wave, timeoutMs, filters) { emit(it) }
}
}
/**
* The deeper, still-honest check pair: READ (a real limit-[readLimit] REQ the
* relay must query its store for) and WRITE (one ephemeral [RelayProbeWriteTest]
* event signed by [signer], the monitor key, timed to its OK). Everything is a
* direct observation — nothing is copied from the relay's NIP-11 self-claims.
*
* Run it against relays ALREADY PROVEN LIVE — typically [Result.reachable] of a
* [probe] that just ran, while the pool's sockets are still open. On a warm
* socket both numbers are honest NIP-66 rtts (`rtt-read`, `rtt-write`); against
* a cold relay they silently include the dial, so don't.
*
* A write REJECTION is still a measurement: `OK false` proves the write path
* works and documents policy ([ReadWriteVerdict.writeMessage] keeps the NIP-01
* machine-readable reason; [toDiscoveryEventTemplate] maps `auth-required:` and
* `pow:` to `R` tags). Only silence leaves [ReadWriteVerdict.writeAccepted] null.
*/
suspend fun readWriteCheck(
relays: Collection<NormalizedRelayUrl>,
signer: NostrSigner,
timeoutMs: Long = 15_000,
waveSize: Int = 1000,
readLimit: Int = 1,
readKinds: List<Int>? = listOf(0),
): Map<NormalizedRelayUrl, ReadWriteVerdict> {
val out = HashMap<NormalizedRelayUrl, ReadWriteVerdict>()
val distinct = relays.toSet()
for ((waveIndex, wave) in distinct.chunked(waveSize.coerceAtLeast(1)).withIndex()) {
val reads = HashMap<NormalizedRelayUrl, Long>()
probeWave(wave, timeoutMs, readTestFilter(readLimit, readKinds)) { reads[it.relay] = it.rttEoseMs }
// A distinct event id per wave (createdAt has second granularity, so the
// content must vary) keeps a straggler OK from an earlier wave's relays
// from ever matching this wave's confirmation window.
val event = signer.sign(RelayProbeWriteTest.build(content = "NIP-66 write probe $waveIndex"))
val writes = client.publishAndCollectResults(event, wave.toSet(), (timeoutMs / 1000).coerceAtLeast(1))
for (relay in wave) {
// Only a real OK (true or false) counts as an answer; transport
// failures and silence leave the write side unobserved.
val answered = writes[relay]?.takeUnless { it.isTransportFailure || it.message == PublishResult.NO_RESPONSE }
out[relay] =
ReadWriteVerdict(
relay = relay,
rttReadMs = reads[relay] ?: -1,
rttWriteMs = answered?.elapsedMs ?: -1,
writeAccepted = answered?.accepted,
writeMessage = answered?.message,
)
}
}
return out
}
private suspend fun probeWave(
wave: List<NormalizedRelayUrl>,
timeoutMs: Long,
): List<Verdict> {
filters: List<Filter>,
onVerdict: suspend (Verdict) -> Unit,
) {
val mark = TimeSource.Monotonic.markNow()
val waveSet = wave.toHashSet()
val openRtt = ConcurrentMap<NormalizedRelayUrl, Long>()
@@ -178,22 +293,7 @@ class RelayProber(
}
}
client.addConnectionListener(connListener)
try {
client.subscribe(subId, wave.associateWith { PROBE_FILTERS }, subListener)
val remaining = wave.toMutableSet()
withTimeoutOrNull(timeoutMs) {
while (remaining.isNotEmpty()) {
remaining.remove(terminals.receive())
}
}
} finally {
client.unsubscribe(subId)
client.removeConnectionListener(connListener)
terminals.close()
}
return wave.map { relay ->
fun verdictOf(relay: NormalizedRelayUrl): Verdict {
val opened = openRtt[relay]
val answered = eoseMs[relay]
val error = errors[relay]
@@ -202,7 +302,7 @@ class RelayProber(
// Only a connect failure, or silence with no socket, is dead.
val cannot = error?.startsWith("cannot:") == true
val reachable = !cannot && (opened != null || answered != null || error != null)
Verdict(
return Verdict(
relay = relay,
reachable = reachable,
rttOpenMs = opened ?: -1,
@@ -210,13 +310,54 @@ class RelayProber(
error = error,
)
}
client.addConnectionListener(connListener)
try {
client.subscribe(subId, wave.associateWith { filters }, subListener)
val remaining = wave.toMutableSet()
while (remaining.isNotEmpty()) {
val left = timeoutMs - mark.elapsedNow().inWholeMilliseconds
if (left <= 0) break
val relay = withTimeoutOrNull(left) { terminals.receive() } ?: break
// The emission happens OUTSIDE the timeout window so a collector that
// suspends on a verdict can never be cancelled mid-emission and lose it.
if (remaining.remove(relay)) onVerdict(verdictOf(relay))
}
// Whatever is left resolved nothing by the deadline: dead if the socket
// never opened, reachable-but-slow if it did.
for (relay in remaining) onVerdict(verdictOf(relay))
} finally {
client.unsubscribe(subId)
client.removeConnectionListener(connListener)
terminals.close()
}
}
companion object {
// A filter no event can match (ids are 64-hex of a hash): the relay answers
// with an immediate EOSE and never streams a payload. Same trick as the
// crawler's warm pool.
private val PROBE_FILTERS = listOf(Filter(ids = listOf("0".repeat(64))))
/**
* A filter no event can match (ids are 64-hex of a hash): the relay answers
* with an immediate EOSE and never streams a payload. Same trick as the
* crawler's warm pool. This is the default check — pure liveness.
*/
val LIVENESS_FILTERS = listOf(Filter(ids = listOf("0".repeat(64))))
/**
* A REQ the relay must actually WORK for: query its store and stream up to
* [limit] real events before the EOSE. Pass to [probe]/[probeFlow] as
* [filters] to turn [Verdict.rttEoseMs] into a genuine read test rather
* than a liveness ping — the time still counts from the wave start (dial
* included), so compare it against [Verdict.rttOpenMs], not across waves.
*
* [kinds] defaults to kind 0: purpose relays (purplepag.es) reject any REQ
* that names no kind with `blocked: filters must specify at least one kind`,
* and practically every relay stores SOME profile — so a kind-0, limit-1
* query works everywhere. Pass a different list to probe a specific shelf,
* or null for a kind-less query on relays known to allow one.
*/
fun readTestFilter(
limit: Int = 1,
kinds: List<Int>? = listOf(0),
) = listOf(Filter(kinds = kinds, limit = limit))
/**
* The relay universe the local store knows: every read/write relay advertised
@@ -263,3 +404,43 @@ class RelayProber(
}
}
}
/**
* This verdict as an UNSIGNED NIP-66 kind:30166 Relay Discovery template — the d-tag
* is the normalized relay url; sign it with the consumer's own monitor key (per
* NIP-66 a monitor is its own identity, so the prober never signs on its own).
*
* Only facts a probe actually observed are tagged:
* - `n` network type inferred from the url (clearnet/tor/i2p);
* - `rtt-open` when the relay was reachable — the measured WS-upgrade round trip,
* or 0 for "reachable, latency not observed" (liveness is the tag's PRESENCE);
* - `rtt-read`/`rtt-write` when a [RelayProber.readWriteCheck] result is passed
* as [readWrite] and actually measured that side;
* - `R auth` when the relay answered a probe with a NIP-42 `auth-required`
* (CLOSED on the REQ, or OK-false on the write) and `R pow` when the write
* was refused with a `pow:` reason — observed walls, not NIP-11 claims.
*
* NIP-11-derived tags (`N` supported NIPs, `k` kinds, `T` type) are deliberately
* absent: those are the relay's self-claims, and asserting them under a monitor
* signature without a per-NIP compliance test would launder claims into
* measurements. [Verdict.rttEoseMs] is likewise never written as `rtt-read` — it
* is wave-relative (dial + TLS + queueing), so the honest read number only comes
* from [readWrite] (or the [RelayObserver]/[RelayMonitor] real-traffic path).
*/
fun RelayProber.Verdict.toDiscoveryEventTemplate(
createdAt: Long = TimeUtils.now(),
readWrite: RelayProber.ReadWriteVerdict? = null,
): EventTemplate<RelayDiscoveryEvent> =
RelayDiscoveryEvent.build(relay, createdAt = createdAt) {
networkType(RelayReachabilityStore.networkTypeOf(relay))
if (reachable) rtt(RttType.OPEN, rttOpenMs.coerceAtLeast(0))
if (readWrite != null) {
if (readWrite.rttReadMs >= 0) rtt(RttType.READ, readWrite.rttReadMs)
if (readWrite.rttWriteMs >= 0) rtt(RttType.WRITE, readWrite.rttWriteMs)
}
val authWalled =
error?.startsWith("closed:auth-required") == true ||
readWrite?.writeMessage?.startsWith("auth-required") == true
if (authWalled) requirement("auth")
if (readWrite?.writeMessage?.startsWith("pow:") == true) requirement("pow")
}
@@ -22,8 +22,7 @@ package com.vitorpamplona.quartz.nip77Negentropy
import com.vitorpamplona.negentropy.storage.IStorage
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
import kotlin.concurrent.atomics.AtomicBoolean
import kotlin.concurrent.atomics.ExperimentalAtomicApi
import com.vitorpamplona.quartz.utils.concurrent.PlatformLock
/**
* Always-current `(created_at, id)` index for NIP-77 negentropy — the
@@ -54,13 +53,12 @@ import kotlin.concurrent.atomics.ExperimentalAtomicApi
* sealed storage, so one snapshot backs any number of concurrent
* sessions and stays valid even if the live index mutates after.
*
* Thread-safety: all operations take a short spin lock (same pattern as
* `LiveEventStore`'s replay dedup). Mutations arrive from the store's
* single writer; snapshots from any REQ coroutine.
* Thread-safety: all operations take a short parking lock ([PlatformLock]).
* Mutations arrive from the store's single writer; snapshots from any REQ
* coroutine.
*/
@OptIn(ExperimentalAtomicApi::class)
class LiveNegentropyIndex {
private val lock = AtomicBoolean(false)
private val lock = PlatformLock()
/** Sorted by (createdAt, id). Only touched under [locked]. */
private var entries = ArrayList<IdAndTime>()
@@ -74,14 +72,17 @@ class LiveNegentropyIndex {
private var cachedSnapshot: IStorage? = null
private var cachedGeneration = -1L
// Parks rather than busy-waits. This lock's critical sections are FAR longer than a
// handful of map ops — [rebuild] sorts the whole entry list and snapshotting builds a
// storage — so a spinning waiter would burn a core for the duration of a sort while
// concurrent ingest threads pile up. Same defect that produced the client-side ANR
// documented on PlatformLock; see quartz/.../prodbench/SpinLockConvoyBenchmark.kt.
private inline fun <R> locked(block: () -> R): R {
while (lock.exchange(true)) {
while (lock.load()) { }
}
lock.lock()
try {
return block()
} finally {
lock.store(false)
lock.unlock()
}
}
@@ -0,0 +1,99 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils
/**
* Pure-Kotlin IPv4 literal parsing and range classification, the companion to [Ipv6].
*
* Exists because asking "is this host private?" with `url.contains("192.168.")` is wrong in
* both directions: it misses `10.0.0.5` and `172.16.3.4` (so a LAN relay gets `wss://` and is
* dialed through Tor) and it matches `192.168.evil.com`, a perfectly registrable domain (so a
* hostile relay url can exempt itself from Tor). Parsing the host and testing the range is the
* only form of the question that has a right answer.
*/
object Ipv4 {
/** Parses a dotted quad in `[from, to)`, or null when the region is not one. */
fun parse(
text: String,
from: Int,
to: Int,
): ByteArray? {
// Cheapest possible rejection of a DNS host: a literal always starts with a digit.
if (from >= to || text[from] !in '0'..'9') return null
val out = ByteArray(4)
return if (parseInto(text, from, to, out, 0)) out else null
}
/**
* Parses `a.b.c.d` in `[from, to)` into four bytes at [at]. Leading zeros are rejected —
* they invite the octal reading that makes `010.1.1.1` ambiguous across resolvers.
*/
fun parseInto(
text: String,
from: Int,
to: Int,
out: ByteArray,
at: Int,
): Boolean {
var i = from
for (octet in 0 until 4) {
if (octet > 0) {
if (i >= to || text[i] != '.') return false
i++
}
var value = 0
var digits = 0
while (i < to && text[i] in '0'..'9') {
if (digits == 3) return false
if (digits == 1 && value == 0) return false // leading zero
value = value * 10 + (text[i] - '0')
digits++
i++
}
if (digits == 0 || value > 255) return false
out[at + octet] = value.toByte()
}
return i == to
}
/** `127.0.0.0/8` — the whole loopback block, not just 127.0.0.1. */
fun isLoopback(bytes: ByteArray): Boolean = octet(bytes, 0) == 127
/** RFC 1918: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`. */
fun isPrivate(bytes: ByteArray): Boolean {
val first = octet(bytes, 0)
val second = octet(bytes, 1)
return first == 10 ||
(first == 172 && second in 16..31) ||
(first == 192 && second == 168)
}
/** `169.254.0.0/16` — link-local / APIPA, reachable only on the local segment. */
fun isLinkLocal(bytes: ByteArray): Boolean = octet(bytes, 0) == 169 && octet(bytes, 1) == 254
/** `0.0.0.0/8` — "this network"; never a routable relay. */
fun isUnspecified(bytes: ByteArray): Boolean = octet(bytes, 0) == 0
private fun octet(
bytes: ByteArray,
at: Int,
) = bytes[at].toInt() and 0xFF
}
@@ -0,0 +1,245 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils
/**
* Pure-Kotlin IPv6 literal parsing, RFC 5952 canonical formatting and address
* classification. No `java.net`, so it works on every KMP target.
*
* Exists because relay identity is a *string*: [com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl]
* is the key of the connection pool, the relay-list sets, the NIP-11 cache and every
* per-relay stat map. RFC 4291 lets one address be written many ways
* (`[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]` are the same
* host), and without folding them the app treats one relay as two — two sockets, two REQ
* sets, two rows in the UI. The canonical form here matches what OkHttp renders, so the
* key the app stores is the host it actually dials.
*/
object Ipv6 {
/** Longest legal literal is 45 chars (`::ffff:` + dotted quad is shorter than 8 full groups). */
private const val MAX_LITERAL = 45
/**
* Parses a bracket-less, zone-less IPv6 literal into its 16 bytes, or null when [address]
* is not a valid literal. Accepts `::` compression and a trailing dotted quad
* (`::ffff:192.168.1.1`).
*/
fun parse(address: String): ByteArray? {
val len = address.length
if (len < 2 || len > MAX_LITERAL) return null
val out = ByteArray(16)
// Bytes written so far, counting from the left. When a `::` is present the bytes after
// it are written contiguously here and shifted to the right end at the very end.
var fill = 0
var gapAt = -1
var i = 0
if (address[0] == ':') {
if (address[1] != ':') return null
gapAt = 0
i = 2
if (i == len) return out
}
while (true) {
val groupStart = i
var value = 0
var digits = 0
while (i < len) {
val digit = hexDigit(address[i])
if (digit < 0) break
if (digits == 4) return null
value = (value shl 4) or digit
digits++
i++
}
if (i < len && address[i] == '.') {
// Trailing dotted quad: occupies the last four bytes, so nothing may follow it.
if (fill > 12) return null
if (!Ipv4.parseInto(address, groupStart, len, out, fill)) return null
fill += 4
i = len
break
}
if (digits == 0) return null
if (fill + 2 > 16) return null
out[fill++] = (value ushr 8).toByte()
out[fill++] = value.toByte()
if (i == len) break
if (address[i] != ':') return null
i++
if (i == len) return null // a single trailing ':' is not a valid literal
if (address[i] == ':') {
if (gapAt >= 0) return null // only one `::` allowed
gapAt = fill
i++
if (i == len) break
}
}
if (gapAt < 0) {
if (fill != 16) return null
} else {
// `::` must stand for at least one omitted group.
if (fill == 16) return null
val tail = fill - gapAt
for (k in tail - 1 downTo 0) {
out[16 - tail + k] = out[gapAt + k]
out[gapAt + k] = 0
}
}
return out
}
/**
* RFC 5952 text form: lowercase hex, no leading zeros, and the longest run of two or more
* zero groups replaced by `::` (leftmost run wins a tie). IPv4-mapped addresses keep their
* dotted tail. This is byte-for-byte what OkHttp prints for the same address.
*/
fun format(bytes: ByteArray): String {
require(bytes.size == 16) { "An IPv6 address is 16 bytes, got ${bytes.size}" }
var bestStart = -1
var bestLen = 0
var i = 0
while (i < 16) {
if (bytes[i] == ZERO && bytes[i + 1] == ZERO) {
val runStart = i
var j = i
while (j < 16 && bytes[j] == ZERO && bytes[j + 1] == ZERO) j += 2
if (j - runStart > bestLen) {
bestLen = j - runStart
bestStart = runStart
}
i = j
} else {
i += 2
}
}
// A single zero group is written as `0`, never as `::`.
if (bestLen < 4) {
bestStart = -1
bestLen = 0
}
val out = StringBuilder(39)
// ::ffff:a.b.c.d — IPv4-mapped addresses read as IPv4 everywhere else, so keep them that way.
if (bestStart == 0 && bestLen == 10 && bytes[10] == ALL_ONES && bytes[11] == ALL_ONES) {
out.append("::ffff:")
appendIpv4(out, bytes, 12)
return out.toString()
}
i = 0
while (i < 16) {
if (i == bestStart) {
out.append(':')
i += bestLen
if (i == 16) out.append(':')
} else {
if (i > 0) out.append(':')
out.append(group(bytes, i).toString(16))
i += 2
}
}
return out.toString()
}
/**
* Canonicalizes a bracket-less literal, preserving any `%zone` suffix verbatim (in URLs the
* zone arrives percent-encoded, e.g. `fe80::1%25wlan0`). Returns null when [address] is not
* a valid literal.
*/
fun canonicalizeOrNull(address: String): String? {
val zoneAt = address.indexOf('%')
if (zoneAt < 0) return parse(address)?.let(::format)
val bytes = parse(address.substring(0, zoneAt)) ?: return null
return format(bytes) + address.substring(zoneAt)
}
/**
* True when [address] is a valid bracket-less literal.
*
* The two-colon gate is what keeps this off the normalizer's hot path: every schemeless
* `host:port` reaching [com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer]
* has exactly one colon, and a literal needs at least two, so the common case answers
* without entering [parse] and allocating its 16-byte buffer.
*/
fun isLiteral(address: String): Boolean {
val firstColon = address.indexOf(':')
if (firstColon < 0 || address.indexOf(':', firstColon + 1) < 0) return false
// A zone id is part of the literal (`fe80::1%25eth0`), not a reason to reject it.
val zoneAt = address.indexOf('%')
return parse(if (zoneAt < 0) address else address.substring(0, zoneAt)) != null
}
/** `::1` — the IPv6 loopback, twin of 127.0.0.1. */
fun isLoopback(bytes: ByteArray): Boolean {
for (i in 0 until 15) if (bytes[i] != ZERO) return false
return bytes[15] == ONE
}
/** `fe80::/10` — link-local, only meaningful on the interface it came from. */
fun isLinkLocal(bytes: ByteArray): Boolean = bytes[0] == FE.toByte() && (bytes[1].toInt() and 0xC0) == 0x80
/** `fc00::/7` — unique local addresses, the IPv6 twin of 192.168.0.0/16. */
fun isUniqueLocal(bytes: ByteArray): Boolean = (bytes[0].toInt() and 0xFE) == 0xFC
/**
* `0200::/7` — the range Yggdrasil derives node addresses (`0200::/8`) and subnets
* (`0300::/8`) from. Formally deprecated NSAP space, so nothing else routes here: an
* address in this range is reachable only through a running mesh interface, is already
* end-to-end encrypted by the overlay, and can never hold a CA-issued certificate.
*/
fun isOverlayMesh(bytes: ByteArray): Boolean = (bytes[0].toInt() and 0xFE) == 0x02
private fun group(
bytes: ByteArray,
at: Int,
) = ((bytes[at].toInt() and 0xFF) shl 8) or (bytes[at + 1].toInt() and 0xFF)
private fun appendIpv4(
out: StringBuilder,
bytes: ByteArray,
from: Int,
) {
for (k in 0 until 4) {
if (k > 0) out.append('.')
out.append(bytes[from + k].toInt() and 0xFF)
}
}
private fun hexDigit(c: Char): Int =
when (c) {
in '0'..'9' -> c - '0'
in 'a'..'f' -> c - 'a' + 10
in 'A'..'F' -> c - 'A' + 10
else -> -1
}
private const val FE = 0xFE
private const val ZERO = 0.toByte()
private const val ONE = 1.toByte()
private const val ALL_ONES = 0xFF.toByte()
}
@@ -61,6 +61,17 @@ expect class ConcurrentMap<K : Any, V : Any>() {
remap: (old: V, new: V) -> V,
): V
/**
* Removes [key] and returns the value it held, or null when absent.
*
* CAUTION: if the removed value owns a lock that callers acquire, removal
* breaks mutual exclusion — a thread holding the old value's lock and a
* thread that re-created the entry are no longer excluding each other. Keep
* such locks in a structure whose identity is stable (see
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`).
*/
fun remove(key: K): V?
fun size(): Int
/** A point-in-time copy of the entries — safe to iterate without holding a lock. */
@@ -0,0 +1,75 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils.concurrent
/**
* A blocking mutual-exclusion lock whose waiters **park** (yield the core to the
* scheduler) instead of busy-waiting.
*
* Why this exists: `commonMain` has no `java.util.concurrent.locks.Lock`, so the
* relay client previously hand-rolled a spin lock over an `AtomicBoolean`. A
* busy-wait is only ever correct when the holder cannot be descheduled while
* holding the lock — and on Android that assumption is false. A production ANR on
* a Pixel 8 (`anr_2026-08-03-12-55-26-256`, Amethyst 1.13.1) caught the exact
* failure: the thread holding the lock was parked in `WaitingForGcToComplete`
* while **51 of 52** runnable relay-dispatch threads sat in the inlined spin loop,
* burning 596% CPU (6 of the phone's 9 cores) waiting for a holder that could not
* be scheduled to release it. The UI thread, needing to allocate, then waited on
* the same GC for over 5s and Android killed the frame with
* "Input dispatching timed out".
*
* Measured on `SpinLockConvoyBenchmark` (12-core dev machine — a phone is worse):
* the spin lock delivered 138M critical sections/s uncontended but only 1.2M/s
* with 52 contenders (0.86%, a 116x collapse), and an *unrelated* allocating
* thread's p90 latency went from 22µs to 10ms. Parking removes the CPU burn: a
* waiter costs one context switch instead of a whole core.
*
* Splits the same way [ConcurrentMap] does:
* - JVM / Android → `ReentrantLock` (real parking via `AbstractQueuedSynchronizer`).
* - Apple → `NSRecursiveLock`, which also parks. The relay client genuinely runs
* on iOS, so this must not spin — same choice commons' `KmpLock` already made.
* - Linux → a spin, since Kotlin/Native ships no parking lock and there is no
* Foundation; linuxX64 is a build/CI target, not a host for the many-relay
* workload. Swap in a pthread mutex if that changes.
*
* (`commons` has an equivalent `KmpLock`, but `commons` depends on `quartz` and not
* the reverse, so quartz cannot use it — keep the two in sync by hand.)
*
* Unlike the primitive it replaces, the JVM/Android actual is **reentrant**, so an
* accidental re-entry degrades into a no-op rather than a self-deadlock. Callers
* should still keep I/O and listener callbacks outside the critical section — that
* discipline is about holding the lock briefly, not about avoiding a hang.
*/
expect class PlatformLock() {
fun lock()
fun unlock()
}
/** Runs [block] holding [this]. Inline so hot paths allocate no closure. */
inline fun <R> PlatformLock.withLock(block: () -> R): R {
lock()
try {
return block()
} finally {
unlock()
}
}
@@ -0,0 +1,186 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrl
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* [RelayUrlNormalizer.isLocalHost] and [RelayUrlNormalizer.isOnion] decide whether a relay is
* exempt from Tor, so they must read the authority and nothing else. Relay urls arrive from
* other people — NIP-65 lists, relay hints, `r` tags — so a url whose *path* can flip those
* answers is a url that can drop a Tor user's protection.
*/
class RelayUrlAuthorityAnchoringTest {
@Test
fun aPathCannotMakeAForeignHostLookLocal() {
listOf(
"wss://evil.example.com/127.0.0.1",
"wss://evil.example.com/localhost",
"wss://evil.example.com/192.168.1.1",
"wss://evil.example.com/umbrel",
"wss://evil.example.com/x.local/y",
"wss://evil.example.com/[fd00::1]",
"wss://evil.example.com/[::1]",
"wss://evil.example.com/?q=127.0.0.1",
"wss://evil.example.com/?q=[::1]",
).forEach {
assertFalse(RelayUrlNormalizer.isLocalHost(it), "$it must not read as localhost")
}
}
@Test
fun aPathCannotMakeAForeignHostLookLikeAnOverlayOrOnion() {
assertFalse(RelayUrlNormalizer.isOverlayNetwork("wss://evil.example.com/[201:d0e:9ba5:8bbc::1]"))
assertFalse(RelayUrlNormalizer.isOnion("wss://evil.example.com/?u=http://nos.lol/.onion/"))
assertFalse(RelayUrlNormalizer.isOnion("wss://evil.example.com/abc.onion"))
}
@Test
fun realLocalAndOnionHostsStillMatch() {
listOf(
"ws://127.0.0.1:8080/",
"ws://localhost:4869/",
"ws://umbrel:4848/",
"ws://192.168.1.100:8080/",
"ws://myrelay.local:8080/",
"ws://myrelay.local/",
"ws://foo.localhost:8080/",
"ws://[::1]:4869/",
"ws://[fd12:3456::1]:8080/",
"ws://[fe80::1]/",
).forEach {
assertTrue(RelayUrlNormalizer.isLocalHost(it), "$it must read as localhost")
}
// schemeless, as fix() sees it before choosing ws:// vs wss://
assertTrue(RelayUrlNormalizer.isLocalHost("127.0.0.1:8080"))
assertTrue(RelayUrlNormalizer.isLocalHost("umbrel:4848"))
}
/**
* `.onion:8080` never matched the old `.onion/` test, so an onion relay on an explicit port
* was not recognized as onion — it skipped the forced-Tor branch and its hostname went to
* the clearnet DNS resolver.
*/
@Test
fun onionRelaysOnAnExplicitPortAreRecognized() {
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion:8080/"))
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion/"))
assertTrue(RelayUrlNormalizer.isOnion("abc123.onion:8080"))
assertTrue(RelayUrlNormalizer.isOnion("abc123.onion"))
// and it now gets ws:// like any other onion relay
assertEquals("ws://abc123.onion:8080/", "abc123.onion:8080".normalizeRelayUrl().url)
assertFalse(RelayUrlNormalizer.isOnion("wss://notonion.example.com/"))
}
/** Canonicalization must never rewrite anything outside the authority. */
@Test
fun canonicalizationLeavesPathsAndQueriesAlone() {
assertEquals(
"wss://evil.example.com/x[0:0:0:0:0:0:0:1]y",
"wss://evil.example.com/x[0:0:0:0:0:0:0:1]y".normalizeRelayUrl().url,
)
// ...while still folding a real IPv6 authority
assertEquals(
"wss://[::1]/x[0:0:0:0:0:0:0:1]y",
"wss://[0:0:0:0:0:0:0:1]/x[0:0:0:0:0:0:0:1]y".normalizeRelayUrl().url,
)
}
/**
* Private IPv4 was substring-matched, which was wrong in both directions.
*/
@Test
fun allPrivateIpv4RangesCountAsLocal() {
listOf(
"ws://127.0.0.1:8080/",
"ws://127.1.2.3:8080/",
"ws://10.0.0.5:4869/",
"ws://172.16.3.4:4869/",
"ws://172.31.255.1/",
"ws://192.168.1.5:4869/",
"ws://169.254.1.1:4869/",
"ws://0.0.0.0:4869/",
).forEach {
assertTrue(RelayUrlNormalizer.isLocalHost(it), "$it must read as localhost")
}
// a LAN relay therefore gets ws://, not a wss:// that can never hold a certificate
assertEquals("ws://10.0.0.5:4869/", "10.0.0.5:4869".normalizeRelayUrl().url)
}
@Test
fun publicIpv4AndPrivateLookalikeDomainsAreNotLocal() {
listOf(
"wss://127.0.0.1.evil.com/",
"wss://192.168.evil.com/",
"wss://10.0.0.5.evil.com/",
"wss://8.8.8.8:4869/",
"wss://172.32.0.1/",
"wss://193.168.1.5/",
"wss://relay.damus.io/",
"wss://notlocalhost.example.com/",
"wss://mylocalhost.io/",
).forEach {
assertFalse(RelayUrlNormalizer.isLocalHost(it), "$it must not read as localhost")
}
}
/**
* Host names are case-insensitive (RFC 4343), and `fix()` asks these questions before the
* RFC 3986 pass folds the case — so a case-sensitive test handed `LOCALHOST:8080` and
* `ABC.ONION:8080` a `wss://` scheme neither host can serve.
*/
@Test
fun hostTestsAreCaseInsensitive() {
assertTrue(RelayUrlNormalizer.isLocalHost("wss://LocalHost:8080/"))
assertTrue(RelayUrlNormalizer.isLocalHost("LOCALHOST:8080"))
assertTrue(RelayUrlNormalizer.isLocalHost("wss://MyRelay.LOCAL/"))
assertTrue(RelayUrlNormalizer.isOnion("wss://ABC123.ONION/"))
assertTrue(RelayUrlNormalizer.isOnion("ABC.ONION:8080"))
assertEquals("ws://localhost:8080/", "LOCALHOST:8080".normalizeRelayUrl().url)
assertEquals("ws://abc.onion:8080/", "ABC.ONION:8080".normalizeRelayUrl().url)
}
/** RFC 1034's fully-qualified form ends in a dot; it names the same host. */
@Test
fun trailingDotFqdnIsTheSameHost() {
assertTrue(RelayUrlNormalizer.isLocalHost("wss://localhost./"))
assertTrue(RelayUrlNormalizer.isLocalHost("wss://myrelay.local./"))
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion./"))
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion.:8080/"))
}
/**
* A `://` inside a path is not a scheme separator; only a real RFC 3986 scheme starts the
* authority. Otherwise the path gets read as the host.
*/
@Test
fun aColonSlashSlashInThePathIsNotASchemeSeparator() {
assertFalse(RelayUrlNormalizer.isLocalHost("relay.example.com/x://127.0.0.1"))
assertFalse(RelayUrlNormalizer.isOnion("nos.lol/?u=x://abc.onion"))
// a real scheme still starts the authority
assertTrue(RelayUrlNormalizer.isLocalHost("wss://127.0.0.1/x://evil.com"))
}
}
@@ -52,4 +52,93 @@ class RelayUrlFormatterTest {
fun weirdRelay() {
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://relay%20list%20to%20discover%20the%20user's%20content"))
}
@Test
fun trailingPercentTwentyIsTrimmedButBareTrailingZeroIsNot() {
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom%20")?.url)
// urls merely ending in '%', '2' or '0' (every port ending in zero) must pass untouched
assertEquals("wss://nostr.mom:3030/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom:3030")?.url)
assertEquals("wss://nostr.mom:8020/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom:8020")?.url)
}
@Test
fun httpWithPathIsNotARelay() {
// Mastodon/bridge actor urls from `proxy` tags: web resources, not relays
assertNull(RelayUrlNormalizer.normalizeOrNull("https://mastodon.social/users/amanita_muscaria"))
assertNull(RelayUrlNormalizer.normalizeOrNull("https://fosstodon.org/ap/users/115532410310000993"))
assertNull(RelayUrlNormalizer.normalizeOrNull("http://example.com/relay"))
assertNull(RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/?author=0"))
assertNull(RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/#section"))
// but bare hosts still convert, with or without port and trailing slash
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom")?.url)
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/")?.url)
assertEquals("wss://nostr.mom:4443/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom:4443/")?.url)
assertEquals("ws://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("http://nostr.mom")?.url)
}
@Test
fun wsWithPathIsStillARelay() {
assertEquals("wss://relay.nostr.band/all", RelayUrlNormalizer.normalizeOrNull("wss://relay.nostr.band/all")?.url)
assertEquals(
"wss://bostr.lecturify.net/?accept=0,1",
RelayUrlNormalizer.normalizeOrNull("wss://bostr.lecturify.net/?accept=0,1")?.url,
)
}
@Test
fun brokenSchemeGarbage() {
assertNull(RelayUrlNormalizer.normalizeOrNull("wss:"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://https//nostr.watch/relay/nostr.21crypto.ch"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://https://lockbox.fiatjaf.com"))
assertNull(RelayUrlNormalizer.normalizeOrNull("ws://http//nos.lol"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://://plebstr.com"))
}
@Test
fun nostrUriIsNotARelay() {
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr://nrelay1qqxhwumn8ghj77tpvf6jumt9e2ckgn/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr://npub1dwy079xmpz7mk02kvz6wan49h02635umk32aa4ufek8t8mjxv58qy2nr22/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr:nrelay1qq8k2cnfwejhyum99eek7cmfv9kqsm7sdm"))
}
@Test
fun addressablePointerIsNotARelay() {
assertNull(RelayUrlNormalizer.normalizeOrNull("31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr"))
}
@Test
fun authorityGarbage() {
// userinfo, percent-encoding and commas never appear in a real relay host
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://catuaba@plebs.place/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://africa.nostr.joburg%0A/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://bitcoiner,social/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://#web3/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("name@domain.com"))
}
@Test
fun interiorWhitespaceAndBackslashes() {
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://nos lol"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://nos.lol/ wss:/nostr.land/ avatar wss:/nostr.wine/"))
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://\\\\relay.damus.io/"))
}
@Test
fun invisibleCharactersAreStripped() {
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("wss://\u200Bnos.lol")?.url)
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("\uFEFFwss://nos.lol")?.url)
}
@Test
fun protocolRelativeUrls() {
assertEquals("wss://relay.most.pub/", RelayUrlNormalizer.normalizeOrNull("//relay.most.pub/")?.url)
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("//nos.lol/")?.url)
}
@Test
fun ipv6AndLanHostsStillWork() {
assertEquals("ws://[31b:6f20:c7f2:3ddf::3221]/", RelayUrlNormalizer.normalizeOrNull("ws://[31b:6f20:c7f2:3ddf::3221]/")?.url)
assertEquals("ws://geyser-relay:7777/", RelayUrlNormalizer.normalizeOrNull("ws://geyser-relay:7777/")?.url)
}
}
@@ -0,0 +1,385 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.accessories
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.utils.TimeUtils
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
import kotlin.test.assertSame
import kotlin.test.assertTrue
/**
* A paged relay has no memory of what it already sent, so without a band every
* restart re-downloads its whole corpus. These pin the band arithmetic and, more
* importantly, the cases where a band must NOT be used — a stale band silently
* skips events, which is a worse failure than re-reading them.
*/
class SyncCoverageTest {
private val relay = RelayUrlNormalizer.normalize("wss://relay.example")
private val other = RelayUrlNormalizer.normalize("wss://other.example")
private val profiles = Filter(kinds = listOf(0))
private fun now(): Long = TimeUtils.now()
// ---- the band arithmetic ----------------------------------------------
@Test
fun `with nothing recorded the whole filter is fetched`() {
val c = SyncCoverage()
assertEquals(listOf(profiles), c.legs(relay, profiles))
}
@Test
fun `a recorded band is fetched around rather than through`() {
val c = SyncCoverage()
c.record(relay, profiles, observedMin = 1_700_001_000L, observedMax = 1_700_002_000L, paged = true)
val legs = c.legs(relay, profiles)
assertEquals(2, legs.size, "one leg older than the band and one newer")
assertEquals(1_700_001_000L, legs[0].until, "older leg stops AT the band floor")
assertNull(legs[0].since, "and reaches as far back as the filter allows")
assertEquals(1_700_002_000L, legs[1].since, "newer leg starts AT its ceiling")
assertNull(legs[1].until)
}
@Test
fun `an event sharing the band boundary second is still reachable`() {
// A paged relay cuts pages by count, so a boundary can fall inside a run
// of events sharing one created_at. Excluding the edge would strand the
// rest of that second in no leg at all, while the band called it covered.
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
val legs = c.legs(relay, profiles)
fun reachable(t: Long) = legs.any { (it.since ?: Long.MIN_VALUE) <= t && t <= (it.until ?: Long.MAX_VALUE) }
assertTrue(reachable(1_700_001_000L), "the band floor second must be re-read")
assertTrue(reachable(1_700_002_000L), "and its ceiling second")
assertTrue(reachable(1_700_000_999L), "below the band")
assertTrue(reachable(1_700_002_001L), "above it")
// Only the interior is skipped, which is the entire point.
assertTrue(!reachable(1_700_001_500L), "the covered interior is not re-read")
}
@Test
fun `successive runs widen the band rather than replacing it`() {
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
// A later run reaches further back and picks up newer events.
c.record(relay, profiles, 1_700_000_500L, 1_700_002_500L, paged = true)
val band = c.band(relay, profiles)!!
assertEquals(1_700_000_500L, band.minCreatedAt)
assertEquals(1_700_002_500L, band.maxCreatedAt)
}
@Test
fun `a capped relay walks further back on each run`() {
// The case that makes this worth having: a relay that only ever answers
// with its newest N events. Each run starts below the last one's floor.
val c = SyncCoverage()
c.record(relay, profiles, 1_700_009_000L, 1_700_010_000L, paged = true)
assertEquals(1_700_009_000L, c.legs(relay, profiles)[0].until)
c.record(relay, profiles, 1_700_008_000L, 1_700_008_999L, paged = true)
assertEquals(1_700_008_000L, c.legs(relay, profiles)[0].until)
}
// ---- when a band must not be used --------------------------------------
@Test
fun `a negentropy sync that reported no outcome records nothing`() {
// Only a sync that says how far it reconciled earns a band; a bare
// paged=false call carries no claim to record.
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = false)
assertNull(c.band(relay, profiles))
assertEquals(listOf(profiles), c.legs(relay, profiles))
}
// ---- coverage: what a finished reconcile earns -------------------------
@Test
fun `a finished reconcile is in sync through the instant it started`() {
// Not through the newest event it happened to see: "the relay had nothing
// newer" and "we never asked" must not record the same thing.
val c = SyncCoverage()
val startedAt = now() - 60
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = false, reconciledThrough = startedAt)
val band = c.band(relay, profiles)!!
assertTrue(band.complete)
assertEquals(startedAt, band.maxCreatedAt)
}
@Test
fun `a reconcile that downloaded nothing still records coverage`() {
// The empty case is the WHOLE point: nothing came back because we already
// have it, and that is exactly when the next run should ask for a sliver.
val c = SyncCoverage()
val startedAt = now() - 60
c.record(relay, profiles, null, null, paged = false, reconciledThrough = startedAt)
val leg = c.legs(relay, profiles).single()
assertEquals(startedAt, leg.since)
assertNull(leg.until)
}
@Test
fun `a complete band drops its older leg while a paged one keeps it`() {
val reconciled = SyncCoverage()
reconciled.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_002_000L)
val only = reconciled.legs(relay, profiles).single()
assertEquals(1_700_002_000L, only.since)
val walked = SyncCoverage()
walked.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
assertEquals(2, walked.legs(relay, profiles).size, "a paged walk says nothing about what it never asked for")
}
@Test
fun `a deeper floor re-opens history below a complete band`() {
// A reconcile only compared down to the window it ran against. When
// the operator raises the backfill window, the span below the band's
// recorded floor is ground nobody ever asked for.
val c = SyncCoverage()
c.record(relay, profiles, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_002_000L)
assertEquals(1, c.legs(relay, profiles, floor = 1_700_000_000L).size, "same floor: nothing older to ask")
val legs = c.legs(relay, profiles, floor = 1_600_000_000L)
assertEquals(2, legs.size, "a deeper floor re-opens the older span")
assertEquals(1_700_000_000L, legs[0].until, "up to the floor the reconcile actually compared")
assertEquals(1_700_002_000L, legs[1].since)
}
// ---- the periodic full re-walk -----------------------------------------
@Test
fun `a band stops narrowing once it is older than the resync period`() {
val c = SyncCoverage(fullResyncSeconds = 60)
c.record(relay, profiles, null, null, paged = false, reconciledThrough = now() - 3600)
// Recorded 'now' whatever the created_at claim, so age it by rewriting.
c.record(relay, profiles, null, null, paged = false, reconciledThrough = now())
assertEquals(1, c.legs(relay, profiles).size, "fresh band still narrows")
val stale = SyncCoverage(fullResyncSeconds = 0)
stale.record(relay, profiles, null, null, paged = false, reconciledThrough = now())
assertSame(profiles, stale.legs(relay, profiles).single(), "a band past its period re-walks everything")
}
@Test
fun `the re-walk replaces the old claim instead of widening it`() {
// Widening would carry the stale band's floor forward forever and the
// periodic pass would never actually reset anything.
val c = SyncCoverage(fullResyncSeconds = 0)
c.record(relay, profiles, 1_700_000_000L, 1_700_001_000L, paged = true)
c.record(relay, profiles, 1_700_005_000L, 1_700_006_000L, paged = true)
val band = c.band(relay, profiles)!!
assertEquals(1_700_005_000L, band.minCreatedAt, "the second pass walked everything; its span is the whole picture")
}
// ---- the shared snapshot window ----------------------------------------
@Test
fun `covering window collapses to the oldest ceiling once everyone is caught up`() {
val c = SyncCoverage()
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
c.record(other, profiles, null, null, paged = false, reconciledThrough = 1_700_003_000L)
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other), profiles).since)
}
@Test
fun `one relay that has never synced puts the window back to the whole filter`() {
// It genuinely needs everything — narrowing the shared snapshot would
// reconcile it against ids we never looked up.
val c = SyncCoverage()
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
// The filter itself, unnarrowed — identity, since Filter has no equals.
assertSame(profiles, c.coveringWindow(listOf(relay, other), profiles))
assertSame(profiles, c.coveringWindow(emptyList(), profiles))
}
@Test
fun `one shared window serves a whole stream of relays`() {
// Every url in a stream shares that stream's filter, so a backfill can
// take ONE snapshot for all of them instead of walking the identical
// range once per relay for byte-identical answers.
val c = SyncCoverage()
val third = RelayUrlNormalizer.normalize("wss://third.example")
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
c.record(other, profiles, null, null, paged = false, reconciledThrough = 1_700_003_000L)
c.record(third, profiles, null, null, paged = false, reconciledThrough = 1_700_007_000L)
// The hungriest of them sets the floor; the other two re-read a little.
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other, third), profiles).since)
}
@Test
fun `a fully covered relay does not widen the shared window`() {
// A complete band past a bounded filter's ceiling needs no legs at
// all. The best case must not force the snapshot back to the whole
// filter — that would make full coverage cost the most.
val window = Filter(kinds = listOf(0), since = 1_700_000_000L, until = 1_700_005_000L)
val c = SyncCoverage()
c.record(relay, window, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_009_000L)
c.record(other, window, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_003_000L)
assertEquals(0, c.legs(relay, window).size, "covered past the ceiling: nothing to ask")
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other), window).since)
}
@Test
fun `a relay with an older gap also widens the shared window`() {
val c = SyncCoverage()
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
c.record(other, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
assertSame(profiles, c.coveringWindow(listOf(relay, other), profiles))
}
@Test
fun `an empty fetch records nothing`() {
// No events says nothing about what the relay holds, only that this
// window was empty — recording it would fabricate coverage.
val c = SyncCoverage()
c.record(relay, profiles, null, null, paged = true)
assertNull(c.band(relay, profiles))
}
@Test
fun `one misdated event does not cost a relay its whole band`() {
// A single future-dated stamp among hundreds of thousands must not fail
// a check applied to the aggregate. Screening per event keeps the rest.
val c = SyncCoverage()
val far = now() + 400L * 86_400
val observed = listOf(1_700_001_000L, far, 1_700_002_000L, 0L)
val plausible = observed.filter { SyncCoverage.isPlausible(it) }
c.record(relay, profiles, plausible.min(), plausible.max(), paged = true)
val band = c.band(relay, profiles)!!
assertEquals(1_700_001_000L, band.minCreatedAt)
assertEquals(1_700_002_000L, band.maxCreatedAt)
}
@Test
fun `changing the filter starts over`() {
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
// Widening the kinds means the old band skipped events it never fetched.
val wider = Filter(kinds = listOf(0, 10002))
assertEquals(listOf(wider), c.legs(relay, wider), "a new filter has no band")
assertNull(c.band(relay, wider))
// ...and the original is untouched, so reverting resumes where it was.
assertEquals(1_700_001_000L, c.band(relay, profiles)!!.minCreatedAt)
}
@Test
fun `each relay keeps its own band`() {
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
assertEquals(listOf(profiles), c.legs(other, profiles))
}
// ---- the filter's own bounds still win ---------------------------------
@Test
fun `a bounded filter never widens past its own since and until`() {
val bounded = Filter(kinds = listOf(0), since = 1_700_001_000L, until = 1_700_005_000L)
val c = SyncCoverage()
c.record(relay, bounded, 1_700_002_000L, 1_700_003_000L, paged = true)
val legs = c.legs(relay, bounded)
assertEquals(2, legs.size)
assertEquals(1_700_001_000L, legs[0].since, "the older leg keeps the configured floor")
assertEquals(1_700_002_000L, legs[0].until)
assertEquals(1_700_003_000L, legs[1].since)
assertEquals(1_700_005_000L, legs[1].until, "the newer leg keeps the configured ceiling")
}
@Test
fun `a fully covered bounded filter re-reads only its two edge seconds`() {
// Inclusive edges mean "covered" can never quite mean "ask for nothing":
// the two boundary seconds are always re-read, because that is the only
// way to catch a run of same-second events a page boundary cut in half.
val bounded = Filter(kinds = listOf(0), since = 1_700_001_000L, until = 1_700_005_000L)
val c = SyncCoverage()
c.record(relay, bounded, 1_700_001_000L, 1_700_005_000L, paged = true)
val legs = c.legs(relay, bounded)
assertEquals(2, legs.size)
assertEquals(1_700_001_000L to 1_700_001_000L, legs[0].since to legs[0].until, "the floor second only")
assertEquals(1_700_005_000L to 1_700_005_000L, legs[1].since to legs[1].until, "the ceiling second only")
}
// ---- persistence hooks --------------------------------------------------
@Test
fun `export and restore round-trip the bands`() {
val c = SyncCoverage()
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
val reopened = SyncCoverage()
reopened.restore(c.export())
val band = reopened.band(relay, profiles)!!
assertEquals(1_700_001_000L, band.minCreatedAt)
assertEquals(1_700_002_000L, band.maxCreatedAt)
}
@Test
fun `onChange fires when a band changes so persistence can mark dirty`() {
var changes = 0
val c = SyncCoverage(onChange = { changes++ })
c.record(relay, profiles, null, null, paged = true)
assertEquals(0, changes, "an empty fetch records nothing and must not dirty the store")
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
assertEquals(1, changes)
}
@Test
fun `the same filter instance is fingerprinted once`() {
// Filter.toJson() runs to tens of thousands of characters for an
// author-scoped filter, and a fan-out keys once per relay per cycle.
val big = Filter(kinds = listOf(30382), authors = (1..500).map { it.toString(16).padStart(64, '0') })
val c = SyncCoverage()
c.record(relay, big, 1_700_001_000L, 1_700_002_000L, paged = true)
// Same instance, many lookups: still one band, and cheap.
repeat(50) { c.legs(relay, big) }
assertEquals(1_700_001_000L, c.band(relay, big)!!.minCreatedAt)
// An equal-but-distinct instance keys the same way; it just misses the cache.
val copy = Filter(kinds = listOf(30382), authors = (1..500).map { it.toString(16).padStart(64, '0') })
assertEquals(1_700_001_000L, c.band(relay, copy)?.minCreatedAt, "identity caching must not change the key")
}
}
@@ -0,0 +1,155 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNull
import kotlin.test.assertTrue
class PagingWindowProgressTest {
private fun assertClose(
expected: Double,
actual: Double?,
message: String? = null,
) {
assertTrue(actual != null && kotlin.math.abs(expected - actual) < 0.001, "${message ?: ""} expected $expected got $actual")
}
@Test
fun `progress is the paged share of the time window`() {
val p = PagingWindowProgress()
p.begin("a", top = 1_000L, bottom = 0L)
assertClose(0.0, p.fraction(), "nothing paged yet")
p.mark("a", 750L)
assertClose(0.25, p.fraction())
p.mark("a", 100L)
assertClose(0.90, p.fraction())
}
@Test
fun `a page that jumps backwards cannot un-advance the pagination`() {
// Pages arrive from one relay in order, but nothing in the protocol
// guarantees it, and a percentage that goes DOWN is worse than one that
// is slightly wrong — it reads as the sync having lost ground.
val p = PagingWindowProgress()
p.begin("a", top = 1_000L, bottom = 0L)
p.mark("a", 200L)
p.mark("a", 900L)
assertClose(0.80, p.fraction(), "the later higher until is ignored")
}
@Test
fun `windows average rather than sum`() {
// Two relays each paging their own window: one done and one untouched
// is half way — not 100% as summing would give.
val p = PagingWindowProgress()
p.begin("a", top = 1_000L, bottom = 0L)
p.begin("b", top = 500L, bottom = 0L)
p.mark("a", 0L)
assertClose(0.5, p.fraction())
}
@Test
fun `a finished window leaves the average`() {
val p = PagingWindowProgress()
p.begin("a", top = 1_000L, bottom = 0L)
p.begin("b", top = 1_000L, bottom = 0L)
p.mark("b", 500L)
p.finish("a")
assertClose(0.5, p.fraction(), "only b is still paging")
p.finish("b")
assertNull(p.fraction(), "nothing paging means no number to report")
}
@Test
fun `a group prefix scopes the numbers to its own paginations`() {
// One instance serves many concurrent paginations; without the scope two
// streams would print each other's percentages.
val p = PagingWindowProgress()
p.begin("streamA|wss://r1", top = 1_000L, bottom = 0L)
p.begin("streamB|wss://r2", top = 1_000L, bottom = 0L)
p.mark("streamA|wss://r1", 0L)
assertClose(1.0, p.fraction("streamA"))
assertClose(0.0, p.fraction("streamB"))
assertClose(0.5, p.fraction())
}
@Test
fun `an inverted window is not a pagination`() {
// A leg whose since is above its until asks for a range nothing can be
// in. Dividing by that span would produce infinities on the status line.
val p = PagingWindowProgress()
p.begin("a", top = 100L, bottom = 900L)
assertNull(p.fraction())
}
@Test
fun `a single-second window is a pagination`() {
// Coverage legs re-read a band's edge second: since == until is a
// real, one-second range, not an inverted one.
val p = PagingWindowProgress()
p.begin("a", top = 500L, bottom = 500L)
assertClose(0.0, p.fraction(), "tracked from its start")
p.mark("a", 500L)
assertClose(0.0, p.fraction(), "still at its only second")
p.finish("a")
assertNull(p.fraction())
}
@Test
fun `no ETA before the estimate means anything`() {
val p = PagingWindowProgress()
p.begin("a", top = 1_000_000L, bottom = 0L)
p.mark("a", 999_000L)
// 0.1% in: extrapolating here yields days-long ETAs from connect
// latency alone, which is worse than printing nothing.
assertNull(p.etaMs(), "too early to extrapolate")
}
@Test
fun `ETA extrapolates from the rate achieved so far`() {
var clock = 1_000_000L
val p = PagingWindowProgress(nowMillis = { clock })
p.begin("a", top = 1_000L, bottom = 0L)
p.mark("a", 500L)
// Half way after six seconds: whatever has elapsed is also what remains.
clock += 6_000
assertEquals(6_000L, p.etaMs())
}
}
@@ -0,0 +1,93 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.client.pool
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class FiltersChangedTest {
private fun authors(vararg a: String) = Filter(authors = a.toList())
@Test
fun emptyListsAreUnchanged() {
assertFalse(FiltersChanged.needsToResendRequest(emptyList(), emptyList()))
}
@Test
fun differentSizesNeedResend() {
assertTrue(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("a"), authors("b"))))
}
@Test
fun identicalSingleFilterDoesNotNeedResend() {
assertFalse(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("a"))))
}
@Test
fun changedFirstFilterNeedsResend() {
assertTrue(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("b"))))
}
/**
* Regression: the loop used a non-local `return` on the first iteration, so only
* filters[0] was ever compared. A subscription whose first filter was unchanged
* reported "no resend needed" no matter what happened to the rest, and silently
* went stale — the relay kept serving the old filter set.
*/
@Test
fun changedSecondFilterNeedsResend() {
val old = listOf(authors("a"), authors("b"))
val new = listOf(authors("a"), authors("CHANGED"))
assertTrue(FiltersChanged.needsToResendRequest(old, new))
}
@Test
fun changedLastOfManyNeedsResend() {
val old = listOf(authors("a"), authors("b"), authors("c"), authors("d"))
val new = listOf(authors("a"), authors("b"), authors("c"), authors("CHANGED"))
assertTrue(FiltersChanged.needsToResendRequest(old, new))
}
@Test
fun allIdenticalOfManyDoesNotNeedResend() {
val old = listOf(authors("a"), authors("b"), authors("c"))
val new = listOf(authors("a"), authors("b"), authors("c"))
assertFalse(FiltersChanged.needsToResendRequest(old, new))
}
/** `since` moving forward is deliberately NOT a resend trigger, on any index. */
@Test
fun sinceMovingForwardOnLaterFilterDoesNotNeedResend() {
val old = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 100))
val new = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 200))
assertFalse(FiltersChanged.needsToResendRequest(old, new))
}
/** ...but moving backwards in time is, including on a later filter. */
@Test
fun sinceMovingBackwardsOnLaterFilterNeedsResend() {
val old = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 200))
val new = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 100))
assertTrue(FiltersChanged.needsToResendRequest(old, new))
}
}
@@ -0,0 +1,125 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip11RelayInfo
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertNull
class CachedNip11FetcherTest {
private val relay = RelayUrlNormalizer.normalize("wss://nostr.example.com")
/** Counts network hits; serves [doc] or throws when [failing]. */
private class FakeFetcher : Nip11Fetcher {
var calls = 0
var failing = false
var doc = Nip11RelayInformation(name = "v1")
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation {
calls++
if (failing) throw Nip11FetchException("boom")
return doc
}
}
private fun cached(
delegate: FakeFetcher,
clock: () -> Long,
) = CachedNip11Fetcher(delegate, ttlSeconds = 100, errorTtlSeconds = 10, now = clock)
@Test
fun freshSuccessIsServedFromCache() =
runBlocking {
val net = FakeFetcher()
var now = 0L
val fetcher = cached(net) { now }
assertEquals("v1", fetcher.fetch(relay).name)
now = 99
assertEquals("v1", fetcher.fetch(relay).name)
assertEquals(1, net.calls, "second fetch inside the TTL must not touch the network")
}
@Test
fun successExpiresAfterTtl() =
runBlocking {
val net = FakeFetcher()
var now = 0L
val fetcher = cached(net) { now }
fetcher.fetch(relay)
net.doc = Nip11RelayInformation(name = "v2")
now = 100
assertEquals("v2", fetcher.fetch(relay).name)
assertEquals(2, net.calls)
}
@Test
fun failureIsCachedForItsOwnShorterTtl() =
runBlocking {
val net = FakeFetcher().apply { failing = true }
var now = 0L
val fetcher = cached(net) { now }
assertFailsWith<Nip11FetchException> { fetcher.fetch(relay) }
now = 9
assertFailsWith<Nip11FetchException> { fetcher.fetch(relay) }
assertEquals(1, net.calls, "a fresh failure must be served from cache, not re-fetched")
now = 10
net.failing = false
assertEquals("v1", fetcher.fetch(relay).name)
assertEquals(2, net.calls, "an expired failure must be retried")
}
@Test
fun invalidateForcesAFreshFetch() =
runBlocking {
val net = FakeFetcher()
val fetcher = cached(net) { 0 }
fetcher.fetch(relay)
fetcher.invalidate(relay)
fetcher.fetch(relay)
assertEquals(2, net.calls)
}
@Test
fun cachedOrNullNeverTouchesTheNetwork() =
runBlocking {
val net = FakeFetcher()
var now = 0L
val fetcher = cached(net) { now }
assertNull(fetcher.cachedOrNull(relay))
assertEquals(0, net.calls)
fetcher.fetch(relay)
assertEquals("v1", fetcher.cachedOrNull(relay)?.name)
now = 100
assertNull(fetcher.cachedOrNull(relay), "a stale hit must not be served")
assertEquals(1, net.calls)
}
}
@@ -0,0 +1,150 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip50Search
import com.vitorpamplona.quartz.buzz.agentProfiles.AgentProfileEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.nip17Dm.messages.ChatMessageEvent
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
import com.vitorpamplona.quartz.nip35Torrents.TorrentEvent
import com.vitorpamplona.quartz.nip89AppHandlers.definition.AppDefinitionEvent
import com.vitorpamplona.quartz.nipB0WebBookmarks.WebBookmarkEvent
import kotlin.test.Test
import kotlin.test.assertEquals
class SearchFieldExtractorTest {
private val alice = "a1".repeat(32)
@Test
fun kind0DecomposesIntoTheProfileRoles() {
val content = """{"name":"vitor","display_name":"Vitor P","about":"builds nostr","nip05":"vitor@vitorpamplona.com","lud16":"me@wallet.com","website":"https://vitorpamplona.com","picture":"https://x/y.jpg"}"""
val fields = SearchFieldExtractor.extract(MetadataEvent("1".repeat(64), alice, 1L, emptyArray(), content, ""))
assertEquals(
IndexableFields.Profile(
name = "vitor",
displayName = "Vitor P",
about = "builds nostr",
nip05 = "vitor@vitorpamplona.com",
lud16 = "me@wallet.com",
website = "https://vitorpamplona.com",
),
fields,
)
}
@Test
fun longFormDecomposesIntoTitleSummaryHashtagsContent() {
val tags = arrayOf(arrayOf("d", "post"), arrayOf("title", "My Post"), arrayOf("summary", "tl;dr"), arrayOf("t", "nostr"), arrayOf("t", "search"))
val fields = SearchFieldExtractor.extract(LongTextNoteEvent("2".repeat(64), alice, 1L, tags, "the whole article", ""))
assertEquals(IndexableFields.Tiered(primary = listOf("My Post"), secondary = listOf("tl;dr"), text = "the whole article", hashtags = listOf("nostr", "search")), fields)
}
@Test
fun notesUseTheSubjectAndHashtags() {
val tags = arrayOf(arrayOf("subject", "meetup"), arrayOf("t", "brazil"))
val fields = SearchFieldExtractor.extract(TextNoteEvent("3".repeat(64), alice, 1L, tags, "see you there", ""))
assertEquals(IndexableFields.Tiered(primary = listOf("meetup"), text = "see you there", hashtags = listOf("brazil")), fields)
}
@Test
fun locationTagsAreCarriedRawLikeHashtags() {
val tags = arrayOf(arrayOf("location", "Rio de Janeiro"))
val fields = SearchFieldExtractor.extract(TextNoteEvent("4".repeat(64), alice, 1L, tags, "gm", ""))
assertEquals(IndexableFields.Tiered(text = "gm", locations = listOf("Rio de Janeiro")), fields)
}
@Test
fun torrentsIndexFileNamesAndTrackers() {
val tags =
arrayOf(
arrayOf("title", "Great Torrent"),
arrayOf("file", "episode1.mkv"),
arrayOf("file", "episode2.mkv"),
arrayOf("tracker", "https://tracker.example.com"),
)
val fields = SearchFieldExtractor.extract(TorrentEvent("5".repeat(64), alice, 1L, tags, "a series", ""))
assertEquals(
IndexableFields.Tiered(
primary = listOf("Great Torrent"),
// Unjoined: the backend decides how file names are indexed.
secondary = listOf("episode1.mkv", "episode2.mkv"),
text = "a series",
websites = listOf("https://tracker.example.com"),
),
fields,
)
}
@Test
fun webBookmarksAreFindableByTheirUrl() {
// NIP-B0: the d tag carries the URL scheme-less; url() re-adds https://.
val tags = arrayOf(arrayOf("d", "vitorpamplona.com/post"), arrayOf("title", "A Post"))
val fields = SearchFieldExtractor.extract(WebBookmarkEvent("6".repeat(64), alice, 1L, tags, "", ""))
assertEquals(IndexableFields.Tiered(primary = listOf("A Post"), websites = listOf("https://vitorpamplona.com/post")), fields)
}
@Test
fun appHandlerMetadataReusesTheProfileRoles() {
val content = """{"name":"CoolApp","about":"an app","website":"https://coolapp.example"}"""
val fields = SearchFieldExtractor.extract(AppDefinitionEvent("7".repeat(64), alice, 1L, arrayOf(arrayOf("d", "x")), content, ""))
assertEquals(IndexableFields.Profile(name = "CoolApp", about = "an app", website = "https://coolapp.example"), fields)
}
@Test
fun nonSearchableKindsExtractNothing() {
// Kind 7 reactions are not SearchableEvent.
val reaction = Event("8".repeat(64), alice, 1L, 7, emptyArray(), "+", "")
assertEquals(IndexableFields.None, SearchFieldExtractor.extract(reaction))
}
@Test
fun unmappedSearchableKindsFallBackToTheTextTier() {
val fields = SearchFieldExtractor.extract(ChatMessageEvent("a".repeat(64), alice, 1L, emptyArray(), "hello group", ""))
assertEquals(IndexableFields.Tiered(text = "hello group"), fields)
}
@Test
fun blankContentKindsNormalizeToNone() {
val fields = SearchFieldExtractor.extract(TextNoteEvent("b".repeat(64), alice, 1L, emptyArray(), " ", ""))
assertEquals(IndexableFields.None, fields)
}
@Test
fun unparseableBuzzContentStillIndexesItsHashtags() {
// The branch finds no text, but the tiers() funnel still carries the
// event's raw tags — the one subtle reachability seam of the port.
val tags = arrayOf(arrayOf("t", "agents"))
val fields = SearchFieldExtractor.extract(AgentProfileEvent("c".repeat(64), alice, 1L, tags, "not json", ""))
assertEquals(IndexableFields.Tiered(hashtags = listOf("agents")), fields)
}
@Test
fun profileShapesNeverGetHashtagFolding() {
// Hashtags/locations are filled only by the tiers() funnel, which
// profile branches never use: a kind-0 with t-tags stays a pure
// profile (and an empty one normalizes to None).
val tags = arrayOf(arrayOf("t", "nostr"))
val fields = SearchFieldExtractor.extract(MetadataEvent("9".repeat(64), alice, 1L, tags, "{}", ""))
assertEquals(IndexableFields.None, fields)
}
}
@@ -199,4 +199,148 @@ class SearchQueryTest {
assertSame(plain, mixed[0])
assertEquals("bitcoin", mixed[1].search)
}
// ---- quoted phrases and -word exclusions --------------------------------
@Test
fun quotedSpanBecomesPhrase() {
val q = SearchQuery.parse("best \"nostr apps\" today")
assertEquals("best today", q.terms)
assertEquals(listOf("nostr apps"), q.phrases)
assertTrue(q.notPhrases.isEmpty())
assertTrue(q.hasText)
}
@Test
fun negatedQuotedSpanBecomesPhraseExclusion() {
val q = SearchQuery.parse("pizza -\"pineapple pizza\"")
assertEquals("pizza", q.terms)
assertEquals(listOf("pineapple pizza"), q.notPhrases)
assertTrue(q.phrases.isEmpty())
}
@Test
fun minusWordBecomesExclusion() {
val q = SearchQuery.parse("pizza -pineapple")
assertEquals("pizza", q.terms)
assertEquals(listOf("pineapple"), q.notTerms)
// Exclusions alone are not required text.
assertFalse(SearchQuery.parse("-pineapple").hasText)
}
@Test
fun loneMinusStaysATerm() {
val q = SearchQuery.parse("a - b")
assertEquals("a - b", q.terms)
assertTrue(q.notTerms.isEmpty())
}
@Test
fun allLeadingDashesAreStripped() {
assertEquals(listOf("word"), SearchQuery.parse("--word").notTerms)
}
@Test
fun quotesProtectExtensionShapedTokens() {
// The quote pass runs BEFORE the extension pass, so a quoted
// extension-shaped token is a phrase, not an extension.
val q = SearchQuery.parse("\"include:spam\"")
assertEquals(listOf("include:spam"), q.phrases)
assertFalse(q.includeSpam)
assertTrue(q.extensions.isEmpty())
}
@Test
fun spanEndingInExtensionKeepsTrailingExclusion() {
// Quote-blind extension parsing would eat the closing quote of
// "pizza include:spam" and swallow the trailing -word; the quote-first
// order keeps the exclusion an exclusion.
val q = SearchQuery.parse("\"pizza include:spam\" -pineapple")
assertEquals(listOf("pizza include:spam"), q.phrases)
assertEquals(listOf("pineapple"), q.notTerms)
assertTrue(q.extensions.isEmpty())
}
@Test
fun minusOnExtensionShapedTokenExcludesTheLiteral() {
// There is no `-extension` syntax: keys are strictly a-z, so the `-`
// makes the whole token an excluded literal.
val q = SearchQuery.parse("pizza -include:spam")
assertEquals(listOf("include:spam"), q.notTerms)
assertFalse(q.includeSpam)
}
@Test
fun unclosedQuoteRunsToEnd() {
val q = SearchQuery.parse("\"nostr apps today")
assertEquals(listOf("nostr apps today"), q.phrases)
assertEquals("", q.terms)
}
@Test
fun midTokenQuoteStaysOrdinary() {
val q = SearchQuery.parse("don\"t panic")
assertEquals("don\"t panic", q.terms)
assertTrue(q.phrases.isEmpty())
}
@Test
fun emptySpansAreDropped() {
val q = SearchQuery.parse("a \"\" b -\"\"")
assertEquals("a b", q.terms)
assertTrue(q.phrases.isEmpty())
assertTrue(q.notPhrases.isEmpty())
}
@Test
fun phrasesComposeWithExtensions() {
val q = SearchQuery.parse("\"nostr apps\" best domain:example.com -spam")
assertEquals("best", q.terms)
assertEquals(listOf("nostr apps"), q.phrases)
assertEquals(listOf("spam"), q.notTerms)
assertEquals("example.com", q.domain)
}
@Test
fun toSearchStringRoundTripsFullGrammar() {
val q = SearchQuery.parse("best \"nostr apps\" -spam -\"bad phrase\" domain:example.com")
assertEquals("best \"nostr apps\" -spam -\"bad phrase\" domain:example.com", q.toSearchString())
}
@Test
fun allDashTokensAreDroppedNotEmptied() {
// "--" strips to nothing; surfacing an empty exclusion would
// round-trip into a required "-" term.
val q = SearchQuery.parse("a --")
assertEquals("a", q.terms)
assertTrue(q.notTerms.isEmpty())
assertEquals("a", SearchQuery.parse(q.toSearchString()).terms)
assertEquals("", SearchQuery.stripExtensions("-- include:spam"))
}
@Test
fun consecutiveSpansEachLift() {
val q = SearchQuery.parse("\"a\"\"b\" -\"c\"")
assertEquals(listOf("a", "b"), q.phrases)
assertEquals(listOf("c"), q.notPhrases)
assertEquals("", q.terms)
}
@Test
fun textAfterClosingQuoteIsItsOwnTerm() {
// The lifted span's place stays a token boundary for what follows.
val q = SearchQuery.parse("\"a b\"c")
assertEquals(listOf("a b"), q.phrases)
assertEquals("c", q.terms)
}
@Test
fun stripExtensionsKeepsPhrasesAndExclusions() {
assertEquals(
"best \"nostr apps\" -spam",
SearchQuery.stripExtensions("best \"nostr apps\" -spam language:en"),
)
// No extensions -> the original string comes back untouched.
assertEquals("best \"nostr apps\" -spam", SearchQuery.stripExtensions("best \"nostr apps\" -spam"))
}
}
@@ -0,0 +1,176 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
/**
* An outbox list is five figures of urls and mostly corpses. These pin the two
* rules that make the difference: failures count per HOST (or a filtering
* relay's hundreds of per-user urls never add up to anything), and a host that
* has ever delivered is never dropped (or a fan-out this wide sheds working
* relays on a race).
*/
class HostStrikesTest {
private fun url(u: String) = RelayUrlNormalizer.normalize(u)
// ---- the authority key --------------------------------------------------
@Test
fun `per-user path urls on one host share an authority`() {
val a = HostStrikes.authorityOf("wss://filter.nostr.wine/npub1aaaa?broadcast=true")
val b = HostStrikes.authorityOf("wss://filter.nostr.wine/npub1bbbb")
assertEquals("filter.nostr.wine", a)
assertEquals(a, b)
assertEquals(a, HostStrikes.authorityOf("wss://filter.nostr.wine"))
}
@Test
fun `a subdomain is not folded into its parent`() {
// Different servers. Shedding the filtering one must never take out the
// bare host, which may be perfectly open.
assertTrue(
HostStrikes.authorityOf("wss://filter.nostr.wine/npub1x") !=
HostStrikes.authorityOf("wss://nostr.wine"),
)
}
@Test
fun `the port is part of the authority`() {
assertEquals("relay.example.com:443", HostStrikes.authorityOf("wss://relay.example.com:443/npub1z"))
assertTrue(
HostStrikes.authorityOf("wss://example.com:443") != HostStrikes.authorityOf("wss://example.com:8080"),
)
}
// ---- striking -----------------------------------------------------------
@Test
fun `one host is struck out by failures spread across its many urls`() {
// The whole point: no single url would ever reach the threshold alone.
val h = HostStrikes()
h.strike(url("wss://filter.example/npub1aaa"))
h.strike(url("wss://filter.example/npub1bbb"))
assertFalse(h.isDead(url("wss://filter.example/npub1ccc")), "two strikes is not yet a verdict")
h.strike(url("wss://filter.example/npub1ccc"))
assertTrue(h.isDead(url("wss://filter.example/npub1ddd")), "the host is out — including urls never tried")
}
@Test
fun `eviction returns a verdict exactly once for publishing`() {
// The eviction is the only finding that will ever exist about the sibling
// urls under this host: from here they are skipped without being dialled,
// so nothing observes them again. It must surface exactly once — silent
// would publish nothing, repeated would rewrite the record every strike.
val h = HostStrikes()
assertNull(h.strike(url("wss://filter.example/npub1")), "one strike is not a verdict")
assertNull(h.strike(url("wss://filter.example/npub2")), "two is not either")
val evicted = h.strike(url("wss://filter.example/npub3"))
assertNotNull(evicted, "the third strike is the finding")
assertEquals("filter.example", evicted.authority)
assertEquals(3, evicted.strikes)
assertNull(h.strike(url("wss://filter.example/npub4")), "already evicted — do not report it again")
}
@Test
fun `a host that has delivered is never evicted so nothing is published`() {
val h = HostStrikes()
h.produced(url("wss://busy.example/npubY"))
repeat(5) { assertNull(h.strike(url("wss://busy.example/npub$it")), "ever-produced outranks any strike") }
}
@Test
fun `striking one host leaves every other alone`() {
val h = HostStrikes()
repeat(5) { h.strike(url("wss://filter.example/npub$it")) }
assertTrue(h.isDead(url("wss://filter.example/npub1")))
assertFalse(h.isDead(url("wss://example.com")))
assertFalse(h.isDead(url("wss://other.example")))
}
@Test
fun `a host that ever delivered is never dead whichever way the race lands`() {
// At a hundred relays in flight one worker can strike an authority out at
// the same instant another is receiving from it. Ever-produced must win in
// both orders, which is why it overrides rather than clearing strikes.
val strikeFirst = HostStrikes()
repeat(3) { strikeFirst.strike(url("wss://busy.example/npub$it")) }
assertTrue(strikeFirst.isDead(url("wss://busy.example/npubX")))
strikeFirst.produced(url("wss://busy.example/npubY"))
assertFalse(strikeFirst.isDead(url("wss://busy.example/npubX")), "a delivery revives the whole host")
val produceFirst = HostStrikes()
produceFirst.produced(url("wss://busy.example/npubY"))
repeat(5) { produceFirst.strike(url("wss://busy.example/npub$it")) }
assertFalse(produceFirst.isDead(url("wss://busy.example/npubX")), "later strikes cannot bury it")
}
@Test
fun `a zero strike limit disables eviction entirely`() {
val h = HostStrikes(strikeLimit = 0)
repeat(50) { h.strike(url("wss://filter.example/npub$it")) }
assertFalse(h.isDead(url("wss://filter.example/npub1")))
}
// ---- what a previous run already learned ---------------------------------
@Test
fun `a relay a previous run proved dead is skipped without dialling`() {
val gone = url("wss://gone.example")
val h = HostStrikes(knownDead = setOf(gone))
assertTrue(h.isDead(gone))
assertFalse(h.isDead(url("wss://alive.example")))
}
@Test
fun `a known-dead relay that answers anyway is believed over the record`() {
// Relays come back. A TTL'd record is "not now" and never "never again":
// a delivery this cycle must beat what an earlier one wrote down.
val back = url("wss://back.example")
val h = HostStrikes(knownDead = setOf(back))
h.produced(back)
assertFalse(h.isDead(back))
}
// ---- what gets written back ---------------------------------------------
@Test
fun `only relays actually dialled are reported and delivery clears a failure`() {
val h = HostStrikes()
val good = url("wss://good.example")
val bad = url("wss://bad.example")
h.strike(bad)
h.strike(good)
h.produced(good)
assertEquals(setOf(good), h.reachable)
assertEquals(setOf(bad), h.unreachable, "a relay that later delivered is not reported dead")
}
}
@@ -0,0 +1,465 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
import com.vitorpamplona.quartz.nip01Core.relay.client.EmptyNostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.OkMessage
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.Command
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerInternal
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.test.currentTime
import kotlinx.coroutines.test.runTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNull
import kotlin.test.assertTrue
/**
* Pins [RelayProber.probeFlow]'s streaming contract: an answering relay's verdict
* is emitted the moment its terminal arrives, while silent relays only resolve at
* the wave deadline — and pins the observed-facts-only tag set of
* [toDiscoveryEventTemplate].
*/
@OptIn(ExperimentalCoroutinesApi::class)
class RelayProberFlowTest {
/** Captures the probe subscription and publish so the test can play the relays. */
private class ScriptedClient : INostrClient by EmptyNostrClient() {
var listener: SubscriptionListener? = null
var sentFilters: Map<NormalizedRelayUrl, List<Filter>>? = null
var published: Event? = null
val connListeners = mutableListOf<RelayConnectionListener>()
override fun subscribe(
subId: String,
filters: Map<NormalizedRelayUrl, List<Filter>>,
listener: SubscriptionListener?,
) {
this.listener = listener
this.sentFilters = filters
}
override fun publish(
event: Event,
relayList: Set<NormalizedRelayUrl>,
) {
published = event
}
override fun addConnectionListener(listener: RelayConnectionListener) {
connListeners += listener
}
override fun removeConnectionListener(listener: RelayConnectionListener) {
connListeners -= listener
}
/** Plays a relay's OK answer for the published event to every armed listener. */
fun answerOk(
relay: NormalizedRelayUrl,
success: Boolean,
message: String,
) {
val ok = OkMessage(published!!.id, success, message)
connListeners.toList().forEach { it.onIncomingMessage(FakeRelayClient(relay), "", ok) }
}
}
private class FakeRelayClient(
override val url: NormalizedRelayUrl,
) : IRelayClient {
override fun connect() = Unit
override fun needsToReconnect() = false
override fun connectAndSyncFiltersIfDisconnected(ignoreRetryDelays: Boolean) = Unit
override fun isConnected() = true
override fun sendOrConnectAndSync(cmd: Command) = Unit
override fun sendIfConnected(cmd: Command) = Unit
override fun disconnect() = Unit
}
private val fast = RelayUrlNormalizer.normalize("wss://fast.example.com")
private val silent = RelayUrlNormalizer.normalize("wss://silent.example.com")
private val walled = RelayUrlNormalizer.normalize("wss://walled.example.com")
@Test
fun answeringRelayStreamsBeforeTheWaveDeadline() =
runTest {
val client = ScriptedClient()
val arrivals = mutableListOf<Pair<RelayProber.Verdict, Long>>()
val collector =
launch {
RelayProber(client)
.probeFlow(listOf(fast, silent), timeoutMs = 10_000)
.collect { arrivals += it to currentTime }
}
launch {
delay(200)
client.listener!!.onEose(fast, null)
}
collector.join()
assertEquals(listOf(fast, silent), arrivals.map { it.first.relay })
// The EOSE'd relay resolved when it answered, not at the deadline …
val (fastVerdict, fastAt) = arrivals[0]
assertTrue(fastVerdict.reachable)
assertTrue(fastAt < 1_000, "verdict should stream at answer time, arrived at ${fastAt}ms")
// … while the silent one waited out the wave and is dead (socket never opened).
val (silentVerdict, silentAt) = arrivals[1]
assertFalse(silentVerdict.reachable)
assertTrue(silentAt >= 10_000, "silent relay must wait for the deadline, arrived at ${silentAt}ms")
}
@Test
fun authWalledRelayIsReachableWithTheWallRecorded() =
runTest {
val client = ScriptedClient()
val arrivals = mutableListOf<RelayProber.Verdict>()
val collector =
launch {
RelayProber(client)
.probeFlow(listOf(walled), timeoutMs = 10_000)
.collect { arrivals += it }
}
launch {
delay(100)
client.listener!!.onClosed("auth-required: sign in first", walled, null)
}
collector.join()
assertEquals(1, arrivals.size)
assertTrue(arrivals[0].reachable, "an auth wall is an app-level answer: the relay works")
assertEquals("closed:auth-required: sign in first", arrivals[0].error)
}
@Test
fun connectFailureStreamsImmediatelyAsDead() =
runTest {
val client = ScriptedClient()
val arrivals = mutableListOf<Pair<RelayProber.Verdict, Long>>()
val collector =
launch {
RelayProber(client)
.probeFlow(listOf(fast), timeoutMs = 10_000)
.collect { arrivals += it to currentTime }
}
launch {
delay(50)
client.listener!!.onCannotConnect(fast, "dns failure", null)
}
collector.join()
val (verdict, at) = arrivals.single()
assertFalse(verdict.reachable)
assertEquals("cannot:dns failure", verdict.error)
assertTrue(at < 1_000, "a failed dial must not wait for the deadline, arrived at ${at}ms")
}
// ------------------------------------------------------------------
// Check options — liveness default, read-test override, write-test event
// ------------------------------------------------------------------
@Test
fun livenessFilterIsTheDefaultCheck() =
runTest {
val client = ScriptedClient()
val collector =
launch {
RelayProber(client).probeFlow(listOf(fast), timeoutMs = 1_000).collect {}
}
launch {
delay(10)
assertEquals(RelayProber.LIVENESS_FILTERS, client.sentFilters!![fast])
client.listener!!.onEose(fast, null)
}
collector.join()
}
@Test
fun readTestFilterIsSentWhenChosen() =
runTest {
val client = ScriptedClient()
val collector =
launch {
RelayProber(client)
.probeFlow(listOf(fast), timeoutMs = 1_000, filters = RelayProber.readTestFilter())
.collect {}
}
launch {
delay(10)
val sent = client.sentFilters!![fast]!!.single()
assertEquals(1, sent.limit, "read test defaults to limit 1")
assertEquals(listOf(0), sent.kinds, "read test defaults to kind 0 — accepted by purpose relays too")
assertNull(sent.ids, "read test must query real events, not the impossible id")
client.listener!!.onEose(fast, null)
}
collector.join()
}
@Test
fun writeTestEventIsEphemeralAndSelfExpiring() {
val template = RelayProbeWriteTest.build(createdAt = 5000)
assertEquals(20166, template.kind)
assertTrue(template.kind in 20000..29999, "the write probe must be an ephemeral kind")
assertTrue(listOf("expiration", "5060") in template.tags.map { it.toList() })
}
// ------------------------------------------------------------------
// readWriteCheck — honest read + write measurements, nothing claimed
// ------------------------------------------------------------------
private suspend fun ScriptedClient.playReadThenWrite(
relay: NormalizedRelayUrl,
ok: Boolean?,
okMessage: String = "",
) {
delay(50)
listener!!.onEose(relay, null) // read phase answers
while (published == null) delay(10) // write phase begins
if (ok != null) answerOk(relay, ok, okMessage)
}
@Test
fun readWriteCheckMeasuresBothSides() =
runTest {
val client = ScriptedClient()
val signer = NostrSignerInternal(KeyPair())
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
val check =
launch {
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 5_000)
}
launch { client.playReadThenWrite(fast, ok = true) }
check.join()
val verdict = result!![fast]!!
assertTrue(verdict.rttReadMs >= 0, "an answered read must be measured")
assertTrue(verdict.rttWriteMs >= 0, "an answered write must be measured")
assertEquals(true, verdict.writeAccepted)
assertEquals(20166, client.published!!.kind, "the write test must use the ephemeral probe event")
}
@Test
fun writeRejectionIsAnAnswerNotAFailure() =
runTest {
val client = ScriptedClient()
val signer = NostrSignerInternal(KeyPair())
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
val check =
launch {
result = RelayProber(client).readWriteCheck(listOf(walled), signer, timeoutMs = 5_000)
}
launch { client.playReadThenWrite(walled, ok = false, okMessage = "pow: 28 bits needed") }
check.join()
val verdict = result!![walled]!!
assertEquals(false, verdict.writeAccepted, "OK false is a measured policy answer")
assertEquals("pow: 28 bits needed", verdict.writeMessage)
assertTrue(verdict.rttWriteMs >= 0, "a rejection is still a round trip")
}
@Test
fun foreignRelayOkDoesNotEndTheWriteConfirmationEarly() =
runTest {
// A relay OUTSIDE the checked set answering with the same event id (a
// straggler from an earlier wave that got the same probe event) must not
// count toward the confirmation window — before the relayList guard in
// publishAndCollectResults, it ended the wait early and misreported the
// real relay as silent.
val client = ScriptedClient()
val signer = NostrSignerInternal(KeyPair())
val foreign = RelayUrlNormalizer.normalize("wss://foreign.example.com")
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
val check =
launch {
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 5_000)
}
launch {
delay(50)
client.listener!!.onEose(fast, null)
while (client.published == null) delay(10)
client.answerOk(foreign, true, "")
delay(100)
client.answerOk(fast, true, "")
}
check.join()
val verdict = result!![fast]!!
assertEquals(true, verdict.writeAccepted, "the listed relay's OK must still be awaited and recorded")
assertNull(result!![foreign], "the foreign relay must not appear in the result")
}
@Test
fun silentWriteLeavesTheWriteSideUnobserved() =
runTest {
val client = ScriptedClient()
val signer = NostrSignerInternal(KeyPair())
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
val check =
launch {
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 2_000)
}
launch { client.playReadThenWrite(fast, ok = null) }
check.join()
val verdict = result!![fast]!!
assertTrue(verdict.rttReadMs >= 0)
assertNull(verdict.writeAccepted, "silence is not evidence about the write path")
assertEquals(-1, verdict.rttWriteMs)
}
// ------------------------------------------------------------------
// toDiscoveryEventTemplate — only observed facts become tags
// ------------------------------------------------------------------
private fun tagsOf(template: EventTemplate<*>) = template.tags.map { it.toList() }
@Test
fun reachableVerdictTemplateCarriesLivenessAndNetwork() {
val template =
RelayProber
.Verdict(fast, reachable = true, rttOpenMs = 150, rttEoseMs = 480, error = null)
.toDiscoveryEventTemplate(createdAt = 1000)
val tags = tagsOf(template)
assertEquals(30166, template.kind)
assertEquals(1000, template.createdAt)
assertTrue(listOf("d", fast.url) in tags)
assertTrue(listOf("n", "clearnet") in tags)
assertTrue(listOf("rtt-open", "150") in tags)
// rtt-eose is wave-relative (dial + queue + read) — never published as rtt-read.
assertNull(tags.firstOrNull { it[0] == "rtt-read" })
}
@Test
fun deadVerdictTemplateHasNoRttOpen() {
val template =
RelayProber
.Verdict(silent, reachable = false, rttOpenMs = -1, rttEoseMs = -1, error = "cannot:timeout")
.toDiscoveryEventTemplate()
val tags = tagsOf(template)
assertTrue(listOf("d", silent.url) in tags)
// Liveness is the PRESENCE of rtt-open; a dead record must not carry one.
assertNull(tags.firstOrNull { it[0] == "rtt-open" })
}
@Test
fun reachableWithoutMeasuredLatencyWritesZeroFlag() {
val template =
RelayProber
.Verdict(fast, reachable = true, rttOpenMs = -1, rttEoseMs = 300, error = null)
.toDiscoveryEventTemplate()
// 0 = "reachable, latency not observed": the flag form, never an invented number.
assertTrue(listOf("rtt-open", "0") in tagsOf(template))
}
@Test
fun observedAuthWallBecomesARequirementTag() {
val template =
RelayProber
.Verdict(walled, reachable = true, rttOpenMs = 90, rttEoseMs = -1, error = "closed:auth-required: sign in")
.toDiscoveryEventTemplate()
assertTrue(listOf("R", "auth") in tagsOf(template))
}
@Test
fun policyClosedIsNotAnAuthRequirement() {
val template =
RelayProber
.Verdict(walled, reachable = true, rttOpenMs = 90, rttEoseMs = -1, error = "closed:blocked: not welcome")
.toDiscoveryEventTemplate()
assertNull(tagsOf(template).firstOrNull { it[0] == "R" })
}
@Test
fun readWriteResultsBecomeRttTags() {
val verdict = RelayProber.Verdict(fast, reachable = true, rttOpenMs = 100, rttEoseMs = 300, error = null)
val readWrite = RelayProber.ReadWriteVerdict(fast, rttReadMs = 40, rttWriteMs = 55, writeAccepted = true, writeMessage = "")
val tags = tagsOf(verdict.toDiscoveryEventTemplate(readWrite = readWrite))
assertTrue(listOf("rtt-read", "40") in tags)
assertTrue(listOf("rtt-write", "55") in tags)
}
@Test
fun unobservedReadWriteSidesStayUntagged() {
val verdict = RelayProber.Verdict(fast, reachable = true, rttOpenMs = 100, rttEoseMs = -1, error = null)
val readWrite = RelayProber.ReadWriteVerdict(fast, rttReadMs = -1, rttWriteMs = -1, writeAccepted = null, writeMessage = null)
val tags = tagsOf(verdict.toDiscoveryEventTemplate(readWrite = readWrite))
assertNull(tags.firstOrNull { it[0] == "rtt-read" })
assertNull(tags.firstOrNull { it[0] == "rtt-write" })
}
@Test
fun writeRejectionReasonsBecomeRequirementTags() {
val verdict = RelayProber.Verdict(walled, reachable = true, rttOpenMs = 100, rttEoseMs = -1, error = null)
val pow = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "pow: 28 bits needed")
assertTrue(listOf("R", "pow") in tagsOf(verdict.toDiscoveryEventTemplate(readWrite = pow)))
val auth = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "auth-required: sign in")
assertTrue(listOf("R", "auth") in tagsOf(verdict.toDiscoveryEventTemplate(readWrite = auth)))
val blocked = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "blocked: not welcome")
assertNull(tagsOf(verdict.toDiscoveryEventTemplate(readWrite = blocked)).firstOrNull { it[0] == "R" })
}
@Test
fun onionRelayIsTaggedTor() {
val onion = RelayUrlNormalizer.normalize("ws://someonionaddressabcdefghijklmnop.onion")
val template =
RelayProber
.Verdict(onion, reachable = true, rttOpenMs = 900, rttEoseMs = -1, error = null)
.toDiscoveryEventTemplate()
assertTrue(listOf("n", "tor") in tagsOf(template))
}
}
@@ -0,0 +1,139 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNull
import kotlin.test.assertTrue
class Ipv6Test {
private fun canonical(address: String) = Ipv6.canonicalizeOrNull(address)
@Test
fun rfc5952CanonicalForm() {
// leading zeros suppressed, hex lowercased
assertEquals("201:d0e:9ba5:8bbc::1", canonical("201:0d0e:9ba5:8bbc:0000:0000:0000:0001"))
assertEquals("201:d0e:9ba5:8bbc::1", canonical("201:D0E:9BA5:8BBC::1"))
assertEquals("2001:db8::1", canonical("2001:0DB8:0000:0000:0000:0000:0000:0001"))
// already canonical stays put
assertEquals("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5", canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5"))
assertEquals("::", canonical("::"))
assertEquals("::1", canonical("0:0:0:0:0:0:0:1"))
}
@Test
fun singleZeroGroupIsNotCompressed() {
// RFC 5952 §4.2.2: `::` must not stand for a single group.
assertEquals("2001:db8:0:1:1:1:1:1", canonical("2001:db8:0:1:1:1:1:1"))
}
@Test
fun longestZeroRunWinsAndTiesGoLeft() {
assertEquals("2001:0:0:1::1", canonical("2001:0:0:1:0:0:0:1"))
// equal runs of two groups: the leftmost is the one compressed
assertEquals("2001::1:1:0:0:1", canonical("2001:0:0:1:1:0:0:1"))
}
@Test
fun ipv4MappedKeepsDottedTail() {
assertEquals("::ffff:192.168.1.1", canonical("::ffff:192.168.1.1"))
assertEquals("::ffff:127.0.0.1", canonical("::FFFF:127.0.0.1"))
// an embedded quad that is not ipv4-mapped collapses to plain hex
assertEquals("::c0a8:101", canonical("::192.168.1.1"))
}
@Test
fun zoneIdIsPreservedVerbatim() {
// In URLs the zone arrives percent-encoded.
assertEquals("fe80::1%25wlan0", canonical("fe80:0000:0000:0000:0000:0000:0000:0001%25wlan0"))
}
@Test
fun rejectsMalformedLiterals() {
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2")) // too few groups
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5:1234")) // too many
assertNull(canonical("201::9ba5::1")) // two `::`
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5:")) // trailing colon
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:gggg")) // non-hex
assertNull(canonical("201:00d0e:9ba5:8bbc::1")) // five-digit group
assertNull(canonical("192.168.1.1")) // ipv4
assertNull(canonical("localhost"))
assertNull(canonical("::ffff:192.168.1")) // short quad
assertNull(canonical("::ffff:010.1.1.1")) // leading zero in quad
assertNull(canonical("0:0:0:0:0:0:0:0:0"))
}
@Test
fun compressionMustCoverAtLeastOneGroup() {
// A `::` that stands for nothing is not a legal literal.
assertNull(canonical("1:2:3:4:5:6:7::8"))
}
@Test
fun classifiesYggdrasilAndPrivateRanges() {
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("201:d0e:9ba5:8bbc::1")!!), "0200::/8 node address")
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("300:1b5d:d0e9:ba58::1")!!), "0300::/8 subnet address")
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("2ff::1")!!))
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("2001:db8::1")!!), "documentation range is clearnet")
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("400::1")!!), "just past 0200::/7")
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("::1")!!))
assertTrue(Ipv6.isLoopback(Ipv6.parse("::1")!!))
assertFalse(Ipv6.isLoopback(Ipv6.parse("::2")!!))
assertFalse(Ipv6.isLoopback(Ipv6.parse("::")!!))
assertTrue(Ipv6.isLinkLocal(Ipv6.parse("fe80::1")!!))
assertTrue(Ipv6.isLinkLocal(Ipv6.parse("febf::1")!!))
assertFalse(Ipv6.isLinkLocal(Ipv6.parse("fec0::1")!!))
assertTrue(Ipv6.isUniqueLocal(Ipv6.parse("fd00::1")!!))
assertTrue(Ipv6.isUniqueLocal(Ipv6.parse("fc00::1")!!))
assertFalse(Ipv6.isUniqueLocal(Ipv6.parse("fe00::1")!!))
}
@Test
fun isLiteralDiscriminatesAgainstNonAddresses() {
assertTrue(Ipv6.isLiteral("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5"))
assertTrue(Ipv6.isLiteral("201:d0e:9ba5:8bbc::1"))
// Things a relay-url field realistically receives, none of which may pass as an address.
assertFalse(Ipv6.isLiteral("relay.example.com:8080"))
assertFalse(Ipv6.isLiteral("wss:"))
assertFalse(Ipv6.isLiteral("localhost:4869"))
assertFalse(Ipv6.isLiteral("31990:abcdef:mydtag"), "addressable event pointer")
assertFalse(Ipv6.isLiteral("abcd:1234"))
assertFalse(Ipv6.isLiteral("nos.lol"))
}
@Test
fun roundTripsEveryFormOfTheSameAddress() {
val forms =
listOf(
"201:d0e:9ba5:8bbc:0:0:0:1",
"201:0d0e:9ba5:8bbc:0000:0000:0000:0001",
"201:d0e:9ba5:8bbc::1",
"201:D0E:9BA5:8BBC::0001",
)
val canonicalForms = forms.map { canonical(it) }.toSet()
assertEquals(setOf("201:d0e:9ba5:8bbc::1"), canonicalForms)
}
}
@@ -71,6 +71,40 @@ class ConcurrentCollectionsTest {
assertEquals(4, m["k"])
}
@Test
fun mapRemoveReturnsOldValueAndDeletes() {
val map = ConcurrentMap<String, Int>()
map["a"] = 1
map["b"] = 2
assertEquals(1, map.remove("a"))
assertNull(map["a"])
assertEquals(1, map.size())
assertEquals(2, map["b"])
}
@Test
fun mapRemoveAbsentKeyIsNullAndNoOp() {
val map = ConcurrentMap<String, Int>()
map["b"] = 2
assertNull(map.remove("missing"))
assertEquals(1, map.size())
assertEquals(2, map["b"])
}
@Test
fun mapRemoveThenGetOrPutRecreates() {
// The lifecycle PoolRequests needs: connecting()/disconnected() drop a relay's
// wire state, and the next REQ re-creates it. A stale value must never survive.
val map = ConcurrentMap<String, Int>()
map.getOrPut("relay") { 1 }
map.remove("relay")
assertEquals(9, map.getOrPut("relay") { 9 })
assertEquals(9, map["relay"])
}
@Test
fun mapSnapshotIsDetached() {
val m = ConcurrentMap<String, Int>()
@@ -0,0 +1,60 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip11RelayInfo
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.coroutines.executeAsync
/**
* OkHttp-backed [Nip11Fetcher]: GETs the relay's https url with
* `Accept: application/nostr+json` and parses the document. The client is
* resolved per relay so callers can route Tor/proxy relays through a different
* OkHttp instance (the same seam Amethyst's Nip11Retriever uses).
*/
class OkHttpNip11Fetcher(
private val okHttpClient: (NormalizedRelayUrl) -> OkHttpClient,
) : Nip11Fetcher {
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation =
withContext(Dispatchers.IO) {
val request =
Request
.Builder()
.header("Accept", "application/nostr+json")
.url(relay.toHttp())
.build()
okHttpClient(relay).newCall(request).executeAsync().use { response ->
if (!response.isSuccessful) {
throw Nip11FetchException("HTTP ${response.code} fetching NIP-11 from ${relay.url}")
}
val body = response.body.string()
if (!body.startsWith("{")) {
throw Nip11FetchException("Not a NIP-11 document from ${relay.url}")
}
Nip11RelayInformation.fromJson(body)
}
}
}
@@ -0,0 +1,48 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
/**
* Whether a failure may be published as "this relay is unreachable".
*
* The distinction matters because the answer is PUBLISHED: a negative NIP-66
* record is a signed, public statement about someone else's server. A relay
* that completes a handshake and then hangs up mid-page is emphatically
* reachable, and an exception thrown by the caller's own code says nothing
* about the relay at all. So this asks only about the connection itself —
* name resolution, routing, refusal, TLS.
*
* Unknown failures stay quiet: the cost of silence is one retry next cycle,
* the cost of being wrong is a false record carrying the monitor's signature.
*/
object Unreachability {
fun proves(e: Exception): Boolean =
when (e) {
is java.net.UnknownHostException,
is java.net.ConnectException,
is java.net.NoRouteToHostException,
is java.net.PortUnreachableException,
is javax.net.ssl.SSLHandshakeException,
-> true
else -> false
}
}
@@ -49,6 +49,8 @@ actual class ConcurrentMap<K : Any, V : Any> {
remap: (old: V, new: V) -> V,
): V = map.merge(key, value) { old, new -> remap(old, new) }!!
actual fun remove(key: K): V? = map.remove(key)
actual fun size(): Int = map.size
actual fun snapshot(): Map<K, V> = HashMap(map)
@@ -0,0 +1,35 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils.concurrent
import java.util.concurrent.locks.ReentrantLock
// ReentrantLock parks contended waiters through AbstractQueuedSynchronizer, so a
// thread waiting on a holder that lost its core (GC, preemption) costs one context
// switch rather than a spinning core. Non-fair on purpose: fairness would add a
// handoff per acquisition and the critical sections here are microseconds long.
actual class PlatformLock {
private val lock = ReentrantLock()
actual fun lock() = lock.lock()
actual fun unlock() = lock.unlock()
}
@@ -0,0 +1,156 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isLocalHost
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOverlayNetwork
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrl
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
import okhttp3.Request
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertNotNull
import kotlin.test.assertTrue
/**
* Relay URLs on an Yggdrasil overlay, end to end through the normalizer.
*
* Yggdrasil gives every node an IPv6 address inside `0200::/7` (nodes in `0200::/8`, subnets in
* `0300::/8`), with no DNS and no CA-issuable certificate. A relay on the mesh is therefore
* always a **bracketed IPv6 literal over plain `ws://`**.
*
* The differential assertions against OkHttp are the point of this file living in
* `jvmAndroidTest`: OkHttp is what actually dials the socket, so a normalized url that
* disagrees with OkHttp's own canonical host is a relay the app tracks under a name it does
* not connect to.
*/
class YggdrasilCompatCharacterizationTest {
// Same node, several legal RFC 4291 spellings of one address.
private val canonical = "ws://[201:d0e:9ba5:8bbc::1]:8080"
private val expanded = "ws://[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]:8080"
private val uppercase = "ws://[201:D0E:9BA5:8BBC::1]:8080"
private fun host(url: String) =
Request
.Builder()
.url(url)
.build()
.url.host
@Test
fun bracketedLiteralsSurviveNormalizationAndReachOkHttp() {
val n = canonical.normalizeRelayUrl()
assertEquals("ws://[201:d0e:9ba5:8bbc::1]:8080/", n.url)
assertEquals("201:d0e:9ba5:8bbc::1", host(n.url))
// NIP-11 / relay-icon fetches derive their http url from the same string.
assertEquals("http://[201:d0e:9ba5:8bbc::1]:8080/", n.toHttp())
}
@Test
fun yggdrasilSubnetAddressesWork() {
assertEquals("ws://[300:1b5d:d0e9:ba58::1]:4848/", "ws://[300:1b5d:d0e9:ba58::1]:4848".normalizeRelayUrl().url)
}
/**
* Every legal spelling of one address collapses to one [NormalizedRelayUrl] — the key of the
* connection pool, the relay-list sets, the NIP-11 cache and the per-relay stat maps. Without
* this the app dials one relay twice and shows it twice.
*/
@Test
fun everySpellingOfOneAddressIsOneRelay() {
val identities = listOf(canonical, expanded, uppercase).map { it.normalizeRelayUrl() }.toSet()
assertEquals(setOf("ws://[201:d0e:9ba5:8bbc::1]:8080/"), identities.map { it.url }.toSet())
// ...and that one identity is the host OkHttp dials for all of them.
assertEquals(setOf("201:d0e:9ba5:8bbc::1"), listOf(canonical, expanded, uppercase).map { host(it) }.toSet())
}
@Test
fun normalizedIdentityAlwaysMatchesTheHostOkHttpDials() {
listOf(
"ws://[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]:8080",
"ws://[300:1b5d:d0e9:ba58:0:0:0:1]:4848",
"ws://[2001:0DB8:0000:0000:0000:0000:0000:0001]:7777",
"ws://[::1]:4869",
).forEach { raw ->
val normalized = raw.normalizeRelayUrl().url
assertEquals(host(normalized), host(raw), "identity for $raw disagrees with the dialed host")
}
}
/**
* A schemeless overlay address defaults to `ws://`: no CA issues certificates for
* `0200::/7`, so `wss://` could only ever fail its handshake. The mesh already encrypts
* end to end, so this is not a downgrade.
*/
@Test
fun schemelessOverlayAddressDefaultsToWs() {
assertEquals("ws://[201:d0e:9ba5:8bbc::1]:8080/", "[201:d0e:9ba5:8bbc::1]:8080".normalizeRelayUrl().url)
assertTrue("ws://[201:d0e:9ba5:8bbc::1]:8080/".normalizeRelayUrl().isOverlayNetwork())
// A clearnet IPv6 relay keeps requiring TLS.
assertEquals("wss://[2001:db8::1]:8080/", "[2001:db8::1]:8080".normalizeRelayUrl().url)
assertFalse("wss://[2001:db8::1]:8080/".normalizeRelayUrl().isOverlayNetwork())
}
/**
* `::1`, `fc00::/7` and `fe80::/10` are the IPv6 twins of 127.0.0.1 and 192.168., so they
* answer [isLocalHost] the same way — no TLS, no Tor, never advertised to the network.
*/
@Test
fun ipv6LoopbackAndPrivateRangesCountAsLocalHost() {
assertEquals("ws://[::1]:4869/", "[::1]:4869".normalizeRelayUrl().url)
assertTrue("ws://[::1]:4869/".normalizeRelayUrl().isLocalHost())
assertTrue("ws://[fd12:3456::1]:8080/".normalizeRelayUrl().isLocalHost(), "unique local address")
assertTrue("ws://[fe80::1]:8080/".normalizeRelayUrl().isLocalHost(), "link local address")
assertFalse("wss://[2001:db8::1]:8080/".normalizeRelayUrl().isLocalHost(), "clearnet ipv6")
// An overlay relay is reachable across the mesh, so it is NOT localhost.
assertFalse("ws://[201:d0e:9ba5:8bbc::1]:8080/".normalizeRelayUrl().isLocalHost())
}
/**
* `yggdrasilctl getSelf` prints the address unbracketed, which is what a user pastes into
* the "add a relay" field. It is bracketed automatically rather than rejected.
*/
@Test
fun bareUnbracketedLiteralIsBracketedAutomatically() {
assertEquals(
"ws://[201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5]/",
"201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5".normalizeRelayUrl().url,
)
assertEquals("ws://[201:d0e:9ba5:8bbc::1]/", "201:d0e:9ba5:8bbc::1".normalizeRelayUrl().url)
assertNotNull(RelayUrlNormalizer.normalizeOrNull("[201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5]:8080"))
}
/**
* Auto-bracketing must not swallow the other colon-bearing strings that reach the
* normalizer. Only a string that parses as a whole IPv6 address is bracketed.
*/
@Test
fun autoBracketingDoesNotCaptureNonAddresses() {
assertEquals("wss://relay.example.com:8080/", "relay.example.com:8080".normalizeRelayUrl().url)
assertEquals("ws://localhost:4869/", "localhost:4869".normalizeRelayUrl().url)
// addressable-event pointer, not a relay
assertEquals(null, RelayUrlNormalizer.normalizeOrNull("31990:abcdef:mydtag"))
// two hex-looking groups are a host and a port, not an address
assertEquals("wss://abcd:1234/", "abcd:1234".normalizeRelayUrl().url)
}
}
@@ -0,0 +1,108 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils
import okhttp3.HttpUrl.Companion.toHttpUrl
import java.net.InetAddress
import kotlin.random.Random
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Differential tests for [Ipv6] against the two parsers that actually matter at runtime: the
* JDK's (what `InetAddress` will do with the host) and OkHttp's (what dials the socket).
*
* A hand-written address parser is exactly the kind of code that passes its own examples and
* then disagrees with the real world on the hundredth input, so this pins it against
* references over a deterministic random corpus rather than against more of my own examples.
*/
class Ipv6DifferentialTest {
/**
* Parses through the JDK. IPv4-mapped literals come back as an `Inet4Address` of 4 bytes,
* so they are widened back to the 16-byte mapped form — [Ipv6] keeps them at 16 bytes,
* which is also what OkHttp does.
*/
private fun jdkBytes(literal: String): ByteArray {
val raw = InetAddress.getByName("[$literal]").address
if (raw.size == 16) return raw
return ByteArray(16).also {
it[10] = 0xFF.toByte()
it[11] = 0xFF.toByte()
raw.copyInto(it, 12)
}
}
private fun randomAddresses(count: Int): List<ByteArray> {
val rnd = Random(20260805)
return List(count) {
ByteArray(16) { rnd.nextInt(256).toByte() }.also { bytes ->
// Sprinkle zero runs so every `::` compression path gets exercised.
val runStart = rnd.nextInt(8) * 2
val runLen = rnd.nextInt(1, 5) * 2
for (k in runStart until minOf(16, runStart + runLen)) bytes[k] = 0
}
}
}
@Test
fun ourTextParsesToTheSameBytesInTheJdk() {
randomAddresses(4000).forEach { bytes ->
val text = Ipv6.format(bytes)
assertTrue(Ipv6.parse(text)!!.contentEquals(bytes), "our own round trip failed for $text")
assertTrue(jdkBytes(text).contentEquals(bytes), "the JDK reads $text as a different address")
}
}
@Test
fun theJdksTextParsesBackThroughUs() {
randomAddresses(2000).forEach { bytes ->
val jdkText = InetAddress.getByAddress(bytes).hostAddress!!
assertTrue(Ipv6.parse(jdkText)?.contentEquals(bytes) == true, "we cannot read the JDK's own rendering: $jdkText")
}
}
/**
* The canonical form is the app's relay identity, so it has to equal the host OkHttp shows
* for the same address — otherwise the app keys a relay under a name it does not dial.
*/
@Test
fun ourCanonicalFormMatchesOkHttp() {
randomAddresses(2000).forEach { bytes ->
val text = Ipv6.format(bytes)
assertEquals("http://[$text]/".toHttpUrl().host, text)
}
}
@Test
fun expandedSpellingsCollapseOntoOkHttpsHost() {
randomAddresses(500).forEach { bytes ->
// The fully expanded, zero-padded, uppercase spelling of the same address.
val expanded =
(0 until 8).joinToString(":") { g ->
val value = ((bytes[g * 2].toInt() and 0xFF) shl 8) or (bytes[g * 2 + 1].toInt() and 0xFF)
value.toString(16).padStart(4, '0').uppercase()
}
assertEquals(Ipv6.format(bytes), Ipv6.canonicalizeOrNull(expanded))
assertEquals("http://[$expanded]/".toHttpUrl().host, Ipv6.canonicalizeOrNull(expanded))
}
}
}
@@ -120,6 +120,7 @@ class ConcurrentIngestLossTest {
val accepted = ConcurrentHashMap.newKeySet<String>()
val rejected = AtomicInteger()
val failed = AtomicInteger()
val window = Semaphore(200)
val done = CompletableDeferred<Unit>()
val remaining = AtomicInteger(events.size)
@@ -130,6 +131,7 @@ class ConcurrentIngestLossTest {
when (outcome) {
is IEventStore.InsertOutcome.Accepted -> accepted.add(e.id)
is IEventStore.InsertOutcome.Rejected -> rejected.incrementAndGet()
is IEventStore.InsertOutcome.Failed -> failed.incrementAndGet()
}
window.release()
if (remaining.decrementAndGet() == 0) done.complete(Unit)
@@ -154,7 +156,7 @@ class ConcurrentIngestLossTest {
}
}
val stored = store.count(Filter())
println(" submitted=${events.size} accepted=${accepted.size} rejected=${rejected.get()} stored=$stored lostAcceptedRegular=$lost")
println(" submitted=${events.size} accepted=${accepted.size} rejected=${rejected.get()} failed=${failed.get()} stored=$stored lostAcceptedRegular=$lost")
ingest.close()
queueJob.cancel()
@@ -0,0 +1,316 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.RequestSubscriptionState
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import org.junit.Test
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicLong
import java.util.concurrent.locks.ReentrantLock
/**
* Compares the three candidate designs for `PoolRequests`' subscription-state lock,
* under the real production topology: R relay-consumer coroutines on a
* limited-parallelism dispatcher (`Dispatchers.IO` = 64) all delivering EVENTs for a
* small number of subscription ids.
*
* - **PER_SUB_BLOCKING** — today: one blocking lock per subId, shared by every relay
* that sub runs on. Waiters park, which fixed the CPU burn, but a parked waiter
* still OCCUPIES its dispatcher thread.
* - **PER_SUB_MUTEX** — kotlinx `Mutex`: a waiter *suspends* and releases its thread.
* Requires making the whole listener chain `suspend` (262 overrides, 110 call sites).
* - **STRIPED** — one lock per (subId, relay). Every critical section in PoolRequests
* is already scoped to a single relay, so this removes the contention outright and
* needs no API change.
*
* The metric that matters is NOT lock throughput — it is whether unrelated work can
* still get a thread while the lock is contended. `bystanderLatency` models that: a
* task that never touches the lock, submitted to the same dispatcher.
*/
class LockDesignComparisonBenchmark {
private val relays = 191
private val dispatcherThreads = 64
private val durationMs = 3000L
/** One relay's slice of a subscription's state — all real fields are relay-keyed. */
private class PerRelay {
val lock = ReentrantLock()
var status: Int = 0
var filters: List<String>? = null
var lastKnown: List<String>? = null
}
private class Striped {
val perRelay = ConcurrentHashMap<Int, PerRelay>()
inline fun <R> withLock(
relay: Int,
block: (PerRelay) -> R,
): R {
val s = perRelay.computeIfAbsent(relay) { PerRelay() }
s.lock.lock()
try {
return block(s)
} finally {
s.lock.unlock()
}
}
}
private class PerSubBlocking {
val lock = ReentrantLock()
val status = HashMap<Int, Int>()
val filters = HashMap<Int, List<String>>()
}
private class PerSubMutex {
val mutex = Mutex()
val status = HashMap<Int, Int>()
val filters = HashMap<Int, List<String>>()
}
private fun report(
name: String,
subs: Int,
ops: Long,
bystanderSamples: List<Long>,
) {
val sorted = bystanderSamples.sorted()
val p50 = sorted[sorted.size / 2] / 1000.0
val p99 = sorted[(sorted.size * 99) / 100] / 1000.0
val max = sorted.last() / 1000.0
println(
"%-18s subs=%-3d ops/s=%,10d bystander p50=%8.1fus p99=%9.1fus max=%9.1fus (n=%d)".format(
name,
subs,
ops * 1000 / durationMs,
p50,
p99,
max,
sorted.size,
),
)
}
@Test
fun compareDesigns() {
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
println("compareDesigns skipped. Run with -PprodRelayBench=1 to enable.")
return
}
println("relays=$relays dispatcherThreads=$dispatcherThreads window=${durationMs}ms")
println("bystander = a task that NEVER touches the lock, on the same dispatcher.")
println("Lower bystander latency = the lock is not stealing dispatcher threads.\n")
for (subs in listOf(1, 4, 16)) {
runPerSubBlocking(subs)
runPerSubMutex(subs)
runStriped(subs)
runRealStriped(subs)
println()
}
}
private fun runPerSubBlocking(subs: Int) =
runBlocking {
@Suppress("DEPRECATION")
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
val states = Array(subs) { PerSubBlocking() }
val stop = AtomicBoolean(false)
val ops = AtomicLong(0)
val bystander = ArrayList<Long>()
val jobs =
(0 until relays).map { relay ->
launch(dispatcher) {
var n = 0L
while (!stop.get()) {
val s = states[relay % subs]
s.lock.lock()
try {
s.status[relay] = 1
s.filters[relay] = SAMPLE
s.status[relay]
} finally {
s.lock.unlock()
}
n++
// Models `for (message in incomingMessages)`: every real
// iteration suspends, letting the dispatcher multiplex.
kotlinx.coroutines.yield()
}
ops.addAndGet(n)
}
}
val by =
launch(dispatcher) {
while (!stop.get()) {
val t = System.nanoTime()
kotlinx.coroutines.yield()
bystander.add(System.nanoTime() - t)
}
}
Thread.sleep(durationMs)
stop.set(true)
jobs.forEach { it.join() }
by.join()
report("PER_SUB_BLOCKING", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
}
private fun runPerSubMutex(subs: Int) =
runBlocking {
@Suppress("DEPRECATION")
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
val states = Array(subs) { PerSubMutex() }
val stop = AtomicBoolean(false)
val ops = AtomicLong(0)
val bystander = ArrayList<Long>()
val jobs =
(0 until relays).map { relay ->
launch(dispatcher) {
var n = 0L
while (!stop.get()) {
val s = states[relay % subs]
s.mutex.withLock {
s.status[relay] = 1
s.filters[relay] = SAMPLE
s.status[relay]
}
n++
// Models `for (message in incomingMessages)`: every real
// iteration suspends, letting the dispatcher multiplex.
kotlinx.coroutines.yield()
}
ops.addAndGet(n)
}
}
val by =
launch(dispatcher) {
while (!stop.get()) {
val t = System.nanoTime()
kotlinx.coroutines.yield()
bystander.add(System.nanoTime() - t)
}
}
Thread.sleep(durationMs)
stop.set(true)
jobs.forEach { it.join() }
by.join()
report("PER_SUB_MUTEX", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
}
private fun runStriped(subs: Int) =
runBlocking {
@Suppress("DEPRECATION")
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
val states = Array(subs) { Striped() }
val stop = AtomicBoolean(false)
val ops = AtomicLong(0)
val bystander = ArrayList<Long>()
val jobs =
(0 until relays).map { relay ->
launch(dispatcher) {
var n = 0L
while (!stop.get()) {
states[relay % subs].withLock(relay) { s ->
s.status = 1
s.filters = SAMPLE
s.lastKnown = SAMPLE
}
n++
// Models `for (message in incomingMessages)`: every real
// iteration suspends, letting the dispatcher multiplex.
kotlinx.coroutines.yield()
}
ops.addAndGet(n)
}
}
val by =
launch(dispatcher) {
while (!stop.get()) {
val t = System.nanoTime()
kotlinx.coroutines.yield()
bystander.add(System.nanoTime() - t)
}
}
Thread.sleep(durationMs)
stop.set(true)
jobs.forEach { it.join() }
by.join()
report("STRIPED", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
}
/**
* Same topology, but driving the REAL shipped [RequestSubscriptionState] (striped per
* relay) instead of a prototype — so the measured win is a property of the code that
* ships, not of this file.
*/
private fun runRealStriped(subs: Int) =
runBlocking {
@Suppress("DEPRECATION")
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
val states = Array(subs) { RequestSubscriptionState<Int>() }
val stop = AtomicBoolean(false)
val ops = AtomicLong(0)
val bystander = ArrayList<Long>()
val filters = listOf(Filter(kinds = listOf(1)))
val jobs =
(0 until relays).map { relay ->
launch(dispatcher) {
var n = 0L
val state = states[relay % subs]
while (!stop.get()) {
state.withLock(relay) {
state.onNewEvent(relay)
state.currentState(relay)
state.onOpenReq(relay, filters)
state.lastKnownFilterStates(relay)
}
n++
kotlinx.coroutines.yield()
}
ops.addAndGet(n)
}
}
val by =
launch(dispatcher) {
while (!stop.get()) {
val t = System.nanoTime()
kotlinx.coroutines.yield()
bystander.add(System.nanoTime() - t)
}
}
Thread.sleep(durationMs)
stop.set(true)
jobs.forEach { it.join() }
by.join()
report("REAL_STRIPED", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
}
companion object {
private val SAMPLE = listOf("kinds:1", "authors:abc")
}
}
@@ -0,0 +1,242 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.RequestSubscriptionState
import org.junit.Assert.assertTrue
import org.junit.Test
import java.util.concurrent.CountDownLatch
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicLong
import kotlin.concurrent.thread
/**
* Reproduces the production ANR seen on a Pixel 8 (2026-08-03, anr_2026-08-03-12-55-26-256):
* 191 live relay sockets feed EVENT frames for the same handful of subscription ids, so
* dozens of DefaultDispatcher workers pile onto ONE [RequestSubscriptionState] busy-wait
* lock. In that trace 51 of 52 runnable workers sat in the inlined spin loop while the
* single lock holder was parked in `WaitingForGcToComplete`.
*
* Two things are measured:
* - [contendedThroughput]: aggregate critical sections/s as the contender count grows past
* the core count (the "negative scaling" the 2026-07-02 plan measured with 4 feeders,
* re-run at production fan-out).
* - [victimLatencyUnderSpin]: what an UNRELATED thread (stand-in for the UI thread) sees
* while the spinners run — this is the ANR mechanism, not the lock throughput.
*/
class SpinLockConvoyBenchmark {
private val cores = Runtime.getRuntime().availableProcessors()
/**
* Every waiter targets ONE reference on purpose. The lock is striped per relay, so
* spreading threads over distinct relays would put them on different stripes and
* this benchmark would measure nothing — the point here is the primitive's behaviour
* when contention DOES land on a single stripe.
*/
private val hotRelay = 0
/** Mirrors the real critical section: a handful of map reads/writes, no I/O. */
private fun criticalSection(
state: RequestSubscriptionState<Int>,
relay: Int,
) {
state.onNewEvent(relay)
state.currentState(relay)
state.lastKnownFilterStates(relay)
}
@Test
fun contendedThroughput() {
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
println("contendedThroughput skipped. Run with -PprodRelayBench=1 to enable.")
return
}
println("cores = $cores")
println("threads | ops/s | vs 1 thread")
var baseline = 0.0
for (threads in listOf(1, 2, 4, 8, 16, 32, 52)) {
val state = RequestSubscriptionState<Int>()
val stop = AtomicBoolean(false)
val ops = AtomicLong(0)
val start = CountDownLatch(1)
val workers =
(0 until threads).map { id ->
thread {
start.await()
var local = 0L
while (!stop.get()) {
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
local++
}
ops.addAndGet(local)
}
}
val t0 = System.nanoTime()
start.countDown()
Thread.sleep(2000)
stop.set(true)
workers.forEach { it.join() }
val secs = (System.nanoTime() - t0) / 1e9
val rate = ops.get() / secs
if (threads == 1) baseline = rate
println("%7d | %9.0f | %.2fx".format(threads, rate, rate / baseline))
}
}
/**
* REGRESSION GUARD: contended waiters on [RequestSubscriptionState.withLock] must PARK,
* never busy-wait. Fails if the lock is ever turned back into a spin lock.
*
* This is the signature that identified the production ANR: in
* `anr_2026-08-03-12-55-26-256`, 37 of 52 runnable workers sat at one obfuscated line of
* `PoolRequests.onIncomingMessage` and 12 more at one line of `syncState$lambda$0` — all
* `state=R`, `sCount=0`, burning 596% CPU while the lock holder was stuck in
* `WaitingForGcToComplete`. With a spin lock this test observes ~40/40 waiters RUNNABLE;
* with a parking lock it observes 0.
*/
@Test
fun contendedWaitersParkInsteadOfSpinning() {
val spinners = 40
val state = RequestSubscriptionState<Int>()
val stop = AtomicBoolean(false)
val start = CountDownLatch(1)
val holder =
thread(name = "holder") {
start.await()
while (!stop.get()) {
state.withLock(hotRelay) {
val t = System.nanoTime()
while (System.nanoTime() - t < 5_000_000) { /* hold 5ms */ }
}
}
}
val threads =
(0 until spinners).map { id ->
thread(name = "spinner-$id") {
start.await()
while (!stop.get()) {
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
}
}
}
start.countDown()
Thread.sleep(300)
// Sample every spinner's stack, exactly like an ANR dump would.
val points = HashMap<String, Int>()
var runnable = 0
threads.forEach { t ->
if (t.state == Thread.State.RUNNABLE) runnable++
val top = t.stackTrace.firstOrNull { it.className.contains("SpinLockConvoyBenchmark") || it.className.contains("RequestSubscriptionState") }
if (top != null) {
val key = "${top.className.substringAfterLast('.')}.${top.methodName}:${top.lineNumber}"
points[key] = (points[key] ?: 0) + 1
}
}
stop.set(true)
threads.forEach { it.join() }
holder.join()
println("sampled $spinners waiters: RUNNABLE=$runnable, distinct program points=${points.size}")
points.entries.sortedByDescending { it.value }.forEach { println(" ${it.value}x ${it.key}") }
// A parked waiter reports WAITING, so the parking lock measures ~0 here while a
// spin lock measures ~100%. The line sits at 50% deliberately: it separates the
// two cases by a mile and leaves headroom on a loaded CI box, where a few waiters
// can legitimately be mid-acquire when we sample.
assertTrue(
"Expected contended waiters to park, but $runnable/$spinners were RUNNABLE — " +
"RequestSubscriptionState.withLock looks like it is busy-waiting again. " +
"See PlatformLock's kdoc: this is what caused anr_2026-08-03-12-55-26-256.",
runnable <= spinners / 2,
)
}
@Test
fun victimLatencyUnderSpin() {
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
println("victimLatencyUnderSpin skipped. Run with -PprodRelayBench=1 to enable.")
return
}
// One holder that briefly stalls inside the critical section (in production: a GC
// pause, or simply being descheduled). Everyone else spins.
val spinners = 52
val state = RequestSubscriptionState<Int>()
val stop = AtomicBoolean(false)
fun measureVictim(label: String) {
// The "UI thread": allocates and measures its own scheduling latency.
val samples = ArrayList<Long>()
repeat(200) {
val t = System.nanoTime()
// trivial allocation work, like a Compose semantics traversal step
val junk = ArrayList<String>(64)
repeat(64) { i -> junk.add("node$i") }
Thread.yield()
samples.add(System.nanoTime() - t)
}
samples.sort()
println(
"%-22s p50=%6.0fus p90=%7.0fus p99=%8.0fus max=%8.0fus".format(
label,
samples[samples.size / 2] / 1000.0,
samples[(samples.size * 90) / 100] / 1000.0,
samples[(samples.size * 99) / 100] / 1000.0,
samples.last() / 1000.0,
),
)
}
measureVictim("idle (no spinners)")
val start = CountDownLatch(1)
val holder =
thread {
start.await()
while (!stop.get()) {
state.withLock(hotRelay) {
// Simulate the holder losing its core / waiting on GC mid-section.
val t = System.nanoTime()
while (System.nanoTime() - t < 2_000_000) { /* 2ms stall */ }
}
Thread.sleep(1)
}
}
val threads =
(0 until spinners).map { id ->
thread {
start.await()
while (!stop.get()) {
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
}
}
}
start.countDown()
Thread.sleep(500)
measureVictim("$spinners spinners")
stop.set(true)
threads.forEach { it.join() }
holder.join()
measureVictim("after (no spinners)")
}
}
@@ -0,0 +1,70 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
import java.io.EOFException
import java.net.ConnectException
import java.net.SocketTimeoutException
import java.net.UnknownHostException
import javax.net.ssl.SSLHandshakeException
import kotlin.test.Test
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* What a monitor is willing to SAY about someone else's relay. A negative
* NIP-66 record is signed and public, so only failures of the connection
* itself may be published as "unreachable".
*/
class UnreachabilityTest {
private fun proves(e: Exception) = Unreachability.proves(e)
@Test
fun `a connection that never opened is unreachable`() {
assertTrue(proves(UnknownHostException("no such host")))
assertTrue(proves(ConnectException("connection refused")))
assertTrue(proves(SSLHandshakeException("cert expired")))
}
@Test
fun `a relay that hung up mid-transfer is not unreachable`() {
// A relay that answers the handshake in 50ms and then sends EOFException
// part-way through a large page is reachable; it declined to finish a
// query. Publishing "unreachable" would be a false statement about a
// working server.
assertFalse(proves(EOFException("stream closed")))
}
@Test
fun `our own bug is never the relay's fault`() {
assertFalse(proves(ConcurrentModificationException()))
assertFalse(proves(NullPointerException()))
assertFalse(proves(ClassCastException("HashMap\$Node cannot be cast")))
}
@Test
fun `an unrecognised failure stays quiet`() {
// Conservative on purpose: staying quiet costs one retry next cycle,
// being wrong costs a false record carrying the monitor's signature.
assertFalse(proves(SocketTimeoutException("read timed out")))
assertFalse(proves(RuntimeException("something new")))
}
}
@@ -0,0 +1,44 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.utils.concurrent
import kotlin.concurrent.atomics.AtomicBoolean
import kotlin.concurrent.atomics.ExperimentalAtomicApi
// Linux/JVM-less target: Kotlin/Native's stdlib ships no parking lock and there is
// no Foundation here, so this keeps a test-and-test-and-set spin. Correct but not
// scalable. Acceptable ONLY because linuxX64 is a build/CI target for quartz, not a
// host for the many-relay client workload whose contention motivated the parking
// actuals on jvmAndroid and Apple. If that ever changes, swap in a pthread mutex.
@OptIn(ExperimentalAtomicApi::class)
actual class PlatformLock {
private val held = AtomicBoolean(false)
actual fun lock() {
while (held.exchange(true)) {
while (held.load()) { }
}
}
actual fun unlock() {
held.store(false)
}
}
@@ -74,6 +74,16 @@ actual class ConcurrentMap<K : Any, V : Any> {
}
}
actual fun remove(key: K): V? {
while (true) {
val cur = ref.load()
val old = cur[key] ?: return null
val copy = HashMap(cur)
copy.remove(key)
if (ref.compareAndSet(cur, copy)) return old
}
}
actual fun size(): Int = ref.load().size
actual fun snapshot(): Map<K, V> = HashMap(ref.load())
+11
View File
@@ -29,15 +29,26 @@
# amethyst-desktop-1.08.0-linux-x64.AppImage
# amethyst-desktop-1.08.0-linux-x64.flatpak
# amethyst-desktop-1.08.0-linux-x64.tar.gz
# amethyst-desktop-1.08.0-linux-arm64.deb
# amethyst-desktop-1.08.0-linux-arm64.rpm
# amethyst-desktop-1.08.0-linux-arm64.AppImage
# amethyst-desktop-1.08.0-linux-arm64.flatpak
# amethyst-desktop-1.08.0-linux-arm64.tar.gz
# amy-1.08.0-macos-arm64.tar.gz
# amy-1.08.0-macos-x64.tar.gz
# amy-1.08.0-linux-x64.tar.gz
# amy-1.08.0-linux-x64.deb
# amy-1.08.0-linux-x64.rpm
# amy-1.08.0-linux-arm64.tar.gz
# amy-1.08.0-linux-arm64.deb
# amy-1.08.0-linux-arm64.rpm
# geode-1.08.0-macos-arm64.tar.gz
# geode-1.08.0-linux-x64.tar.gz
# geode-1.08.0-linux-x64.deb
# geode-1.08.0-linux-x64.rpm
# geode-1.08.0-linux-arm64.tar.gz
# geode-1.08.0-linux-arm64.deb
# geode-1.08.0-linux-arm64.rpm
#
# Two assets break the family/arch shape on purpose: the no-JRE jar bundles for
# Homebrew-core are pure JVM bytecode (no bundled runtime), so a single