mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-08-05 06:14:37 +00:00
Compare commits
42
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f9bef87160 | ||
|
|
a58289b62c | ||
|
|
70d51cc98b | ||
|
|
067d68b89c | ||
|
|
129401bdaf | ||
|
|
8f8713d825 | ||
|
|
21b02d780f | ||
|
|
d3bd7a45b9 | ||
|
|
a5d2d153c8 | ||
|
|
3a702add57 | ||
|
|
5ec35c7772 | ||
|
|
9a4f6c6cdd | ||
|
|
c767298368 | ||
|
|
ad3ce4d1a1 | ||
|
|
0d0117061a | ||
|
|
fd5bd994a1 | ||
|
|
a7eec1d605 | ||
|
|
d9a58950ec | ||
|
|
ba7cc72dd2 | ||
|
|
156176f5fe | ||
|
|
1a25557bdb | ||
|
|
b48b87a60b | ||
|
|
cdbd550405 | ||
|
|
c029b271e7 | ||
|
|
4081ef1681 | ||
|
|
804b850095 | ||
|
|
42fc73f6cb | ||
|
|
19d8e3069d | ||
|
|
9231195890 | ||
|
|
564cafedc4 | ||
|
|
e54a546d10 | ||
|
|
b46063713e | ||
|
|
d54e54b48a | ||
|
|
2b2e47256b | ||
|
|
7b29f57526 | ||
|
|
a110ce0a30 | ||
|
|
ba78eb599e | ||
|
|
5896ae5eff | ||
|
|
fdc68e801a | ||
|
|
30742ab352 | ||
|
|
1622bd7109 | ||
|
|
ebd9a163bc |
@@ -85,3 +85,30 @@ skills verified clean):
|
||||
`ParseReturn.entity` (the `Nip19Parser.Return.*` sealed class never existed);
|
||||
Event Store section corrected from "Android only" to commonMain/all platforms
|
||||
with the real `store.sqlite.EventStore` import and suspend generic `query<T>`.
|
||||
|
||||
## Phase 4 (2026-08): Store-implementer skills (external consumer request)
|
||||
|
||||
Three skills added at the request of an external Quartz consumer
|
||||
(vespa-eventstore — a server-side `IEventStore` on Vespa that asserts result
|
||||
parity against the SQLite store in CI). All three document the
|
||||
**store/relay-implementer's perspective**, which `quartz-integration` and
|
||||
`nostr-expert` (client-side) did not cover. Requirements doc: the skill-requests
|
||||
file reviewed 2026-08-04; the requester's items #4 (storage-lifecycle-nips) was
|
||||
folded into `event-store-semantics` per their own recommendation, and #5
|
||||
(relay-server/geode policies) was declined as not currently needed.
|
||||
|
||||
- **`event-store-semantics/`** — the `IEventStore`/SQLite-store behavioral
|
||||
contract as named rules (STORE-Fxx/Wxx/Dxx/Cxx/Sxx/Nxx) with a semantics
|
||||
changelog for pin-bump review. Written from `QueryBuilder`,
|
||||
`MergeQueryExecutor`, the seven `*Module.kt` files, and `IEventStore` KDoc.
|
||||
- **`nip85-trusted-assertions/`** — the NIP-85 model (10040/30382/30383/30384/
|
||||
30385), full tag vocabulary with value semantics, authorization conventions,
|
||||
worked JSON examples, stability notes.
|
||||
- **`searchable-events/`** — the `SearchableEvent` contract + maintenance
|
||||
mandate, with `references/searchable-kinds.md` holding the exhaustive
|
||||
kind → class → `indexableContent()` table (126 classes / 129 kinds) that
|
||||
external search engines diff at version bumps.
|
||||
|
||||
Follow-ups suggested but not implemented: a shared JSON test-vector corpus for
|
||||
filter semantics (testFixtures both the SQLite tests and external parity suites
|
||||
could run), and a snapshot test pinning the searchable-kind set.
|
||||
|
||||
@@ -0,0 +1,325 @@
|
||||
---
|
||||
name: event-store-semantics
|
||||
description: The authoritative behavioral contract of Quartz's event stores — `IEventStore` and its reference SQLite implementation (`nip01Core/store/sqlite/`). Use when implementing or asserting parity with a Quartz event store (external engines like Vespa, the filesystem store, geode), answering filter-semantics questions (since/until inclusivity, tag OR/AND, multi-filter limits, ordering tiebreaks), or working on the write-path rules for replaceable/addressable supersession, NIP-09 deletions, NIP-40 expiration, NIP-62 vanish, NIP-45 counts, or NIP-50 search inside the store. Every behavior has a named rule id (STORE-Fxx/Wxx/Dxx/Sxx/Cxx) so downstream implementations can annotate divergences precisely.
|
||||
---
|
||||
|
||||
# Event Store Semantics — the `IEventStore` / SQLite-store contract
|
||||
|
||||
The SQLite `EventStore` (`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/`)
|
||||
is the de-facto **reference implementation** of what a Quartz event store must do. Other
|
||||
implementations — the in-repo filesystem store (`nip01Core/store/fs/`, held to parity by
|
||||
`quartz/src/jvmTest/.../store/fs/FsParityTest.kt`) and external engines (e.g. a Vespa-backed
|
||||
store) — reimplement its *observable behavior* and assert parity in CI. This skill states that
|
||||
behavior as **named, numbered decisions** so a parity divergence becomes a lookup, not an
|
||||
archaeology session through `QueryBuilder`/`MergeQueryExecutor`.
|
||||
|
||||
Every rule below was verified against the code as of this skill's last update. When you change
|
||||
store behavior, **update the rule here in the same PR** and add a line to the
|
||||
[Semantics changelog](#semantics-changelog) — downstream implementations pin Quartz by commit and
|
||||
review pin bumps against this file.
|
||||
|
||||
## Key files
|
||||
|
||||
| Concern | File |
|
||||
|---|---|
|
||||
| Public contract (KDoc is normative) | `nip01Core/store/IEventStore.kt` |
|
||||
| High-level store (owns pool + planner) | `sqlite/EventStore.kt`, `sqlite/SQLiteEventStore.kt` |
|
||||
| Filter → SQL, ordering, limits, counts | `sqlite/QueryBuilder.kt` |
|
||||
| k-way merge fast path (feed shapes) | `sqlite/MergeQueryExecutor.kt` |
|
||||
| Schema, tag hashing, immutability | `sqlite/EventIndexesModule.kt`, `sqlite/TagNameValueHasher.kt`, `sqlite/SeedModule.kt` |
|
||||
| Replaceable / addressable supersession | `sqlite/ReplaceableModule.kt`, `sqlite/AddressableModule.kt` |
|
||||
| NIP-09 / NIP-40 / NIP-62 / ephemeral | `sqlite/DeletionRequestModule.kt`, `sqlite/ExpirationModule.kt`, `sqlite/RightToVanishModule.kt`, `sqlite/EphemeralModule.kt` |
|
||||
| NIP-50 FTS | `sqlite/FullTextSearchModule.kt` (see also the `searchable-events` skill) |
|
||||
| Index/feature toggles | `sqlite/IndexingStrategy.kt` (client default) and geode's `RelayIndexingStrategy.kt` (relay preset) |
|
||||
| Operational README | `sqlite/README.md` (concurrency, pragmas, maintenance) |
|
||||
|
||||
Executable spec: the test suites in
|
||||
`quartz/src/commonTest/.../store/sqlite/` (`BasicTest`, `ReplaceableTest`, `AddressableTest`,
|
||||
`DeletionTest`, `ExpirationTest`, `RightToVanishTest`, `SearchTest`, `SearchRelevanceOrderTest`,
|
||||
`MergeQueryCorrectnessTest`, `TagMergeCorrectnessTest`, `QueryAssemblerTest`,
|
||||
`SnapshotIdsForNegentropyTest`, `FilterMatcherTest`, …). If a rule here ever contradicts a test,
|
||||
the test wins — and this file has a bug to fix.
|
||||
|
||||
## Kind classes (used throughout)
|
||||
|
||||
- **Replaceable**: kind `0`, kind `3`, and `10000 ≤ kind < 20000`.
|
||||
- **Ephemeral**: `20000 ≤ kind < 30000`.
|
||||
- **Addressable**: `30000 ≤ kind < 40000`.
|
||||
- Everything else is a regular event.
|
||||
|
||||
---
|
||||
|
||||
## Filter matching (STORE-F)
|
||||
|
||||
**STORE-F01 — `since`/`until` are both inclusive.** `since` compiles to
|
||||
`created_at >= ?`, `until` to `created_at <= ?` (`QueryBuilder` uses
|
||||
`greaterThanOrEquals`/`lessThanOrEquals` everywhere). An event with
|
||||
`created_at == since == until` matches.
|
||||
|
||||
**STORE-F02 — `ids` and `authors` are exact-match only.** They compile to `=`/`IN` against the
|
||||
full 64-char hex columns. **NIP-01 prefix matching is NOT supported** anywhere in the store.
|
||||
(`Filter`'s constructor logs an error for non-64-char ids/authors but still sends them; they
|
||||
simply never match.)
|
||||
|
||||
**STORE-F03 — tag filter combination.** Within one tag name, values are **OR**
|
||||
(`tag_hash IN (…)`). Across different tag names in the same filter, conditions are **AND**
|
||||
(each extra name becomes another `event_tags` self-join). `tagsAll` (NIP-91 `&x` syntax) demands
|
||||
**every listed value** be present on the event — one join + equality per value — and composes by
|
||||
AND with any plain `tags` in the same filter.
|
||||
|
||||
**STORE-F04 — only single-letter tag names are indexed (by default).**
|
||||
`DefaultIndexingStrategy.shouldIndex` indexes a tag iff `tag.size >= 2 && tag[0].length == 1`.
|
||||
A filter on a multi-letter tag name (`#title`, `#alt`) matches **nothing** in the SQLite store.
|
||||
Deployments can widen `shouldIndex`, but the stock contract is single-letter-only.
|
||||
|
||||
**STORE-F05 — `d` is special-cased out of the tag index.** `#d` values are matched against the
|
||||
`event_headers.d_tag` column, not `event_tags` (`Filter.toFilterWithDTags()`). Consequences:
|
||||
`#d` works on addressable events (which populate `d_tag`); when all `kinds` are addressable the
|
||||
query adds `kind >= 30000 AND kind < 40000` to pin the addressable index. **Only use `#d` via
|
||||
plain `tags`.** A `#d` under `tagsAll` is handled inconsistently: on the simple (no other
|
||||
tags/search) path it degrades to OR semantics (`toFilterWithDTags` folds it into `dTags`), and
|
||||
when `tags["d"]` is also present it is dropped entirely; on the tag-join path it is ignored.
|
||||
(An event has one d-tag, so AND-across-values could never match anyway.)
|
||||
|
||||
**STORE-F06 — tag and author matching in the tag path is hash-based.** `event_tags` stores a
|
||||
64-bit MurmurHash3 of `(tag name, value)` keyed by a per-database random seed (`SeedModule`,
|
||||
`TagNameValueHasher`); the p/e/a-owner columns are hashes too. There is **no post-verification**
|
||||
of hash matches, so a hash collision would return a false positive. Probability is negligible in
|
||||
practice but nonzero — a parity harness comparing against an exact-match engine should know this
|
||||
is the one place the reference can (theoretically) over-match.
|
||||
|
||||
**STORE-F07 — multiple filters are a union with dedup; `limit` is per-filter.** Each filter
|
||||
becomes its own row-id subquery with its **own** `ORDER BY … LIMIT`; branches are combined with
|
||||
SQL `UNION` (dedup by row). There is **no global limit** — a 3-filter query with limits
|
||||
10/20/30 can return up to 60 events, presented in one merged `created_at DESC` ordering. NIP-45
|
||||
counts and negentropy snapshots dedup the same way (`SELECT DISTINCT` / `UNION`).
|
||||
|
||||
**STORE-F08 — result ordering.** Non-search queries order `created_at DESC`. The `id ASC`
|
||||
tiebreak on equal `created_at` is applied **only when
|
||||
`IndexingStrategy.useAndIndexIdOnOrderBy = true`** — which is `false` in the client default
|
||||
**and** in geode's relay preset. So by default, same-second ordering is unspecified (SQLite
|
||||
returns them in storage order). Any newest-N is valid; a parity suite must not assert
|
||||
same-`created_at` order unless it configures the flag. One extra caveat with the flag ON: the
|
||||
`MergeQueryExecutor` tag-stream path still yields same-second ties in rowid order (its cursors
|
||||
run off `event_tags`, which has no id column) — a valid newest-N that may differ byte-for-byte
|
||||
from the single-SQL ordering.
|
||||
|
||||
**STORE-F09 — the merge fast path returns the same *set*.** Single-filter queries of the shape
|
||||
"authors (+kinds) + limit" or "one `#x` IN-list (+kinds) + limit" (≤2048 streams) route through
|
||||
`MergeQueryExecutor`, a k-way newest-first merge over per-(kind,author) / per-(tag-value,kind)
|
||||
index cursors with dedup by id on the tag shape. This is an optimization, not a semantics
|
||||
change — `MergeQueryCorrectnessTest`/`TagMergeCorrectnessTest` assert set-equality with the
|
||||
single-SQL plan (ordering caveat per STORE-F08).
|
||||
|
||||
**STORE-F10 — empty filter.** `query(Filter())` / `count(Filter())` match **everything**
|
||||
(`Filter.isEmpty()` → the "everything" query). `delete(Filter())` is deliberately asymmetric:
|
||||
it deletes **nothing** and returns 0, so a stray empty filter can't wipe the store (documented
|
||||
on `QueryBuilder.delete`).
|
||||
|
||||
**STORE-F11 — empty lists (`kinds = emptyList()` etc.) are a client error with inconsistent
|
||||
handling; don't rely on either outcome.** On the single-filter simple path an empty list
|
||||
renders as `1 = 0` → matches nothing. But `Filter.isEmpty()` treats empty lists the same as
|
||||
`null`, so on the multi-filter union path such a filter contributes no subquery — and a list of
|
||||
*only* empty-list filters degrades to the match-everything query. Known quirk; treat
|
||||
empty-list filters as invalid input rather than replicating this shape.
|
||||
|
||||
**STORE-F12 — `limit` edge cases.** `limit = 0` compiles to `LIMIT 0` → zero rows.
|
||||
`limit = null` means unbounded. Negative limits are not defended against (don't send them).
|
||||
|
||||
**STORE-F13 — the in-memory matcher is a separate (simpler) implementation.**
|
||||
`Filter.match(event)` (`FilterMatcher`) is used for live-stream matching, not storage queries;
|
||||
it checks ids/authors/kinds/tags/tagsAll/since/until but not `search` or `limit`. Parity work
|
||||
targets the SQL semantics above, not `FilterMatcher`.
|
||||
|
||||
---
|
||||
|
||||
## Write path (STORE-W)
|
||||
|
||||
Inserts run every module in one transaction: header+tags → NIP-09 side effects → expiration
|
||||
row → FTS row → vanish side effects. A trigger `RAISE(ABORT, …)` rejects the whole row with the
|
||||
messages quoted below (they surface as the NIP-01 `OK false` reason).
|
||||
|
||||
**STORE-W01 — replaceable supersession.** Unique index on `(kind, pubkey)` for replaceable
|
||||
kinds. A `BEFORE INSERT` trigger deletes any stored version that is *older* — meaning
|
||||
`created_at` smaller, **or equal `created_at` with lexicographically larger id** (NIP-01
|
||||
lowest-id-wins). Inserting a version that is *not* newer under that ordering leaves the stored
|
||||
row in place and fails the unique index → rejected (`UNIQUE constraint failed`). Net contract:
|
||||
exactly one version stored; newest wins; ties broken by lowest id; older re-inserts blocked.
|
||||
|
||||
**STORE-W02 — addressable supersession.** Same as W01 with unique index
|
||||
`(kind, pubkey, d_tag)` over `30000 ≤ kind < 40000`. Nuance: `d_tag` is populated from the
|
||||
*parsed* event class (`AddressableEvent.dTag()`); an addressable-range kind whose class doesn't
|
||||
parse as `AddressableEvent` stores `d_tag NULL`, and SQLite treats NULLs as distinct in unique
|
||||
indexes — such events don't supersede each other. An event with no `d` tag parses as `dTag() = ""`
|
||||
(empty string), which *does* dedupe normally.
|
||||
|
||||
**STORE-W03 — ephemeral events are never stored but are acked as accepted.**
|
||||
`insert()` returns silently and `batchInsert` reports `Accepted` for `20000 ≤ kind < 30000`
|
||||
without writing (the live relay stream still broadcasts them). A DB-level backstop trigger
|
||||
(`blocked: cannot store ephemeral events`) rejects any that sneak past the app-level check.
|
||||
|
||||
**STORE-W04 — expired events are rejected at insert.** App-level check
|
||||
(`event.isExpired()`) plus a trigger on the expiration-row insert
|
||||
(`blocked: this event is expired` when `expiration <= unixepoch()`). Single-event `insert`
|
||||
**throws**; `batchInsert` returns `Rejected`.
|
||||
|
||||
**STORE-W05 — expiry is enforced at insert and by sweep, NOT at query time.** Events with a
|
||||
future `expiration` store a row in `event_expirations`. Nothing filters them out of queries
|
||||
after the timestamp passes: **a query between expiry and the next `deleteExpiredEvents()` sweep
|
||||
returns the expired event.** Operators run the sweep periodically (README recommends ~15 min).
|
||||
Re-inserting an already-expired event after the sweep is rejected per W04.
|
||||
|
||||
**STORE-W06 — GiftWrap ownership is the recipient.** For kind 1059 the store computes
|
||||
`pubkey_owner_hash` from the `p`-tag recipient (falling back to the random signer key if
|
||||
absent). All owner-scoped machinery — NIP-09 re-insert blocking, NIP-62 vanish deletion and
|
||||
blocking — operates on that owner hash, so **a user's deletions/vanish remove giftwraps
|
||||
addressed to them**, even though the wrap's `pubkey` is a one-time key. (Consequently GiftWraps
|
||||
are also excluded from `authorsMissingOutbox()`.)
|
||||
|
||||
**STORE-W07 — immutability.** `event_headers`/`event_tags` rows are never updated
|
||||
(`BEFORE UPDATE` triggers abort). All supersession is delete + insert; `event_tags`,
|
||||
`event_expirations`, `event_vanish`, and the FTS row follow the header by
|
||||
`ON DELETE CASCADE` / trigger.
|
||||
|
||||
**STORE-W08 — batch insert.** One outer transaction, one SAVEPOINT per row: a bad row rolls
|
||||
back alone and reports `Rejected(reason)`; the rest commit. If the **outer commit** fails, every
|
||||
entry is treated as `Rejected` (the `IEventStore.batchInsert` contract). Outcomes are returned
|
||||
in input order; OK frames pair by event id, not order.
|
||||
|
||||
---
|
||||
|
||||
## Deletion lifecycle — NIP-09 / NIP-62 (STORE-D)
|
||||
|
||||
**STORE-D01 — delete by id.** A kind-5's `e` tags delete stored events with those ids **whose
|
||||
owner is the kind-5's author** (`pubkey_owner_hash` match — recipient for giftwraps per W06).
|
||||
The id path has **no timestamp condition**: it deletes the target regardless of the relative
|
||||
`created_at` values.
|
||||
|
||||
**STORE-D02 — delete by address.** A kind-5's `a` tags delete events at that
|
||||
`(kind, pubkey, d_tag)` coordinate with `created_at <= deletion.created_at` — **inclusive**; a
|
||||
version newer than the deletion survives. Only coordinates whose pubkey equals the kind-5's
|
||||
author are honored. Replaceable coordinates (`kind:pubkey:` with no d-tag) get the same
|
||||
`created_at <=` treatment against `(kind, pubkey)`.
|
||||
|
||||
**STORE-D03 — cross-author kind-5s are stored but inert.** A deletion naming someone else's
|
||||
events is inserted like any regular event (it may be useful to other relays/clients) but its
|
||||
delete pass removes zero rows and creates no blocking.
|
||||
|
||||
**STORE-D04 — re-insert blocking.** A `BEFORE INSERT` trigger rejects
|
||||
(`blocked: a deletion event exists`) any event whose id (`e`-hash) **or** address (`a`-hash) is
|
||||
named by a stored kind-5 from the same owner with `deletion.created_at >= event.created_at`.
|
||||
Note the asymmetry with D01: a *backdated* id-deletion (older `created_at` than its target)
|
||||
still deletes on arrival, but would not block a later re-insert.
|
||||
|
||||
**STORE-D05 — a kind-5 CAN delete another kind-5, and doing so un-blocks its targets.**
|
||||
Nothing excludes kind 5 from the id path (D01). Deleting a deletion removes its tombstone rows
|
||||
from `event_tags`, so events it had deleted become re-insertable. **Status: known quirk, not a
|
||||
considered decision.** NIP-09 leaves it open; at least one external implementation
|
||||
(vespa-eventstore) deliberately diverges by treating deletion-of-a-deletion as a no-op, which is
|
||||
the safer reading (tombstones shouldn't be revocable). If you change this, update this rule and
|
||||
the changelog — parity suites key off it.
|
||||
|
||||
**STORE-D06 — NIP-62 vanish is relay-scoped.** A kind-62 only cascades when
|
||||
`shouldVanishFrom(relay)` — its `relay` tags name this store's `relay` URL or `ALL_RELAYS`.
|
||||
(A store constructed with `relay = null` matches only `ALL_RELAYS` requests.) Out-of-scope
|
||||
vanish events are stored as regular events with no side effects.
|
||||
|
||||
**STORE-D07 — vanish scope and horizon.** An in-scope vanish deletes every event whose
|
||||
**owner** (W06) is the vanishing pubkey with `created_at < vanish.created_at` (strict — the
|
||||
vanish event itself survives), and blocks inserts of owned events with
|
||||
`created_at <= vanish.created_at` (`blocked: a request to vanish event exists`; note blocking is
|
||||
inclusive where deletion is strict). Newer vanish requests supersede older ones per pubkey
|
||||
(unique on `pubkey_hash`).
|
||||
|
||||
**STORE-D08 — manual deletes.** `delete(id)` removes one row unconditionally (no blocking
|
||||
created). `delete(filter)` deletes matching rows honoring per-filter limits, with the F10
|
||||
empty-filter no-op guard. Neither creates re-insert blocking — only stored kind-5/kind-62
|
||||
events do that.
|
||||
|
||||
---
|
||||
|
||||
## NIP-45 count (STORE-C)
|
||||
|
||||
**STORE-C01 — count = size of the deduped match set, honoring per-filter limits.** Single
|
||||
filter: `COUNT(*)` over that filter's row-id subquery (including its `LIMIT`, so
|
||||
`count(Filter(kinds=…, limit=10))` is at most 10). Multiple filters: branches are `UNION`ed
|
||||
(dedup) **before** counting — an event matching several filters counts once. FTS-off + search
|
||||
term → 0 (F-series search rules apply).
|
||||
|
||||
---
|
||||
|
||||
## NIP-50 search inside the store (STORE-S)
|
||||
|
||||
The indexing surface (which kinds are searchable, what text they contribute) is the
|
||||
`searchable-events` skill; these rules are the store's query-side contract.
|
||||
|
||||
**STORE-S01 — extension stripping at the store boundary.** Every filter-accepting method runs
|
||||
`strippingSearchExtensions()`: NIP-50 `key:value` tokens (`include:spam`, `domain:…`, …) are
|
||||
removed before FTS. Unsupported extensions are **ignored, never matched as literal text and
|
||||
never match-nothing** — an extensions-only search collapses to an unconstrained query. Stores
|
||||
that *do* implement extensions receive the raw string through the relay layer and parse it with
|
||||
`nip50Search.SearchQuery.parse` (see the `IEventStore` KDoc).
|
||||
|
||||
**STORE-S02 — relevance ordering.** Search results order by FTS5 `bm25` rank (best match
|
||||
first), with `created_at DESC` only as tiebreak; the `LIMIT` keeps the most *relevant* N, not
|
||||
the newest N. A multi-filter REQ is relevance-ordered only when **every** filter carries a
|
||||
search term (best/min rank per event across branches); mixing search and non-search filters
|
||||
falls back to `created_at DESC`.
|
||||
|
||||
**STORE-S03 — search combines by AND with the structural parts** (ids/authors/kinds/tags/
|
||||
since/until) of the same filter — an FTS `MATCH` join on top of the normal conditions.
|
||||
|
||||
**STORE-S04 — search grammar is SQLite FTS5 `MATCH`.** The raw (post-strip) string is passed to
|
||||
FTS5, so implicit-AND terms, `"phrase queries"`, `OR`, and `prefix*` follow FTS5 semantics.
|
||||
Tokenization details live in `FullTextSearchModule` (see `searchable-events`).
|
||||
|
||||
**STORE-S05 — FTS off.** With `IndexingStrategy.indexFullTextSearch = false`: a filter with a
|
||||
non-empty search term matches **nothing** (query/count/delete alike); an empty-string search
|
||||
imposes no constraint. Everything else is unchanged.
|
||||
|
||||
**STORE-S06 — deferred FTS.** Relays may set `deferFullTextSearchIndexing = true` (geode does):
|
||||
tokenization moves off the insert path to a watermark-driven catch-up
|
||||
(`needsFtsCatchUp`/`ftsCatchUp`), and search queries drain the backlog first — so NIP-50
|
||||
results are exactly as fresh as the synchronous path.
|
||||
|
||||
---
|
||||
|
||||
## Negentropy / NIP-77 (STORE-N)
|
||||
|
||||
**STORE-N01 —** `snapshotIdsForNegentropy(filters)` returns `(created_at, id)` pairs under the
|
||||
**same filter semantics as `query`** (per-filter limits included, multi-filter dedup), order
|
||||
unspecified (negentropy re-sorts). `maxEntries` returns up to `maxEntries + 1` as an overflow
|
||||
sentinel. `liveNegentropySnapshot` serves full-corpus NEG-OPENs from an in-memory index when
|
||||
`maintainLiveNegentropyIndex` is on; the delta plumbing in `SQLiteEventStore` keeps it exact
|
||||
across replaceable displacement, kind-5s, and vanish (invalidate-and-rebuild for the
|
||||
non-itemizable cases).
|
||||
|
||||
---
|
||||
|
||||
## Configuration presets
|
||||
|
||||
- **Client default** (`DefaultIndexingStrategy()`): FTS on (synchronous), optional indexes off,
|
||||
`useAndIndexIdOnOrderBy` off, no live negentropy index.
|
||||
- **Relay preset** (geode's `relayIndexingStrategy()`): adds created_at-alone, pubkey-alone and
|
||||
tag+kind+pubkey indexes, defers FTS, maintains the live negentropy index — still leaves
|
||||
`useAndIndexIdOnOrderBy` off.
|
||||
- Flag-gated indexes are runtime config, not schema: flipping one on an existing DB builds the
|
||||
index on next open (`ensureOptionalIndexes`), no migration.
|
||||
|
||||
## For parity implementers
|
||||
|
||||
- Treat the rule ids above as the vocabulary for divergence notes
|
||||
(e.g. "diverges from STORE-D05: we no-op deletion-of-a-deletion").
|
||||
- The commonTest suites are the executable spec; `FsParityTest` shows the in-repo pattern for
|
||||
holding a second engine to it.
|
||||
- Remember F06 (hash-based tag matching) and F08 (unordered same-second ties by default) when
|
||||
diffing results byte-for-byte — both are places where a "divergence" may be the reference's
|
||||
own slack, not your bug.
|
||||
|
||||
## Semantics changelog
|
||||
|
||||
Add one line per behavior change, newest first: `YYYY-MM-DD <short sha> <rule id> — what changed`.
|
||||
|
||||
- 2026-08-04 (baseline) — rules F01–F13, W01–W08, D01–D08, C01, S01–S06, N01 written from the
|
||||
code at the time this skill was introduced. Changes before this date are not itemized;
|
||||
archaeology starts at `git log` on `nip01Core/store/`.
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
name: nip85-trusted-assertions
|
||||
description: The NIP-85 trusted-assertions model in Quartz (`nip85TrustedAssertions/`) — kind 10040 trust-provider lists, kind 30382 contact cards / user assertions, 30383 event assertions, 30384 addressable assertions, 30385 external-id assertions. Use when building or parsing these events, working with the typed tags (RankTag, HopsTag, FollowerCountTag, ServiceProviderTag/ServiceType, …), wiring a consumer that resolves a 10040 provider entry to the 30382s it signs, ranking on assertion values, or touching the GrapeRank publisher, contact-card nicknames, or the trust projection of an external store.
|
||||
---
|
||||
|
||||
# NIP-85 Trusted Assertions — the Quartz model
|
||||
|
||||
Package: `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip85TrustedAssertions/`.
|
||||
NIP-85 is still an evolving spec; **this package is the operative definition** of what
|
||||
Amethyst-family software writes and reads. This skill states the model (who signs what about
|
||||
whom), the exact kind/d-tag/tag vocabulary, and what consumers may — and may not — assume.
|
||||
|
||||
## The model in one paragraph
|
||||
|
||||
An **assertion is signed by the asserting party** (a trust provider service, or the user
|
||||
themself) **about a subject named in the d-tag**. All assertion kinds are addressable, so
|
||||
"latest card by provider P about subject S" is just the addressable coordinate
|
||||
`(kind, P, S)` and supersession is standard NIP-01 latest-wins. Discovery is the observer's
|
||||
**kind 10040 list**: each entry says *"for metric M on kind K, I trust provider P — fetch
|
||||
their assertions at relay R"*. Quartz enforces none of this cryptographically beyond normal
|
||||
event signatures; the 10040→assertion link is **consumer-side convention** (see
|
||||
"Authorization" below).
|
||||
|
||||
## Kind map
|
||||
|
||||
| Kind | Class | Kind class | d-tag = the subject | Content |
|
||||
|---|---|---|---|---|
|
||||
| 10040 | `list/TrustProviderListEvent` | replaceable | *(none — always `""`)* | NIP-44 private provider entries (optional) |
|
||||
| 30382 | `users/ContactCardEvent` | addressable | **target user's pubkey** (hex) | NIP-44 private tags (petname/summary/emoji) |
|
||||
| 30383 | `events/EventAssertionEvent` | addressable | **target event id** (hex) | `""` |
|
||||
| 30384 | `addressables/AddressableAssertionEvent` | addressable | **target coordinate** `kind:pubkey:dtag` | `""` |
|
||||
| 30385 | `externalIds/ExternalIdAssertionEvent` | addressable | **external identifier** (e.g. `isbn:978-0-13-468599-1`) | `""` |
|
||||
|
||||
Addresses: `ContactCardEvent.createAddress(owner, target)` → `Address(30382, owner, target)`
|
||||
(owner = signer, target = subject). `TrustProviderListEvent.createAddress(pubKey)` uses
|
||||
`FIXED_D_TAG = ""`. `AssertionEventTest.eventKindsAreCorrect` pins all five numbers.
|
||||
|
||||
`ContactCardEvent` is also a `SearchableEvent` — it indexes only the **public** petname/summary
|
||||
tags plus topics; the encrypted card content is intentionally never indexed.
|
||||
|
||||
## The 10040 provider entry (`ServiceProviderTag` / `ServiceType`)
|
||||
|
||||
There is **no fixed tag name**: `tag[0]` *is* the service string.
|
||||
|
||||
```json
|
||||
["30382:rank", "<provider pubkey, 64 hex>", "wss://nip85.brainstorm.world"]
|
||||
```
|
||||
|
||||
- `ServiceType(kind, type)` parses/renders `"<kind>:<type>"` — kind must be an int, the first
|
||||
`:` splits, colons in the remainder stay in `type`. `ServiceType.isOfKind` is the
|
||||
allocation-free prefix check.
|
||||
- `ServiceProviderTag.parse` requires ≥3 elements, non-empty service, 64-char pubkey
|
||||
(length-only check), and a **normalizable relay URL** (`RelayUrlNormalizer.normalizeOrNull`) —
|
||||
entries failing any check are silently dropped, which is what keeps foreign tags like
|
||||
`["client","nostria"]` out (regression-tested in `ServiceTypeParserTest`).
|
||||
- Entries may be **public** (tag array) or **private** (NIP-44 content); `create`/`add` take
|
||||
`isPrivate`. `remove` always needs decryption and strips from both sides by parsed-value
|
||||
equality.
|
||||
- `object ProviderTypes` (`list/tags/ServiceType.kt`) enumerates the *known* service types —
|
||||
`30382:rank`, `30382:followers`, `30382:first_created_at`, per-metric `30383:*`/`30384:*`/
|
||||
`30385:*`, etc. It is an **open vocabulary**: real 10040s in the wild (see the fiatjaf →
|
||||
brainstorm fixture in `commonTest/.../nip85TrustedAssertions/ServiceParser.kt`) carry types
|
||||
Quartz doesn't enumerate (`30382:personalizedGrapeRank_influence`, `30382:hops`,
|
||||
`30382:verifiedFollowersCount`, …). Parse any `kind:type`; special-case only what you rank on.
|
||||
|
||||
## Authorization — what a consumer may assume
|
||||
|
||||
- **A 30382 (or 30383/…) is meaningful to an observer only if its author is listed in the
|
||||
observer's 10040 for a matching service type.** Quartz does not enforce this; the consuming
|
||||
code does. The in-repo pattern is `commons/.../model/nip85TrustedAssertions/UserCardsCache.kt`:
|
||||
`rankFlow(trustProviderList)` picks the received card whose **author pubkey equals the
|
||||
provider entry's pubkey** and reads `rank()` from it. Assertions from unlisted signers are
|
||||
simply ignored for trust purposes (they may still be stored; dropping them — as an external
|
||||
store's orphan sweep does — is a legitimate storage policy, not a protocol rule).
|
||||
- What an entry authorizes is scoped by its `ServiceType`: `30382:rank` authorizes that
|
||||
provider's user-rank cards, nothing else. Amethyst models this as one provider slot per
|
||||
metric (`liveUserRankProvider`, `liveUserFollowerCount` in
|
||||
`amethyst/.../model/trustedAssertions/TrustProviderListState.kt`).
|
||||
- **Multi-provider combination is unprescribed.** When two listed providers assert different
|
||||
ranks, there is no spec'd merge; Amethyst avoids the question by selecting one provider per
|
||||
metric slot. Consumers choose their own policy — document it.
|
||||
- The relay URL in the entry is a **fetch hint, and it is honored**:
|
||||
`amethyst/.../UserCardsSubAssembler.kt` subscribes for cards at the provider's declared relay
|
||||
(`kinds=[30382], authors=[provider], #d=[targets]`).
|
||||
|
||||
### The dual use of kind 30382
|
||||
|
||||
The same kind serves two roles, distinguished **by author**:
|
||||
|
||||
1. **Provider WoT cards** — signed by a trust provider; public metric tags (`rank`,
|
||||
`followers`, `hops`, …); this is what 10040 discovery points at.
|
||||
2. **The account's own contact cards (nicknames, NIP-81-style)** — signed by the account,
|
||||
one per target user. The petname, summary, and their NIP-30 emoji mappings **always live in
|
||||
the NIP-44 encrypted content, never in public tags** (`ContactCardEvent.build`/
|
||||
`updatePetNameAndSummary` strip stray public copies; asserted by `ContactCardPetNameTest`).
|
||||
`commons/.../ContactCardsState.kt` keys everything on `author == account` and ignores
|
||||
provider cards.
|
||||
|
||||
## Tag vocabulary and value semantics
|
||||
|
||||
All tag classes share one shape: `TAG_NAME` + `parse(tag)` (null on wrong name/empty/non-numeric
|
||||
value — a bad tag is *dropped*, never an error) + `assemble(value)` → `[name, value.toString()]`.
|
||||
**A missing tag means "unknown" (`null` accessor), never zero.** There is deliberately no range
|
||||
validation (rank isn't clamped, hours aren't checked against 0–23, counts may be negative) —
|
||||
consumers must defend.
|
||||
|
||||
**On 30382** (`users/tags/`, accessors on `ContactCardEvent` and as `TagArray` extensions in
|
||||
`users/TagArrayExt.kt` so they also work on decrypted private arrays):
|
||||
|
||||
| Tag name | Accessor | Type | Semantics |
|
||||
|---|---|---|---|
|
||||
| `rank` | `rank()` | Int | Provider-relative score; higher is better. GrapeRank publishes `round(score × 100)` (so 0–100 in practice), but nothing enforces a scale — treat it as comparable only *within one provider*. |
|
||||
| `followers` | `followerCount()` | Int | Follower count as the provider computes it (cumulative, provider-defined). |
|
||||
| `hops` | `hops()` | Int | Shortest follow-path length **from the observer the provider computed for** to the subject (1 = directly followed). Mirrors Brainstorm GrapeRank's `hops`. The only tag with KDoc. |
|
||||
| `first_created_at` | `firstCreatedAt()` | Long | Unix seconds of subject's earliest known event. |
|
||||
| `post_cnt` / `reply_cnt` / `reactions_cnt` | `postCount()` etc. | Int | Activity counts. |
|
||||
| `zap_amt_recd` / `zap_amt_sent` | `zapAmountReceived()`/`…Sent()` | Long | Sats. |
|
||||
| `zap_cnt_recd` / `zap_cnt_sent` | `zapCountReceived()`/`…Sent()` | Int | Counts. |
|
||||
| `zap_avg_amt_day_recd` / `zap_avg_amt_day_sent` | `zapAvgAmountDay…()` | Long | Sats/day averages. |
|
||||
| `reports_cnt_recd` / `reports_cnt_sent` | `reportsCount…()` | Int | NIP-56 report counts. |
|
||||
| `t` (repeatable) | `topics()` | List\<String> | Subject's topics/interests. |
|
||||
| `active_hours_start` / `active_hours_end` | `activeHours…()` | Int | Hour-of-day; **no timezone is specified in code** — treat as provider-defined (UTC in practice) and unclamped. |
|
||||
| `petname` / `summary` | `petName()`/`summary()` | String | Nickname fields — conventionally private (see dual use above). |
|
||||
|
||||
**On 30383/30384** (`tags/`, shared): `rank`, `comment_cnt`, `quote_cnt`, `repost_cnt`,
|
||||
`reaction_cnt`, `zap_cnt` (Int) and `zap_amount` (Long, sats).
|
||||
**On 30385**: only `rank`, `comment_cnt`, `reaction_cnt`.
|
||||
|
||||
## Building and parsing (use the typed helpers, not raw `arrayOf`)
|
||||
|
||||
```kotlin
|
||||
// Provider list: declare a rank provider (this is what `amy graperank register` does)
|
||||
val tag = ServiceProviderTag(ProviderTypes.rank, providerPubkeyHex, relayUrl)
|
||||
val list = TrustProviderListEvent.create(tag, isPrivate = false, signer)
|
||||
// or append to an existing one:
|
||||
val updated = TrustProviderListEvent.add(existing, tag, isPrivate = false, signer)
|
||||
val providers: List<ServiceProviderTag> = updated.serviceProviders() // public
|
||||
val private = updated.privateTags(signer)?.serviceProviders() // private side
|
||||
|
||||
// Provider-style contact card (public metrics) — the GrapeRankPublisher pattern:
|
||||
val card = ContactCardEvent.create(
|
||||
targetUser = subjectPubkey,
|
||||
signer = providerSigner,
|
||||
publicInitializer = {
|
||||
rank(87)
|
||||
followers(1234)
|
||||
hops(2)
|
||||
},
|
||||
)
|
||||
card.aboutUser() // d-tag → subject pubkey
|
||||
card.rank() // 87
|
||||
|
||||
// Event assertion: unsigned template only (30383/84/85 have build(), no create())
|
||||
val template = EventAssertionEvent.build(targetEventId) {
|
||||
rank(12)
|
||||
reactionCount(40)
|
||||
zapAmount(2100)
|
||||
}
|
||||
val signed = signer.sign(template)
|
||||
```
|
||||
|
||||
## Worked end-to-end example
|
||||
|
||||
Observer `O` trusts provider `P` for user ranks (kind 10040, replaceable, by `O`):
|
||||
|
||||
```json
|
||||
{ "kind": 10040, "pubkey": "<O>",
|
||||
"tags": [
|
||||
["30382:rank", "<P>", "wss://nip85.brainstorm.world"],
|
||||
["30382:followers", "<P>", "wss://nip85.brainstorm.world"]
|
||||
],
|
||||
"content": "" }
|
||||
```
|
||||
|
||||
Provider `P` asserts about subject `S` (kind 30382, addressable at `30382:<P>:<S>`):
|
||||
|
||||
```json
|
||||
{ "kind": 30382, "pubkey": "<P>",
|
||||
"tags": [
|
||||
["d", "<S>"],
|
||||
["rank", "87"], ["followers", "1234"], ["hops", "2"]
|
||||
],
|
||||
"content": "" }
|
||||
```
|
||||
|
||||
`P` asserts about an event `E` (kind 30383, addressable at `30383:<P>:<E>`):
|
||||
|
||||
```json
|
||||
{ "kind": 30383, "pubkey": "<P>",
|
||||
"tags": [["d", "<E>"], ["rank", "12"], ["reaction_cnt", "40"], ["zap_amount", "2100"]],
|
||||
"content": "" }
|
||||
```
|
||||
|
||||
Consumption chain: read `O`'s 10040 → entry matching `ServiceType(30382, "rank")` → subscribe
|
||||
`{kinds:[30382], authors:["<P>"], "#d":["<S>", …]}` at the hinted relay → newest card per
|
||||
address wins → `rank()`.
|
||||
|
||||
Literal fixtures: `quartz/src/commonTest/.../nip85TrustedAssertions/ServiceParser.kt` (a real
|
||||
10040 — fiatjaf's, pointing at the Brainstorm provider) and `AssertionEventTest.kt` (all four
|
||||
assertion kinds with every tag populated).
|
||||
|
||||
## Freshness / supersession
|
||||
|
||||
Assertions are addressable: **latest per `(kind, author, d-tag)` wins**; there is no expiry tag
|
||||
convention and **no prescribed refresh cadence** — staleness policy is the consumer's.
|
||||
Writers should avoid churn: `GrapeRankPublisher` re-signs a card only when
|
||||
`(rank, followers, hops)` actually changed, and retracts with a NIP-09 kind-5 carrying the
|
||||
card's `a`-tag (`30382:<provider>:<target>`).
|
||||
|
||||
## Stability notes (as of 2026-08)
|
||||
|
||||
- **Settled** (shipped consumers on both ends): the kind map; `ServiceProviderTag` entry shape;
|
||||
`rank`/`followers`/`hops` on 30382; petname/summary-in-encrypted-content; 10040 relay-hint
|
||||
consumption.
|
||||
- **Written but lightly consumed** (parse, but gate ranking features carefully): the activity/
|
||||
zap/report count tags, `active_hours_*` (no timezone semantics), 30383/30384/30385 (builders +
|
||||
tests exist; no in-repo publisher yet).
|
||||
- **Known warts**: `ServiceProviderTag.assemble(id: ServiceProviderTag)` infers `Array<Any>` —
|
||||
dead code, don't use it; `SummaryTag.assemble(ip:)`/`ActiveHours*Tag.assemble(count:)` params
|
||||
are misnamed; the tests live under `commonTest/.../experimental/nip85TrustedAssertions/`
|
||||
(stale path); `TrustProviderListEvent` extends the addressable base, so a stray on-wire `d`
|
||||
tag is reflected by `dTag()` even though the convention is `""`.
|
||||
|
||||
## Where it's consumed (reading list)
|
||||
|
||||
- **Publisher**: `quartz/.../experimental/graperank/GrapeRankPublisher.kt` (canonical 30382
|
||||
writer), `cli/.../graperank/` (`amy graperank register|unregister|providers|publish`).
|
||||
- **Client model**: `commons/.../model/nip85TrustedAssertions/` (`ContactCardsState`,
|
||||
`UserCardsCache`, `ContactCardDecryptionCache`, `TrustProviderListDecryptionCache`),
|
||||
`amethyst/.../model/trustedAssertions/TrustProviderListState.kt`.
|
||||
- **Relay plumbing**: `commons/.../relayClient/assemblers/ContactCardFilters.kt`,
|
||||
`amethyst/.../reqCommand/user/watchers/UserCardsSubAssembler.kt`.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
name: searchable-events
|
||||
description: The NIP-50 indexing surface of Quartz — the `SearchableEvent` interface, which event kinds are searchable, exactly what text each kind's `indexableContent()` contributes, how the SQLite/filesystem stores consume it, and the NIP-50 `SearchQuery` extension grammar plus `SearchRelayListEvent` (kind 10007). Use when making a kind searchable, changing what a kind indexes, diffing the searchable set at a Quartz version bump (external search engines mirror this table), debugging why an event is or isn't found by search, or working with search extensions (`include:spam`, `domain:`, …).
|
||||
---
|
||||
|
||||
# Searchable Events — the NIP-50 indexing surface
|
||||
|
||||
## The contract
|
||||
|
||||
`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip50Search/SearchableEvent.kt`:
|
||||
|
||||
```kotlin
|
||||
interface SearchableEvent {
|
||||
fun indexableContent(): String
|
||||
}
|
||||
```
|
||||
|
||||
One method; marker and extractor in one. An event kind is searchable **iff** its event class
|
||||
implements this interface **and** the class is wired into `EventFactory` (the stores probe
|
||||
searchability by kind through `EventFactory.create` — an unwired implementor is invisible).
|
||||
|
||||
Rules every implementation follows (keep them when adding one):
|
||||
|
||||
- **Plain text out.** Return the human-meaningful fields joined with `"\n"` (a handful of
|
||||
metadata-ish kinds use `" "`); no markup stripping is performed — markdown/asciidoc content
|
||||
goes in raw, JSON-content kinds (kind 0 metadata, marketplace stalls, channel info) **parse
|
||||
first and join the extracted fields**, never the raw JSON.
|
||||
- **Never throw, never null.** There is no defensive wrapper at any call site; a throw aborts
|
||||
the insert transaction. Parsed-JSON implementations use `?.let { … } ?: ""`.
|
||||
- **Only public data.** Encrypted content stays out (e.g. kind 30382 contact cards index only
|
||||
the public petname/summary/topics, never the NIP-44 payload).
|
||||
- Typical shapes: `content` alone (~33 kinds); `listOfNotNull(title(), content)`;
|
||||
`listOfNotNull(title(), summary(), content)`; lists index `title() + description()`.
|
||||
|
||||
## The full kind table
|
||||
|
||||
**`references/searchable-kinds.md`** in this skill holds the authoritative table — every
|
||||
implementor with its kind number, class, and the exact `indexableContent()` expression
|
||||
(126 concrete classes / 129 kind values as of 2026-08). Diff that file at a version bump to
|
||||
answer "did the searchable set or any kind's indexed text change?".
|
||||
|
||||
Notables that surprise people:
|
||||
|
||||
- **Kind 9735 (zap receipt) indexes the embedded zap request's content**
|
||||
(`zapRequest?.content.orEmpty()`) — receipts are searchable by the zapper's comment.
|
||||
- **Kind 0 / 31990** index many profile fields space-joined (name, about, nip05, lud16,
|
||||
website, picture URL, …).
|
||||
- **Kind 30063 is claimed twice** (`ReleaseArtifactSetEvent` in nip51Lists and the experimental
|
||||
`SoftwareReleaseEvent`); `EventFactory` resolves 30063 to `ReleaseArtifactSetEvent`, so
|
||||
`title()\ndescription()` is what actually gets indexed — `SoftwareReleaseEvent.indexableContent()`
|
||||
is dead on the store path.
|
||||
- Poll kinds (1068, 6969) append each option label on its own line.
|
||||
|
||||
## MANDATORY maintenance when you touch this surface
|
||||
|
||||
Adding `SearchableEvent` to a kind, removing it, or changing any `indexableContent()` body:
|
||||
|
||||
1. **Update `references/searchable-kinds.md`** in the same PR (external search engines — e.g.
|
||||
the Vespa-backed store's `SearchExtractors` — mirror this table at pin bumps; a silent
|
||||
change ships them stale search results).
|
||||
2. **Remember existing databases don't reindex themselves.** Old rows keep their old (or
|
||||
missing) FTS text until `IEventStore.reindexFullTextSearch()` runs — the KDoc on that method
|
||||
is the contract. App-side, schedule the resumable overload after shipping such a change.
|
||||
3. New implementors must be **registered in `EventFactory`** or the reindex scan and kind
|
||||
pre-filter (`FullTextSearchModule.isSearchableKind`) will never see them.
|
||||
|
||||
Eligibility policy: a kind becomes searchable when it carries human-authored, human-meaningful
|
||||
text (titles, bodies, names, descriptions). Pure-machine kinds (reactions, follow lists, zaps
|
||||
minus their comment, relay lists) stay out to keep the index small.
|
||||
|
||||
## How the stores consume it
|
||||
|
||||
**SQLite** (`nip01Core/store/sqlite/FullTextSearchModule.kt`):
|
||||
`CREATE VIRTUAL TABLE event_fts USING fts5(content, content='', contentless_delete=1)` —
|
||||
contentless, `rowid` = `event_headers.row_id`, an `AFTER DELETE` trigger keeps it in sync. On
|
||||
insert (when FTS is on and not deferred): `if (event is SearchableEvent)` → bind
|
||||
`event.indexableContent()` — the only method ever called. Tokenization is entirely SQLite's
|
||||
default FTS5 `unicode61`; queries are always a bound `event_fts MATCH ?` (never concatenated),
|
||||
ordered by bm25 `rank` then `created_at DESC`. Query-side semantics (relevance ordering,
|
||||
extension stripping, FTS-off behavior, deferred catch-up) are rules STORE-S01…S06 in the
|
||||
`event-store-semantics` skill.
|
||||
|
||||
**Filesystem store** (`jvmMain/.../store/fs/FsIndexer.kt` + `FsSearchTokenizer.kt`): tokenizes
|
||||
`indexableContent()` itself, approximating `unicode61` (split on non-letter/digit, lowercase);
|
||||
the same tokenizer runs on queries so drift cancels.
|
||||
|
||||
## NIP-50 client side
|
||||
|
||||
**`SearchQuery`** (`nip50Search/SearchQuery.kt`) — typed parse of the `search` filter string
|
||||
into `terms` + `extensions`. A whitespace token is an extension iff it looks like
|
||||
`lowercasekey:value` (the value not starting with `//`, so URLs stay free text); duplicate keys
|
||||
keep the last; unknown extensions are preserved (`extension(key)`). Typed accessors:
|
||||
`includeSpam`, `domain`, `language`, `sentiment`, `nsfw`. `stripExtensions()` /
|
||||
`Filter.strippingSearchExtensions()` is the bridge the built-in stores use — unsupported
|
||||
extensions are **ignored** (NIP-50), so an extensions-only search collapses to an unconstrained
|
||||
query, never match-nothing. A server-side store that implements its own extensions
|
||||
(`observer:`, `sort:rank`, …) receives the raw string (see the `IEventStore` KDoc) and should
|
||||
parse with `SearchQuery.parse` so its syntax stays compatible with what clients send.
|
||||
|
||||
**`SearchRelayListEvent`** — **kind 10007**, the user's search-relay list (NIP-51-style, public
|
||||
tags + NIP-44 private tags; *not* a `SearchableEvent` itself). Client consumption:
|
||||
`commons/.../actions/SearchActions.kt`, bootstrap defaults in
|
||||
`commons/.../account/AccountBootstrapEvents.kt`.
|
||||
|
||||
Don't confuse it with `commons/.../commons/search/SearchQuery.kt` — an app-level local-feed
|
||||
query model (authors/kinds/hashtags/or-terms), unrelated to the NIP-50 wire string.
|
||||
|
||||
## Tests (executable spec)
|
||||
|
||||
- `commonTest/.../nip50Search/SearchQueryTest.kt` — the extension grammar, token by token.
|
||||
- `commonTest/.../store/sqlite/SearchTest.kt` — per-kind indexing (kind 0 profile fields,
|
||||
40/41 channel JSON, 31924/30617), extension-token ignoring, reindex/resumable-reindex,
|
||||
FTS cleanup on replaceable rotation.
|
||||
- `commonTest/.../store/sqlite/SearchRelevanceOrderTest.kt` — bm25-before-recency ordering,
|
||||
limit-after-score, multi-filter rank union.
|
||||
- `commonTest/.../store/sqlite/NoFullTextSearchTest.kt` — FTS-off contract.
|
||||
- `jvmTest/.../store/fs/FsSearchTest.kt` — tokenizer parity for the filesystem store.
|
||||
@@ -0,0 +1,166 @@
|
||||
# Searchable kinds — the authoritative implementor table
|
||||
|
||||
Every concrete `SearchableEvent` implementor in Quartz, with the exact `indexableContent()`
|
||||
expression. **Update this file in the same PR as any change to the searchable set or to an
|
||||
`indexableContent()` body** (see SKILL.md). Verified against the code 2026-08-04.
|
||||
|
||||
Counts: 126 concrete classes covering 129 kind values (`GitStatusEvent` spans 4 kinds;
|
||||
kind 30063 has a collision — see the footnote). File paths are under
|
||||
`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/`.
|
||||
|
||||
Separator legend: **NL** = `joinToString("\n")`, **SP** = `joinToString(" ")`.
|
||||
|
||||
| Kind | Class | Package | `indexableContent()` |
|
||||
|---|---|---|---|
|
||||
| 0 | MetadataEvent | nip01Core/metadata | `contactMetaData()?.let { listOfNotNull(it.name, it.displayName, it.about, it.nip05, it.lud06, it.lud16, it.website, it.picture, it.banner).joinToString(" ") } ?: ""` (SP) |
|
||||
| 1 | TextNoteEvent | nip10Notes | `listOfNotNull(subject(), content)` NL |
|
||||
| 9 | ChatEvent | nipC7Chats | `content` |
|
||||
| 11 | ThreadEvent | nip7DThreads | `listOfNotNull(title(), content)` NL |
|
||||
| 14 | ChatMessageEvent | nip17Dm/messages | `content` |
|
||||
| 20 | PictureEvent | nip68Picture | `listOfNotNull(title(), content)` NL |
|
||||
| 21 | VideoNormalEvent | nip71Video | inherited `RegularVideoEvent`: `listOfNotNull(title(), content)` NL |
|
||||
| 22 | VideoShortEvent | nip71Video | inherited `RegularVideoEvent`: `listOfNotNull(title(), content)` NL |
|
||||
| 24 | PublicMessageEvent | nipA4PublicMessages | `content` |
|
||||
| 40 | ChannelCreateEvent | nip28PublicChat/admin | `channelInfo().let { listOfNotNull(it.name, it.about, it.picture).joinToString(" ") }` (SP) |
|
||||
| 41 | ChannelMetadataEvent | nip28PublicChat/admin | same as kind 40 (SP) |
|
||||
| 42 | ChannelMessageEvent | nip28PublicChat/message | `content` |
|
||||
| 54 | PodcastEpisodeEvent | nipF4Podcasts/episode | `listOfNotNull(title(), description(), content)` NL |
|
||||
| 1010 | TextNoteModificationEvent | experimental/edits | `listOfNotNull(content, summary())` NL (content first) |
|
||||
| 1063 | FileHeaderEvent | nip94FileMetadata | `listOfNotNull(summary(), content)` NL |
|
||||
| 1065 | FileStorageHeaderEvent | experimental/nip95/header | `listOfNotNull(summary())` NL |
|
||||
| 1068 | PollEvent | nip88Polls/poll | `buildString { append(content); options().forEach { append('\n').append(it.label) } }` |
|
||||
| 1111 | CommentEvent | nip22Comments | `(listOf(content) + tags.hashtags())` NL |
|
||||
| 1163 | ProfileGalleryEntryEvent | experimental/profileGallery | `listOfNotNull(summary())` NL |
|
||||
| 1301 | WorkoutRecordEvent | experimental/fitness/workout | `listOfNotNull(title(), content)` NL |
|
||||
| 1311 | LiveActivitiesChatMessageEvent | nip53LiveActivities/chat | `(listOf(content) + tags.hashtags())` NL |
|
||||
| 1312 | LiveActivitiesRaidEvent | nip53LiveActivities/raid | `content` |
|
||||
| 1313 | LiveActivitiesClipEvent | nip53LiveActivities/clip | `listOfNotNull(title(), content)` NL |
|
||||
| 1315 | RoadEventReportEvent | experimental/roadstr/report | `content` |
|
||||
| 1337 | CodeSnippetEvent | nipC0CodeSnippets | `listOfNotNull(snippetName(), snippetDescription(), content)` NL |
|
||||
| 1617 | GitPatchEvent | nip34Git/patch | `content` |
|
||||
| 1618 | GitPullRequestEvent | nip34Git/pr | `listOfNotNull(subject(), content)` NL |
|
||||
| 1621 | GitIssueEvent | nip34Git/issue | `listOfNotNull(subject(), content)` NL |
|
||||
| 1622 | GitReplyEvent | nip34Git/reply | `content` |
|
||||
| 1630–1633 | GitStatusEvent | nip34Git/status | `content` (open/applied/closed/draft) |
|
||||
| 1808 | AudioHeaderEvent | experimental/audio/header | `content` |
|
||||
| 1985 | LabelEvent | nip32Labeling | `(listOf(content) + labels().map { it.label }).filter { it.isNotEmpty() }` NL |
|
||||
| 2003 | TorrentEvent | nip35Torrents | `listOfNotNull(title(), content)` NL |
|
||||
| 2004 | TorrentCommentEvent | nip35Torrents | `content` |
|
||||
| 2473 | BirdDetectionEvent | experimental/birdstar | `listOfNotNull(summary(), speciesName())` NL |
|
||||
| 3302 | ConcordChatEditEvent | concord/cord03Channels | `content` |
|
||||
| 5050 | NIP90TextGenerationRequestEvent | nip90Dvms/textGeneration | `inputs().filter { it.type == "prompt" \|\| it.type == "text" }.joinToString(" ") { it.value }` (SP) |
|
||||
| 5100 | NIP90ImageGenerationRequestEvent | nip90Dvms/imageGeneration | `listOfNotNull(prompt(), negativePrompt()).joinToString(" ")` (SP) |
|
||||
| 5129 | NappletSnapshotEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
|
||||
| 5250 | NIP90TextToSpeechRequestEvent | nip90Dvms/textToSpeech | `text() ?: ""` |
|
||||
| 5302 | NIP90ContentSearchRequestEvent | nip90Dvms/contentSearch | `searchQuery() ?: ""` |
|
||||
| 5303 | NIP90PeopleSearchRequestEvent | nip90Dvms/peopleSearch | `searchQuery() ?: ""` |
|
||||
| 6969 | ZapPollEvent | experimental/zapPolls | `buildString { append(content); pollOptionsArray().forEach { append('\n').append(it.descriptor) } }` |
|
||||
| 8333 | OnchainZapEvent | nipBCOnchainZaps/zap | `content` |
|
||||
| 9002 | EditMetadataEvent | nip29RelayGroups/moderation | `(listOfNotNull(name(), about()) + hashtags())` NL |
|
||||
| 9041 | GoalEvent | nip75ZapGoals | `listOfNotNull(summary(), content)` NL |
|
||||
| 9321 | NutzapEvent | nip61Nutzaps/nutzap | `content` |
|
||||
| 9734 | LnZapRequestEvent | nip57Zaps | `content` |
|
||||
| 9735 | LnZapEvent | nip57Zaps | `zapRequest?.content.orEmpty()` — indexes the **embedded 9734's** content |
|
||||
| 9736 | Bolt12ZapEvent | nipB1Bolt12Zaps/zap | `content` |
|
||||
| 9737 | Bolt12ZapIntentEvent | nipB1Bolt12Zaps/intent | `content` |
|
||||
| 9802 | HighlightEvent | nip84Highlights | `listOfNotNull(comment(), context(), content)` NL |
|
||||
| 10003 | BookmarkListEvent | nip51Lists/bookmarkList | `listOfNotNull(title())` NL |
|
||||
| 10100 | AgentProfileEvent | buzz/agentProfiles | `profileOrNull()?.let { listOfNotNull(it.name, it.displayName).joinToString("\n") } ?: ""` |
|
||||
| 10154 | PodcastMetadataEvent | nipF4Podcasts/metadata | `listOfNotNull(title(), description())` NL |
|
||||
| 11871 | AttestorProficiencyEvent | experimental/attestations/proficiency | `listOfNotNull(description())` NL |
|
||||
| 12473 | BirdexEvent | experimental/birdstar | `(listOfNotNull(summary()) + speciesNames())` NL |
|
||||
| 15128 | RootSiteEvent | nip5aStaticWebsites | `listOfNotNull(title(), description())` NL |
|
||||
| 15129 | RootNappletEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
|
||||
| 30000 | PeopleListEvent | nip51Lists/peopleList | `listOfNotNull(titleOrName(), description())` NL |
|
||||
| 30001 | OldBookmarkListEvent | nip51Lists/bookmarkList | `listOfNotNull(title())` NL |
|
||||
| 30002 | RelaySetEvent | nip51Lists/relaySets | `listOfNotNull(title(), description())` NL |
|
||||
| 30003 | LabeledBookmarkListEvent | nip51Lists/labeledBookmarkList | `listOfNotNull(titleOrName(), description())` NL |
|
||||
| 30004 | ArticleCurationSetEvent | nip51Lists/articleCurationSet | `listOfNotNull(title(), description())` NL |
|
||||
| 30005 | VideoCurationSetEvent | nip51Lists/videoCurationSet | `listOfNotNull(title(), description())` NL |
|
||||
| 30006 | PictureCurationSetEvent | nip51Lists/pictureCurationSet | `listOfNotNull(title(), description())` NL |
|
||||
| 30009 | BadgeDefinitionEvent | nip58Badges/definition | `listOfNotNull(name(), description(), content)` NL |
|
||||
| 30015 | InterestSetEvent | nip51Lists/interestSet | `(listOfNotNull(title(), description()) + publicHashtags())` NL |
|
||||
| 30017 | StallEvent | nip15Marketplace/stall | `stallData()?.let { listOfNotNull(it.name, it.description).joinToString("\n") } ?: ""` |
|
||||
| 30018 | ProductEvent | nip15Marketplace/product | `productData()?.let { (listOfNotNull(it.name, it.description) + categories()).joinToString("\n") } ?: ""` |
|
||||
| 30019 | MarketplaceEvent | nip15Marketplace/marketplace | `marketplaceData()?.let { listOfNotNull(it.name, it.about).joinToString("\n") } ?: ""` |
|
||||
| 30020 | AuctionEvent | nip15Marketplace/auction | `auctionData()?.let { (listOfNotNull(it.name, it.description) + tags.hashtags()).joinToString("\n") } ?: ""` |
|
||||
| 30023 | LongTextNoteEvent | nip23LongContent | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 30030 | EmojiPackEvent | nip30CustomEmoji/pack | `listOfNotNull(titleOrName(), description(), content)` NL |
|
||||
| 30054 | Podcasting20EpisodeEvent | nipXXPodcasting20/episode | `(listOfNotNull(title(), description(), content) + topics())` NL |
|
||||
| 30055 | Podcasting20TrailerEvent | nipXXPodcasting20/trailer | `listOfNotNull(title(), content)` NL |
|
||||
| 30063 | ReleaseArtifactSetEvent † | nip51Lists/releaseArtifactSet | `listOfNotNull(title(), description())` NL |
|
||||
| 30175 | PersonaEvent | buzz/apPersonas | `personaOrNull()?.let { listOfNotNull(it.displayName, it.systemPrompt).joinToString("\n") } ?: ""` |
|
||||
| 30176 | TeamEvent | buzz/teams | `teamOrNull()?.let { listOfNotNull(it.name, it.description, it.instructions).joinToString("\n") } ?: ""` |
|
||||
| 30177 | ManagedAgentEvent | buzz/managedAgents | `agentOrNull()?.let { listOfNotNull(it.name, it.systemPrompt).joinToString("\n") } ?: ""` |
|
||||
| 30267 | AppCurationSetEvent | nip51Lists/appCurationSet | `listOfNotNull(title(), description())` NL |
|
||||
| 30296 | InteractiveStoryPrologueEvent | experimental/interactiveStories | inherited base: `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 30297 | InteractiveStorySceneEvent | experimental/interactiveStories | inherited base: `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 30311 | LiveActivitiesEvent | nip53LiveActivities/streaming | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 30312 | MeetingSpaceEvent | nip53LiveActivities/meetingSpaces | `listOfNotNull(room(), summary(), content)` NL |
|
||||
| 30313 | MeetingRoomEvent | nip53LiveActivities/meetingSpaces | `listOfNotNull(title(), summary())` NL |
|
||||
| 30315 | StatusEvent | nip38UserStatus | `content` |
|
||||
| 30382 | ContactCardEvent | nip85TrustedAssertions/users | `(listOfNotNull(petName(), summary()) + topics())` NL — public tags only, never the NIP-44 content |
|
||||
| 30402 | ClassifiedsEvent | nip99Classifieds | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 30617 | GitRepositoryEvent | nip34Git/repository | `listOfNotNull(name(), description(), content)` NL |
|
||||
| 30620 | WorkflowDefEvent | buzz/workflow | `listOfNotNull(name(), content)` NL |
|
||||
| 30817 | NipTextEvent | experimental/nipsOnNostr | `listOfNotNull(title(), content)` NL |
|
||||
| 30818 | WikiNoteEvent | nip54Wiki | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 31337 | AudioTrackEvent | experimental/audio/track | `listOfNotNull(subject())` NL |
|
||||
| 31871 | AttestationEvent | experimental/attestations/attestation | `content` |
|
||||
| 31872 | AttestationRequestEvent | experimental/attestations/request | `content` |
|
||||
| 31873 | AttestorRecommendationEvent | experimental/attestations/recommendation | `listOfNotNull(description())` NL |
|
||||
| 31890 | FeedDefinitionEvent | feedDefinition | `title().orEmpty()` |
|
||||
| 31922 | CalendarDateSlotEvent | nip52Calendar/appt/day | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 31923 | CalendarTimeSlotEvent | nip52Calendar/appt/time | `listOfNotNull(title(), summary(), content)` NL |
|
||||
| 31924 | CalendarEvent | nip52Calendar/calendar | `listOfNotNull(title(), content)` NL |
|
||||
| 31925 | CalendarRSVPEvent | nip52Calendar/rsvp | `content` |
|
||||
| 31990 | AppDefinitionEvent | nip89AppHandlers/definition | `appMetaData()?.let { listOfNotNull(it.name, it.username, it.displayName, it.about, it.nip05, it.lud06, it.lud16, it.website, it.picture, it.banner, it.image).joinToString(" ") } ?: ""` (SP) |
|
||||
| 32267 | SoftwareApplicationEvent | experimental/nip82SoftwareApps/application | `listOfNotNull(name(), summary(), content)` NL |
|
||||
| 33401 | ExerciseTemplateEvent | experimental/fitness/workout | `listOfNotNull(title(), content)` NL |
|
||||
| 33863 | FundraiserEvent | experimental/agora | `listOfNotNull(title(), content)` NL |
|
||||
| 34139 | MusicPlaylistEvent | experimental/music/playlist | `listOfNotNull(title(), description(), content)` NL |
|
||||
| 34235 | VideoHorizontalEvent | nip71Video | inherited `AddressableVideoEvent`: `listOfNotNull(title(), content)` NL |
|
||||
| 34236 | VideoVerticalEvent | nip71Video | inherited `AddressableVideoEvent`: `listOfNotNull(title(), content)` NL |
|
||||
| 34550 | CommunityDefinitionEvent | nip72ModCommunities/definition | `listOfNotNull(name(), description(), rules(), content)` NL |
|
||||
| 35128 | NamedSiteEvent | nip5aStaticWebsites | `listOfNotNull(title(), description())` NL |
|
||||
| 35129 | NamedNappletEvent | nip5dNapplets | `listOfNotNull(title(), description())` NL |
|
||||
| 36787 | MusicTrackEvent | experimental/music/track | `listOfNotNull(title(), artist(), album(), content)` NL |
|
||||
| 38000 | MintRecommendationEvent | nip87Ecash/recommendation | `content` |
|
||||
| 38192 | Ps1SaveEvent | experimental/ps1saves | `listOfNotNull(summary(), saveTitle(), region(), filename())` NL |
|
||||
| 38383 | P2POrderEvent | nip69P2pOrderEvents | `(listOfNotNull(makerName(), currency()) + paymentMethods().orEmpty()).joinToString(" ")` (SP) |
|
||||
| 39000 | GroupMetadataEvent | nip29RelayGroups/metadata | `listOfNotNull(name(), about())` NL |
|
||||
| 39089 | FollowListEvent | nip51Lists/followList | `listOfNotNull(title(), description())` NL |
|
||||
| 39092 | MediaStarterPackEvent | nip51Lists/mediaStarterPack | `listOfNotNull(title(), description())` NL |
|
||||
| 39701 | WebBookmarkEvent | nipB0WebBookmarks | `listOfNotNull(title(), description())` NL |
|
||||
| 40002 | StreamMessageV2Event | buzz/stream | `content` |
|
||||
| 40100 | CanvasEvent | buzz/stream | `content` |
|
||||
| 45001 | ForumPostEvent | buzz/forum | `content` |
|
||||
| 45003 | ForumCommentEvent | buzz/forum | `content` |
|
||||
| 48106 | HuddleGuidelinesEvent | buzz/huddles | `content` |
|
||||
|
||||
† **Kind 30063 collision:** `experimental/nip82SoftwareApps/release/SoftwareReleaseEvent` also
|
||||
declares `KIND = 30063` and implements `SearchableEvent` (`content`), but `EventFactory` maps
|
||||
30063 to `ReleaseArtifactSetEvent`, so on every store path kind 30063 indexes
|
||||
`title()\ndescription()`. If the factory mapping ever changes, this table changes with it.
|
||||
|
||||
## Abstract bases (no kind of their own)
|
||||
|
||||
| Base class | Body | Concrete kinds |
|
||||
|---|---|---|
|
||||
| `InteractiveStoryBaseEvent` | `listOfNotNull(title(), summary(), content)` NL | 30296, 30297 |
|
||||
| `AddressableVideoEvent` | `listOfNotNull(title(), content)` NL | 34235, 34236 |
|
||||
| `RegularVideoEvent` | `listOfNotNull(title(), content)` NL | 21, 22 |
|
||||
|
||||
## How to regenerate / verify this table
|
||||
|
||||
```bash
|
||||
# All implementor files:
|
||||
grep -rln "override fun indexableContent" quartz/src/commonMain
|
||||
# For each, pair the KIND constant with the indexableContent() body.
|
||||
# Searchability on the store path additionally requires EventFactory registration:
|
||||
grep -n "<ClassName>" quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/utils/EventFactory.kt
|
||||
```
|
||||
|
||||
A CI-diffable snapshot test (assert the set of kinds whose `EventFactory` product implements
|
||||
`SearchableEvent` against a checked-in list) would make this table impossible to go stale —
|
||||
suggested follow-up, not yet implemented.
|
||||
@@ -26,8 +26,12 @@ env:
|
||||
# bundle deps — that fights jpackage's self-contained JRE (libjvm.so has
|
||||
# $ORIGIN RPATH so ldd can't resolve it standalone). appimagetool only
|
||||
# embeds the AppDir as-is, which is what we actually want.
|
||||
APPIMAGETOOL_URL: https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-x86_64.AppImage
|
||||
APPIMAGETOOL_SHA256: 46fdd785094c7f6e545b61afcfb0f3d98d8eab243f644b4b17698c01d06083d1
|
||||
#
|
||||
# Both arch binaries come from the same appimagetool release so their SHA256
|
||||
# values move in lockstep on version bumps.
|
||||
APPIMAGETOOL_VERSION: '1.9.0'
|
||||
APPIMAGETOOL_SHA256_X86_64: 46fdd785094c7f6e545b61afcfb0f3d98d8eab243f644b4b17698c01d06083d1
|
||||
APPIMAGETOOL_SHA256_AARCH64: 04f45ea45b5aa07bb2b071aed9dbf7a5185d3953b11b47358c1311f11ea94a96
|
||||
|
||||
jobs:
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -38,11 +42,17 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# Linux legs run on x64 and arm64 GitHub-hosted runners (the
|
||||
# ubuntu-24.04-arm label is a standard free public-repo runner as of
|
||||
# early 2025). jpackage / jlink / Compose Multiplatform 1.11 all
|
||||
# produce host-native artifacts — no cross-compilation needed.
|
||||
include:
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "packageReleaseDmg" }
|
||||
- { os: windows-latest, arch: x64, family: windows, tasks: "packageReleaseMsi createReleaseDistributable" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "packageReleaseDmg" }
|
||||
- { os: windows-latest, arch: x64, family: windows, tasks: "packageReleaseMsi createReleaseDistributable" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
|
||||
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "packageReleaseDeb packageReleaseRpm" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
|
||||
- { os: ubuntu-24.04-arm, arch: arm64, family: linux-portable, tasks: "createReleaseAppImage createReleaseDistributable" }
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 60 # linux-portable leg also downloads the freedesktop runtime + builds the Flatpak bundle
|
||||
defaults:
|
||||
@@ -93,13 +103,21 @@ jobs:
|
||||
set -euo pipefail
|
||||
# appimagetool 1.9.0 validates the .desktop file via desktop-file-validate.
|
||||
sudo apt-get update && sudo apt-get install -y desktop-file-utils
|
||||
curl -fsSL --retry 3 "$APPIMAGETOOL_URL" -o desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
|
||||
actual=$(sha256sum desktopApp/packaging/appimage/appimagetool-x86_64.AppImage | awk '{print $1}')
|
||||
if [[ "$actual" != "$APPIMAGETOOL_SHA256" ]]; then
|
||||
echo "::error::appimagetool SHA256 mismatch. Expected $APPIMAGETOOL_SHA256, got $actual"
|
||||
# Map runner arch → upstream AppImage suffix (x86_64 / aarch64).
|
||||
case "${{ matrix.arch }}" in
|
||||
x64) TOOL_ARCH=x86_64 ; EXPECTED_SHA="$APPIMAGETOOL_SHA256_X86_64" ;;
|
||||
arm64) TOOL_ARCH=aarch64; EXPECTED_SHA="$APPIMAGETOOL_SHA256_AARCH64" ;;
|
||||
*) echo "::error::unsupported arch for AppImage: ${{ matrix.arch }}"; exit 1 ;;
|
||||
esac
|
||||
URL="https://github.com/AppImage/appimagetool/releases/download/${APPIMAGETOOL_VERSION}/appimagetool-${TOOL_ARCH}.AppImage"
|
||||
DEST="desktopApp/packaging/appimage/appimagetool-${TOOL_ARCH}.AppImage"
|
||||
curl -fsSL --retry 3 "$URL" -o "$DEST"
|
||||
actual=$(sha256sum "$DEST" | awk '{print $1}')
|
||||
if [[ "$actual" != "$EXPECTED_SHA" ]]; then
|
||||
echo "::error::appimagetool SHA256 mismatch for $TOOL_ARCH. Expected $EXPECTED_SHA, got $actual"
|
||||
exit 1
|
||||
fi
|
||||
chmod +x desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
|
||||
chmod +x "$DEST"
|
||||
|
||||
# Flatpak tooling + the freedesktop runtime/sdk the manifest pins
|
||||
# (runtime-version is greped from the manifest so this never drifts).
|
||||
@@ -208,12 +226,13 @@ jobs:
|
||||
run: |
|
||||
set -euo pipefail
|
||||
VER="${{ steps.ver.outputs.version }}"
|
||||
ARCH="${{ matrix.arch }}"
|
||||
APP="desktopApp/build/compose/binaries/main-release/app"
|
||||
mkdir -p desktopApp/build/portable
|
||||
if [[ "${{ matrix.family }}" == "windows" ]]; then
|
||||
( cd "$APP" && 7z a -tzip "../../../../portable/amethyst-desktop-${VER}-windows-x64.zip" Amethyst/ )
|
||||
( cd "$APP" && 7z a -tzip "../../../../portable/amethyst-desktop-${VER}-windows-${ARCH}.zip" Amethyst/ )
|
||||
else
|
||||
( cd "$APP" && tar czf "../../../../portable/amethyst-desktop-${VER}-linux-x64.tar.gz" Amethyst/ )
|
||||
( cd "$APP" && tar czf "../../../../portable/amethyst-desktop-${VER}-linux-${ARCH}.tar.gz" Amethyst/ )
|
||||
fi
|
||||
|
||||
# Flatpak bundle: wraps the same createReleaseDistributable tree the
|
||||
@@ -230,6 +249,17 @@ jobs:
|
||||
PKG="desktopApp/packaging/flatpak"
|
||||
APP_ID="com.vitorpamplona.amethyst.Desktop"
|
||||
OUT="desktopApp/build/flatpak"
|
||||
# AppImage-style arch names for the bundle filename.
|
||||
case "${{ matrix.arch }}" in
|
||||
x64) BUNDLE_ARCH=x86_64 ; GST_TRIPLET=x86_64-linux-gnu ;;
|
||||
arm64) BUNDLE_ARCH=aarch64 ; GST_TRIPLET=aarch64-linux-gnu ;;
|
||||
*) echo "::error::unsupported arch for Flatpak: ${{ matrix.arch }}"; exit 1 ;;
|
||||
esac
|
||||
# Rewrite the arch-specific GStreamer plugin path in the manifest
|
||||
# (checked-in default is x86_64-linux-gnu). Idempotent — the sed only
|
||||
# matches the original triplet.
|
||||
sed -i "s|/usr/lib/x86_64-linux-gnu/gstreamer-1.0|/usr/lib/${GST_TRIPLET}/gstreamer-1.0|g" \
|
||||
"${PKG}/${APP_ID}.yml"
|
||||
# Inject the AppStream <release> entry for this build (the checked-in
|
||||
# metainfo deliberately carries none — CI is the source of truth).
|
||||
sed -i "s|<releases>|<releases>\n <release version=\"${VER}\" date=\"$(date -u +%F)\" />|" \
|
||||
@@ -241,7 +271,7 @@ jobs:
|
||||
"${OUT}/build-dir" \
|
||||
"${PKG}/${APP_ID}.yml"
|
||||
flatpak build-bundle "${OUT}/repo" \
|
||||
"${OUT}/Amethyst-${VER}-x86_64.flatpak" \
|
||||
"${OUT}/Amethyst-${VER}-${BUNDLE_ARCH}.flatpak" \
|
||||
"$APP_ID" \
|
||||
--runtime-repo=https://dl.flathub.org/repo/flathub.flatpakrepo
|
||||
ls -la "$OUT"
|
||||
@@ -325,8 +355,9 @@ jobs:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "amyImage" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "amyImage" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
|
||||
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "amyImage jpackageDeb jpackageRpm" }
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 45 # macOS leg also codesigns + notarizes the jlink image
|
||||
defaults:
|
||||
@@ -574,8 +605,9 @@ jobs:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "geodeImage" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
|
||||
- { os: macos-14, arch: arm64, family: macos, tasks: "geodeImage" }
|
||||
- { os: ubuntu-latest, arch: x64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
|
||||
- { os: ubuntu-24.04-arm, arch: arm64, family: linux, tasks: "geodeImage jpackageDeb jpackageRpm" }
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 45 # macOS leg also codesigns + notarizes the jlink image
|
||||
defaults:
|
||||
|
||||
@@ -49,9 +49,17 @@ jobs:
|
||||
# package, installs it, and verifies the process stays alive for 10s.
|
||||
# Catches ProGuard stripping (JNI, reflection), missing jlink modules
|
||||
# (java.management, java.prefs), and native lib bundling issues.
|
||||
#
|
||||
# Runs on both x64 and arm64 hosted runners so release-time arm64 breakage
|
||||
# (e.g. ProGuard rules missing an arch-specific reflection root) is caught
|
||||
# at PR time instead of on the tag build.
|
||||
# -------------------------------------------------------------------------
|
||||
release-deb-launch:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, ubuntu-24.04-arm]
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- name: Checkout code
|
||||
@@ -139,5 +147,6 @@ jobs:
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: Release DEB (smoke-tested)
|
||||
# Artifact names must be unique across a run — disambiguate per arch.
|
||||
name: Release DEB (smoke-tested, ${{ matrix.os }})
|
||||
path: desktopApp/build/compose/binaries/main-release/deb/*.deb
|
||||
|
||||
+8
-5
@@ -57,9 +57,12 @@ Install appimagetool locally (CI fetches its own — SHA-verified):
|
||||
# Debian/Ubuntu — appimagetool calls desktop-file-validate on the .desktop entry
|
||||
sudo apt-get install -y desktop-file-utils
|
||||
|
||||
curl -fsSL -o desktopApp/packaging/appimage/appimagetool-x86_64.AppImage \
|
||||
https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-x86_64.AppImage
|
||||
chmod +x desktopApp/packaging/appimage/appimagetool-x86_64.AppImage
|
||||
# createReleaseAppImage picks appimagetool-<arch>.AppImage matching the JVM's
|
||||
# os.arch — fetch the one for your host (x86_64 on Intel/AMD, aarch64 on ARM).
|
||||
ARCH="$(uname -m)"
|
||||
curl -fsSL -o "desktopApp/packaging/appimage/appimagetool-${ARCH}.AppImage" \
|
||||
"https://github.com/AppImage/appimagetool/releases/download/1.9.0/appimagetool-${ARCH}.AppImage"
|
||||
chmod +x "desktopApp/packaging/appimage/appimagetool-${ARCH}.AppImage"
|
||||
```
|
||||
|
||||
---
|
||||
@@ -110,8 +113,8 @@ are **not** required to build Amethyst from the committed sources.
|
||||
| Windows MSI | `./gradlew :desktopApp:packageReleaseMsi` | `desktopApp/build/compose/binaries/main-release/msi/Amethyst-*.msi` |
|
||||
| Linux `.deb` | `./gradlew :desktopApp:packageReleaseDeb` | `desktopApp/build/compose/binaries/main-release/deb/amethyst_*.deb` |
|
||||
| Linux `.rpm` | `./gradlew :desktopApp:packageReleaseRpm` | `desktopApp/build/compose/binaries/main-release/rpm/amethyst-*.rpm` |
|
||||
| Linux AppImage | `./gradlew :desktopApp:createReleaseAppImage` | `desktopApp/build/appimage/Amethyst-*-x86_64.AppImage` |
|
||||
| Linux Flatpak | `flatpak-builder` over `createReleaseDistributable` output — see [`desktopApp/packaging/flatpak/README.md`](desktopApp/packaging/flatpak/README.md) | `desktopApp/build/flatpak/Amethyst-*-x86_64.flatpak` (CI) |
|
||||
| Linux AppImage | `./gradlew :desktopApp:createReleaseAppImage` | `desktopApp/build/appimage/Amethyst-*-<arch>.AppImage` (x86_64 or aarch64, from host) |
|
||||
| Linux Flatpak | `flatpak-builder` over `createReleaseDistributable` output — see [`desktopApp/packaging/flatpak/README.md`](desktopApp/packaging/flatpak/README.md) | `desktopApp/build/flatpak/Amethyst-*-<arch>.flatpak` (CI; x86_64 or aarch64) |
|
||||
| Windows `.zip` portable | See below (inline `7z`) | — |
|
||||
| Linux `.tar.gz` portable | See below (inline `tar`) | — |
|
||||
|
||||
|
||||
+4
-2
@@ -67,7 +67,9 @@ class RoleBasedHttpClientBuilder(
|
||||
normalizedUrl: String,
|
||||
final: Boolean,
|
||||
): Boolean =
|
||||
if (RelayUrlNormalizer.isLocalHost(normalizedUrl)) {
|
||||
if (RelayUrlNormalizer.isLocalHost(normalizedUrl) || RelayUrlNormalizer.isOverlayNetwork(normalizedUrl)) {
|
||||
// Overlay-mesh hosts (0200::/7) are reachable only through the local mesh
|
||||
// interface — Tor cannot route the range, so proxying only breaks the fetch.
|
||||
false
|
||||
} else if (RelayUrlNormalizer.isOnion(normalizedUrl)) {
|
||||
true
|
||||
@@ -113,7 +115,7 @@ class RoleBasedHttpClientBuilder(
|
||||
isOnionRelaysActive: Boolean,
|
||||
final: Boolean,
|
||||
): Boolean =
|
||||
if (RelayUrlNormalizer.isLocalHost(normalizedUrl)) {
|
||||
if (RelayUrlNormalizer.isLocalHost(normalizedUrl) || RelayUrlNormalizer.isOverlayNetwork(normalizedUrl)) {
|
||||
false
|
||||
} else if (RelayUrlNormalizer.isOnion(normalizedUrl)) {
|
||||
isOnionRelaysActive
|
||||
|
||||
+22
@@ -170,6 +170,7 @@ fun RelayUrlEditField(
|
||||
nav: INav,
|
||||
) {
|
||||
var url by remember { mutableStateOf("") }
|
||||
var isInvalid by remember { mutableStateOf(false) }
|
||||
|
||||
fun submitRelay() {
|
||||
if (url.isNotBlank()) {
|
||||
@@ -177,7 +178,13 @@ fun RelayUrlEditField(
|
||||
if (relay != null) {
|
||||
onNewRelay(relay)
|
||||
url = ""
|
||||
isInvalid = false
|
||||
relaySuggestions.reset()
|
||||
} else {
|
||||
// Without this the Add button is a silent no-op, which reads as a broken button.
|
||||
// Bare IPv6 literals are the common way to land here: an overlay-mesh address
|
||||
// pasted straight out of `yggdrasilctl getSelf` needs brackets to carry a port.
|
||||
isInvalid = true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -189,8 +196,23 @@ fun RelayUrlEditField(
|
||||
value = url,
|
||||
onValueChange = {
|
||||
url = it
|
||||
isInvalid = false
|
||||
relaySuggestions.processInput(it)
|
||||
},
|
||||
isError = isInvalid,
|
||||
// Null, not an empty lambda: a non-null slot reserves its line height even when it
|
||||
// draws nothing, which would pad the field permanently for every user.
|
||||
supportingText =
|
||||
if (isInvalid) {
|
||||
{
|
||||
Text(
|
||||
text = stringRes(R.string.relay_url_not_valid),
|
||||
color = MaterialTheme.colorScheme.error,
|
||||
)
|
||||
}
|
||||
} else {
|
||||
null
|
||||
},
|
||||
placeholder = {
|
||||
Text(
|
||||
text = "server.com",
|
||||
|
||||
@@ -332,7 +332,7 @@
|
||||
<string name="concord_leave_title">क्या समुदाय छोडें।</string>
|
||||
<string name="concord_leave_message">क्या %1$s छोडें। इसे हटाया जाएगा इस लेखा की सूची से तथा आपके यन्त्रों पर समचरणीकरण रुक जाएगा। समुदाय को सूचित नहीं किया जाएगा। तथा आपको उसके सदस्य कार्यसूची से हटाया नहीं जाएगा। सन्देश जिनका आप अरहस्यीकरण नहीं कर सकेंगे सम्भाव्यतः पुनःप्राप्तव्य नहीं होंगे। तथा आप केवल नए आमन्त्रण के साथ लौट सकेंगे।</string>
|
||||
<string name="concord_leave_owner_warning">आपने इस समुदाय को बनाया। छोड जाने से यह मिटेगा नहीं। किसी अन्य के हाथ सौंपा नहीं जाएगा। परन्तु स्वत्वधारी कुंचिका जो आपकी सूची में हैं वह हटाया जाएगा। आप आगे से इसका प्रबन्धन नहीं कर पाएँगे।</string>
|
||||
<string name="concord_edit_relays_desc">जहाँ इस समुदाय के रहस्यीकृत पत्रों का प्रकाशन तथा पठन किया जाता है।</string>
|
||||
<string name="concord_edit_relays_desc">जहाँ इस समुदाय के रहस्यीकृत समतलों का प्रकाशन तथा पठन किया जाता है।</string>
|
||||
<string name="concord_dissolved_read_only">इस समुदाय को विघटित किया गया है तथा अब पठनेवशक्य है। आप इसका इतिहास पढ सकते हैं परन्तु कोई नए सन्देश नहीं भेज सकते।</string>
|
||||
<string name="concord_typing_one">%1$s टंकण मध्य…</string>
|
||||
<string name="concord_typing_two">%1$s तथा %2$s टंकण मध्य…</string>
|
||||
@@ -1233,7 +1233,7 @@
|
||||
<string name="nest_participant_unmute">मौन हटाएँ</string>
|
||||
<string name="nest_force_mute_note">सम्भाव्यतः उन ग्राहकों द्वारा उपेक्षित जो आज्ञा का सम्मान नहीं करते।</string>
|
||||
<string name="nest_confirm_kick_title">क्या शाला से निष्कासित करें।</string>
|
||||
<string name="nest_confirm_kick_body">%1$s को ध्वनि तल से हटाए जाएँगे तथा सहभागी सूची से भी। वे पुनः जुड सकते हैं यदि वे शाला योजक प्राप्त कर लें।</string>
|
||||
<string name="nest_confirm_kick_body">%1$s को ध्वनि समतल से हटाए जाएँगे तथा सहभागी सूची से भी। वे पुनः जुड सकते हैं यदि वे शाला योजक प्राप्त कर लें।</string>
|
||||
<string name="nest_confirm_kick_confirm">पदप्रहार</string>
|
||||
<string name="nest_confirm_force_mute_title">क्या वक्ता को मौन करें।</string>
|
||||
<string name="nest_confirm_force_mute_body">%1$s के ग्राहक को अपना ध्वनिग्राहक मौन करने का अनुरोध करता है। कुछ ग्राहक इस आदेश की उपेक्षा कर सकते हैं।</string>
|
||||
@@ -1942,14 +1942,95 @@
|
||||
</plurals>
|
||||
<!-- Expanded-only breakdown of the always-on notification. Counts overlap: one relay commonly
|
||||
serves several jobs at once, so these deliberately sum to more than the relay count. -->
|
||||
<plurals name="relay_purpose_line">
|
||||
<item quantity="one">%1$s \u00b7 %2$d पुनःप्रसारक</item>
|
||||
<item quantity="other">%1$s \u00b7 %2$d पुनःप्रसारक</item>
|
||||
</plurals>
|
||||
<string name="relay_purpose_browsing">जालभ्रमण</string>
|
||||
<string name="relay_purpose_media">ध्वनिचित्राभिलेख</string>
|
||||
<string name="relay_purpose_tags">विषयसूचक</string>
|
||||
<string name="relay_purpose_topics">विषय सूची</string>
|
||||
<string name="relay_purpose_thread">वार्तालाप</string>
|
||||
<string name="relay_purpose_search">खोज</string>
|
||||
<string name="relay_purpose_referenced">लुप्त घटनाओं को ढूँढें</string>
|
||||
<string name="relay_purpose_engagement">घटना अवलोकन</string>
|
||||
<string name="relay_explain_referenced">घटनाओं को विभेदक अनुसार ले आता है जिसका उल्लेख आपके पटल पर अमुक करता है पर जिसकी प्राप्ती अभी नहीं हुई। एक उद्धरण अथवा एक प्रत्युत्तर का पूर्वपत्र अथवा एक सूत्र का मूल।</string>
|
||||
<string name="relay_explain_engagement">घटनाओं का अवलोकन करता है जो वर्तमान में प्रदर्शित हो रहे हैं नए प्रत्युत्तर प्रतिक्रियाएँ उद्धरण ज्साप तथा वृत्तान्तों के लिए जिससे गिनतियों का नवीकरण होता है जब आप पढ रहे हैं।</string>
|
||||
<string name="relay_purpose_add_ons">संलग्न</string>
|
||||
<string name="relay_purpose_relay_info">पुनःप्रसारक जानकारी</string>
|
||||
<string name="relay_purpose_other">अन्य</string>
|
||||
<!-- Activity labels, used where the app's own noun for the data type already means something
|
||||
else to the user: "Outbox Relays" is their own relay list in settings, and "Profile" is
|
||||
their profile screen. Naming the job avoids an active misreading. -->
|
||||
<string name="relay_purpose_relay_list_finder">पुनःप्रसारक सूची खोजकर्ता</string>
|
||||
<string name="relay_purpose_observing_profiles">परिचय अवलोकन</string>
|
||||
<string name="relay_purpose_your_account">लेखा जानकारी</string>
|
||||
<string name="relay_purpose_home_feed">मुख्य सूचनावली</string>
|
||||
<string name="relay_purpose_relay_groups">पुनःप्रसारक समूह</string>
|
||||
<plurals name="active_subs_groups">
|
||||
<item quantity="one">%1$d समूह</item>
|
||||
<item quantity="other">%1$d समूह</item>
|
||||
</plurals>
|
||||
<string name="relay_purpose_ephemeral_chats">अस्थायी चर्चाएँ</string>
|
||||
<string name="relay_purpose_geohash_chats">स्थानीय चर्चाएँ</string>
|
||||
<string name="relay_purpose_live_chat">वर्तमानप्रवाह चर्चा</string>
|
||||
<string name="relay_explain_relay_groups">निप॰२९ समूह जिनसे आप जुडे हैं। प्रत्येक समूह एक जालावास पुनःप्रसारक में रहता है। इसलिए क्रमक प्रत्येक पुनःप्रसारक से संयोजन करता है जो आपके किसी समूह का जालावास है।</string>
|
||||
<string name="relay_explain_ephemeral_chats">चर्चाशालाएँ जो कोई इतिहास नहीं रखते। सन्देश केवल तब तक रहते हैं जब तक आप संयोजित हैं। इसलिए ये ग्राहकता बनाए रखते हैं कुछ भी प्राप्त होने के लिए।</string>
|
||||
<string name="relay_explain_geohash_chats">स्थान आधारित शालाएँ उन क्षेत्रों के लिए जिनका आप अनुगमन करते हैं। पृष्ट उन पुनःप्रसारको से जो इनके जालावास हैं।</string>
|
||||
<string name="relay_explain_live_chat">चर्चा तथा ज्साप उद्देश्य जो वर्तमानप्रवाहों से संलग्न हैं जिन्हें आप खोले हुए हैं अथवा अनुगमन करते हैं।</string>
|
||||
<string name="relay_purpose_dm_inbox">सीधासन्देश आगतपेटिका</string>
|
||||
<string name="relay_purpose_your_wallet">धनकोष</string>
|
||||
<string name="relay_purpose_nutzap_inbox">नटज्साप आगतपेटिका</string>
|
||||
<string name="relay_purpose_mint_directory">टकसाल निर्देशिका</string>
|
||||
<string name="relay_purpose_nwc">धनकोष संयोजन</string>
|
||||
<string name="relay_purpose_community_chats">समुदाय चर्चाएँ</string>
|
||||
<string name="relay_purpose_community_feeds">समुदाय सूचनावलियाँ</string>
|
||||
<!-- How each subscription actually works, shown on the Active Subscriptions screen.
|
||||
Describe the real strategy, not the intent — these are read by people trying to explain
|
||||
a relay count they think is too high. -->
|
||||
<string name="relay_explain_notifications">आपके आगतपेटिका पुनःप्रसारक तथा कुछ अल्पमात्रा परिभ्रमणवर्ती दृष्टान्त पुनःप्रसारक जिनपर आपके अनुचरित पत्र प्रकाशित करते हैं। यदि कोई उल्लेख अन्यत्र भेजा गया।</string>
|
||||
<string name="relay_explain_direct_messages">आपके सीधासन्देश पुनःप्रसारक। जहाँ उपहारकोषयुक्त सन्देश भेजे जाते हैं।</string>
|
||||
<string name="relay_explain_public_chats">मुख्य पुनःप्रसारक प्रत्येक चर्चा का जिन्हें आप खोल रखे हैं अथवा जिनसे आप जुड चुके हैं।</string>
|
||||
<string name="relay_explain_community_chats">पुनःप्रसारक जिनपर प्रत्येक समुदाय अपने समतलों को प्रकाशित करते हैं।</string>
|
||||
<string name="relay_explain_encrypted_groups">समूह सन्देश तथा कुंचिकापेटलियाँ। प्रत्येक समूह के पुनःप्रसारकों पर।</string>
|
||||
<string name="relay_explain_live_rooms">शाला के पुनःप्रसारक। जब वह खुला हो।</string>
|
||||
<string name="relay_explain_account_data">आपके अपने परिचय तथा स्थापना विकल्प तथा पाण्डुलिपियाँ। आपके मुख्य पुनःप्रसारकों पर।</string>
|
||||
<string name="relay_explain_profiles">वर्तमानतः पटल पर लोगों के परिचय।</string>
|
||||
<string name="relay_explain_relay_lists">खोजता है किन पुनःप्रसारकों पर प्रत्येक व्यक्ति प्रकाशन करता है। जिससे कि उनके पत्र सम्यक स्थल से प्राप्प हो।</string>
|
||||
<string name="relay_explain_follows">अनुचरण सूचियाँ। आपकी सूचनावली तथा आपका विश्वासजाल का निर्माण के लिए उपयुक्त।</string>
|
||||
<string name="relay_explain_moderation">वृत्तान्त जो आपके अनुचरितों ने लिखा वर्तमानतः आपके पटल पर दिखनेवाले परिचयों के विषय में। पृष्ट प्रत्येक पुनःप्रसारक से जिनपर वे पत्र प्रकाशन करते हैं।</string>
|
||||
<string name="relay_purpose_reports_from_follows">अनुचरित से वृत्तान्त</string>
|
||||
<string name="relay_explain_wallet">आपके अपने धनकोष घटनाएँ। पुनःपठित उन पुनःप्रसारकों से जिनपर आपने उनके प्रकाशन किए।</string>
|
||||
<string name="relay_explain_nutzap_inbox">सुनता है आपके नटज्साप पुनःप्रसारकों पर तथा आपके आगतपेटिका तथा सीधासन्देश पुनःप्रसारकों पर। जिससे कि कोई भी भुगतान छूट ना जाए।</string>
|
||||
<string name="relay_explain_mint_directory">पुनःप्रसारकों का वीक्षण करता है यह देखने के लिए कि कौनसे टकसाल हैं तथा लोग किनकी अनुशम्सा करते हैं।</string>
|
||||
<string name="relay_explain_nwc">आपके संयोजित धनकोष से सूचनाएँ।</string>
|
||||
<string name="active_subs_title">सक्रिय पुनःप्रसारक ग्राहकताएँ</string>
|
||||
<!-- Two countable nouns, so two plurals composed at the call site rather than one string with
|
||||
two %d in it: filter and relay decline independently in Slavic/Baltic/Semitic languages. -->
|
||||
<plurals name="active_subs_filters">
|
||||
<item quantity="one">%1$d छलनी</item>
|
||||
<item quantity="other">%1$d छलनियाँ</item>
|
||||
</plurals>
|
||||
<plurals name="active_subs_relays">
|
||||
<item quantity="one">%1$d पुनःप्रसारक</item>
|
||||
<item quantity="other">%1$d पुनःप्रसारक</item>
|
||||
</plurals>
|
||||
<plurals name="active_subs_untagged">
|
||||
<item quantity="one">%1$d छलनी आरोपित नहीं अब तक</item>
|
||||
<item quantity="other">%1$d छलनियाँ आरोपित नहीं अब तक</item>
|
||||
</plurals>
|
||||
<string name="active_subs_pair">%1$s \u00b7 %2$s</string>
|
||||
<string name="active_subs_unattributed">किसी लेखा प्रति आरोपित नहीं</string>
|
||||
<string name="active_subs_no_entity">सभी</string>
|
||||
<string name="active_subs_scope_global">सभी</string>
|
||||
<string name="active_subs_scope_follows">आपके द्वारा अनुचरित लोग</string>
|
||||
<string name="active_subs_scope_authors">चयनित लोगों की सूची</string>
|
||||
<string name="active_subs_scope_muted">मौनकृत लोग</string>
|
||||
<string name="active_subs_scope_all_communities">आपके समुदाय</string>
|
||||
<string name="active_subs_scope_algo">एक प्रिय कलनविधि सूचनावली</string>
|
||||
<string name="active_subs_share">%1$dप्रतिशतप्रतिशत सब में से</string>
|
||||
<string name="active_subs_search_keywords">ग्राहकताएँ छलनियाँ पुनःप्रसारक अनुरोध अनु॰ संयोजन क्यों निदानतन्त्र</string>
|
||||
<string name="relay_explain_home">आपके अनुचरितों के पत्र। पठित उन पुनःप्रसारकों से जिनपर उनमें से प्रत्येक प्रकाशन करते हैं।</string>
|
||||
<string name="always_on_notif_connecting">आगतपेटिका पुनःप्रसारकों के साथ संयोजन किया जा रहा है \u2026</string>
|
||||
<string name="always_on_notif_setting_title">सदैव सक्रिय सूचना सेवा</string>
|
||||
<string name="always_on_notif_setting_description">अनवरत संयोजन बनाए रखता है आपके आगतपेटिका पुनःप्रसारकों के साथ तत्काल सूचना वितरण के लिए। एक स्थायी सूचना दिखाता है। विद्युत्कोष का अधिक उपयोग करता है पर निश्चित करता है कि आप कभी भी सन्देश नहीं खोएँगे।</string>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -214,6 +214,7 @@
|
||||
<string name="connection_success_rate_description">Percentage of successful connections to the relay</string>
|
||||
<string name="search_and_add_a_user">Search and add user</string>
|
||||
<string name="add_a_relay">Add a Relay</string>
|
||||
<string name="relay_url_not_valid">Not a valid relay address. Use a host name, or an IP address in brackets (for example [201:d0e:9ba5:8bbc::1]:8080).</string>
|
||||
<string name="my_name">My @tag name</string>
|
||||
<string name="display_name">Display Name</string>
|
||||
<string name="my_display_name">My display name</string>
|
||||
|
||||
@@ -28,6 +28,7 @@ import com.vitorpamplona.quartz.marmot.mip00KeyPackages.KeyPackageRelayListEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrlOrNull
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
|
||||
import com.vitorpamplona.quartz.nip11RelayInfo.Nip11RelayInformation
|
||||
@@ -50,6 +51,7 @@ import com.vitorpamplona.quartz.nip65RelayList.tags.AdvertisedRelayType
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.reachability.RelayProber
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
import java.io.File
|
||||
|
||||
/**
|
||||
* `amy relay …` — manage every relay list this account maintains, mirroring
|
||||
@@ -112,10 +114,12 @@ object RelayCommands {
|
||||
| relay info URL fetch + print a relay's NIP-11 info document (stateless)
|
||||
| relay probe [--timeout SECS] relay census: mass-connect every relay the store
|
||||
| [--concurrency N] knows and record live/dead + measured rtt-open
|
||||
| into the reachability cache (NIP-66 kind:30166),
|
||||
| [--file PATH] into the reachability cache (NIP-66 kind:30166),
|
||||
| so reachability-aware commands (graperank crawl/
|
||||
| refresh) skip dead relays and wait once
|
||||
| (--timeout: per wave, default 15s)
|
||||
| (--timeout: per wave, default 15s; --file: also
|
||||
| probe candidate urls, one per line, each run
|
||||
| through the relay url normalizer first)
|
||||
""".trimMargin()
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
@@ -337,12 +341,41 @@ object RelayCommands {
|
||||
// Relays dialed at once; --relay-concurrency accepted as the alias the
|
||||
// graperank verbs spell it with.
|
||||
val waveSize = args.intFlag("concurrency", args.intFlag("relay-concurrency", Context.defaultPreconnectCap))
|
||||
// Optional external candidate list: one raw url per line, run through the
|
||||
// same RelayUrlNormalizer the app uses, so a probe doubles as a census of
|
||||
// how a corpus of relay hints normalizes (rejects are counted, not dialed).
|
||||
val fromFile = args.flag("file")
|
||||
args.rejectUnknown()
|
||||
|
||||
var fileRaw = 0
|
||||
var fileRejected = 0
|
||||
var fileOnion = 0
|
||||
val fileRelays = HashSet<NormalizedRelayUrl>()
|
||||
if (fromFile != null) {
|
||||
val candidates = File(fromFile)
|
||||
if (!candidates.canRead()) return Output.error("bad_args", "cannot read --file $fromFile")
|
||||
candidates.forEachLine { line ->
|
||||
if (line.isBlank()) return@forEachLine
|
||||
fileRaw++
|
||||
val normalized = line.normalizeRelayUrlOrNull()
|
||||
if (normalized == null) {
|
||||
fileRejected++
|
||||
} else if (RelayUrlNormalizer.isOnion(normalized.url)) {
|
||||
fileOnion++
|
||||
} else {
|
||||
fileRelays.add(normalized)
|
||||
}
|
||||
}
|
||||
System.err.println(
|
||||
"[relay-probe] $fromFile: $fileRaw urls → ${fileRelays.size} unique clearnet relays " +
|
||||
"($fileRejected rejected by the normalizer, $fileOnion onion skipped)",
|
||||
)
|
||||
}
|
||||
|
||||
Context.openOrAnonymous(dataDir).use { ctx ->
|
||||
ctx.prepare()
|
||||
val cached = ctx.reachability.snapshot()
|
||||
val universe = RelayProber.knownRelayUniverse(ctx.store) + cached.live + cached.dead
|
||||
val universe = RelayProber.knownRelayUniverse(ctx.store) + cached.live + cached.dead + fileRelays
|
||||
if (universe.isEmpty()) {
|
||||
Output.emit(
|
||||
linkedMapOf<String, Any?>(
|
||||
@@ -381,6 +414,10 @@ object RelayCommands {
|
||||
Output.emit(
|
||||
linkedMapOf<String, Any?>(
|
||||
"probed" to result.verdicts.size,
|
||||
"file_urls" to (if (fromFile != null) fileRaw else null),
|
||||
"file_normalized" to (if (fromFile != null) fileRelays.size else null),
|
||||
"file_rejected" to (if (fromFile != null) fileRejected else null),
|
||||
"file_onion_skipped" to (if (fromFile != null) fileOnion else null),
|
||||
"reachable" to result.reachable.size,
|
||||
"dead" to result.dead.size,
|
||||
"closed_by_policy" to authWalled,
|
||||
|
||||
@@ -1,17 +1,95 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<resources>
|
||||
<!-- Login & Auth -->
|
||||
<string name="login_title">Üdvözöljük az Amethystben</string>
|
||||
<string name="login_subtitle">Jelentkezzen be Nostr a-fiókjába</string>
|
||||
<string name="login_subtitle_desktop">Asztali Nostr-kliens</string>
|
||||
<string name="login_card_title">Jelentkezzen be Nostr-a kulcsával</string>
|
||||
<string name="login_card_subtitle">nsec a teljes hozzáféréshez, bunker:// a távoli aláíróhoz, vagy npub a csak olvasható módhoz</string>
|
||||
<string name="login_with_key">Bejelentkezés kulccsal</string>
|
||||
<string name="login_button">Bejelentkezés</string>
|
||||
<string name="login_generate_new">Új kulcs előállítása</string>
|
||||
<string name="login_generate_button">Új előállítása</string>
|
||||
<string name="login_key_hint">Adja meg a privát kulcsát (nsec) vagy a nyilvános kulcsát (npub)</string>
|
||||
<string name="login_key_label">nsec, bunker:// vagy npub</string>
|
||||
<string name="login_key_placeholder">nsec1… / bunker://… / npub1…</string>
|
||||
<string name="login_show_key">Kulcs megjelenítése</string>
|
||||
<string name="login_hide_key">Kulcs elrejtése</string>
|
||||
<!-- New Key Warning -->
|
||||
<string name="new_key_warning_title">FONTOS: Mentse el a kulcsait!</string>
|
||||
<string name="new_key_warning_message">A titkos kulcsa (nsec) az EGYETLEN módja annak, hogy hozzáférjen a fiókjához. Ha elveszíti, akkor a fiókja végleg elvész. Mentse el egy biztonságos helyre!</string>
|
||||
<string name="new_key_public_label">Nyilvános kulcs (megosztható):</string>
|
||||
<string name="new_key_secret_label">Titkos kulcs (SOHA ne ossza meg!):</string>
|
||||
<string name="new_key_continue_button">Elmentettem a kulcsaimat, folytatás</string>
|
||||
<!-- Common Actions -->
|
||||
<string name="action_copy">Másolás</string>
|
||||
<string name="action_paste">Beillesztés</string>
|
||||
<string name="action_cancel">Mégse</string>
|
||||
<string name="action_ok">OK</string>
|
||||
<string name="action_save">Mentés</string>
|
||||
<string name="action_delete">Törlés</string>
|
||||
<string name="action_share">Megosztás</string>
|
||||
<!-- Errors -->
|
||||
<string name="error_invalid_key">Érvénytelen kulcsformátum. Ellenőrizze, és próbálja újra.</string>
|
||||
<string name="error_network">Hálózati hiba. Ellenőrizze a kapcsolatot.</string>
|
||||
<string name="error_generic">Hiba történt. Próbálja újra.</string>
|
||||
<!-- Loading & Empty States -->
|
||||
<string name="action_refresh">Frissítés</string>
|
||||
<string name="action_try_again">Próbálja újra</string>
|
||||
<string name="feed_empty">A hírfolyam üres</string>
|
||||
<string name="error_loading_feed">Hiba történt a hírfolyam betöltésekor: %s</string>
|
||||
<!-- Placeholder Screens -->
|
||||
<string name="screen_search_title">Keresés</string>
|
||||
<string name="screen_search_description">Keressen felhasználókat, bejegyzéseket és kulcsszavakat.</string>
|
||||
<string name="screen_messages_title">Üzenetek</string>
|
||||
<string name="screen_messages_description">Az Ön titkosított közvetlen üzenetei itt fognak megjelenni.</string>
|
||||
<string name="screen_notifications_title">Értesítések</string>
|
||||
<string name="screen_notifications_description">Az említések, válaszok és reakciók itt fognak megjelenni.</string>
|
||||
<!-- Accessibility -->
|
||||
<string name="accessibility_user_avatar">Felhasználó profilképe</string>
|
||||
<string name="accessibility_navigate">Navigáció</string>
|
||||
<!-- Relay history paging (shared feed markers + status card) -->
|
||||
<string name="chats_history_loading_label">Betöltés:</string>
|
||||
<string name="chats_history_fully_loaded_label">Teljesen betöltve:</string>
|
||||
<string name="chats_history_fully_loaded">(teljesen betöltve)</string>
|
||||
<string name="chats_history_by_relay">Előzmények átjátszónként</string>
|
||||
<string name="chats_history_stalled_retry">Újra megpróbálja, amint újra megnyitja ezt a képernyőt</string>
|
||||
<string name="chats_history_older">%1$s korábbi üzenet</string>
|
||||
<string name="chats_history_all_caught_up">Naprakész</string>
|
||||
<string name="chats_history_reached_start">Elérte a(z) %1$s üzeneteinek elejét</string>
|
||||
<string name="chats_history_subtitle">%1$s · %2$s · betöltve ekkortól: %3$s</string>
|
||||
<string name="chats_history_subtitle_no_date">%1$s · %2$s</string>
|
||||
<string name="chats_history_waiting">várakozás erre: %1$s</string>
|
||||
<string name="chats_history_incomplete">Néhány átjátszó nem válaszolt</string>
|
||||
<string name="chats_history_incomplete_sub">%1$s nem érhető el · koppintson a részletekért</string>
|
||||
<string name="chats_history_relays_title">%1$s · előzmények átjátszónként</string>
|
||||
<string name="chats_history_relay_since">ekkortól: %1$s</string>
|
||||
<string name="action_dismiss">Eltüntetés</string>
|
||||
<plurals name="chats_history_relays">
|
||||
<item quantity="one">%1$d átjátszó</item>
|
||||
<item quantity="other">%1$d relé</item>
|
||||
</plurals>
|
||||
<!-- Notes & Replies -->
|
||||
<string name="replying_to">válasz neki: </string>
|
||||
<!-- Static sites (NIP-5A) & napplets (NIP-5D) feed card -->
|
||||
<string name="nsite_title">nOldal: %1$s</string>
|
||||
<string name="napplet_card_title">nKisalkalmazás: %1$s</string>
|
||||
<string name="napplet_card_kind">nKisalkalmazás</string>
|
||||
<string name="nsite_website_kind">nOldal</string>
|
||||
<string name="napplet_card_permissions">Amihez hozzáférhet</string>
|
||||
<string name="nsite_root_site">Gyökéroldal</string>
|
||||
<string name="nsite_source">Forrás:</string>
|
||||
<string name="nsite_servers">Kiszolgálók:</string>
|
||||
<string name="nsite_open">Megnyitás</string>
|
||||
<!-- Custom emoji suggestions (NIP-30) -->
|
||||
<string name="use_direct_url">Közvetlen webcím használata</string>
|
||||
<!-- Nicknames (NIP-85 contact cards) -->
|
||||
<string name="nickname_dialog_title">Becenév</string>
|
||||
<string name="nickname_dialog_explainer">Ez jelenik meg Önnek ezen felhasználó neve helyett az alkalmazásban bárhol. Titkosítva tárolódik el a kapcsolatkártyájára: csak Ön olvashatja. Írjon be kettőspontot (:) az egyéni emodzsik használatához.</string>
|
||||
<string name="nickname_label">Becenév</string>
|
||||
<string name="nickname_summary_label">Privát megjegyzés erről a felhasználóról</string>
|
||||
<string name="nickname_save">Mentés</string>
|
||||
<string name="nickname_cancel">Mégse</string>
|
||||
<string name="git_status_open">Nyitva</string>
|
||||
<string name="git_status_merged">Beolvasztva</string>
|
||||
<string name="git_status_closed">Lezárva</string>
|
||||
@@ -53,6 +131,7 @@
|
||||
<string name="road_event_traffic_jam">Forgalmi dugó</string>
|
||||
<string name="road_event_unknown">Útesemény</string>
|
||||
<string name="podcast_value_zap_split_hint">Az erre küldött Zapek megoszlanak a következők között:</string>
|
||||
<string name="podcast_value_split_percent">%1$d%%</string>
|
||||
<string name="podcast_value_for_value">Értéket az értékért</string>
|
||||
<string name="relay_monitor_rtt_open">Megnyitás</string>
|
||||
<string name="relay_monitor_rtt_read">Olvasás </string>
|
||||
@@ -61,6 +140,7 @@
|
||||
<string name="relay_monitor_relay_type">Típus</string>
|
||||
<string name="relay_monitor_requirements">Követelmények</string>
|
||||
<string name="relay_monitor_supported_nips">Támogatott NIP-ek</string>
|
||||
<string name="relay_monitor_ms">%1$d ms</string>
|
||||
<string name="relay_discovery_accepted_kinds">Elfogadott típusok</string>
|
||||
<string name="relay_discovery_geohash">Helyszín</string>
|
||||
<string name="calendar_rsvp_going">Ott leszek</string>
|
||||
|
||||
+10
@@ -81,6 +81,16 @@ interface NotificationSettings {
|
||||
|
||||
fun setEnabled(v: Boolean)
|
||||
|
||||
/**
|
||||
* True iff the user has taken an explicit action to disable
|
||||
* notifications (i.e. flipped the master switch OFF at some point).
|
||||
* Used by the Settings screen to distinguish "master switch is off
|
||||
* because it defaults to off on first launch" from "master switch
|
||||
* is off because the user asked for it to be off". Only the former
|
||||
* gets auto-enabled when the OS permission check passes.
|
||||
*/
|
||||
fun wasExplicitlyDisabled(): Boolean
|
||||
|
||||
fun setKindToggle(
|
||||
kind: NotifKind,
|
||||
v: Boolean,
|
||||
|
||||
+6
@@ -23,6 +23,7 @@ package com.vitorpamplona.amethyst.commons.tor
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isLocalHost
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOnion
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOverlayNetwork
|
||||
|
||||
class TorRelayEvaluation(
|
||||
val torSettings: TorRelaySettings,
|
||||
@@ -36,6 +37,11 @@ class TorRelayEvaluation(
|
||||
} else {
|
||||
if (relay.isLocalHost()) {
|
||||
false
|
||||
} else if (relay.isOverlayNetwork()) {
|
||||
// An overlay-mesh relay (0200::/7, e.g. Yggdrasil) is reachable only through the
|
||||
// local mesh interface: Tor cannot route the range at all, so proxying it would
|
||||
// guarantee failure rather than privacy. The overlay already encrypts end to end.
|
||||
false
|
||||
} else if (relay.isOnion()) {
|
||||
// .onion is only reachable over Tor regardless of any other classification.
|
||||
torSettings.onionRelaysViaTor
|
||||
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.tor
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* An overlay-mesh relay (`0200::/7`, e.g. Yggdrasil) must never be dialed through the Tor SOCKS
|
||||
* proxy: Tor cannot route the range, so proxying guarantees failure rather than privacy. The
|
||||
* overlay already encrypts end to end and authenticates the peer by its key-derived address.
|
||||
*/
|
||||
class YggdrasilTorRoutingTest {
|
||||
private val yggdrasilRelay = NormalizedRelayUrl("ws://[201:d0e:9ba5:8bbc::1]:8080/")
|
||||
private val yggdrasilSubnetRelay = NormalizedRelayUrl("ws://[300:1b5d:d0e9:ba58::1]:4848/")
|
||||
private val lanRelay = NormalizedRelayUrl("ws://192.168.1.100:8080/")
|
||||
private val ulaRelay = NormalizedRelayUrl("ws://[fd12:3456::1]:8080/")
|
||||
private val clearnetIpv6Relay = NormalizedRelayUrl("wss://[2001:db8::1]:8080/")
|
||||
|
||||
private fun evaluation(newViaTor: Boolean) =
|
||||
TorRelayEvaluation(
|
||||
torSettings =
|
||||
TorRelaySettings(
|
||||
torType = TorType.INTERNAL,
|
||||
onionRelaysViaTor = true,
|
||||
dmRelaysViaTor = true,
|
||||
newRelaysViaTor = newViaTor,
|
||||
trustedRelaysViaTor = false,
|
||||
moneyOperationsViaTor = false,
|
||||
),
|
||||
trustedRelayList = emptySet(),
|
||||
dmRelayList = emptySet(),
|
||||
)
|
||||
|
||||
@Test
|
||||
fun overlayRelaysAreNeverTorifiedEvenWhenNewRelaysViaTorIsOn() {
|
||||
val eval = evaluation(newViaTor = true)
|
||||
assertFalse(eval.useTor(yggdrasilRelay), "0200::/8 node address must not be proxied")
|
||||
assertFalse(eval.useTor(yggdrasilSubnetRelay), "0300::/8 subnet address must not be proxied")
|
||||
assertFalse(eval.useTor(lanRelay), "LAN relay stays off Tor")
|
||||
assertFalse(eval.useTor(ulaRelay), "IPv6 unique local address stays off Tor")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun clearnetIpv6RelaysStillFollowTheTorSetting() {
|
||||
// The overlay exemption must not leak into ordinary IPv6 relays.
|
||||
assertTrue(evaluation(newViaTor = true).useTor(clearnetIpv6Relay))
|
||||
assertFalse(evaluation(newViaTor = false).useTor(clearnetIpv6Relay))
|
||||
}
|
||||
}
|
||||
+13
@@ -53,8 +53,20 @@ class PreferencesNotificationSettings(
|
||||
override fun setEnabled(v: Boolean) {
|
||||
_enabled.value = v
|
||||
prefs.putBoolean(KEY_ENABLED, v)
|
||||
// Track explicit user intent so the Settings screen can auto-enable
|
||||
// on next visit for users who never touched the switch, while
|
||||
// respecting users who deliberately turned it off. Only false
|
||||
// → "explicit disable"; going from off to on clears the flag so
|
||||
// subsequent auto-enable heuristics work normally.
|
||||
if (v) {
|
||||
prefs.remove(KEY_EXPLICITLY_DISABLED)
|
||||
} else {
|
||||
prefs.putBoolean(KEY_EXPLICITLY_DISABLED, true)
|
||||
}
|
||||
}
|
||||
|
||||
override fun wasExplicitlyDisabled(): Boolean = prefs.getBoolean(KEY_EXPLICITLY_DISABLED, false)
|
||||
|
||||
override fun setKindToggle(
|
||||
kind: NotifKind,
|
||||
v: Boolean,
|
||||
@@ -92,6 +104,7 @@ class PreferencesNotificationSettings(
|
||||
companion object {
|
||||
const val NODE = "com/vitorpamplona/amethyst/notifications"
|
||||
private const val KEY_ENABLED = "enabled"
|
||||
private const val KEY_EXPLICITLY_DISABLED = "explicitly_disabled"
|
||||
private const val KEY_DND_UNTIL = "dnd_until"
|
||||
private const val KEY_PREVIEW = "preview_in_toast"
|
||||
|
||||
|
||||
+86
@@ -0,0 +1,86 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.moderation.notifications
|
||||
|
||||
import org.junit.Assert.assertFalse
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
import java.util.UUID
|
||||
import java.util.prefs.Preferences
|
||||
|
||||
/**
|
||||
* Pins the semantics of the new [NotificationSettings.wasExplicitlyDisabled]
|
||||
* flag added to unblock the "Enable OS notifications button doesn't work on
|
||||
* desktop" bug's second failure mode.
|
||||
*
|
||||
* The Settings screen auto-enables the master notifications switch when it
|
||||
* detects that the OS permission is fine and the switch is off. That's the
|
||||
* common path for users who click the "Enable OS notifications" button and
|
||||
* expect it to fully take effect (button label promise). But it MUST NOT
|
||||
* override users who deliberately turned notifications off. The
|
||||
* [wasExplicitlyDisabled] flag is how we distinguish the two.
|
||||
*/
|
||||
class PreferencesNotificationSettingsExplicitDisableTest {
|
||||
private fun freshNode(): Preferences {
|
||||
// Use a UUID-scoped node so tests never share state and never
|
||||
// pollute the real user prefs on the machine running CI/dev builds.
|
||||
return Preferences.userRoot().node("amethyst-test-" + UUID.randomUUID())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `fresh install defaults to not-explicitly-disabled`() {
|
||||
val settings = PreferencesNotificationSettings(freshNode())
|
||||
assertFalse(
|
||||
"First launch must not look like a deliberate opt-out; otherwise auto-enable stays off forever",
|
||||
settings.wasExplicitlyDisabled(),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `turning off marks explicitly disabled`() {
|
||||
val prefs = freshNode()
|
||||
val settings = PreferencesNotificationSettings(prefs)
|
||||
settings.setEnabled(false)
|
||||
assertTrue(settings.wasExplicitlyDisabled())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `turning on clears the explicit-disable flag`() {
|
||||
val prefs = freshNode()
|
||||
val settings = PreferencesNotificationSettings(prefs)
|
||||
settings.setEnabled(false)
|
||||
assertTrue(settings.wasExplicitlyDisabled())
|
||||
settings.setEnabled(true)
|
||||
assertFalse(
|
||||
"Toggling back on must clear the flag so subsequent OFF->auto-enable cycles work",
|
||||
settings.wasExplicitlyDisabled(),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `flag persists across new instances on the same prefs node`() {
|
||||
val prefs = freshNode()
|
||||
PreferencesNotificationSettings(prefs).setEnabled(false)
|
||||
// Second instance opens the same node \u2014 flag must survive process restart.
|
||||
val reopened = PreferencesNotificationSettings(prefs)
|
||||
assertTrue(reopened.wasExplicitlyDisabled())
|
||||
}
|
||||
}
|
||||
@@ -246,18 +246,30 @@ compose.desktop {
|
||||
// - amethyst.png 512x512 icon
|
||||
//
|
||||
// appimagetool binary is fetched by CI (SHA-verified) into
|
||||
// desktopApp/packaging/appimage/ as appimagetool-x86_64.AppImage.
|
||||
// desktopApp/packaging/appimage/ as appimagetool-<arch>.AppImage.
|
||||
// The arch is selected at task-execution time from the host JVM's os.arch, so
|
||||
// the same task builds the correct AppImage on both x86_64 and aarch64 hosts.
|
||||
// BUILDING.md documents local-dev fetch.
|
||||
val createReleaseAppImage by tasks.registering(Exec::class) {
|
||||
group = "compose desktop"
|
||||
description = "Package createReleaseDistributable output into a Linux AppImage via appimagetool."
|
||||
dependsOn("createReleaseDistributable")
|
||||
|
||||
// AppImage's ARCH env accepts the Linux kernel arch names: x86_64 / aarch64
|
||||
// / armhf / i686. jpackage produces host-native binaries, so mirror the
|
||||
// host JVM arch. Do not read the property inside doFirst — it needs to be
|
||||
// resolved at configuration time so outputs.file() below is stable.
|
||||
val hostArch = when (val a = System.getProperty("os.arch").lowercase()) {
|
||||
"amd64", "x86_64" -> "x86_64"
|
||||
"aarch64", "arm64" -> "aarch64"
|
||||
else -> a
|
||||
}
|
||||
|
||||
val distDir = layout.buildDirectory.dir("compose/binaries/main-release/app/Amethyst")
|
||||
val appDir = layout.buildDirectory.dir("appimage/Amethyst.AppDir")
|
||||
val outFile = layout.buildDirectory.file("appimage/Amethyst-$appVersion-x86_64.AppImage")
|
||||
val outFile = layout.buildDirectory.file("appimage/Amethyst-$appVersion-$hostArch.AppImage")
|
||||
val toolRoot = layout.projectDirectory.dir("packaging/appimage")
|
||||
val appimagetool = toolRoot.file("appimagetool-x86_64.AppImage")
|
||||
val appimagetool = toolRoot.file("appimagetool-$hostArch.AppImage")
|
||||
|
||||
inputs.dir(distDir)
|
||||
inputs.dir(toolRoot)
|
||||
@@ -292,7 +304,7 @@ val createReleaseAppImage by tasks.registering(Exec::class) {
|
||||
appDir.get().asFile.absolutePath,
|
||||
outFile.get().asFile.absolutePath,
|
||||
)
|
||||
environment("ARCH", "x86_64")
|
||||
environment("ARCH", hostArch)
|
||||
// Bypass FUSE requirement on CI runners (ubuntu-latest lacks libfuse.so.2).
|
||||
// AppImage standard env var: extracts + runs without mounting.
|
||||
environment("APPIMAGE_EXTRACT_AND_RUN", "1")
|
||||
|
||||
@@ -7,7 +7,10 @@
|
||||
# (Equivalent packages on Fedora/Arch.)
|
||||
set -eu
|
||||
HERE="$(dirname "$(readlink -f "${0}")")"
|
||||
export LD_LIBRARY_PATH="${HERE}/usr/lib:${HERE}/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH:-}"
|
||||
# Multiarch lib path is set by the host, not baked at build time — same
|
||||
# AppRun works in both x86_64 and aarch64 AppImages.
|
||||
GNU_TRIPLET="$(uname -m)-linux-gnu"
|
||||
export LD_LIBRARY_PATH="${HERE}/usr/lib:${HERE}/usr/lib/${GNU_TRIPLET}:${LD_LIBRARY_PATH:-}"
|
||||
export PATH="${HERE}/usr/bin:${PATH}"
|
||||
export APPDIR="${HERE}"
|
||||
exec "${HERE}/usr/bin/Amethyst" "$@"
|
||||
|
||||
@@ -28,8 +28,12 @@ used two ways:
|
||||
manifest (archive source pinned to the release tarball URL + sha256, with
|
||||
`x-checker-data` so Flathub's update bot bumps it), its own metainfo
|
||||
(carries the permanent `<releases>` history Flathub requires), desktop
|
||||
entry, icon, and `flathub.json` (`only-arches: x86_64` — we publish no
|
||||
aarch64 tarball, and jpackage can't cross-compile one)
|
||||
entry, icon, and `flathub.json` — currently gated to `only-arches:
|
||||
x86_64` so the Flathub build machinery never tries the aarch64 tarball
|
||||
before we've validated it end-to-end on Flathub's aarch64 builders. GitHub
|
||||
releases already ship aarch64 flatpak bundles (built from the same source
|
||||
tree on `ubuntu-24.04-arm`); flipping `only-arches` to include `aarch64`
|
||||
is the follow-up once we've smoke-tested a Flathub aarch64 build.
|
||||
|
||||
## Local build
|
||||
|
||||
|
||||
+42
-6
@@ -39,13 +39,17 @@ import androidx.compose.material3.Switch
|
||||
import androidx.compose.material3.Text
|
||||
import androidx.compose.material3.TextButton
|
||||
import androidx.compose.runtime.Composable
|
||||
import androidx.compose.runtime.LaunchedEffect
|
||||
import androidx.compose.runtime.collectAsState
|
||||
import androidx.compose.runtime.getValue
|
||||
import androidx.compose.runtime.mutableStateOf
|
||||
import androidx.compose.runtime.remember
|
||||
import androidx.compose.runtime.rememberCoroutineScope
|
||||
import androidx.compose.runtime.setValue
|
||||
import androidx.compose.runtime.snapshotFlow
|
||||
import androidx.compose.ui.Alignment
|
||||
import androidx.compose.ui.Modifier
|
||||
import androidx.compose.ui.platform.LocalWindowInfo
|
||||
import androidx.compose.ui.unit.dp
|
||||
import com.vitorpamplona.amethyst.commons.moderation.notifications.HostOs
|
||||
import com.vitorpamplona.amethyst.commons.moderation.notifications.NotifKind
|
||||
@@ -112,7 +116,7 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
|
||||
dispatcher?.nativeAvailable?.collectAsState()
|
||||
?: remember { mutableStateOf(false) }
|
||||
)
|
||||
val coroutineScope = androidx.compose.runtime.rememberCoroutineScope()
|
||||
val coroutineScope = rememberCoroutineScope()
|
||||
var testStatus by remember { mutableStateOf<String?>(null) }
|
||||
var requestingPermission by remember { mutableStateOf(false) }
|
||||
var sendingTest by remember { mutableStateOf(false) }
|
||||
@@ -120,18 +124,36 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
|
||||
// Re-sync permission state whenever this screen enters composition
|
||||
// and whenever the window regains focus — user may have toggled
|
||||
// Amethyst in System Settings → Notifications while we were open.
|
||||
val windowInfo = androidx.compose.ui.platform.LocalWindowInfo.current
|
||||
androidx.compose.runtime.LaunchedEffect(dispatcher) {
|
||||
val windowInfo = LocalWindowInfo.current
|
||||
LaunchedEffect(dispatcher) {
|
||||
dispatcher?.refreshPermission()
|
||||
}
|
||||
androidx.compose.runtime.LaunchedEffect(dispatcher, windowInfo) {
|
||||
androidx.compose.runtime
|
||||
.snapshotFlow { windowInfo.isWindowFocused }
|
||||
LaunchedEffect(dispatcher, windowInfo) {
|
||||
snapshotFlow { windowInfo.isWindowFocused }
|
||||
.collect { focused ->
|
||||
if (focused) dispatcher?.refreshPermission()
|
||||
}
|
||||
}
|
||||
|
||||
// Covers the two paths the earlier fix (e9475dd0) missed: the OS
|
||||
// permission was already granted in a prior session (an older
|
||||
// build asked, or the user allowed Amethyst in System Settings
|
||||
// directly), and Windows/Linux, where permissionState is
|
||||
// NotApplicable from startup. On both, permissionState is not
|
||||
// NotRequested, so the "Enable OS notifications" button never
|
||||
// renders, yet the master switch is still OFF from first-launch
|
||||
// defaults — leaving no affordance that both explains the problem
|
||||
// and fixes it. Auto-enable when the permission is fine, the
|
||||
// master switch is off, and the user has never explicitly turned
|
||||
// it off; [NotificationSettings.wasExplicitlyDisabled] is what
|
||||
// keeps a deliberate opt-out from being overruled.
|
||||
LaunchedEffect(permissionState, enabled) {
|
||||
val allowed = permissionState == PermissionState.Granted || permissionState == PermissionState.NotApplicable
|
||||
if (allowed && !enabled && !settings.wasExplicitlyDisabled()) {
|
||||
settings.setEnabled(true)
|
||||
}
|
||||
}
|
||||
|
||||
PlatformStatusCard(
|
||||
host = host,
|
||||
nativeAvailable = nativeAvailable,
|
||||
@@ -219,6 +241,20 @@ fun NotificationSettingsScreen(onBack: (() -> Unit)? = null) {
|
||||
}
|
||||
}
|
||||
PermissionState.Granted, PermissionState.NotApplicable -> {
|
||||
// Recovery affordance for the deliberate-opt-out path.
|
||||
// The LaunchedEffect above already re-enables the
|
||||
// switch for anyone who never touched it, so in
|
||||
// practice the only state that still reaches here with
|
||||
// `enabled == false` is an explicit opt-out. Guarding
|
||||
// on `!enabled` alone (rather than also calling
|
||||
// wasExplicitlyDisabled) keeps this a pure Compose
|
||||
// state read and leaves the button visible for the one
|
||||
// frame before the effect runs.
|
||||
if (!enabled) {
|
||||
OutlinedButton(
|
||||
onClick = { settings.setEnabled(true) },
|
||||
) { Text("Turn on desktop notifications") }
|
||||
}
|
||||
OutlinedButton(
|
||||
onClick = {
|
||||
if (sendingTest) return@OutlinedButton
|
||||
|
||||
@@ -84,27 +84,34 @@
|
||||
"Dutch"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "summoner001",
|
||||
"languages": [
|
||||
"Hungarian"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "maxblake2015",
|
||||
"languages": [
|
||||
"Polish"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "rajs19420616",
|
||||
"languages": [
|
||||
"Hindi"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "vitorpamplona",
|
||||
"languages": [
|
||||
"Czech",
|
||||
"German",
|
||||
"Polish",
|
||||
"Portuguese, Brazilian",
|
||||
"Swedish"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "rajs19420616",
|
||||
"languages": [
|
||||
"Hindi"
|
||||
]
|
||||
},
|
||||
{
|
||||
"user": "greenart7c3",
|
||||
"languages": []
|
||||
@@ -265,10 +272,6 @@
|
||||
"user": "D4rkFIow",
|
||||
"languages": []
|
||||
},
|
||||
{
|
||||
"user": "summoner001",
|
||||
"languages": []
|
||||
},
|
||||
{
|
||||
"user": "fiddleway",
|
||||
"languages": []
|
||||
|
||||
@@ -112,6 +112,15 @@ require_auth = false
|
||||
# the future. Enforced by RejectFutureEventsPolicy.
|
||||
# reject_future_seconds = 1800
|
||||
|
||||
# Path for the JSON file that remembers what the [[mirror]] catch-up
|
||||
# has already synced (per upstream, per scope), so a restart resumes
|
||||
# instead of re-downloading each upstream's whole backfill window.
|
||||
# Defaults to "<database file>.sync-coverage.json" next to the event
|
||||
# store; only written when the store itself is file-backed (an
|
||||
# in-memory store keeps no resume state — saved coverage would
|
||||
# describe events that no longer exist).
|
||||
# mirror_sync_state_file = "/var/lib/geode/events.db.sync-coverage.json"
|
||||
|
||||
[authorization]
|
||||
# Allow / deny lists. Allow is a permissive ceiling; deny still
|
||||
# removes specific entries inside it. Enforced by Pubkey/KindAllowDenyPolicy.
|
||||
|
||||
@@ -28,6 +28,7 @@ import com.vitorpamplona.geode.config.StaticConfig
|
||||
import com.vitorpamplona.geode.mirror.MirrorDirection
|
||||
import com.vitorpamplona.geode.mirror.MirrorUpstream
|
||||
import com.vitorpamplona.geode.mirror.MirrorWorker
|
||||
import com.vitorpamplona.geode.mirror.SyncCoverageFile
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
@@ -44,6 +45,7 @@ import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.NdjsonImportExport
|
||||
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
|
||||
import com.vitorpamplona.quartz.nip77Negentropy.NegentropySettings
|
||||
import com.vitorpamplona.quartz.utils.Log
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
@@ -223,7 +225,8 @@ private fun runImport(args: Array<String>) {
|
||||
}
|
||||
System.err.println(
|
||||
"geode import: read=${stats.read} imported=${stats.imported} " +
|
||||
"rejected=${stats.rejected} invalid-sig=${stats.invalid} malformed=${stats.malformed} " +
|
||||
"rejected=${stats.rejected} failed=${stats.failed} " +
|
||||
"invalid-sig=${stats.invalid} malformed=${stats.malformed} " +
|
||||
"→ ${ctx.dbFile ?: "(in-memory — not persisted; pass --db)"}",
|
||||
)
|
||||
} finally {
|
||||
@@ -411,6 +414,27 @@ private fun serve(args: Array<String>) {
|
||||
require(upstreams.none { it.url.displayUrl() == advertisedIdentity }) {
|
||||
"[[mirror]] must not list this relay's own URL ($advertisedUrl)"
|
||||
}
|
||||
// Resume state for the mirror catch-up, following the admin state-file
|
||||
// convention: next to the event database unless configured. Keyed to the
|
||||
// store's ACTUAL persistence, not to `database.file` being set: a
|
||||
// volatile store with a persistent coverage file would claim, on the
|
||||
// next boot, that an empty database already holds the backfill window —
|
||||
// and the mirror would never fetch it.
|
||||
val sqliteBackend =
|
||||
config.database.backend
|
||||
.trim()
|
||||
.lowercase() in StoreFactory.SQLITE_BACKEND_KEYWORDS
|
||||
val persistentLocation =
|
||||
a.opt("--db") ?: config.database.file?.takeUnless { sqliteBackend && config.database.in_memory }
|
||||
val syncCoverage =
|
||||
if (upstreams.isEmpty() || persistentLocation == null) {
|
||||
if (upstreams.isNotEmpty() && config.options.mirror_sync_state_file != null) {
|
||||
Log.w("Main") { "mirror_sync_state_file ignored: the event store is in-memory, so saved coverage would outlive the events it describes" }
|
||||
}
|
||||
null
|
||||
} else {
|
||||
SyncCoverageFile(File(config.options.mirror_sync_state_file ?: "$persistentLocation.sync-coverage.json"))
|
||||
}
|
||||
val mirror =
|
||||
if (upstreams.isEmpty()) {
|
||||
null
|
||||
@@ -425,6 +449,9 @@ private fun serve(args: Array<String>) {
|
||||
// for the historical window, then live REQ tail. Auto-falls back
|
||||
// to paged REQ for upstreams without NIP-77.
|
||||
negentropyBackfill = true,
|
||||
// Without NIP-77 the catch-up is a paged re-download; the
|
||||
// coverage bands remember what previous boots already walked.
|
||||
coverage = syncCoverage?.coverage,
|
||||
).also { it.start() }
|
||||
}
|
||||
|
||||
@@ -462,6 +489,8 @@ private fun serve(args: Array<String>) {
|
||||
// queue and store beneath them shut down.
|
||||
runCatching { maintenanceScope.cancel() }
|
||||
runCatching { mirror?.close() }
|
||||
// After the mirror, so the final flush carries the last bands.
|
||||
runCatching { syncCoverage?.close() }
|
||||
runCatching { server.stop() }
|
||||
runCatching { relay.close() }
|
||||
},
|
||||
|
||||
@@ -185,6 +185,16 @@ data class StaticConfig(
|
||||
* that don't offer NIP-50 at all (strfry, for example).
|
||||
*/
|
||||
val full_text_search: Boolean = true,
|
||||
/**
|
||||
* Where the mirror catch-up's resume state lives: the per-upstream
|
||||
* `created_at` coverage bands (quartz's `SyncCoverage`). Without it
|
||||
* every restart re-syncs each upstream's whole backfill window —
|
||||
* a full re-download for an upstream without NIP-77. Defaults to
|
||||
* `<database file>.sync-coverage.json` when the store is
|
||||
* file-backed; an in-memory store keeps no resume state (bands
|
||||
* only pay off across restarts).
|
||||
*/
|
||||
val mirror_sync_state_file: String? = null,
|
||||
)
|
||||
|
||||
/**
|
||||
|
||||
@@ -23,8 +23,11 @@ package com.vitorpamplona.geode.mirror
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.NostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.NegentropySyncException
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.SyncCoverage
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.fetchAllPages
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropyReconcile
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySyncOrFetch
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.negentropySync
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.EventMessage
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.ReqCmd
|
||||
@@ -40,12 +43,15 @@ import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.Job
|
||||
import kotlinx.coroutines.SupervisorJob
|
||||
import kotlinx.coroutines.cancel
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.channels.trySendBlocking
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import okhttp3.OkHttpClient
|
||||
import java.time.Duration
|
||||
import java.util.concurrent.atomic.AtomicLong
|
||||
@@ -154,11 +160,18 @@ class MirrorWorker(
|
||||
* historical window before the live REQ tail takes over. The geode binary
|
||||
* turns this on (see `Main`); it defaults **off** so the many existing
|
||||
* MirrorWorker tests keep exercising the pure live-REQ path unchanged.
|
||||
* When on, `negentropySyncOrFetch` automatically falls back to paged REQ
|
||||
* against an upstream that doesn't speak NIP-77 — so "either mode" is
|
||||
* transparent and needs no separate toggle.
|
||||
* When on, the catch-up automatically falls back to paged REQ against an
|
||||
* upstream that doesn't speak NIP-77 — so "either mode" is transparent
|
||||
* and needs no separate toggle.
|
||||
*/
|
||||
private val negentropyBackfill: Boolean = false,
|
||||
/**
|
||||
* Resume memory for the down catch-up, shared across upstreams and — via
|
||||
* [SyncCoverageFile] — across restarts. Null keeps the old behavior:
|
||||
* every boot re-syncs the whole backfill window, which for an upstream
|
||||
* without NIP-77 is a full re-download.
|
||||
*/
|
||||
private val coverage: SyncCoverage? = null,
|
||||
) : AutoCloseable {
|
||||
private val scope = CoroutineScope(Dispatchers.IO + SupervisorJob())
|
||||
|
||||
@@ -201,6 +214,14 @@ class MirrorWorker(
|
||||
/** Events the store rejected — mostly duplicate replays after a reconnect. */
|
||||
val rejected = AtomicLong(0)
|
||||
|
||||
/**
|
||||
* Good events the store could not write ([IEventStore.InsertOutcome.Failed]).
|
||||
* The store's fault, not the event's, and nothing re-offers them — counted
|
||||
* apart from [rejected] so a schema drift reads as store damage rather
|
||||
* than as upstreams sending junk.
|
||||
*/
|
||||
val failed = AtomicLong(0)
|
||||
|
||||
/**
|
||||
* Deliveries dropped before ever reaching the store: events outside
|
||||
* the [MirrorUpstream.filter] scope (an upstream answering outside
|
||||
@@ -300,6 +321,13 @@ class MirrorWorker(
|
||||
rejected.incrementAndGet()
|
||||
Log.d("MirrorWorker") { "rejected ${msg.event.id}: ${outcome.reason}" }
|
||||
}
|
||||
is IEventStore.InsertOutcome.Failed -> {
|
||||
// The store's error, not the event's: the event
|
||||
// was good and nothing will re-offer it. Louder
|
||||
// than a rejection on purpose.
|
||||
failed.incrementAndGet()
|
||||
Log.w("MirrorWorker") { "store failed ${msg.event.id}: ${outcome.reason}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (e: CancellationException) {
|
||||
@@ -399,9 +427,9 @@ class MirrorWorker(
|
||||
* **client-paced** so a fast upstream can't overrun the sink: a plain REQ
|
||||
* backfill of a large set dies here (strfry kills a slow REQ client once its
|
||||
* unsent-outbound buffer crosses `maxPendingOutboundBytes`), which is exactly
|
||||
* why this uses negentropy. [INostrClient.negentropySyncOrFetch] falls back
|
||||
* to paged REQ automatically when the upstream doesn't speak NIP-77, so the
|
||||
* mirror is compatible with either kind of upstream with no config.
|
||||
* why this uses negentropy. When the upstream doesn't speak NIP-77 the
|
||||
* catch-up falls back to paged REQ, so the mirror is compatible with
|
||||
* either kind of upstream with no config.
|
||||
*
|
||||
* A failure here is non-fatal: the live subscription keeps the mirror current
|
||||
* and the reconnect watermark narrows any residual gap.
|
||||
@@ -412,16 +440,26 @@ class MirrorWorker(
|
||||
initialSince: Long,
|
||||
until: Long,
|
||||
) {
|
||||
val catchUpFilter = scopedBase.copy(since = initialSince, until = until)
|
||||
// Reconcile against what we already hold in this window → download only
|
||||
// the diff (like `strfry sync`). No store wired → empty local set → the
|
||||
// whole window is downloaded and the store's unique-id constraint dedups.
|
||||
val localEntries = store?.snapshotIdsForNegentropy(listOf(catchUpFilter)) ?: emptyList()
|
||||
// Resume memory: coverage is keyed on the STABLE scoped filter, never
|
||||
// on the boot window — whose since/until change every start and would
|
||||
// never match a stored band. The window itself rides along as the
|
||||
// floor argument, so a band recorded against a shallower window
|
||||
// re-opens the older span when the operator deepens the backfill.
|
||||
val legs =
|
||||
coverage
|
||||
?.legs(up.url, scopedBase, initialSince)
|
||||
?.mapNotNull { clampToWindow(it, initialSince, until) }
|
||||
?: listOf(scopedBase.copy(since = initialSince, until = until))
|
||||
if (legs.isEmpty()) {
|
||||
Log.i("MirrorWorker") { "catch-up from ${up.url.url}: window already covered - nothing outside the synced band" }
|
||||
return
|
||||
}
|
||||
|
||||
// Bounded hand-off → one ingest consumer. `onEvent` can't suspend, so it
|
||||
// blocks here when the sink falls behind; because negentropySyncOrFetch's
|
||||
// own delivery pipeline is bounded, that backpressure reaches all the way
|
||||
// to the upstream — no unbounded buffering (unlike the live-tail path).
|
||||
// blocks here when the sink falls behind; because negentropySync's own
|
||||
// delivery pipeline is bounded (and a paged REQ is paced by its pages),
|
||||
// that backpressure reaches all the way to the upstream — no unbounded
|
||||
// buffering (unlike the live-tail path).
|
||||
val handoff = Channel<Event>(capacity = CATCHUP_HANDOFF)
|
||||
val consumer =
|
||||
scope.launch {
|
||||
@@ -431,6 +469,10 @@ class MirrorWorker(
|
||||
when (outcome) {
|
||||
IEventStore.InsertOutcome.Accepted -> accepted.incrementAndGet()
|
||||
is IEventStore.InsertOutcome.Rejected -> rejected.incrementAndGet()
|
||||
is IEventStore.InsertOutcome.Failed -> {
|
||||
failed.incrementAndGet()
|
||||
Log.w("MirrorWorker") { "store failed ${event.id}: ${outcome.reason}" }
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (e: CancellationException) {
|
||||
@@ -443,24 +485,99 @@ class MirrorWorker(
|
||||
}
|
||||
|
||||
try {
|
||||
val result =
|
||||
client.negentropySyncOrFetch(
|
||||
relay = up.url,
|
||||
filter = catchUpFilter,
|
||||
localEntries = localEntries,
|
||||
onEvent = { event ->
|
||||
// Same containment as the live path: even a trusted
|
||||
// upstream may only inject events inside the declared scope.
|
||||
if (up.filter == null || up.filter.match(event)) {
|
||||
handoff.trySendBlocking(event)
|
||||
} else {
|
||||
filtered.incrementAndGet()
|
||||
var downloaded = 0
|
||||
var paged = false
|
||||
for (leg in legs) {
|
||||
// Reconcile against what we already hold in this leg → download
|
||||
// only the diff (like `strfry sync`). No store wired → empty
|
||||
// local set → the whole leg is downloaded and the store's
|
||||
// unique-id constraint dedups.
|
||||
val localEntries = store?.snapshotIdsForNegentropy(listOf(leg)) ?: emptyList()
|
||||
// Coverage is stamped from when the local ids were read — that
|
||||
// is the state the relay is being compared against.
|
||||
val syncStartedAt = TimeUtils.now()
|
||||
var seenMin: Long? = null
|
||||
var seenMax: Long? = null
|
||||
|
||||
fun observe(event: Event) {
|
||||
// Same containment as the live path: even a trusted
|
||||
// upstream may only inject events inside the declared scope.
|
||||
if (up.filter == null || up.filter.match(event)) {
|
||||
// Only plausible stamps widen a band — one
|
||||
// misdated event must not discard the rest.
|
||||
if (SyncCoverage.isPlausible(event.createdAt)) {
|
||||
seenMin = minOf(seenMin ?: event.createdAt, event.createdAt)
|
||||
seenMax = maxOf(seenMax ?: event.createdAt, event.createdAt)
|
||||
}
|
||||
},
|
||||
)
|
||||
handoff.trySendBlocking(event)
|
||||
} else {
|
||||
filtered.incrementAndGet()
|
||||
}
|
||||
}
|
||||
|
||||
// The two phases run by hand rather than through
|
||||
// negentropySyncOrFetch, for two reasons. The combinator keeps
|
||||
// every delivered id for cross-phase dedup — a multi-million-
|
||||
// event catch-up cannot afford that heap, and the store's
|
||||
// unique-id constraint dedups anyway. And the fallback must
|
||||
// reset the observed span: a half-finished reconcile delivers
|
||||
// events scattered across the whole leg, and a band built from
|
||||
// that scatter would claim interior ranges nobody walked.
|
||||
val legPaged =
|
||||
try {
|
||||
downloaded +=
|
||||
client
|
||||
.negentropySync(
|
||||
relay = up.url,
|
||||
filter = leg,
|
||||
localEntries = localEntries,
|
||||
onEvent = ::observe,
|
||||
).downloaded
|
||||
false
|
||||
} catch (e: NegentropySyncException) {
|
||||
seenMin = null
|
||||
seenMax = null
|
||||
// The watchdog matches negentropySync's default rather
|
||||
// than fetchAllPages' shorter one: a paged catch-up
|
||||
// sits behind the same slow upstreams.
|
||||
downloaded += client.fetchAllPages(up.url, listOf(leg), idleTimeoutMs = 120_000L) { observe(it) }
|
||||
true
|
||||
}
|
||||
paged = paged || legPaged
|
||||
// Recorded per leg, so a failure between legs keeps the ground
|
||||
// the first one gained — and no more: a reconcile compared only
|
||||
// its own leg, so completeness reaches the leg's ceiling, never
|
||||
// "now" while a later leg is still pending. A paged fallback
|
||||
// earns only the span it actually saw, capped at the snapshot
|
||||
// instant so one future-dated event cannot lift the band's
|
||||
// ceiling past what was asked.
|
||||
if (legPaged) {
|
||||
coverage?.record(
|
||||
up.url,
|
||||
scopedBase,
|
||||
seenMin,
|
||||
seenMax?.coerceAtMost(syncStartedAt),
|
||||
paged = true,
|
||||
)
|
||||
} else {
|
||||
val legFloor = leg.since ?: initialSince
|
||||
coverage?.record(
|
||||
up.url,
|
||||
scopedBase,
|
||||
// The compared range starts at the leg's floor whether
|
||||
// or not anything was observed there — that is what a
|
||||
// clean reconcile proves.
|
||||
observedMin = minOf(seenMin ?: legFloor, legFloor),
|
||||
observedMax = null,
|
||||
paged = false,
|
||||
reconciledThrough = minOf(leg.until ?: syncStartedAt, syncStartedAt),
|
||||
)
|
||||
}
|
||||
}
|
||||
Log.i("MirrorWorker") {
|
||||
val how = if (result.pagedFallback) "paged REQ (upstream has no NIP-77)" else "negentropy"
|
||||
"catch-up from ${up.url.url}: ${result.downloaded} events via $how"
|
||||
val how = if (paged) "paged REQ (upstream has no NIP-77)" else "negentropy"
|
||||
val resumed = if (coverage != null && legs.size > 1) " [resumed: ${legs.size} legs outside the synced band]" else ""
|
||||
"catch-up from ${up.url.url}: $downloaded events via $how$resumed"
|
||||
}
|
||||
} catch (e: CancellationException) {
|
||||
throw e
|
||||
@@ -671,11 +788,21 @@ class MirrorWorker(
|
||||
runCatching { client.close() }
|
||||
inbound.close()
|
||||
scope.cancel()
|
||||
// Wait (bounded) for the workers to land: a coverage.record racing
|
||||
// past the state file's final flush would be recorded and lost.
|
||||
// Bounded because a worker parked in a blocking hand-off does not
|
||||
// feel the cancel, and shutdown must not hang on it.
|
||||
runBlocking {
|
||||
withTimeoutOrNull(CLOSE_JOIN_MS) { scope.coroutineContext[Job]?.join() }
|
||||
}
|
||||
okhttp?.dispatcher?.executorService?.shutdown()
|
||||
okhttp?.connectionPool?.evictAll()
|
||||
}
|
||||
|
||||
private companion object {
|
||||
/** How long [close] waits for the worker coroutines to land. */
|
||||
const val CLOSE_JOIN_MS = 5_000L
|
||||
|
||||
/** Matches the Android app's relay-pool WebSocket ping interval. */
|
||||
const val PING_INTERVAL_SECS = 120L
|
||||
|
||||
@@ -701,7 +828,7 @@ class MirrorWorker(
|
||||
|
||||
/**
|
||||
* Depth of the catch-up hand-off between the negentropy download and the
|
||||
* ingest consumer. Small: negentropySyncOrFetch is already internally
|
||||
* ingest consumer. Small: the negentropy download is already internally
|
||||
* backpressured, so this only smooths the seam — the bounded IngestQueue
|
||||
* behind `server.ingest` is the real limiter.
|
||||
*/
|
||||
@@ -720,3 +847,20 @@ class MirrorWorker(
|
||||
const val UP_SYNC_SETTLE_MS = 1_500L
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* [leg] intersected with the boot window `[since, until]`, or null when the
|
||||
* band already covers everything this window could ask. Coverage legs come
|
||||
* off the stable scoped filter and are unbounded on one side; the catch-up
|
||||
* only ever asks inside its own window.
|
||||
*/
|
||||
internal fun clampToWindow(
|
||||
leg: Filter,
|
||||
since: Long,
|
||||
until: Long,
|
||||
): Filter? {
|
||||
val newSince = maxOf(leg.since ?: since, since)
|
||||
val newUntil = minOf(leg.until ?: until, until)
|
||||
if (newSince > newUntil) return null
|
||||
return leg.copy(since = newSince, until = newUntil)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.geode.mirror
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.SyncCoverage
|
||||
import com.vitorpamplona.quartz.utils.Log
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.boolean
|
||||
import kotlinx.serialization.json.buildJsonObject
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import kotlinx.serialization.json.jsonPrimitive
|
||||
import kotlinx.serialization.json.long
|
||||
import kotlinx.serialization.json.put
|
||||
import java.io.File
|
||||
import java.nio.file.AtomicMoveNotSupportedException
|
||||
import java.nio.file.Files
|
||||
import java.nio.file.StandardCopyOption
|
||||
|
||||
/**
|
||||
* File persistence for [SyncCoverage], so the mirror's catch-up resumes
|
||||
* across restarts instead of re-syncing each upstream's whole backfill
|
||||
* window — which, for an upstream without NIP-77, is a full re-download
|
||||
* every boot.
|
||||
*
|
||||
* Same shape as the admin state file: JSON next to the event database,
|
||||
* written via a temp file and an atomic move so a reader never sees a half
|
||||
* map. A daemon timer flushes changed state so progress survives a hard
|
||||
* kill; [close] flushes the rest. A corrupt file starts fresh — the cost of
|
||||
* losing it is one re-sync, the cost of refusing to start is the relay.
|
||||
*/
|
||||
class SyncCoverageFile(
|
||||
private val file: File,
|
||||
flushSeconds: Long = DEFAULT_FLUSH_SECONDS,
|
||||
) : AutoCloseable {
|
||||
@Volatile private var dirty = false
|
||||
|
||||
val coverage = SyncCoverage(onChange = { dirty = true })
|
||||
|
||||
private val flusher: Thread
|
||||
|
||||
init {
|
||||
load()
|
||||
// restore() bypasses onChange, but stay defensive: reopening a file
|
||||
// must never count as a change, or every boot rewrites it.
|
||||
dirty = false
|
||||
flusher =
|
||||
Thread {
|
||||
while (!Thread.currentThread().isInterrupted) {
|
||||
try {
|
||||
Thread.sleep(flushSeconds * 1000)
|
||||
} catch (_: InterruptedException) {
|
||||
return@Thread
|
||||
}
|
||||
flush()
|
||||
}
|
||||
}.apply {
|
||||
isDaemon = true
|
||||
name = "mirror-sync-coverage-flush"
|
||||
start()
|
||||
}
|
||||
}
|
||||
|
||||
/** Write the map if anything changed since the last write. */
|
||||
@Synchronized
|
||||
fun flush() {
|
||||
if (!dirty) return
|
||||
dirty = false
|
||||
save()
|
||||
}
|
||||
|
||||
override fun close() {
|
||||
flusher.interrupt()
|
||||
flush()
|
||||
}
|
||||
|
||||
private fun load() {
|
||||
if (!file.isFile) return
|
||||
runCatching {
|
||||
val root = Json.parseToJsonElement(file.readText()).jsonObject
|
||||
coverage.restore(
|
||||
root.mapValues { (_, v) ->
|
||||
val o = v.jsonObject
|
||||
SyncCoverage.Band(
|
||||
o.getValue("min").jsonPrimitive.long,
|
||||
o.getValue("max").jsonPrimitive.long,
|
||||
o["complete"]?.jsonPrimitive?.boolean ?: false,
|
||||
o["fullAt"]?.jsonPrimitive?.long ?: 0L,
|
||||
)
|
||||
},
|
||||
)
|
||||
}.onFailure {
|
||||
Log.w("SyncCoverageFile") { "could not read ${file.path} (${it.message}); starting fresh" }
|
||||
}
|
||||
}
|
||||
|
||||
@Synchronized
|
||||
private fun save() {
|
||||
runCatching {
|
||||
val doc =
|
||||
buildJsonObject {
|
||||
coverage.export().forEach { (key, band) ->
|
||||
put(
|
||||
key,
|
||||
buildJsonObject {
|
||||
put("min", band.minCreatedAt)
|
||||
put("max", band.maxCreatedAt)
|
||||
put("complete", band.complete)
|
||||
put("fullAt", band.fullAt)
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
file.parentFile?.mkdirs()
|
||||
val tmp = File(file.parentFile ?: File("."), "${file.name}.tmp")
|
||||
tmp.writeText(json.encodeToString(JsonObject.serializer(), doc))
|
||||
// ATOMIC_MOVE requested explicitly: without it the JVM may
|
||||
// legally fall back to copy+delete, and a reader could see a
|
||||
// half map. Same-directory rename, so support is the norm; a
|
||||
// filesystem that truly can't gets the plain move (and the
|
||||
// corrupt-file recovery absorbs the residual risk).
|
||||
try {
|
||||
Files.move(tmp.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE)
|
||||
} catch (_: AtomicMoveNotSupportedException) {
|
||||
Files.move(tmp.toPath(), file.toPath(), StandardCopyOption.REPLACE_EXISTING)
|
||||
}
|
||||
}.onFailure {
|
||||
Log.w("SyncCoverageFile") { "could not write ${file.path}: ${it.message}" }
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
// Pretty-printed: this file is read by a human debugging why an
|
||||
// upstream re-synced.
|
||||
private val json = Json { prettyPrint = true }
|
||||
|
||||
// Often enough that a kill costs little, rare enough to be free.
|
||||
private const val DEFAULT_FLUSH_SECONDS = 30L
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.geode.mirror
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import java.io.File
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* The catch-up's resume memory across restarts. Without it every boot
|
||||
* re-syncs each upstream's whole backfill window — a full re-download for an
|
||||
* upstream without NIP-77. These pin the restart round-trip and the window
|
||||
* clamping that keys bands on the stable filter rather than the sliding
|
||||
* boot window.
|
||||
*/
|
||||
class SyncCoverageFileTest {
|
||||
private val relay = RelayUrlNormalizer.normalize("wss://relay.example")
|
||||
private val profiles = Filter(kinds = listOf(0))
|
||||
|
||||
private fun tempFile(): File {
|
||||
val f = File.createTempFile("sync-coverage", ".json")
|
||||
f.delete()
|
||||
return f
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `bands survive a restart`() {
|
||||
val f = tempFile()
|
||||
SyncCoverageFile(f).use {
|
||||
it.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
}
|
||||
|
||||
// A fresh instance, as a restart would build.
|
||||
SyncCoverageFile(f).use { reopened ->
|
||||
val band = reopened.coverage.band(relay, profiles)!!
|
||||
assertEquals(1_700_001_000L, band.minCreatedAt)
|
||||
assertEquals(1_700_002_000L, band.maxCreatedAt)
|
||||
assertFalse(band.complete)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a complete band survives with its completeness`() {
|
||||
val f = tempFile()
|
||||
SyncCoverageFile(f).use {
|
||||
it.coverage.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_005_000L)
|
||||
}
|
||||
SyncCoverageFile(f).use { reopened ->
|
||||
assertTrue(reopened.coverage.band(relay, profiles)!!.complete)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a corrupt file starts fresh instead of refusing to start`() {
|
||||
val f = tempFile()
|
||||
f.writeText("{ not json")
|
||||
SyncCoverageFile(f).use {
|
||||
assertNull(it.coverage.band(relay, profiles))
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `recording does not write but closing does`() {
|
||||
val f = tempFile()
|
||||
val store = SyncCoverageFile(f)
|
||||
store.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
assertFalse(f.isFile, "a record marks dirty; only a flush writes")
|
||||
store.close()
|
||||
assertTrue(f.isFile, "close flushes")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `reopening without new records does not rewrite the file`() {
|
||||
val f = tempFile()
|
||||
SyncCoverageFile(f).use {
|
||||
it.coverage.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
}
|
||||
val written = f.lastModified()
|
||||
SyncCoverageFile(f).close()
|
||||
assertEquals(written, f.lastModified(), "restoring bands must not mark the store dirty")
|
||||
}
|
||||
|
||||
// ---- the window clamp --------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `an unbanded filter clamps to exactly the boot window`() {
|
||||
val leg = clampToWindow(profiles, since = 1_000L, until = 2_000L)!!
|
||||
assertEquals(1_000L, leg.since)
|
||||
assertEquals(2_000L, leg.until)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a leg outside the window is dropped rather than inverted`() {
|
||||
// The band covers past the window's floor: the older leg would ask
|
||||
// [since..band.min] with since above until — a range nothing can be in.
|
||||
val olderLeg = profiles.copy(until = 500L)
|
||||
assertNull(clampToWindow(olderLeg, since = 1_000L, until = 2_000L))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a leg inside the window keeps its own tighter bound`() {
|
||||
val newerLeg = profiles.copy(since = 1_500L)
|
||||
val clamped = clampToWindow(newerLeg, since = 1_000L, until = 2_000L)!!
|
||||
assertEquals(1_500L, clamped.since, "the band's ceiling wins over the window floor")
|
||||
assertEquals(2_000L, clamped.until)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,221 @@
|
||||
# PoolRequests subscription-state lock: what a suspending Mutex would cost
|
||||
|
||||
**Date:** 2026-08-03
|
||||
**Status:** analysis + measurements; recommendation is NOT to use `Mutex`
|
||||
**Harness:** `quartz/src/jvmTest/.../prodbench/LockDesignComparisonBenchmark.kt`
|
||||
(`./gradlew :quartz:jvmTest --tests "*.LockDesignComparisonBenchmark"`)
|
||||
**Context:** follows the Pixel 8 ANR (`anr_2026-08-03-12-55-26-256`) that replaced
|
||||
`RequestSubscriptionState`'s busy-wait with a parking `PlatformLock`.
|
||||
|
||||
## The question
|
||||
|
||||
The parking lock removed the CPU burn, but it still **blocks** rather than suspends.
|
||||
Relay consumers run as coroutines on `Dispatchers.IO` (limitedParallelism 64) and
|
||||
production has ~191 live relays, so a blocked waiter occupies one of 64 dispatcher
|
||||
threads. kotlinx's scheduler treats IO tasks as blocking and grows the pool when they
|
||||
block — which is why the on-device fix moved CPU but left thread count flat (+7%).
|
||||
|
||||
Would `kotlinx.coroutines.sync.Mutex` (a waiter *suspends*, freeing its thread) be
|
||||
better?
|
||||
|
||||
## What Mutex actually costs
|
||||
|
||||
The lock sits at the bottom of a call chain that is **entirely non-suspend**:
|
||||
|
||||
```
|
||||
OkHttpWebSocket: scope.launch { for (m in incomingMessages) out.onMessage(m) } <- coroutine
|
||||
WebSocketListener.onMessage (non-suspend)
|
||||
BasicRelayClient.MyWebsocketListener.onMessage
|
||||
RelayPool.onIncomingMessage
|
||||
NostrClient.onIncomingMessage (RelayConnectionListener)
|
||||
PoolRequests.onIncomingMessage
|
||||
RequestSubscriptionState.withLock <- the lock
|
||||
SubscriptionListener.onEvent (non-suspend, fans out to the app)
|
||||
```
|
||||
|
||||
`Mutex.lock()` is `suspend`, so every frame above it must become `suspend`. Measured
|
||||
blast radius:
|
||||
|
||||
| interface / API | count |
|
||||
|---|---|
|
||||
| `override fun onEvent(` | 59 |
|
||||
| `override fun onEose(` | 52 |
|
||||
| `override fun onIncomingMessage(` | 35 |
|
||||
| `override fun onCannotConnect(` | 32 |
|
||||
| `override fun onClosed(` | 25 |
|
||||
| `override fun onDisconnected(` | 17 |
|
||||
| `override fun onConnected(` | 16 |
|
||||
| `override fun onSent(` | 14 |
|
||||
| `override fun onConnecting(` | 11 |
|
||||
| others (`onSubscriptionStarted`, …) | 1 |
|
||||
| **total overrides to convert** | **262** |
|
||||
| `.subscribe(` / `.unsubscribe(` call sites | **110** |
|
||||
|
||||
Worse than the count: the *entry points* are not all coroutines. `INostrClient` is
|
||||
non-suspend by design — `subscribe`, `unsubscribe`, `publish`, `syncFilters`,
|
||||
`connect` — and is called from ViewModels, filter assemblers and Compose effects.
|
||||
`PoolRequests.addOrUpdate` / `remove` / `sendToRelayIfChanged` reach the lock from
|
||||
those paths. Making them suspend pushes coroutine scoping into every call site that
|
||||
today just calls `subscribe(...)` synchronously.
|
||||
|
||||
## The cheaper alternative: stripe the lock per relay
|
||||
|
||||
**Enabling invariant (verified by reading all 11 `withLock` bodies):** *every*
|
||||
critical section in `PoolRequests` is scoped to exactly one relay. Each one takes
|
||||
`url` / `relay` / `relay.url` as its key:
|
||||
|
||||
| line | body | key |
|
||||
|---|---|---|
|
||||
| 204 | `state.connecting(url)` | url |
|
||||
| 218 | `state.onOpenReq(relay, cmd.filters)` | relay |
|
||||
| 228 | `state.onSubscriptionClosed(relay)` | relay |
|
||||
| 249 | `onNewEvent` / `currentState` / `lastKnownFilterStates` | relay.url |
|
||||
| 267 | `onEose` + `decideCommandLocked` | relay.url |
|
||||
| 296 | `onClosed` + `recordRefusalIfStructural` + `decideCommandLocked` | relay.url |
|
||||
| 325 | `state.disconnected(url)` | url |
|
||||
| 354 | `isStructurallyRefused` + `onOpenReq` | relay |
|
||||
| 382 | `lastKnownFilterStates(url)` | url |
|
||||
| 403 | `decideCommandLocked(state, subId, relay)` | relay |
|
||||
|
||||
Every field in `RequestSubscriptionState` is a `Map<T, …>` keyed by relay
|
||||
(`subStates`, `filterStates`, `lastKnownFilterStates`, `refusedFilters`,
|
||||
`refusalCounts`). The single cross-relay accessor, `currentFilters()` (no-arg,
|
||||
returns the whole map), has **zero usages** — dead code, delete it.
|
||||
|
||||
So the lock is only shared across relays as an artifact of `mutableMapOf` not being
|
||||
thread-safe. One lock per `(subId, relay)` is semantically equivalent and drops
|
||||
contention from ~191 threads to ~1–2 (that relay's consumer, plus the occasional app
|
||||
thread in `sendToRelayIfChanged`).
|
||||
|
||||
Blast radius: `RequestSubscriptionState` + `PoolRequests` only. Used by 4 production
|
||||
files (`PoolRequests`, `RelayActiveRequestStates`, `RelayReqRefusals`) and 2 tests.
|
||||
**No public API change.**
|
||||
|
||||
## Measurements
|
||||
|
||||
191 relay coroutines on a 64-thread limited-parallelism dispatcher, 3s windows. Each
|
||||
iteration yields, modelling the real per-message suspension point of
|
||||
`for (message in incomingMessages)` — without it the tight loops monopolise the
|
||||
dispatcher and the harness measures itself, not the lock.
|
||||
|
||||
`bystander` = a task that never touches the lock, on the same dispatcher. Its latency
|
||||
answers "is the lock stealing dispatcher threads from unrelated work?"
|
||||
|
||||
| subs | design | ops/s | bystander p50 | p99 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | PER_SUB_BLOCKING (today) | 548,608 | 196.5µs | 770.9µs |
|
||||
| 1 | PER_SUB_MUTEX | 190,672 | **1.9µs** | 30.4µs |
|
||||
| 1 | **STRIPED** | **1,543,037** | 88.8µs | 237.6µs |
|
||||
| 4 | PER_SUB_BLOCKING | 833,090 | 136.1µs | 507.6µs |
|
||||
| 4 | PER_SUB_MUTEX | 523,440 | **3.1µs** | 26.0µs |
|
||||
| 4 | **STRIPED** | **1,499,757** | 92.3µs | 337.5µs |
|
||||
| 16 | PER_SUB_BLOCKING | 1,198,581 | 94.3µs | 444.3µs |
|
||||
| 16 | PER_SUB_MUTEX | 749,487 | **5.0µs** | 16.5µs |
|
||||
| 16 | **STRIPED** | **1,789,509** | 85.7µs | 219.8µs |
|
||||
|
||||
Reading:
|
||||
|
||||
- **STRIPED gives the most throughput** — 2.8× / 1.8× / 1.5× over today — and roughly
|
||||
halves bystander p50. It removes the contention rather than tolerating it.
|
||||
- **MUTEX is the slowest of the three** (0.35× / 0.63× / 0.63× of today): per-acquisition
|
||||
suspend/resume is not free at these rates.
|
||||
- **MUTEX does win bystander latency decisively** (1.9–5.0µs vs 85–196µs, and it collected
|
||||
626k vs 24k samples). But read that honestly — part of the win is that its coroutines
|
||||
spend their time *suspended waiting for the mutex instead of doing work*. Better
|
||||
thread-yielding is partly a symptom of lower throughput, not purely a win.
|
||||
|
||||
## On-device result (SM-T220, playBenchmark, same account, n=3 per design)
|
||||
|
||||
Cold-start burst, 75s window, exact per-thread CPU accounting from `/proc/<tid>/stat`.
|
||||
Warmups matched (~220-255 established sockets, ~200 relay reader threads each time).
|
||||
|
||||
| design | DefaultDispatcher CPU (ms/75s) mean / median / range | GC ms | threads |
|
||||
|---|---|---|---|
|
||||
| spin lock (pre-fix) | 136,743 / 117,750 / 112,230–180,250 | 33,777 | 475 |
|
||||
| parking, one lock per sub | 103,780 / 99,160 / 86,120–130,680 | 30,235 | 510 |
|
||||
| **striped, per (sub, relay)** | **88,953 / 82,160 / 72,860–111,840** | **27,687** | 489 |
|
||||
|
||||
- **Striped vs spin: −35% mean, −30% median CPU, −18% GC — and the ranges do not
|
||||
overlap** (striped max 111,840 < spin min 112,230). That separation is what the
|
||||
parking-only change could not show; its range overlapped the baseline heavily.
|
||||
- **Striped vs parking: −14% mean / −17% median**, ranges still overlap — directional,
|
||||
not conclusive at n=3.
|
||||
- **Thread count is unchanged across all three** (475 / 510 / 489). Expected: the lock
|
||||
is blocking, and kotlinx marks IO tasks blocking and grows the pool. Striping reduces
|
||||
how *often* threads block, not the fact that they can.
|
||||
- Not a false win from broken subscriptions: the feed rendered live notes 3-15 min old
|
||||
with avatars and reaction counts, on 223 sockets / 200 relay reader threads.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Do the striping; do not convert to `Mutex`.**
|
||||
|
||||
Striping attacks the cause — the lock is contended only because it is shared across
|
||||
relays that touch disjoint keys. Once contention is ~0, the blocking-vs-suspending
|
||||
question is moot: *a lock that is never contended never blocks a thread*. It also
|
||||
happens to be the fastest option and is contained to two files, versus 262 overrides
|
||||
and 110 call sites for a design that measures slower.
|
||||
|
||||
`Mutex` only becomes the right answer if a future critical section must genuinely span
|
||||
relays (or do I/O), which none does today.
|
||||
|
||||
## The lock must not live inside a removable entry
|
||||
|
||||
The obvious shape — `ConcurrentMap<T, PerRelayState>` where `PerRelayState` owns both
|
||||
the fields *and* its lock — is **wrong as soon as entries can be removed**.
|
||||
`connecting()` / `disconnected()` genuinely mean "forget this relay's wire state", so
|
||||
they want removal, and then:
|
||||
|
||||
```
|
||||
T1: getOrPut(R) -> stateA ; stateA.lock.lock() // in a critical section
|
||||
T2: disconnected(R) -> remove(R) // stateA is now orphaned
|
||||
T3: getOrPut(R) -> stateB (NEW lock) ; stateB.lock.lock() // acquires immediately
|
||||
-> T1 and T3 are both "in" relay R's critical section, excluding nothing.
|
||||
```
|
||||
|
||||
Two ways out:
|
||||
|
||||
- **(A) never remove; null the fields.** Lock identity is stable, but entries
|
||||
accumulate for every relay a sub has *ever* seen. Bounded but monotonic — and slow
|
||||
monotonic growth in a long-lived process is precisely the class of bug this whole
|
||||
investigation was about.
|
||||
- **(B) stable stripe locks + removable state.** A fixed `Array(N) { PlatformLock() }`
|
||||
indexed by `relay.hashCode()`, never mutated, so lock identity can't change; the
|
||||
`ConcurrentMap<T, RelayState>` entries are then free to be added and removed with
|
||||
exact `connecting`/`disconnected` semantics and no growth.
|
||||
|
||||
**(B) is the recommendation.** With N = 32 and ~191 relays, ~6 relays share a stripe —
|
||||
still a ~32x contention reduction versus today's single lock per sub, with none of the
|
||||
lock-lifetime hazard. `ConcurrentMap.remove` (added 2026-08-03, with tests in
|
||||
`ConcurrentCollectionsTest`) is what makes (B) possible.
|
||||
|
||||
## Implementation sketch
|
||||
|
||||
1. Delete the dead `currentFilters()` (no-arg).
|
||||
2. In `RequestSubscriptionState`, replace the five relay-keyed maps with a single
|
||||
`ConcurrentMap<T, RelayState>` holding the five fields — **no lock inside**.
|
||||
3. Add a fixed `private val stripes = Array(32) { PlatformLock() }` and
|
||||
`withLock(reference)` = `stripes[reference.hashCode().absoluteValue % 32].withLock { }`.
|
||||
The array is never mutated, so lock identity is stable for the object's life.
|
||||
4. `connecting()` / `disconnected()` become `map.remove(reference)` — exact current
|
||||
semantics, no residue.
|
||||
5. Update the 11 `withLock` call sites in `PoolRequests` to pass the relay.
|
||||
`decideCommandLocked`'s "MUST hold the lock" contract becomes "MUST hold *that
|
||||
relay's stripe*" — tighten the kdoc.
|
||||
6. Extend `PoolRequestsRefusalTest` with concurrent multi-relay access on one subId,
|
||||
and add a striped variant to `LockDesignComparisonBenchmark` to confirm the
|
||||
measured win survives stripe collisions.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Stripe collisions serialize unrelated relays.** With 32 stripes and 191 relays this
|
||||
is ~6-way sharing; it is a contention *reduction*, not elimination. If a future
|
||||
profile shows it mattering, raise N — it is a one-line change with no semantic effect.
|
||||
- **Two relays on one stripe must never be locked simultaneously by one thread** — that
|
||||
would self-deadlock (`PlatformLock` is reentrant on JVM/Apple, so same-thread
|
||||
re-entry is survivable, but the invariant should be stated). `PoolRequests` already
|
||||
locks one relay at a time.
|
||||
- The state machine's atomicity comments are load-bearing; the per-relay scoping must
|
||||
be re-verified against any new `withLock` body added between now and the change.
|
||||
- Striping is NOT a fix for a critical section that does I/O or spans relays. If one is
|
||||
ever added, this analysis must be redone.
|
||||
@@ -0,0 +1,154 @@
|
||||
# Amethyst over Yggdrasil (IPv6 overlay)
|
||||
|
||||
Status: **fixed** — the four gaps found in the original assessment are closed. The last
|
||||
section records what was deliberately left alone.
|
||||
|
||||
## What Yggdrasil looks like to the app
|
||||
|
||||
Yggdrasil is an encrypted end-to-end mesh. Every node gets an IPv6 address derived from its
|
||||
public key inside `0200::/7` (nodes in `0200::/8`, subnets in `0300::/8`). Consequences:
|
||||
|
||||
- **No DNS.** A relay on the mesh is addressed as an IPv6 literal, always.
|
||||
- **No certificates.** No CA issues for `0200::/7`, so relays run plain `ws://`. Not a
|
||||
downgrade — the overlay already encrypts end to end and authenticates the peer by an
|
||||
address derived from its public key.
|
||||
- **On Android it is a `VpnService`**, so the app's default network becomes the VPN network.
|
||||
- `0200::/7` is deprecated NSAP space, so nothing else routes there. An address in the range
|
||||
is reachable *only* through a running mesh interface — which is what makes it safe to key
|
||||
behavior off the prefix.
|
||||
|
||||
## What was wrong, and what fixed it
|
||||
|
||||
### 1. One relay, two identities
|
||||
|
||||
`RelayUrlNormalizer` folded hex case but not zero-compression, so
|
||||
`[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]` stayed two distinct
|
||||
`NormalizedRelayUrl`s for one host — while OkHttp collapsed both to the same host when
|
||||
dialing. Since that value keys the connection pool, the relay-list sets, the NIP-11 cache and
|
||||
the per-relay stat maps, the app opened two sockets to one relay and counted it twice.
|
||||
|
||||
**Fix:** new `Ipv6` util (`quartz/utils/Ipv6.kt`) — pure-Kotlin parse, RFC 5952 canonical
|
||||
format and range classification, no `java.net`, so it works on every KMP target.
|
||||
`RelayUrlNormalizer.norm()` now canonicalizes the bracketed host. The canonical form is
|
||||
byte-for-byte what OkHttp renders, so the key the app stores is the host it actually dials —
|
||||
asserted differentially against OkHttp in `YggdrasilCompatCharacterizationTest`.
|
||||
|
||||
Affects every IPv6 relay, not just mesh ones; it only bit Yggdrasil users because on the mesh
|
||||
a literal is the *only* way to name a relay. No migration needed: relay lists are rehydrated
|
||||
from event tags through `normalizeOrNull`, so stored entries fold on load.
|
||||
|
||||
### 2. Schemeless entry defaulted to `wss://`
|
||||
|
||||
`isLocalHost()` knew `127.0.0.1` / `localhost` / `//umbrel:` / `192.168.` / `.local`, so a
|
||||
mesh address fell through to the clearnet default and produced a `wss://` url whose TLS
|
||||
handshake could never succeed.
|
||||
|
||||
**Fix:** new `RelayUrlNormalizer.isOverlayNetwork()` recognizes `0200::/7` and joins
|
||||
`isOnion` / `isLocalHost` in choosing `ws://`. Clearnet IPv6 (`2001:db8::1`) still gets
|
||||
`wss://`.
|
||||
|
||||
`isLocalHost()` separately grew the IPv6 twins of the literals it already knew — `::1`
|
||||
(loopback), `fc00::/7` (unique local, the 192.168. analogue) and `fe80::/10` (link-local).
|
||||
Those are the same question every caller is asking, so a relay on one now correctly skips TLS
|
||||
and Tor and stays out of published relay lists.
|
||||
|
||||
### 3. Unbracketed literal silently rejected
|
||||
|
||||
`yggdrasilctl getSelf` prints the address unbracketed — exactly what gets pasted into "add a
|
||||
relay". Normalization returned null (correctly: it is ambiguous with a scheme) and
|
||||
`RelayUrlEditField.submitRelay` had no else branch, so the Add button did nothing at all.
|
||||
|
||||
**Fix, two halves:**
|
||||
- `fix()` brackets a bare literal automatically, but only when the whole string parses as an
|
||||
IPv6 address — so `31990:hex:dtag` (addressable pointer), `abcd:1234` (host:port) and
|
||||
`relay.example.com:8080` still fall through untouched.
|
||||
- The edit field now sets `isError` and shows `relay_url_not_valid` instead of no-opping.
|
||||
That fixes the dead button for *all* invalid input, not just IPv6.
|
||||
|
||||
### 4. Tor routing broke mesh relays
|
||||
|
||||
`TorRelayEvaluation` classified mesh relays as "new", so with Tor on and the default "new
|
||||
relays via Tor" they were dialed through the SOCKS proxy — which cannot route `0200::/7`.
|
||||
Guaranteed failure, not privacy.
|
||||
|
||||
**Fix:** `useTor()` returns false for `isOverlayNetwork()`, checked right after the localhost
|
||||
branch. Both the Android and desktop relay paths delegate here (`TorRelayState`,
|
||||
`DesktopHttpClient`), so one change covers both. `RoleBasedHttpClientBuilder` got the same
|
||||
treatment for non-relay HTTP (images, previews, NIP-05, money ops). Clearnet IPv6 relays keep
|
||||
following the Tor setting — asserted in `YggdrasilTorRoutingTest`.
|
||||
|
||||
## Audit round: bugs found in the host predicates
|
||||
|
||||
Auditing the change above turned up a family of bugs in `isLocalHost` / `isOnion` that predate
|
||||
it. All shared one root cause — the predicates ran `contains` over the **whole url** instead of
|
||||
parsing the authority — and all are now anchored, parsed and covered by
|
||||
`RelayUrlAuthorityAnchoringTest`.
|
||||
|
||||
These predicates decide whether a relay is exempt from Tor, and relay urls arrive from other
|
||||
people (NIP-65 lists, relay hints, `r` tags), so they are attacker-controlled input.
|
||||
|
||||
| # | Bug | Effect |
|
||||
|---|---|---|
|
||||
| 1 | A path could impersonate the host: `wss://evil.example.com/127.0.0.1` answered `isLocalHost() == true` | Any relay list could hand the app a url that silently dropped its own Tor routing |
|
||||
| 2 | `.onion:8080` never matched the `.onion/` test | An onion relay on an explicit port was not treated as onion — never forced onto Tor, hostname sent to the clearnet DNS resolver |
|
||||
| 3 | `.onion.` / `localhost.` (RFC 1034 fully-qualified form) matched nothing | Same leak as #2, via a different spelling |
|
||||
| 4 | Host tests were case-sensitive, but `fix()` runs *before* the RFC 3986 pass folds case | `LOCALHOST:8080` and `ABC.ONION:8080` were given a `wss://` scheme neither host can serve |
|
||||
| 5 | Private IPv4 was substring-matched | `10.0.0.5`, `172.16.3.4`, `127.1.2.3` were not local (LAN relay got `wss://` and Tor), while `192.168.evil.com` — a registrable domain — was |
|
||||
| 6 | `contains("localhost")` matched `notlocalhost.example.com` | Same Tor exemption as #1, via a registrable domain |
|
||||
| 7 | A `://` inside a path was read as a scheme separator | `relay.com/x://127.0.0.1` read its path as the authority |
|
||||
|
||||
Two bugs were introduced by this branch and caught in the same pass: the IPv6 host lookup had
|
||||
the #1 flaw (`wss://evil.example.com/[fd00::1]` read as localhost), and IPv6 canonicalization
|
||||
could rewrite a **path** (`/x[0:0:0:0:0:0:0:1]y` → `/x[::1]y`), corrupting the url.
|
||||
|
||||
Fixes: a shared `hostStart` / `hostEnd` / `hostEndWithoutPort` trio bounds every test to the
|
||||
authority, strips `:port` and trailing dots, and validates the scheme; private ranges are
|
||||
parsed via the new `Ipv4` util and `Ipv6` rather than substring-matched; comparisons are
|
||||
case-insensitive per RFC 4343; `NormalizedRelayUrl.isOnion()` now delegates instead of keeping
|
||||
a second, weaker copy of the test.
|
||||
|
||||
Performance: no regression, likely a small win. The old form ran six full-string `contains`
|
||||
scans; the new one bounds its work to the authority and rejects a DNS host from an IP parse on
|
||||
a single character. `Ipv6.isLiteral` gained a two-colon gate so the schemeless `host:port` case
|
||||
answers without allocating the parser's 16-byte buffer.
|
||||
|
||||
`Ipv6` itself is pinned by `Ipv6DifferentialTest`: 4000 random addresses round-trip against
|
||||
`java.net.InetAddress` in both directions, and the canonical form is asserted equal to
|
||||
OkHttp's host for the same address — so the relay identity the app stores provably matches the
|
||||
host it dials.
|
||||
|
||||
## Deliberately not changed
|
||||
|
||||
**Mesh relays are still published and recommended.** `AdvertisedRelayInfoTag` (NIP-65) and
|
||||
`RelayListRecommendationProcessor.filterValidRelays` only exclude localhost, so a mesh relay
|
||||
in your relay list is still published to public relays and offered to other users via the
|
||||
outbox model. Two consequences worth a maintainer's decision:
|
||||
|
||||
- Peers not on the mesh dial `[201:…]` and burn reconnect attempts on an unreachable host.
|
||||
- Your Yggdrasil address — a stable, key-derived node identifier — becomes public.
|
||||
|
||||
Onion relays already have precedent for both readings: they *are* published, but
|
||||
`filterValidRelays` gates them behind `hasOnionConnection`. The equivalent for overlay relays
|
||||
would be a `hasMeshConnection` gate. That is a product call about whether mesh relays are
|
||||
meant to be discoverable, so it is flagged rather than decided here.
|
||||
|
||||
## Not verified here
|
||||
|
||||
- **No live socket test.** The analysis container has no IPv6 stack at all (`AF_INET6` →
|
||||
`EAFNOSUPPORT`), so everything is verified below the socket: normalization, OkHttp URL/host
|
||||
agreement, and the Tor routing decision. An on-device run against a real mesh relay is
|
||||
still needed to confirm the happy path end to end.
|
||||
- **Android VPN interaction untested.** `ConnectivityFlow` uses
|
||||
`registerDefaultNetworkCallback`, so it follows the app into the VPN network. Whether
|
||||
`isMeteredOrMobileData()` reads correctly through Yggdrasil's `VpnService` depends on
|
||||
whether that app declares underlying networks — worth checking on device before assuming
|
||||
data-saving mode behaves.
|
||||
- Media loading (Coil) and NIP-05 resolution against mesh hosts were not exercised.
|
||||
|
||||
## Tests
|
||||
|
||||
- `quartz/…/utils/Ipv6Test.kt` — parser, RFC 5952 formatting, range classification.
|
||||
- `quartz/…/relay/YggdrasilCompatCharacterizationTest.kt` — normalization end to end, plus
|
||||
the differential assertions that our identity matches the host OkHttp dials.
|
||||
- `commons/…/tor/YggdrasilTorRoutingTest.kt` — overlay relays never Torified, clearnet IPv6
|
||||
still follows the setting.
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils.concurrent
|
||||
|
||||
import platform.Foundation.NSRecursiveLock
|
||||
|
||||
// NSRecursiveLock parks contended waiters in the kernel, matching the
|
||||
// ReentrantLock semantics of the jvmAndroid actual (and the same choice
|
||||
// commons' KmpLock made for iOS). This must NOT be a spin lock: quartz's
|
||||
// relay client runs here too, and spinning is what produced the Android
|
||||
// ANR documented on the expect declaration.
|
||||
actual class PlatformLock {
|
||||
private val delegate = NSRecursiveLock()
|
||||
|
||||
actual fun lock() = delegate.lock()
|
||||
|
||||
actual fun unlock() = delegate.unlock()
|
||||
}
|
||||
+4
-8
@@ -25,12 +25,13 @@ import com.vitorpamplona.quartz.nip01Core.core.AddressableEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.core.isReplaceable
|
||||
import com.vitorpamplona.quartz.nip01Core.core.supersedes
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.store.ObservableEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.ObservableEventStore.StoreChange
|
||||
import com.vitorpamplona.quartz.nip01Core.store.owner
|
||||
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
|
||||
import com.vitorpamplona.quartz.nip40Expiration.isExpirationBefore
|
||||
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
|
||||
import com.vitorpamplona.quartz.nip62RequestToVanish.RequestToVanishEvent
|
||||
import com.vitorpamplona.quartz.utils.SortedList
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
@@ -342,19 +343,14 @@ class EventStoreProjection<T : Event>(
|
||||
private fun supersedes(
|
||||
new: Event,
|
||||
existing: Event,
|
||||
): Boolean =
|
||||
when {
|
||||
new.createdAt > existing.createdAt -> true
|
||||
new.createdAt < existing.createdAt -> false
|
||||
else -> new.id < existing.id
|
||||
}
|
||||
): Boolean = new.supersedes(existing)
|
||||
|
||||
/**
|
||||
* Owner pubkey for ownership checks (NIP-09 author match,
|
||||
* NIP-62 vanish target). For GiftWrap the owner is the p-tag
|
||||
* recipient; for everything else it's `event.pubKey`.
|
||||
*/
|
||||
private fun ownerOf(event: Event): HexKey = (event as? GiftWrapEvent)?.recipientPubKey() ?: event.pubKey
|
||||
private fun ownerOf(event: Event): HexKey = event.owner()
|
||||
|
||||
/**
|
||||
* created_at DESC, id ASC. Sort keys are frozen at slot
|
||||
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.core
|
||||
|
||||
/**
|
||||
* The NIP-01 replaceable/addressable supersession rule: the winner of an
|
||||
* address is the highest `created_at`, with ties broken by the LEXICALLY
|
||||
* SMALLEST id. True when THIS event beats [existing] — equal events (same id)
|
||||
* do not supersede themselves. One rule, shared by every store; the SQLite
|
||||
* triggers encode the same comparison in SQL.
|
||||
*/
|
||||
fun Event.supersedes(existing: Event): Boolean =
|
||||
when {
|
||||
createdAt > existing.createdAt -> true
|
||||
createdAt < existing.createdAt -> false
|
||||
else -> id < existing.id
|
||||
}
|
||||
+17
-3
@@ -34,6 +34,7 @@ import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.channels.Channel.Factory.UNLIMITED
|
||||
import kotlinx.coroutines.coroutineScope
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
/**
|
||||
* One relay's verdict on a published event: [accepted] plus the reason the
|
||||
@@ -45,6 +46,12 @@ import kotlinx.coroutines.withTimeoutOrNull
|
||||
class PublishResult(
|
||||
val accepted: Boolean,
|
||||
val message: String,
|
||||
/**
|
||||
* Milliseconds from the publish to this relay's OK (true or false — a rejection
|
||||
* is still a measured round trip), or -1 when the relay never answered with an
|
||||
* OK. On an already-open socket this is an honest NIP-66 `rtt-write`.
|
||||
*/
|
||||
val elapsedMs: Long = -1,
|
||||
) {
|
||||
/**
|
||||
* True when this failure came from the transport (never connected,
|
||||
@@ -102,6 +109,7 @@ suspend fun INostrClient.publishAndCollectResults(
|
||||
timeoutInSeconds: Long = 15,
|
||||
): Map<NormalizedRelayUrl, PublishResult> {
|
||||
val resultChannel = Channel<DetailedResult>(UNLIMITED)
|
||||
val mark = TimeSource.Monotonic.markNow()
|
||||
|
||||
Log.d("publishAndConfirm") { "Waiting for ${relayList.size} responses" }
|
||||
|
||||
@@ -133,8 +141,13 @@ suspend fun INostrClient.publishAndCollectResults(
|
||||
|
||||
when (msg) {
|
||||
is OkMessage -> {
|
||||
if (msg.eventId == event.id) {
|
||||
resultChannel.trySend(DetailedResult(relay.url, msg.success, msg.message))
|
||||
// The relayList guard matters, not just the id: the same event may
|
||||
// have been published to OTHER relays by an earlier call (probe
|
||||
// waves, republish), and counting their late OKs here would inflate
|
||||
// receivedResults and end the wait loop before every listed relay
|
||||
// answered — misreporting the missing ones as NO_RESPONSE.
|
||||
if (msg.eventId == event.id && relay.url in relayList) {
|
||||
resultChannel.trySend(DetailedResult(relay.url, msg.success, msg.message, mark.elapsedNow().inWholeMilliseconds))
|
||||
Log.d("publishAndConfirm") { "onSendResponse Received response for ${msg.eventId} from relay ${relay.url} message ${msg.message} success ${msg.success}" }
|
||||
}
|
||||
}
|
||||
@@ -160,7 +173,7 @@ suspend fun INostrClient.publishAndCollectResults(
|
||||
val currentResult = receivedResults[result.relay]
|
||||
// do not override a successful result.
|
||||
if (currentResult == null || !currentResult.accepted) {
|
||||
receivedResults[result.relay] = PublishResult(result.success, result.message)
|
||||
receivedResults[result.relay] = PublishResult(result.success, result.message, result.elapsedMs)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -191,4 +204,5 @@ private class DetailedResult(
|
||||
val relay: NormalizedRelayUrl,
|
||||
val success: Boolean,
|
||||
val message: String,
|
||||
val elapsedMs: Long = -1,
|
||||
)
|
||||
|
||||
+313
@@ -0,0 +1,313 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.accessories
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
|
||||
/**
|
||||
* How much of a filter's history has already been pulled from one relay, so a
|
||||
* restart does not pull it again.
|
||||
*
|
||||
* A negentropy relay needs none of this — reconciliation downloads only the
|
||||
* diff. Most relays lack NIP-77, and a paged fetch ([fetchAllPages]) has no
|
||||
* memory: it re-downloads everything it walked last time, every restart,
|
||||
* forever. So for those, remember the band of `created_at` covered per
|
||||
* (relay, filter), and the next run asks only for the two legs outside it:
|
||||
*
|
||||
* stored band: |<-------- covered -------->|
|
||||
* next fetch: <------| |------>
|
||||
*
|
||||
* Keyed by the WHOLE filter deliberately: any edit to a filter is a new key
|
||||
* with no band, so the next run starts over — the safe direction to be wrong
|
||||
* in, and the intended way to force a re-walk.
|
||||
*
|
||||
* A band does not guarantee completeness (a truncating relay, an event
|
||||
* back-dated into a walked span). The trade is deliberate: re-reading a
|
||||
* corpus on every restart is a certain daily cost, while both holes are
|
||||
* occasional and self-heal on the next filter change or full re-walk.
|
||||
*
|
||||
* Persistence is the caller's: [export] the map on a schedule and [restore]
|
||||
* it at startup. [onChange] fires whenever a band changes, so a persistence
|
||||
* layer can mark itself dirty without polling.
|
||||
*
|
||||
* Not to be confused with the `relay.client.paging` package: its
|
||||
* `RelayLoadingCursors` are in-memory POSITIONS for demand-driven UI paging
|
||||
* within one session, while these are persistent INTERVALS — a claim about
|
||||
* coverage that outlives the process and licenses skipping work.
|
||||
*/
|
||||
class SyncCoverage(
|
||||
// How long a band may narrow work before the whole filter is walked
|
||||
// again. Everything a band claims is a claim about the past; this is how
|
||||
// long to trust it without re-testing.
|
||||
private val fullResyncSeconds: Long = DEFAULT_FULL_RESYNC_SECONDS,
|
||||
private val now: () -> Long = { TimeUtils.now() },
|
||||
private val onChange: () -> Unit = {},
|
||||
) {
|
||||
/**
|
||||
* What is already covered for one (relay, filter) pair.
|
||||
*
|
||||
* [complete] is the difference between "we walked this span" (a paged
|
||||
* fetch) and "we are in sync below this point" (a finished negentropy
|
||||
* reconcile, which compared the whole range). Only a complete band may
|
||||
* skip its older leg.
|
||||
*
|
||||
* [fullAt] is when the last pass that started from nothing finished — the
|
||||
* clock for the periodic re-walk.
|
||||
*/
|
||||
data class Band(
|
||||
val minCreatedAt: Long,
|
||||
val maxCreatedAt: Long,
|
||||
val complete: Boolean = false,
|
||||
val fullAt: Long = 0,
|
||||
)
|
||||
|
||||
private val bands = ConcurrentMap<String, Band>()
|
||||
|
||||
// filter -> its canonical json. Filter.toJson() runs to tens of thousands
|
||||
// of characters for author-scoped filters, and a fan-out keys once per
|
||||
// relay per cycle over the SAME handful of filter instances. Filter
|
||||
// compares by identity, so this map is an identity cache — and an
|
||||
// identity cache retains every distinct instance it is handed. A caller
|
||||
// that rebuilds its filter each cycle would grow it forever, so past
|
||||
// MAX_FINGERPRINTS new instances key correctly but are not cached.
|
||||
private val fingerprints = ConcurrentMap<Filter, String>()
|
||||
|
||||
/**
|
||||
* The filters to actually run now, given what is already covered: the
|
||||
* whole filter when nothing is recorded (or the band went stale),
|
||||
* otherwise the legs outside the band, clamped to the filter's own
|
||||
* `since`/`until`.
|
||||
*
|
||||
* The legs are INCLUSIVE of the band's edges (`until = min`, not
|
||||
* `min - 1`): a page boundary can split a run of events sharing one
|
||||
* `created_at`, and excluding the edge would strand the rest of that
|
||||
* second in no leg at all. The cost is re-reading one second's worth of
|
||||
* events per leg, which a store rejects as duplicates.
|
||||
*/
|
||||
fun legs(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
floor: Long? = null,
|
||||
): List<Filter> {
|
||||
val band = bands[key(url, filter)] ?: return listOf(filter)
|
||||
// Time for another full pass: relays gain old events, and without
|
||||
// this the band's claim is never re-tested.
|
||||
if (isStale(band)) return listOf(filter)
|
||||
val legs = mutableListOf<Filter>()
|
||||
|
||||
// Older: up to and including the band's floor, but not past the
|
||||
// filter's (or, when the filter has no `since`, the caller's
|
||||
// [floor] — a sync window the filter itself must not carry, or it
|
||||
// would change the band's key every run). A complete band compared
|
||||
// its whole range already, but only down to the floor it ran
|
||||
// against: a caller now reaching deeper — a raised backfill window
|
||||
// — re-opens the span below the band.
|
||||
val since = filter.since ?: floor
|
||||
val wantsOlder =
|
||||
if (band.complete) {
|
||||
since != null && since < band.minCreatedAt
|
||||
} else {
|
||||
since == null || band.minCreatedAt >= since
|
||||
}
|
||||
if (wantsOlder) {
|
||||
legs.add(filter.copy(until = minOf(band.minCreatedAt, filter.until ?: Long.MAX_VALUE)))
|
||||
}
|
||||
|
||||
// Newer: from the band's ceiling on, but not past the filter's.
|
||||
if (filter.until == null || band.maxCreatedAt <= filter.until) {
|
||||
legs.add(filter.copy(since = maxOf(band.maxCreatedAt, filter.since ?: Long.MIN_VALUE)))
|
||||
}
|
||||
return legs
|
||||
}
|
||||
|
||||
/**
|
||||
* Widen the band for (url, filter) to include what a completed fetch saw.
|
||||
*
|
||||
* [paged] gates the mechanism: a negentropy sync needs no band, and
|
||||
* recording one would only risk narrowing a future reconciliation.
|
||||
* Nothing is recorded for a fetch that saw no events — an empty result
|
||||
* says nothing about what the relay holds.
|
||||
*
|
||||
* [reconciledThrough] is the strong case: a FINISHED reconcile compared
|
||||
* the filter's whole range, so the caller is in sync up to the instant
|
||||
* the sync STARTED — recorded against that instant rather than the newest
|
||||
* event seen, because "the relay had nothing newer" and "we never asked"
|
||||
* must not look alike.
|
||||
*/
|
||||
fun record(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
observedMin: Long?,
|
||||
observedMax: Long?,
|
||||
paged: Boolean,
|
||||
reconciledThrough: Long? = null,
|
||||
) {
|
||||
if (reconciledThrough != null) {
|
||||
put(url, filter, observedMin ?: reconciledThrough, reconciledThrough, complete = true)
|
||||
return
|
||||
}
|
||||
if (!paged) return
|
||||
// Guarded even though callers should filter with [isPlausible] per
|
||||
// event: a 1970 floor or a far-future ceiling would make the band
|
||||
// claim the whole timeline, and the leg outside it would ask for a
|
||||
// range nothing can be in, forever.
|
||||
if (observedMin == null || observedMax == null) return
|
||||
if (!isPlausible(observedMin, now()) || !isPlausible(observedMax, now())) return
|
||||
put(url, filter, observedMin, observedMax, complete = false)
|
||||
}
|
||||
|
||||
/**
|
||||
* Widen (or reset) the band. A pass that ran because the previous band
|
||||
* had gone stale REPLACES it: it re-walked the whole filter, so its own
|
||||
* span is the complete picture and [Band.fullAt] restarts from here.
|
||||
*/
|
||||
private fun put(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
min: Long,
|
||||
max: Long,
|
||||
complete: Boolean,
|
||||
) {
|
||||
val fresh = Band(min, max, complete, now())
|
||||
bands.merge(key(url, filter), fresh) { old, new ->
|
||||
if (isStale(old)) {
|
||||
new
|
||||
} else {
|
||||
Band(
|
||||
minOf(old.minCreatedAt, new.minCreatedAt),
|
||||
maxOf(old.maxCreatedAt, new.maxCreatedAt),
|
||||
old.complete || new.complete,
|
||||
old.fullAt,
|
||||
)
|
||||
}
|
||||
}
|
||||
onChange()
|
||||
}
|
||||
|
||||
private fun isStale(band: Band): Boolean = now() - band.fullAt >= fullResyncSeconds
|
||||
|
||||
/**
|
||||
* The narrowest single filter that still covers what every one of [urls]
|
||||
* needs — the window a shared negentropy snapshot has to be taken over.
|
||||
*
|
||||
* In steady state every relay carries a complete band and this collapses
|
||||
* to `since = the oldest of their ceilings` — the difference between
|
||||
* snapshotting an id set of millions and one of a few thousand. One relay
|
||||
* that has never synced puts it back to the full filter, correctly: that
|
||||
* relay genuinely needs everything.
|
||||
*/
|
||||
fun coveringWindow(
|
||||
urls: List<NormalizedRelayUrl>,
|
||||
filter: Filter,
|
||||
): Filter {
|
||||
if (urls.isEmpty()) return filter
|
||||
var since = Long.MAX_VALUE
|
||||
for (url in urls) {
|
||||
val legs = legs(url, filter)
|
||||
// Nothing outside its band: this relay asks nothing of the
|
||||
// snapshot at all — the best case must not widen the window.
|
||||
if (legs.isEmpty()) continue
|
||||
// More than one leg means an older gap this relay still wants, so
|
||||
// the snapshot cannot start above the filter's own floor.
|
||||
val only = legs.singleOrNull() ?: return filter
|
||||
val legSince = only.since ?: return filter
|
||||
since = minOf(since, legSince)
|
||||
}
|
||||
// Every relay fully covered: any window would do; the unnarrowed
|
||||
// filter is merely safe, and callers usually skip the sync entirely.
|
||||
return if (since == Long.MAX_VALUE) filter else filter.copy(since = since)
|
||||
}
|
||||
|
||||
/** What is currently covered, for logging and tests. */
|
||||
fun band(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
): Band? = bands[key(url, filter)]
|
||||
|
||||
fun size(): Int = bands.size()
|
||||
|
||||
/** A point-in-time copy of every band, for a persistence layer to write out. */
|
||||
fun export(): Map<String, Band> = bands.snapshot()
|
||||
|
||||
/** Load previously [export]ed bands, e.g. at startup. */
|
||||
fun restore(entries: Map<String, Band>) {
|
||||
for ((key, band) in entries) bands[key] = band
|
||||
}
|
||||
|
||||
/**
|
||||
* The identity of one (relay, filter) pair. [Filter.toJson] is the
|
||||
* protocol's own canonical form, so two filters that mean the same thing
|
||||
* key the same way and any edit keys differently — exactly the "config
|
||||
* changed, start over" rule.
|
||||
*/
|
||||
private fun key(
|
||||
url: NormalizedRelayUrl,
|
||||
filter: Filter,
|
||||
): String {
|
||||
val fingerprint =
|
||||
fingerprints[filter]
|
||||
?: filter.toJson().also {
|
||||
// Bounded: stable callers hit the cache at any size a real
|
||||
// config produces; a caller minting fresh instances just
|
||||
// pays the toJson each time instead of growing the heap.
|
||||
if (fingerprints.size() < MAX_FINGERPRINTS) fingerprints[filter] = it
|
||||
}
|
||||
return "${url.url} $fingerprint"
|
||||
}
|
||||
|
||||
companion object {
|
||||
// More filter instances than any deliberate configuration holds; only
|
||||
// a caller rebuilding filters per cycle ever reaches it.
|
||||
private const val MAX_FINGERPRINTS = 1_000
|
||||
|
||||
/**
|
||||
* A week. Long enough that the narrow path is the normal one, short
|
||||
* enough that anything a band is wrong about is wrong for days, not
|
||||
* forever.
|
||||
*/
|
||||
const val DEFAULT_FULL_RESYNC_SECONDS = 7L * 24 * 60 * 60
|
||||
|
||||
/**
|
||||
* 2020-01-01. Below this a `created_at` is a bug, not a date — the
|
||||
* protocol did not exist. Also the natural floor for measuring a
|
||||
* paged walk's progress when a filter names no `since`.
|
||||
*/
|
||||
const val PLAUSIBLE_FLOOR = 1_577_836_800L
|
||||
|
||||
// Clock skew a relay may legitimately be ahead by. Past this, a
|
||||
// created_at is the author's fiction rather than a time.
|
||||
private const val FUTURE_SKEW_SECONDS = 86_400L
|
||||
|
||||
/**
|
||||
* Whether a `created_at` can be believed as evidence of coverage.
|
||||
* Filter with this per EVENT, not over a leg's aggregate: one
|
||||
* misdated event among hundreds of thousands would otherwise discard
|
||||
* the whole relay's band.
|
||||
*/
|
||||
fun isPlausible(
|
||||
createdAt: Long,
|
||||
now: Long = TimeUtils.now(),
|
||||
): Boolean = createdAt in PLAUSIBLE_FLOOR..(now + FUTURE_SKEW_SECONDS)
|
||||
}
|
||||
}
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
|
||||
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
import kotlin.concurrent.Volatile
|
||||
|
||||
/**
|
||||
* How far a bulk pagination has got, measured on the time axis — the only
|
||||
* axis whose end is known in advance.
|
||||
*
|
||||
* A paged fetch (`fetchAllPages`) has no event denominator: how many events
|
||||
* exist is exactly what it is finding out, so every count-based percentage
|
||||
* degenerates to `downloaded/downloaded = 100%`. The time axis has both ends
|
||||
* before the first request — the filter's `until` (or now) down to its
|
||||
* `since` (or the accessories' `SyncCoverage.PLAUSIBLE_FLOOR`) — with each
|
||||
* page's new `until` cursor reporting the exact position between them. It
|
||||
* needs no COUNT support.
|
||||
*
|
||||
* The estimate assumes events are spread evenly over time, which they are
|
||||
* not — so it errs pessimistic on the tail, and is a bound, not a promise.
|
||||
*
|
||||
* One instance can serve many concurrent paginations: keys are
|
||||
* `"group|name"`, and the group prefix scopes [fraction], [reached] and
|
||||
* [etaMs] so two groups never report each other's numbers.
|
||||
*
|
||||
* Its siblings in this package track different things: [RelayLoadingCursors]
|
||||
* is demand-driven `until`+`limit` paging for one scope (a feed pulling
|
||||
* older pages on demand, no window), and [RelayPagingProgress] is the
|
||||
* per-relay display state derived from it. This class is for a BULK
|
||||
* pagination over a known `[since, until]` window, where "how far through
|
||||
* the window, and when will it finish" is the question.
|
||||
*/
|
||||
class PagingWindowProgress(
|
||||
private val nowMillis: () -> Long = { TimeUtils.nowMillis() },
|
||||
) {
|
||||
private class Window(
|
||||
val top: Long,
|
||||
val bottom: Long,
|
||||
val startedMs: Long,
|
||||
@Volatile var current: Long,
|
||||
)
|
||||
|
||||
private val windows = ConcurrentMap<String, Window>()
|
||||
|
||||
/**
|
||||
* Begin a pagination over `[bottom, top]` seconds. An inverted window is
|
||||
* not one; a single-second window (`top == bottom`) is — coverage legs
|
||||
* that re-read a band's edge second are exactly that shape.
|
||||
*/
|
||||
fun begin(
|
||||
key: String,
|
||||
top: Long,
|
||||
bottom: Long,
|
||||
) {
|
||||
if (top >= bottom) windows[key] = Window(top, bottom, nowMillis(), top)
|
||||
}
|
||||
|
||||
/**
|
||||
* The pagination reached [until]; monotonic, so a page that jumps back
|
||||
* cannot un-advance it. The check-then-set is unsynchronized on purpose:
|
||||
* one pagination is one coroutine, and a display racing a mark can only
|
||||
* ever read a value one page stale.
|
||||
*/
|
||||
fun mark(
|
||||
key: String,
|
||||
until: Long,
|
||||
) {
|
||||
windows[key]?.let {
|
||||
// Clamped to the window's own floor: relays serve events stamped
|
||||
// 0, and one of those would drag the position to the epoch. Below
|
||||
// the floor means the pagination is done, not time travel.
|
||||
val reached = until.coerceAtLeast(it.bottom)
|
||||
if (reached < it.current) it.current = reached
|
||||
}
|
||||
}
|
||||
|
||||
fun finish(key: String) {
|
||||
windows.remove(key)
|
||||
}
|
||||
|
||||
/**
|
||||
* Fraction complete, averaged over every pagination still going in
|
||||
* [group] (or all of them when null) — averaged rather than summed
|
||||
* because each covers its own span, so "half of them done and half at
|
||||
* zero" is 50%.
|
||||
*/
|
||||
fun fraction(group: String? = null): Double? {
|
||||
val live = live(group)
|
||||
if (live.isEmpty()) return null
|
||||
return live.sumOf { w ->
|
||||
val span = (w.top - w.bottom).coerceAtLeast(1)
|
||||
((w.top - w.current).toDouble() / span).coerceIn(0.0, 1.0)
|
||||
} / live.size
|
||||
}
|
||||
|
||||
private fun live(group: String?): List<Window> =
|
||||
if (group == null) {
|
||||
windows.snapshot().values.toList()
|
||||
} else {
|
||||
windows
|
||||
.snapshot()
|
||||
.entries
|
||||
.filter { it.key.startsWith("$group|") }
|
||||
.map { it.value }
|
||||
}
|
||||
|
||||
/** The oldest second [group] has reached, or null when nothing is paging. */
|
||||
fun reached(group: String? = null): Long? = live(group).minOfOrNull { it.current }
|
||||
|
||||
/** Milliseconds left at the rate achieved so far, or null before it means anything. */
|
||||
fun etaMs(group: String? = null): Long? {
|
||||
val f = fraction(group) ?: return null
|
||||
// Under a few percent the extrapolation is dominated by connect time
|
||||
// and produces numbers worse than saying nothing.
|
||||
if (f < 0.02) return null
|
||||
val oldestStart = live(group).minOfOrNull { it.startedMs } ?: return null
|
||||
val elapsed = nowMillis() - oldestStart
|
||||
if (elapsed < 5_000) return null
|
||||
return ((elapsed / f) - elapsed).toLong()
|
||||
}
|
||||
}
|
||||
+8
-4
@@ -29,10 +29,14 @@ object FiltersChanged {
|
||||
): Boolean {
|
||||
if (oldFilters.size != newFilters.size) return true
|
||||
|
||||
oldFilters.forEachIndexed { index, oldFilter ->
|
||||
val newFilter = newFilters.getOrNull(index) ?: return true
|
||||
|
||||
return needsToResendRequest(oldFilter, newFilter)
|
||||
// Every filter must be compared. This used to be a forEachIndexed whose body
|
||||
// `return`ed on the first iteration — a non-local return from this function — so
|
||||
// only filters[0] was ever checked and a subscription whose first filter happened
|
||||
// to be unchanged reported "no resend needed" however much the rest had changed,
|
||||
// leaving the relay serving a stale filter set.
|
||||
// Indexing is safe: the sizes were just proven equal.
|
||||
for (i in oldFilters.indices) {
|
||||
if (needsToResendRequest(oldFilters[i], newFilters[i])) return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
+28
-27
@@ -84,25 +84,26 @@ class PoolRequests(
|
||||
/*
|
||||
* Locking model: every compound access to a subscription's state machine
|
||||
* ([RequestSubscriptionState]) — including the check-then-send decision in
|
||||
* [decideCommandLocked] — runs inside THAT subscription's own lock
|
||||
* ([RequestSubscriptionState.withLock]).
|
||||
* [decideCommandLocked] — runs inside the stripe for THAT (subscription, relay)
|
||||
* pair ([RequestSubscriptionState.withLock], which takes the relay).
|
||||
*
|
||||
* A single subscription can span many relays, and each relay's
|
||||
* socket-reader thread delivers messages into this class concurrently
|
||||
* while the app thread adds/removes subscriptions — so the plain maps
|
||||
* inside [RequestSubscriptionState] are written from several threads at
|
||||
* once. That is both a memory hazard (concurrent map mutation) and a
|
||||
* logic hazard: two threads must never both observe "no REQ in flight"
|
||||
* and both send a REQ for the same sub id.
|
||||
* A single subscription can span many relays, and each relay's socket-reader
|
||||
* thread delivers messages into this class concurrently while the app thread
|
||||
* adds/removes subscriptions. Two threads must never both observe "no REQ in
|
||||
* flight" and both send a REQ for the same (sub, relay).
|
||||
*
|
||||
* The lock is per subscription, not global, because different subIds
|
||||
* share no wire state: EVENT frames for different subs coming from
|
||||
* different relay consumer threads must not serialize on each other (a
|
||||
* global lock here measured negative scaling under 4 concurrent relay
|
||||
* feeders). Listener callbacks and the actual socket sends are ALWAYS
|
||||
* performed outside the lock — they re-enter this class through
|
||||
* [onSent], so holding the lock across them would self-deadlock, and the
|
||||
* lock is non-reentrant. Never hold two subscriptions' locks at once.
|
||||
* The lock is striped PER RELAY rather than per subscription: every critical
|
||||
* section below touches only one relay's slice of the state, so EVENT frames
|
||||
* arriving for the same subId from different relays no longer serialize on each
|
||||
* other. With ~191 relays that previously made a single per-sub lock contended
|
||||
* ~191 threads deep — the production ANR in
|
||||
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`.
|
||||
*
|
||||
* Two rules this file must keep:
|
||||
* - Never hold two relays' stripes (or two subscriptions') at once. Every loop
|
||||
* below locks exactly one relay at a time.
|
||||
* - Listener callbacks and socket sends are ALWAYS performed outside the lock —
|
||||
* they re-enter this class through [onSent].
|
||||
*/
|
||||
|
||||
/**
|
||||
@@ -201,7 +202,7 @@ class PoolRequests(
|
||||
fun onConnecting(url: NormalizedRelayUrl) {
|
||||
// Change states to connecting. One sub's lock at a time.
|
||||
relayState.forEach { subId, state ->
|
||||
state.withLock { state.connecting(url) }
|
||||
state.withLock(url) { state.connecting(url) }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -215,7 +216,7 @@ class PoolRequests(
|
||||
when (cmd) {
|
||||
is ReqCmd -> {
|
||||
subState(cmd.subId).let { state ->
|
||||
state.withLock { state.onOpenReq(relay, cmd.filters) }
|
||||
state.withLock(relay) { state.onOpenReq(relay, cmd.filters) }
|
||||
}
|
||||
desiredSubListeners.get(cmd.subId)?.onSubscriptionStarted(
|
||||
relay = relay.url,
|
||||
@@ -225,7 +226,7 @@ class PoolRequests(
|
||||
|
||||
is CloseCmd -> {
|
||||
subState(cmd.subId).let { state ->
|
||||
state.withLock { state.onSubscriptionClosed(relay) }
|
||||
state.withLock(relay) { state.onSubscriptionClosed(relay) }
|
||||
}
|
||||
desiredSubListeners.get(cmd.subId)?.onSubscriptionClosed(
|
||||
relay = relay.url,
|
||||
@@ -246,7 +247,7 @@ class PoolRequests(
|
||||
var isLive = false
|
||||
var forFilters: List<Filter>? = null
|
||||
relayState.get(msg.subId)?.let { state ->
|
||||
state.withLock {
|
||||
state.withLock(relay.url) {
|
||||
state.onNewEvent(relay.url)
|
||||
isLive = state.currentState(relay.url) == ReqSubStatus.LIVE
|
||||
forFilters = state.lastKnownFilterStates(relay.url)
|
||||
@@ -264,7 +265,7 @@ class PoolRequests(
|
||||
var forFilters: List<Filter>? = null
|
||||
val cmd =
|
||||
relayState.get(msg.subId)?.let { state ->
|
||||
state.withLock {
|
||||
state.withLock(relay.url) {
|
||||
state.onEose(relay.url)
|
||||
forFilters = state.lastKnownFilterStates(relay.url)
|
||||
// Decide (and pre-mark) the resend while still holding the
|
||||
@@ -293,7 +294,7 @@ class PoolRequests(
|
||||
var forFilters: List<Filter>? = null
|
||||
val cmd =
|
||||
relayState.get(msg.subId)?.let { state ->
|
||||
state.withLock {
|
||||
state.withLock(relay.url) {
|
||||
state.onClosed(relay.url)
|
||||
forFilters = state.lastKnownFilterStates(relay.url)
|
||||
recordRefusalIfStructural(state, relay.url, msg.message, forFilters)
|
||||
@@ -322,7 +323,7 @@ class PoolRequests(
|
||||
*/
|
||||
fun onDisconnected(url: NormalizedRelayUrl) {
|
||||
relayState.forEach { subId, state ->
|
||||
state.withLock { state.disconnected(url) }
|
||||
state.withLock(url) { state.disconnected(url) }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -351,7 +352,7 @@ class PoolRequests(
|
||||
if (!filters.isNullOrEmpty()) {
|
||||
val send =
|
||||
subState(subId).let { state ->
|
||||
state.withLock {
|
||||
state.withLock(relay) {
|
||||
if (isStructurallyRefused(state, relay, filters)) {
|
||||
false
|
||||
} else {
|
||||
@@ -379,7 +380,7 @@ class PoolRequests(
|
||||
// These are all my subs.. need to figure out which relays have them
|
||||
val subs = desiredSubs.get(subId)
|
||||
if (subs != null && url in subs.keys) {
|
||||
toNotify.add(subId to state.withLock { state.lastKnownFilterStates(url) })
|
||||
toNotify.add(subId to state.withLock(url) { state.lastKnownFilterStates(url) })
|
||||
}
|
||||
}
|
||||
|
||||
@@ -400,7 +401,7 @@ class PoolRequests(
|
||||
val state = subState(subId)
|
||||
relaysToUpdate.forEach { relay ->
|
||||
// Decide + pre-mark atomically under the sub's lock, then send outside it.
|
||||
val cmd = state.withLock { decideCommandLocked(state, subId, relay) }
|
||||
val cmd = state.withLock(relay) { decideCommandLocked(state, subId, relay) }
|
||||
if (cmd != null) {
|
||||
sync(relay, cmd)
|
||||
}
|
||||
|
||||
+139
-89
@@ -21,84 +21,116 @@
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.reqs
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import kotlin.concurrent.atomics.AtomicBoolean
|
||||
import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
import com.vitorpamplona.quartz.utils.concurrent.PlatformLock
|
||||
|
||||
/**
|
||||
* Manages the State of Subscriptions by logging states as the
|
||||
* subscription progresses.
|
||||
*
|
||||
* Thread-safety: the plain maps below are only touched inside [withLock].
|
||||
* The lock lives HERE — one per subscription — instead of a single global
|
||||
* lock in PoolRequests, because every mutation is scoped to one subId:
|
||||
* relays delivering EVENTs for *different* subscriptions have no shared
|
||||
* state and must not serialize on each other. (A single global spin lock
|
||||
* measured *negative* scaling: 4 relay consumer threads pushed less
|
||||
* aggregate throughput through it than 1 — see
|
||||
* quartz/plans/2026-07-02-nostrclient-receiver-perf.md.)
|
||||
* **Thread-safety: the lock is striped per reference (per relay), not per
|
||||
* subscription.** Every operation below is scoped to a single [reference] — all state
|
||||
* lives in [RelayState] objects held in a [ConcurrentMap] keyed by it — so two relays
|
||||
* delivering EVENTs for the SAME subscription touch disjoint state and no longer
|
||||
* serialize on each other.
|
||||
*
|
||||
* That mattered in production: one subId spans every relay it is subscribed on, so a
|
||||
* single per-subscription lock was contended ~191 threads deep on the Pixel 8 that
|
||||
* produced `anr_2026-08-03-12-55-26-256` (it was a *spin* lock then, which burned 6 of
|
||||
* 9 cores — see [PlatformLock]). Striping removes the contention instead of making
|
||||
* waiting cheaper: measured 1.5-2.8x the throughput of one lock per sub in
|
||||
* `quartz/src/jvmTest/.../prodbench/LockDesignComparisonBenchmark.kt`. Full analysis,
|
||||
* including why a suspending `Mutex` was rejected, is in
|
||||
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`.
|
||||
*
|
||||
* The stripe array is allocated once and NEVER mutated, so a stripe's identity is
|
||||
* stable for this object's whole life. That is load-bearing: if locks lived inside the
|
||||
* per-relay values, a thread holding one while another thread dropped and re-created
|
||||
* that entry would leave both "inside" the critical section excluding nothing.
|
||||
*
|
||||
* Callers MUST NOT hold two references' stripes at once ([PoolRequests] locks one relay
|
||||
* at a time, including inside its all-subs iterations), and MUST keep socket sends and
|
||||
* listener callbacks outside the critical section — they re-enter this class through
|
||||
* `onSent`, and the point of a lock is to be held briefly.
|
||||
*/
|
||||
@OptIn(ExperimentalAtomicApi::class)
|
||||
class RequestSubscriptionState<T> {
|
||||
class RequestSubscriptionState<T : Any> {
|
||||
/**
|
||||
* Tiny non-reentrant spin lock (same primitive as BasicRelayClient's
|
||||
* connecting mutex). Critical sections are a handful of map operations,
|
||||
* never I/O — callers MUST NOT re-enter and MUST NOT hold two
|
||||
* subscriptions' locks at once (PoolRequests locks one sub at a time,
|
||||
* including inside its all-subs iterations).
|
||||
* One reference's (relay's) slice of this subscription's state. Plain `var`s: every
|
||||
* field is written and read under that reference's stripe by [PoolRequests].
|
||||
*
|
||||
* `@PublishedApi internal` only because [withLock] is inline (this sits
|
||||
* on the per-EVENT hot path; inlining avoids a closure allocation per
|
||||
* message) — treat it as private.
|
||||
* Note `RelayActiveRequestStates` uses this class WITHOUT locking. There the fields
|
||||
* may be read stale — but the backing [ConcurrentMap] can no longer be structurally
|
||||
* corrupted the way the plain `HashMap`s this replaced could.
|
||||
*/
|
||||
private class RelayState {
|
||||
/** Null == no REQ state on this relay (fresh, or wiped by connecting/disconnected). */
|
||||
var status: ReqSubStatus? = null
|
||||
|
||||
/** Filters of the REQ currently believed to be in flight. */
|
||||
var filters: List<Filter>? = null
|
||||
|
||||
/**
|
||||
* Survives connect/disconnect so that if new events still arrive we can link
|
||||
* them with the filters the relay was processing.
|
||||
*/
|
||||
var lastKnownFilters: List<Filter>? = null
|
||||
|
||||
/**
|
||||
* Refused-filter memory. Unlike [status]/[filters] — per-connection wire state
|
||||
* wiped by [connecting]/[disconnected] — this SURVIVES reconnects on purpose: a
|
||||
* relay that structurally refuses a filter (a search-only relay CLOSING a plain
|
||||
* kinds REQ, a relay that "does not accept REQs", "too many filters", …) refuses
|
||||
* it again on every new socket, so [PoolRequests.syncState] replaying it each
|
||||
* reconnect is pure waste. [refusalCount] accumulates repeated refusals of the
|
||||
* same shape so a one-off (transient) close isn't mistaken for a structural one.
|
||||
* Cleared on a successful REQ ([onEose]/[onNewEvent]) or when the caller observes
|
||||
* the desired filter meaningfully changed.
|
||||
*/
|
||||
var refusedFilters: List<Filter>? = null
|
||||
|
||||
var refusalCount: Int = 0
|
||||
}
|
||||
|
||||
private val states = ConcurrentMap<T, RelayState>()
|
||||
|
||||
/**
|
||||
* Fixed stripe array — allocated once, never mutated, so lock identity is stable.
|
||||
* [STRIPE_COUNT] stripes over ~191 relays is roughly 6-way sharing: a ~32x
|
||||
* contention reduction versus one lock per subscription. References colliding on a
|
||||
* stripe merely serialize; correctness never depends on N.
|
||||
*/
|
||||
@PublishedApi
|
||||
internal val lock = AtomicBoolean(false)
|
||||
internal val stripes = Array(STRIPE_COUNT) { PlatformLock() }
|
||||
|
||||
inline fun <R> withLock(block: () -> R): R {
|
||||
while (lock.exchange(true)) {
|
||||
// Test-and-test-and-set: spin-read until it looks free (cheaper
|
||||
// on the cache line than hammering exchange), then retry above.
|
||||
while (lock.load()) { }
|
||||
}
|
||||
@PublishedApi
|
||||
internal fun stripeFor(reference: T): PlatformLock = stripes[(reference.hashCode() and 0x7FFFFFFF) % STRIPE_COUNT]
|
||||
|
||||
/**
|
||||
* Runs [block] holding [reference]'s stripe. Inline so the per-EVENT hot path
|
||||
* allocates no closure.
|
||||
*/
|
||||
inline fun <R> withLock(
|
||||
reference: T,
|
||||
block: () -> R,
|
||||
): R {
|
||||
val lock = stripeFor(reference)
|
||||
lock.lock()
|
||||
try {
|
||||
return block()
|
||||
} finally {
|
||||
lock.store(false)
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// Logs the state of each channel to:
|
||||
// 1. inform when an event is received as live
|
||||
// 2. to block REQs being sent before finished (receiving an EOSE or Closed)
|
||||
//
|
||||
// If 2 happens, the relay might send multiple EOSEs in sequence
|
||||
// for the same sub and we won't know which REQ was it for.
|
||||
private val subStates = mutableMapOf<T, ReqSubStatus>()
|
||||
private val filterStates = mutableMapOf<T, List<Filter>>()
|
||||
/** Read-only lookup — never creates an entry. */
|
||||
private fun peek(reference: T): RelayState? = states[reference]
|
||||
|
||||
/**
|
||||
* This cache is used to make sure we know what the relay was processing
|
||||
* before a close or disconnect so that if new events still arrive
|
||||
* we can link them with the appropriate filters.
|
||||
*/
|
||||
private val lastKnownFilterStates = mutableMapOf<T, List<Filter>>()
|
||||
/** Write lookup — creates the entry on first use. */
|
||||
private fun mutable(reference: T): RelayState = states.getOrPut(reference) { RelayState() }
|
||||
|
||||
/**
|
||||
* Refused-filter memory. Unlike [subStates]/[filterStates] above — per-connection
|
||||
* wire state wiped by [connecting]/[disconnected] — this SURVIVES reconnects on
|
||||
* purpose: a relay that structurally refuses a filter (a search-only relay CLOSING
|
||||
* a plain kinds REQ, a relay that "does not accept REQs", "too many filters", …)
|
||||
* refuses it again on every new socket, so [PoolRequests.syncState] replaying it
|
||||
* each reconnect is pure waste. [refusalCounts] accumulates repeated refusals of
|
||||
* the same shape so a one-off (transient) close isn't mistaken for a structural
|
||||
* one. Cleared on a successful REQ ([onEose]/[onNewEvent]) or when the caller
|
||||
* observes the desired filter meaningfully changed.
|
||||
*/
|
||||
private val refusedFilters = mutableMapOf<T, List<Filter>>()
|
||||
private val refusalCounts = mutableMapOf<T, Int>()
|
||||
fun refusedFilters(reference: T) = peek(reference)?.refusedFilters
|
||||
|
||||
fun refusedFilters(reference: T) = refusedFilters[reference]
|
||||
|
||||
fun refusalCount(reference: T) = refusalCounts[reference] ?: 0
|
||||
fun refusalCount(reference: T) = peek(reference)?.refusalCount ?: 0
|
||||
|
||||
/**
|
||||
* Records that [reference] refused [filters]. [sameAsLastRefusal] must be true when
|
||||
@@ -111,73 +143,91 @@ class RequestSubscriptionState<T> {
|
||||
filters: List<Filter>,
|
||||
sameAsLastRefusal: Boolean,
|
||||
) {
|
||||
val state = mutable(reference)
|
||||
if (sameAsLastRefusal) {
|
||||
refusalCounts[reference] = refusalCount(reference) + 1
|
||||
state.refusalCount += 1
|
||||
} else {
|
||||
refusedFilters[reference] = filters
|
||||
refusalCounts[reference] = 1
|
||||
state.refusedFilters = filters
|
||||
state.refusalCount = 1
|
||||
}
|
||||
}
|
||||
|
||||
fun clearRefusal(reference: T) {
|
||||
refusedFilters.remove(reference)
|
||||
refusalCounts.remove(reference)
|
||||
peek(reference)?.let {
|
||||
it.refusedFilters = null
|
||||
it.refusalCount = 0
|
||||
}
|
||||
}
|
||||
|
||||
fun currentFilters() = filterStates
|
||||
fun currentFilters(reference: T) = peek(reference)?.filters
|
||||
|
||||
fun currentFilters(reference: T) = filterStates[reference]
|
||||
fun lastKnownFilterStates(reference: T) = peek(reference)?.lastKnownFilters
|
||||
|
||||
fun lastKnownFilterStates(reference: T) = lastKnownFilterStates[reference]
|
||||
|
||||
fun currentState(reference: T) = subStates[reference]
|
||||
fun currentState(reference: T) = peek(reference)?.status
|
||||
|
||||
fun onNewEvent(reference: T) {
|
||||
val state = mutable(reference)
|
||||
// The relay is serving this REQ (it matched an event), so any past refusal
|
||||
// no longer applies — let it be tried freely again.
|
||||
clearRefusal(reference)
|
||||
if (subStates[reference] == ReqSubStatus.SENT) {
|
||||
subStates[reference] = ReqSubStatus.QUERYING_PAST
|
||||
state.refusedFilters = null
|
||||
state.refusalCount = 0
|
||||
if (state.status == ReqSubStatus.SENT) {
|
||||
state.status = ReqSubStatus.QUERYING_PAST
|
||||
}
|
||||
}
|
||||
|
||||
fun onEose(reference: T) {
|
||||
val state = mutable(reference)
|
||||
// Reaching EOSE means the relay accepted and finished the REQ; clear any refusal.
|
||||
clearRefusal(reference)
|
||||
subStates[reference] = ReqSubStatus.LIVE
|
||||
state.refusedFilters = null
|
||||
state.refusalCount = 0
|
||||
state.status = ReqSubStatus.LIVE
|
||||
}
|
||||
|
||||
fun onClosed(reference: T) {
|
||||
subStates[reference] = ReqSubStatus.CLOSED
|
||||
// Closed messages are usually relays refusing to process a REQ
|
||||
// This message keeps the state of filterStates intact to
|
||||
// avoid sending the same filter, and getting immediately closed,
|
||||
// over and over again.
|
||||
|
||||
// filterStates.remove(reference)
|
||||
// Closed messages are usually relays refusing to process a REQ. This keeps
|
||||
// [RelayState.filters] intact to avoid sending the same filter, and getting
|
||||
// immediately closed, over and over again.
|
||||
mutable(reference).status = ReqSubStatus.CLOSED
|
||||
}
|
||||
|
||||
fun onOpenReq(
|
||||
reference: T,
|
||||
filters: List<Filter>,
|
||||
) {
|
||||
subStates[reference] = ReqSubStatus.SENT
|
||||
filterStates[reference] = filters
|
||||
lastKnownFilterStates[reference] = filters
|
||||
val state = mutable(reference)
|
||||
state.status = ReqSubStatus.SENT
|
||||
state.filters = filters
|
||||
state.lastKnownFilters = filters
|
||||
}
|
||||
|
||||
fun onSubscriptionClosed(reference: T) {
|
||||
subStates[reference] = ReqSubStatus.CLOSED
|
||||
filterStates.remove(reference)
|
||||
val state = mutable(reference)
|
||||
state.status = ReqSubStatus.CLOSED
|
||||
state.filters = null
|
||||
}
|
||||
|
||||
fun connecting(reference: T) {
|
||||
subStates.remove(reference)
|
||||
filterStates.remove(reference)
|
||||
// Wipes per-connection wire state only; lastKnownFilters and the refusal memory
|
||||
// deliberately survive (see [RelayState]).
|
||||
peek(reference)?.let {
|
||||
it.status = null
|
||||
it.filters = null
|
||||
}
|
||||
}
|
||||
|
||||
fun disconnected(reference: T) {
|
||||
subStates.remove(reference)
|
||||
filterStates.remove(reference)
|
||||
peek(reference)?.let {
|
||||
it.status = null
|
||||
it.filters = null
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* Comfortably above the IO dispatcher's thread count (64 / 2) so collisions stay
|
||||
* rare even when many workers are inside this class at once.
|
||||
*/
|
||||
const val STRIPE_COUNT = 32
|
||||
}
|
||||
}
|
||||
|
||||
+4
-3
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.relay.filters
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip01Core.tags.isIndexableTagName
|
||||
import kotlinx.collections.immutable.PersistentMap
|
||||
import kotlinx.collections.immutable.PersistentSet
|
||||
import kotlinx.collections.immutable.persistentHashMapOf
|
||||
@@ -238,7 +239,7 @@ class FilterIndex<S : Any> {
|
||||
s.kinds[event.kind]?.let { result.addAll(it) }
|
||||
if (s.tags.isNotEmpty()) {
|
||||
for (tag in event.tags) {
|
||||
if (tag.size >= 2 && tag[0].length == 1) {
|
||||
if (tag.size >= 2 && isIndexableTagName(tag[0])) {
|
||||
s.tags[tag[0]]?.get(tag[1])?.let { result.addAll(it) }
|
||||
}
|
||||
}
|
||||
@@ -348,14 +349,14 @@ class FilterIndex<S : Any> {
|
||||
if (!filter.tags.isNullOrEmpty()) {
|
||||
val first =
|
||||
filter.tags.entries.firstOrNull {
|
||||
it.key.length == 1 && it.value.isNotEmpty()
|
||||
isIndexableTagName(it.key) && it.value.isNotEmpty()
|
||||
}
|
||||
if (first != null) return first.value.map { TagKey(first.key, it) }
|
||||
}
|
||||
if (!filter.tagsAll.isNullOrEmpty()) {
|
||||
val first =
|
||||
filter.tagsAll.entries.firstOrNull {
|
||||
it.key.length == 1 && it.value.isNotEmpty()
|
||||
isIndexableTagName(it.key) && it.value.isNotEmpty()
|
||||
}
|
||||
if (first != null) return first.value.map { TagKey(first.key, it) }
|
||||
}
|
||||
|
||||
+6
-1
@@ -44,6 +44,11 @@ fun NormalizedRelayUrl.toHttp() =
|
||||
"https://$url"
|
||||
}
|
||||
|
||||
fun NormalizedRelayUrl.isOnion() = url.contains(".onion/")
|
||||
// Delegates rather than re-implementing `contains(".onion/")`: that copy missed
|
||||
// `wss://host.onion:8080/`, so an onion relay on an explicit port was never forced onto Tor.
|
||||
fun NormalizedRelayUrl.isOnion() = RelayUrlNormalizer.isOnion(this.url)
|
||||
|
||||
fun NormalizedRelayUrl.isLocalHost() = RelayUrlNormalizer.isLocalHost(this.url)
|
||||
|
||||
/** True for a relay inside an encrypted IPv6 overlay mesh. See [RelayUrlNormalizer.isOverlayNetwork]. */
|
||||
fun NormalizedRelayUrl.isOverlayNetwork() = RelayUrlNormalizer.isOverlayNetwork(this.url)
|
||||
|
||||
+359
-27
@@ -21,6 +21,8 @@
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.normalizer
|
||||
|
||||
import androidx.collection.LruCache
|
||||
import com.vitorpamplona.quartz.utils.Ipv4
|
||||
import com.vitorpamplona.quartz.utils.Ipv6
|
||||
import com.vitorpamplona.quartz.utils.Log
|
||||
import com.vitorpamplona.quartz.utils.Rfc3986
|
||||
import kotlinx.coroutines.CancellationException
|
||||
@@ -38,15 +40,183 @@ val normalizedUrls = LruCache<String, NormalizationResult>(5000)
|
||||
|
||||
class RelayUrlNormalizer {
|
||||
companion object {
|
||||
fun isLocalHost(url: String) =
|
||||
url.contains("127.0.0.1") ||
|
||||
url.contains("localhost") ||
|
||||
url.contains("//umbrel:") ||
|
||||
url.contains("192.168.") ||
|
||||
url.contains(".local:") ||
|
||||
url.contains(".local/")
|
||||
/**
|
||||
* Every host test below is anchored to the **authority** (`host[:port]`), never to the
|
||||
* whole url. A plain `contains` reads the path and query too, so
|
||||
* `wss://evil.example.com/127.0.0.1` used to answer true here — and since [isLocalHost]
|
||||
* is what exempts a relay from Tor, any relay list could hand the app a url that quietly
|
||||
* dropped its own Tor routing. Relay urls arrive from other people (NIP-65 lists, relay
|
||||
* hints, `r` tags), so they are attacker-controlled input and have to be parsed as such.
|
||||
*
|
||||
* Anchoring is also what makes `host:port` work: `.onion:8080` never matched the old
|
||||
* `.onion/` test, so an onion relay on an explicit port was not recognized as onion at
|
||||
* all and its hostname went to the clearnet DNS resolver.
|
||||
*/
|
||||
fun isLocalHost(url: String): Boolean {
|
||||
val start = hostStart(url)
|
||||
val end = hostEnd(url, start)
|
||||
if (end <= start) return false
|
||||
val hostEnd = hostEndWithoutPort(url, start, end)
|
||||
return isPrivateIpv4(url, start, hostEnd) ||
|
||||
// RFC 6761: `localhost` and anything under it — the same rule SurgeDns applies
|
||||
// when deciding whether a loopback answer is legitimate. A substring test would
|
||||
// also match `notlocalhost.example.com`, which is registrable.
|
||||
regionEquals(url, "localhost", start, hostEnd) ||
|
||||
regionEndsWith(url, ".localhost", start, hostEnd) ||
|
||||
regionEquals(url, "umbrel", start, hostEnd) ||
|
||||
regionEndsWith(url, ".local", start, hostEnd) ||
|
||||
isPrivateIpv6(url, start, end)
|
||||
}
|
||||
|
||||
fun isOnion(url: String) = url.endsWith(".onion") || url.contains(".onion/")
|
||||
/**
|
||||
* The IPv4 ranges that are never a public relay: `127.0.0.0/8` loopback, the RFC 1918
|
||||
* private blocks, `169.254.0.0/16` link-local and `0.0.0.0/8`.
|
||||
*
|
||||
* Parsed rather than substring-matched, which was wrong both ways: `contains("192.168.")`
|
||||
* missed `10.0.0.5` and `172.16.3.4` — so a LAN relay was given `wss://` and dialed
|
||||
* through Tor — while matching `192.168.evil.com`, a registrable domain that could
|
||||
* therefore exempt itself from Tor.
|
||||
*/
|
||||
private fun isPrivateIpv4(
|
||||
url: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): Boolean {
|
||||
val bytes = Ipv4.parse(url, start, end) ?: return false
|
||||
return Ipv4.isLoopback(bytes) ||
|
||||
Ipv4.isPrivate(bytes) ||
|
||||
Ipv4.isLinkLocal(bytes) ||
|
||||
Ipv4.isUnspecified(bytes)
|
||||
}
|
||||
|
||||
fun isOnion(url: String): Boolean {
|
||||
val start = hostStart(url)
|
||||
val end = hostEnd(url, start)
|
||||
if (end <= start) return false
|
||||
return regionEndsWith(url, ".onion", start, hostEndWithoutPort(url, start, end))
|
||||
}
|
||||
|
||||
/**
|
||||
* True for a relay inside an encrypted IPv6 overlay mesh — today `0200::/7`, the range
|
||||
* Yggdrasil derives node addresses and subnets from.
|
||||
*
|
||||
* Unlike [isLocalHost] this is not a private address: it is reachable from anywhere on
|
||||
* the mesh. But it is unreachable *off* the mesh, which has two consequences the relay
|
||||
* stack has to honour — it can never be dialed through a SOCKS/Tor proxy, and it can
|
||||
* never present a CA-issued certificate, so it speaks plain `ws://`. Both are safe:
|
||||
* the overlay already encrypts end to end and authenticates the peer by its address,
|
||||
* which is derived from the peer's public key.
|
||||
*/
|
||||
fun isOverlayNetwork(url: String): Boolean {
|
||||
val start = hostStart(url)
|
||||
val bytes = ipv6HostOf(url, start, hostEnd(url, start)) ?: return false
|
||||
return Ipv6.isOverlayMesh(bytes)
|
||||
}
|
||||
|
||||
/**
|
||||
* The IPv6 twins of the literals in [isLocalHost]: `::1` (127.0.0.1), `fc00::/7` unique
|
||||
* local addresses (192.168.0.0/16) and `fe80::/10` link-local. All three name a host that
|
||||
* only exists on this machine or this LAN, which is what every caller of [isLocalHost]
|
||||
* means by the question — so a relay on one must not be Torified, must not need TLS,
|
||||
* and must not be advertised to the network.
|
||||
*/
|
||||
private fun isPrivateIpv6(
|
||||
url: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): Boolean {
|
||||
val bytes = ipv6HostOf(url, start, end) ?: return false
|
||||
return Ipv6.isLoopback(bytes) || Ipv6.isUniqueLocal(bytes) || Ipv6.isLinkLocal(bytes)
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses the authority of [url] as a bracketed IPv6 literal, dropping any `%zone` suffix.
|
||||
* Returns null — on a single char comparison — for the overwhelmingly common case of a
|
||||
* url with a DNS host.
|
||||
*/
|
||||
private fun ipv6HostOf(
|
||||
url: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): ByteArray? {
|
||||
if (start >= end || url[start] != '[') return null
|
||||
val close = url.indexOf(']', start + 1)
|
||||
if (close < 0 || close >= end || close <= start + 1) return null
|
||||
val zone = url.indexOf('%', start + 1)
|
||||
val addressEnd = if (zone in (start + 1) until close) zone else close
|
||||
return Ipv6.parse(url.substring(start + 1, addressEnd))
|
||||
}
|
||||
|
||||
/**
|
||||
* Index of the first char of the authority: past `://`, or 0 for a schemeless host.
|
||||
*
|
||||
* The `://` only counts when what precedes it is a real RFC 3986 scheme
|
||||
* (`ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )`). Otherwise `relay.com/x://127.0.0.1`
|
||||
* would have its *path* read as the authority and answer true to [isLocalHost].
|
||||
*/
|
||||
private fun hostStart(url: String): Int {
|
||||
val scheme = url.indexOf("://")
|
||||
if (scheme <= 0 || !url[0].isLetter()) return 0
|
||||
for (i in 1 until scheme) {
|
||||
val c = url[i]
|
||||
if (!c.isLetterOrDigit() && c != '+' && c != '-' && c != '.') return 0
|
||||
}
|
||||
return scheme + 3
|
||||
}
|
||||
|
||||
/** Index just past the authority — the first `/`, `?` or `#`, or the end of [url]. */
|
||||
private fun hostEnd(
|
||||
url: String,
|
||||
start: Int,
|
||||
): Int {
|
||||
var i = start
|
||||
while (i < url.length) {
|
||||
val c = url[i]
|
||||
if (c == '/' || c == '?' || c == '#') return i
|
||||
i++
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
/**
|
||||
* [end] trimmed back past a `:port` and any trailing dots, so the host tests see the
|
||||
* name alone. RFC 1034's fully-qualified form ends in a dot (`abc.onion.`), and missing
|
||||
* that spelling on [isOnion] would send a `.onion` name to the clearnet DNS resolver.
|
||||
*/
|
||||
private fun hostEndWithoutPort(
|
||||
url: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): Int {
|
||||
var stop =
|
||||
if (url[start] == '[') {
|
||||
// In an IPv6 authority only the `:` after the `]` can start a port.
|
||||
val close = url.indexOf(']', start + 1)
|
||||
if (close in start until end) close + 1 else end
|
||||
} else {
|
||||
val colon = url.lastIndexOf(':', end - 1)
|
||||
if (colon >= start) colon else end
|
||||
}
|
||||
while (stop > start && url[stop - 1] == '.') stop--
|
||||
return stop
|
||||
}
|
||||
|
||||
// Host names are case-insensitive (RFC 4343), and `fix()` asks these questions *before*
|
||||
// the RFC 3986 pass folds the case — so a case-sensitive test gave `LOCALHOST:8080` and
|
||||
// `ABC.ONION:8080` a `wss://` scheme neither host can ever serve.
|
||||
private fun regionEndsWith(
|
||||
url: String,
|
||||
suffix: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): Boolean = end - start >= suffix.length && url.regionMatches(end - suffix.length, suffix, 0, suffix.length, ignoreCase = true)
|
||||
|
||||
private fun regionEquals(
|
||||
url: String,
|
||||
name: String,
|
||||
start: Int,
|
||||
end: Int,
|
||||
): Boolean = end - start == name.length && url.regionMatches(start, name, 0, name.length, ignoreCase = true)
|
||||
|
||||
fun isRelaySchemePrefix(url: String) = url.length > 6 && url[0] == 'w' && url[1] == 's'
|
||||
|
||||
@@ -82,19 +252,135 @@ class RelayUrlNormalizer {
|
||||
return false
|
||||
}
|
||||
|
||||
private fun norm(url: String) = NormalizedRelayUrl(Rfc3986.normalize(url))
|
||||
private fun norm(url: String) = NormalizedRelayUrl(canonicalizeIpv6Host(Rfc3986.normalize(url)))
|
||||
|
||||
/**
|
||||
* Rewrites a bracketed IPv6 host into its RFC 5952 canonical form.
|
||||
*
|
||||
* RFC 4291 lets one address be spelled many ways, and the RFC 3986 pass only folds hex
|
||||
* case — so `[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]`
|
||||
* survive as two different [NormalizedRelayUrl]s for one host. That value keys the
|
||||
* connection pool, the relay-list sets, the NIP-11 cache and the per-relay stats, so the
|
||||
* app would dial the same relay twice and count it twice. OkHttp canonicalizes to this
|
||||
* exact form when it dials, so folding here makes the stored key the host on the wire.
|
||||
*
|
||||
* Returns [url] itself — no allocation — when there is no literal or it is already
|
||||
* canonical, which is every url with a DNS host.
|
||||
*/
|
||||
private fun canonicalizeIpv6Host(url: String): String {
|
||||
// Anchored to the authority: a `[...]` in a path or query is data, and rewriting it
|
||||
// would silently corrupt the url.
|
||||
val open = hostStart(url)
|
||||
if (open >= url.length || url[open] != '[') return url
|
||||
val close = url.indexOf(']', open + 1)
|
||||
if (close <= open + 1 || close >= hostEnd(url, open)) return url
|
||||
val inner = url.substring(open + 1, close)
|
||||
val canonical = Ipv6.canonicalizeOrNull(inner) ?: return url
|
||||
if (canonical == inner) return url
|
||||
return url.substring(0, open + 1) + canonical + url.substring(close)
|
||||
}
|
||||
|
||||
private fun isInvisible(c: Char) = c == '\u200B' || c == '\u200C' || c == '\u200D' || c == '\u2060' || c == '\uFEFF'
|
||||
|
||||
/**
|
||||
* Scans the authority (host[:port]) that starts at [start] and ends at the first
|
||||
* `/`, `?` or `#`. Returns the end index, or -1 when the authority is empty or
|
||||
* contains characters that never appear in a real relay host (`@` userinfo,
|
||||
* percent-encoding, commas).
|
||||
*/
|
||||
private fun authorityEnd(
|
||||
url: String,
|
||||
start: Int,
|
||||
): Int {
|
||||
if (start >= url.length) return -1
|
||||
var i = start
|
||||
if (url[i] == '[') {
|
||||
// IPv6 literal: defer validation to the RFC 3986 parser
|
||||
while (i < url.length && url[i] != '/' && url[i] != '?' && url[i] != '#') i++
|
||||
return i
|
||||
}
|
||||
while (i < url.length) {
|
||||
val c = url[i]
|
||||
if (c == '/' || c == '?' || c == '#') break
|
||||
if (c == '@' || c == '%' || c == ',') return -1
|
||||
i++
|
||||
}
|
||||
return if (i == start) -1 else i
|
||||
}
|
||||
|
||||
/**
|
||||
* Accepts a ws/wss url whose host starts at [hostStart] if the authority is sane
|
||||
* and the path does not start with `//` (the signature of a second URL or a broken
|
||||
* `https//` pasted after the scheme, e.g. `wss://https//nostr.watch/relay/x`).
|
||||
*/
|
||||
private fun fixWs(
|
||||
url: String,
|
||||
hostStart: Int,
|
||||
): String? {
|
||||
val end = authorityEnd(url, hostStart)
|
||||
if (end < 0) return null
|
||||
if (end + 1 < url.length && url[end] == '/' && url[end + 1] == '/') return null
|
||||
return url
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts an http(s) url to ws(s) only when it is a bare host — nothing after
|
||||
* `host[:port]` but an optional trailing `/`. An http url with a path, query or
|
||||
* fragment (Mastodon actor urls from bridge `proxy` tags, web pages, images) is
|
||||
* a web resource, not a relay: converting it creates a wss:// url that can never
|
||||
* answer and only wastes connection attempts.
|
||||
*/
|
||||
private fun fixHttp(
|
||||
url: String,
|
||||
hostStart: Int,
|
||||
newScheme: String,
|
||||
): String? {
|
||||
val end = authorityEnd(url, hostStart)
|
||||
if (end < 0) return null
|
||||
val bareHost = end == url.length || (end == url.length - 1 && url[end] == '/')
|
||||
if (!bareHost) return null
|
||||
return "$newScheme${url.substring(hostStart)}"
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates a schemeless candidate: the part before the first `/` must look like
|
||||
* `host` or `host:port` — letters, digits, `.`, `-`, `_`, plus at most one `:`
|
||||
* followed by digits only. Rejects addressable-event pointers (`31990:hex:dtag`),
|
||||
* bare scheme leftovers (`wss:`) and anything else that would otherwise be blindly
|
||||
* prefixed with `wss://`.
|
||||
*/
|
||||
private fun isBareHostAndPath(url: String): Boolean {
|
||||
if (url[0] == '[') return true // IPv6 literal: defer to the RFC 3986 parser
|
||||
var i = 0
|
||||
var portStart = -1
|
||||
while (i < url.length) {
|
||||
val c = url[i]
|
||||
if (c == '/') break
|
||||
if (c == ':') {
|
||||
if (portStart >= 0) return false
|
||||
portStart = i + 1
|
||||
} else if (portStart >= 0) {
|
||||
if (c < '0' || c > '9') return false
|
||||
} else if (!c.isLetterOrDigit() && c != '.' && c != '-' && c != '_') {
|
||||
return false
|
||||
}
|
||||
i++
|
||||
}
|
||||
if (i == 0) return false
|
||||
if (portStart >= 0 && portStart == i) return false
|
||||
return true
|
||||
}
|
||||
|
||||
@OptIn(ExperimentalContracts::class)
|
||||
fun fix(rawUrl: String): String? {
|
||||
if (rawUrl.length < 4) return null
|
||||
if (rawUrl.contains("%00")) return null
|
||||
|
||||
// Trim trailing %20 (percent-encoded spaces from malformed event data)
|
||||
val url =
|
||||
rawUrl.trimEnd('%', '2', '0').let { trimmed ->
|
||||
// Only accept if we actually removed a trailing %20 pattern
|
||||
if (trimmed.length < rawUrl.length && rawUrl.endsWith("%20")) trimmed else rawUrl
|
||||
}
|
||||
// Trim trailing %20 (percent-encoded spaces from malformed event data).
|
||||
// The endsWith gate keeps the hot path allocation-free: trimEnd would
|
||||
// copy the string for ANY url merely ending in '%', '2' or '0' — which
|
||||
// includes every port ending in zero ("wss://host:3030").
|
||||
val url = if (rawUrl.endsWith("%20")) rawUrl.trimEnd('%', '2', '0') else rawUrl
|
||||
if (url.length < 4) return null
|
||||
|
||||
// Reject URLs with %20 in the middle — these are garbage
|
||||
@@ -109,17 +395,33 @@ class RelayUrlNormalizer {
|
||||
}
|
||||
}
|
||||
|
||||
val trimmed =
|
||||
var trimmed =
|
||||
if (url[0].isWhitespace() || url[url.length - 1].isWhitespace()) {
|
||||
url.trim()
|
||||
} else {
|
||||
url
|
||||
}
|
||||
|
||||
// Single pass: interior whitespace means multiple urls or prose in one field,
|
||||
// backslashes never appear in a real relay url; both are garbage. Invisible
|
||||
// characters (zero-width spaces, BOM) are copy-paste artifacts — strip them.
|
||||
var hasInvisible = false
|
||||
for (c in trimmed) {
|
||||
if (c == '\\') return null
|
||||
if (c.isWhitespace()) return null
|
||||
if (isInvisible(c)) hasInvisible = true
|
||||
}
|
||||
if (hasInvisible) {
|
||||
trimmed = buildString(trimmed.length) { for (c in trimmed) if (!isInvisible(c)) append(c) }
|
||||
if (trimmed.length < 4) return null
|
||||
}
|
||||
|
||||
// fast for good wss:// urls
|
||||
if (isRelaySchemePrefix(trimmed)) {
|
||||
if (isRelaySchemePrefixSecure(trimmed) || isRelaySchemePrefixInsecure(trimmed)) {
|
||||
return trimmed
|
||||
if (isRelaySchemePrefixSecure(trimmed)) {
|
||||
return fixWs(trimmed, 6)
|
||||
} else if (isRelaySchemePrefixInsecure(trimmed)) {
|
||||
return fixWs(trimmed, 5)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,31 +429,31 @@ class RelayUrlNormalizer {
|
||||
if (isHttpPrefix(trimmed)) {
|
||||
if (isHttpSSuffix(trimmed)) {
|
||||
// https://
|
||||
return "wss://${trimmed.drop(8)}"
|
||||
return fixHttp(trimmed, 8, "wss://")
|
||||
} else if (isHttpSuffix(trimmed)) {
|
||||
// http://
|
||||
return "ws://${trimmed.drop(7)}"
|
||||
return fixHttp(trimmed, 7, "ws://")
|
||||
}
|
||||
}
|
||||
|
||||
// fast for good ww:// urls
|
||||
if (trimmed.startsWith("ww://")) {
|
||||
return "wss://${trimmed.drop(5)}"
|
||||
return fixWs("wss://${trimmed.drop(5)}", 6)
|
||||
}
|
||||
|
||||
// fast for good ww:// urls
|
||||
if (trimmed.startsWith("was://")) {
|
||||
return "wss://${trimmed.drop(6)}"
|
||||
return fixWs("wss://${trimmed.drop(6)}", 6)
|
||||
}
|
||||
|
||||
// fast for good ww:// urls
|
||||
if (trimmed.startsWith("Wws://")) {
|
||||
return "wss://${trimmed.drop(6)}"
|
||||
return fixWs("wss://${trimmed.drop(6)}", 6)
|
||||
}
|
||||
|
||||
// fast for good ww:// urls
|
||||
if (trimmed.startsWith("Wss://")) {
|
||||
return "wss://${trimmed.drop(6)}"
|
||||
return fixWs("wss://${trimmed.drop(6)}", 6)
|
||||
}
|
||||
|
||||
if (trimmed.contains("://")) {
|
||||
@@ -160,10 +462,33 @@ class RelayUrlNormalizer {
|
||||
return null
|
||||
}
|
||||
|
||||
return if (isOnion(trimmed) || isLocalHost(trimmed)) {
|
||||
"ws://$trimmed"
|
||||
// protocol-relative urls (`//host/`) are just missing the scheme
|
||||
val protocolRelative = if (trimmed.startsWith("//")) trimmed.drop(2) else trimmed
|
||||
if (protocolRelative.length < 4) return null
|
||||
|
||||
// A bare IPv6 literal is missing its brackets, not malformed. This is the shape a
|
||||
// user actually has in hand — `yggdrasilctl getSelf` prints the address unbracketed
|
||||
// — and without the brackets `isBareHostAndPath` rejects it below as a host with too
|
||||
// many colons. Only a string that parses as a whole address is bracketed, so an
|
||||
// addressable-event pointer (`31990:hex:dtag`) or a `host:port` still falls through.
|
||||
val bare =
|
||||
if (protocolRelative[0] != '[' && Ipv6.isLiteral(protocolRelative)) {
|
||||
"[$protocolRelative]"
|
||||
} else {
|
||||
protocolRelative
|
||||
}
|
||||
|
||||
if (!isBareHostAndPath(bare)) {
|
||||
Log.d("RelayUrlNormalizer") { "Rejected $url" }
|
||||
return null
|
||||
}
|
||||
|
||||
// Overlay and localhost relays cannot hold a certificate, so wss:// could only ever
|
||||
// fail its handshake. Both carry their own encryption, so ws:// is not a downgrade.
|
||||
return if (isOnion(bare) || isLocalHost(bare) || isOverlayNetwork(bare)) {
|
||||
"ws://$bare"
|
||||
} else {
|
||||
"wss://$trimmed"
|
||||
"wss://$bare"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -186,6 +511,13 @@ class RelayUrlNormalizer {
|
||||
val fixed = fix(url)
|
||||
if (fixed != null) {
|
||||
val normalized = norm(fixed)
|
||||
// the RFC 3986 parser can drop or replace the scheme on odd inputs;
|
||||
// anything that is not ws(s):// at this point cannot be connected to.
|
||||
if (!isRelayUrl(normalized.url)) {
|
||||
Log.d("NormalizedRelayUrl") { "Rejected $url" }
|
||||
normalizedUrls.put(url, NormalizationResult.Error)
|
||||
return null
|
||||
}
|
||||
normalizedUrls.put(url, NormalizationResult.Success(normalized))
|
||||
normalized
|
||||
} else {
|
||||
|
||||
+7
@@ -214,6 +214,13 @@ class RelaySession(
|
||||
is IEventStore.InsertOutcome.Rejected -> {
|
||||
send(OkMessage(cmd.event.id, false, outcome.reason))
|
||||
}
|
||||
|
||||
is IEventStore.InsertOutcome.Failed -> {
|
||||
// The store's error, not the event's — NIP-01's
|
||||
// machine-readable prefix for that is "error:".
|
||||
val reason = outcome.reason
|
||||
send(OkMessage(cmd.event.id, false, if (reason.startsWith("error:")) reason else "error: $reason"))
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (_: ClosedSendChannelException) {
|
||||
|
||||
+12
-5
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.relay.server.backend
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
|
||||
import com.vitorpamplona.quartz.utils.Log
|
||||
import kotlinx.coroutines.CoroutineScope
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
@@ -36,6 +37,7 @@ import kotlin.concurrent.atomics.AtomicBoolean
|
||||
import kotlin.concurrent.atomics.AtomicInt
|
||||
import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
import kotlin.coroutines.CoroutineContext
|
||||
import kotlin.coroutines.cancellation.CancellationException
|
||||
|
||||
/**
|
||||
* Group-commit writer for incoming EVENT publishes.
|
||||
@@ -268,8 +270,8 @@ class IngestQueue(
|
||||
* Run the SQLite transaction for the verified subset of [batch]
|
||||
* and stitch outcomes back to a per-batch-index array. Failed
|
||||
* verifies pre-mark `Rejected` and skip the insert. A whole-batch
|
||||
* commit failure converts every persisted entry to `Rejected`
|
||||
* with the throw message.
|
||||
* commit failure converts every persisted entry to `Failed` with
|
||||
* the throw message — the events were good; the store was not.
|
||||
*/
|
||||
private suspend fun runInsertStage(
|
||||
batch: List<Submission>,
|
||||
@@ -290,10 +292,15 @@ class IngestQueue(
|
||||
} else {
|
||||
try {
|
||||
store.batchInsert(toInsert)
|
||||
} catch (e: CancellationException) {
|
||||
// Shutdown, not a store failure: rethrow so the loop
|
||||
// stops instead of stamping the batch Failed and
|
||||
// carrying on while cancelled.
|
||||
throw e
|
||||
} catch (e: Throwable) {
|
||||
Log.w("IngestQueue") { "batchInsert failed for ${toInsert.size} events: ${e.message}" }
|
||||
val reason = e.message ?: e::class.simpleName ?: "insert failed"
|
||||
List(toInsert.size) { IEventStore.InsertOutcome.Rejected(reason) }
|
||||
val reason = e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED
|
||||
List(toInsert.size) { IEventStore.InsertOutcome.Failed(reason) }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -349,7 +356,7 @@ class IngestQueue(
|
||||
* inserts), so the message is informational, not user-facing.
|
||||
*/
|
||||
private val missingOutcome =
|
||||
IEventStore.InsertOutcome.Rejected("internal error: missing outcome")
|
||||
IEventStore.InsertOutcome.Failed("internal error: missing outcome")
|
||||
|
||||
/**
|
||||
* Cap per batch. Sized to keep per-batch latency low (each
|
||||
|
||||
+4
@@ -180,6 +180,10 @@ class LiveEventStore(
|
||||
is IEventStore.InsertOutcome.Rejected -> {
|
||||
done.completeExceptionally(IllegalStateException(outcome.reason))
|
||||
}
|
||||
|
||||
is IEventStore.InsertOutcome.Failed -> {
|
||||
done.completeExceptionally(IllegalStateException(outcome.reason))
|
||||
}
|
||||
}
|
||||
}
|
||||
done.await()
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.store
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.HexKey
|
||||
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
|
||||
|
||||
/**
|
||||
* The pubkey that CONTROLS this event for ownership checks — NIP-09 deletion
|
||||
* authority and NIP-62 vanish targeting. A gift wrap (kind 1059) is signed by
|
||||
* a random one-time key, so control belongs to its p-tag RECIPIENT (the
|
||||
* recipient deletes the wraps addressed to them); every other event is
|
||||
* controlled by its author. One rule, shared by every store — do not re-derive
|
||||
* it inline.
|
||||
*/
|
||||
fun Event.owner(): HexKey = (this as? GiftWrapEvent)?.recipientPubKey() ?: pubKey
|
||||
+38
-10
@@ -27,6 +27,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
|
||||
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
|
||||
import kotlin.coroutines.cancellation.CancellationException
|
||||
|
||||
/**
|
||||
* Storage contract for Nostr events: insert, filter-query, count, delete,
|
||||
@@ -92,37 +93,64 @@ interface IEventStore : AutoCloseable {
|
||||
|
||||
/**
|
||||
* Per-row outcome from [batchInsert]. The OK frame on the wire is
|
||||
* built from this — `Accepted` becomes `OK true`, `Rejected.reason`
|
||||
* becomes the false reason. NIP-01 says OK pairs to its EVENT by
|
||||
* built from this — `Accepted` becomes `OK true`, the other two
|
||||
* become the false reason. NIP-01 says OK pairs to its EVENT by
|
||||
* id, not by order, so callers may dispatch outcomes in any order.
|
||||
*/
|
||||
sealed class InsertOutcome {
|
||||
data object Accepted : InsertOutcome()
|
||||
|
||||
/**
|
||||
* The EVENT's fault: policy said no — a duplicate, an expired
|
||||
* or invalid event, a blocked author. Final: re-offering the
|
||||
* same event yields the same answer, so dropping it is correct.
|
||||
*/
|
||||
data class Rejected(
|
||||
val reason: String,
|
||||
) : InsertOutcome()
|
||||
|
||||
/**
|
||||
* The STORE's fault: the event was acceptable but could not be
|
||||
* written — schema drift, a failed feed, a resource error. The
|
||||
* event is lost unless the caller re-offers it, and nothing
|
||||
* else will. Callers should count these apart from [Rejected]:
|
||||
* a rising [Rejected] is usually the protocol working
|
||||
* (duplicates on a wide fan-out), while a rising [Failed]
|
||||
* means the store is losing good events.
|
||||
*/
|
||||
data class Failed(
|
||||
val reason: String,
|
||||
) : InsertOutcome()
|
||||
}
|
||||
|
||||
/**
|
||||
* Bulk insert in a single transaction with per-row error isolation.
|
||||
* Returns one outcome per input event in the same order.
|
||||
* Bulk insert with per-row attribution. Returns one outcome per
|
||||
* input event in the same order.
|
||||
*
|
||||
* Implementations must isolate per-row failures so one bad event
|
||||
* doesn't roll back the others (SQLite uses SAVEPOINTs). If the
|
||||
* outer commit itself fails, every entry in the returned list is
|
||||
* `Rejected` with the commit-failure reason.
|
||||
* Implementations must isolate per-row problems so one bad event
|
||||
* never costs the batch: a policy refusal is [InsertOutcome.Rejected],
|
||||
* a store-side write error is [InsertOutcome.Failed] (SQLite uses
|
||||
* SAVEPOINTs for the isolation). Throwing is reserved for failures
|
||||
* with no per-event answer — the engine unreachable, the transaction
|
||||
* never started — and a caller may read a throw as "nothing in this
|
||||
* batch was written".
|
||||
*
|
||||
* Default impl runs each insert in its own transaction — correct
|
||||
* but loses the group-commit win. SQLite overrides this.
|
||||
* but loses the group-commit win — and cannot classify a throw from
|
||||
* [insert], so it reports `Failed`: re-offering a duplicate is
|
||||
* idempotent, while dropping a good event on a transient store
|
||||
* error is not. Implementations that can tell a refusal from a
|
||||
* write error should override and say which.
|
||||
*/
|
||||
suspend fun batchInsert(events: List<Event>): List<InsertOutcome> =
|
||||
events.map { event ->
|
||||
try {
|
||||
insert(event)
|
||||
InsertOutcome.Accepted
|
||||
} catch (e: CancellationException) {
|
||||
throw e
|
||||
} catch (e: Throwable) {
|
||||
InsertOutcome.Rejected(e.message ?: e::class.simpleName ?: "insert failed")
|
||||
InsertOutcome.Failed(e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+7
-3
@@ -52,15 +52,17 @@ object NdjsonImportExport {
|
||||
val imported: Long,
|
||||
/** Events the store rejected — overwhelmingly duplicates (unique-id). */
|
||||
val rejected: Long,
|
||||
/** Good events the store could not write — a store-side error, not the event's. */
|
||||
val failed: Long,
|
||||
/** Events dropped for a bad signature (only when verifying). */
|
||||
val invalid: Long,
|
||||
/** Lines that didn't parse as a NIP-01 event. */
|
||||
val malformed: Long,
|
||||
) {
|
||||
operator fun plus(o: ImportStats) = ImportStats(read + o.read, imported + o.imported, rejected + o.rejected, invalid + o.invalid, malformed + o.malformed)
|
||||
operator fun plus(o: ImportStats) = ImportStats(read + o.read, imported + o.imported, rejected + o.rejected, failed + o.failed, invalid + o.invalid, malformed + o.malformed)
|
||||
|
||||
companion object {
|
||||
val ZERO = ImportStats(0, 0, 0, 0, 0)
|
||||
val ZERO = ImportStats(0, 0, 0, 0, 0, 0)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -81,6 +83,7 @@ object NdjsonImportExport {
|
||||
var read = 0L
|
||||
var imported = 0L
|
||||
var rejected = 0L
|
||||
var failed = 0L
|
||||
var invalid = 0L
|
||||
var malformed = 0L
|
||||
val batch = ArrayList<Event>(batchSize)
|
||||
@@ -91,6 +94,7 @@ object NdjsonImportExport {
|
||||
when (outcome) {
|
||||
IEventStore.InsertOutcome.Accepted -> imported++
|
||||
is IEventStore.InsertOutcome.Rejected -> rejected++
|
||||
is IEventStore.InsertOutcome.Failed -> failed++
|
||||
}
|
||||
}
|
||||
batch.clear()
|
||||
@@ -112,7 +116,7 @@ object NdjsonImportExport {
|
||||
if (batch.size >= batchSize) flush()
|
||||
}
|
||||
flush()
|
||||
return ImportStats(read, imported, rejected, invalid, malformed)
|
||||
return ImportStats(read, imported, rejected, failed, invalid, malformed)
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+2
-2
@@ -103,7 +103,7 @@ class ObservableEventStore(
|
||||
// order. Already-expired ephemerals are dropped (matching
|
||||
// [insert]). Accepted events are emitted on [_changes] only
|
||||
// after the inner batch returns, so a commit failure that
|
||||
// converts everything to Rejected suppresses the emits.
|
||||
// converts everything to Failed suppresses the emits.
|
||||
if (events.isEmpty()) return emptyList()
|
||||
|
||||
val outcomes = arrayOfNulls<IEventStore.InsertOutcome>(events.size)
|
||||
@@ -114,7 +114,7 @@ class ObservableEventStore(
|
||||
if (event.kind.isEphemeral()) {
|
||||
outcomes[i] =
|
||||
if (event.isExpired()) {
|
||||
IEventStore.InsertOutcome.Rejected("blocked: Cannot insert an expired event")
|
||||
IEventStore.InsertOutcome.Rejected(RejectionReason.EXPIRED)
|
||||
} else {
|
||||
IEventStore.InsertOutcome.Accepted
|
||||
}
|
||||
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.store
|
||||
|
||||
/**
|
||||
* The machine-readable insert-rejection vocabulary, shared by every
|
||||
* [IEventStore] implementation so the same condition always rejects with the
|
||||
* same words — a caller tallying [IEventStore.InsertOutcome.Rejected] reasons,
|
||||
* or a relay building `OK false` frames, must never see two stores spell
|
||||
* "duplicate" differently.
|
||||
*
|
||||
* NIP-01: an `OK false` message SHOULD begin with a single-word
|
||||
* machine-readable prefix followed by `:`. The full-sentence constants here
|
||||
* are the standard reasons the built-in stores emit. `replaced:` is not in
|
||||
* NIP-01 but is the de-facto prefix (strfry and others) for a replaceable
|
||||
* event that lost to a stored newer version — distinct from `duplicate:`
|
||||
* (the exact event is already held).
|
||||
*/
|
||||
object RejectionReason {
|
||||
/**
|
||||
* The NIP-01 prefix vocabulary itself lives in
|
||||
* [com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.MachineReadablePrefix]
|
||||
* — use its `format`/`parse` instead of hand-writing prefixes. The one
|
||||
* prefix that enum lacks is the de-facto (not in NIP-01) word for a stale
|
||||
* version of a replaceable/addressable event:
|
||||
*/
|
||||
const val PREFIX_REPLACED = "replaced:"
|
||||
|
||||
// The standard store reasons.
|
||||
const val DUPLICATE = "duplicate: already have this event"
|
||||
const val EXPIRED = "blocked: Cannot insert an expired event"
|
||||
const val DELETED = "blocked: a deletion event exists"
|
||||
const val VANISHED = "blocked: a request to vanish event exists"
|
||||
const val REPLACED = "replaced: a newer version exists"
|
||||
const val INSERT_FAILED = "error: insert failed"
|
||||
}
|
||||
+3
-7
@@ -25,7 +25,7 @@ import com.vitorpamplona.quartz.nip01Core.core.AddressSerializer
|
||||
import com.vitorpamplona.quartz.nip01Core.core.AddressableEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.core.OptimizedJsonMapper
|
||||
import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.store.owner
|
||||
|
||||
class EventIndexesModule(
|
||||
val hasher: (db: SQLiteConnection) -> TagNameValueHasher,
|
||||
@@ -206,12 +206,8 @@ class EventIndexesModule(
|
||||
val kindLong = event.kind.toLong()
|
||||
val pubkeyHash = hasher.hash(event.pubKey)
|
||||
|
||||
val eventOwnerHash =
|
||||
if (event is GiftWrapEvent) {
|
||||
event.recipientPubKey()?.let { hasher.hash(it) } ?: pubkeyHash
|
||||
} else {
|
||||
pubkeyHash
|
||||
}
|
||||
val ownerKey = event.owner()
|
||||
val eventOwnerHash = if (ownerKey == event.pubKey) pubkeyHash else hasher.hash(ownerKey)
|
||||
|
||||
val eTagHash = hasher.hashETag(event.id)
|
||||
|
||||
|
||||
+2
-1
@@ -22,6 +22,7 @@ package com.vitorpamplona.quartz.nip01Core.store.sqlite
|
||||
|
||||
import androidx.sqlite.SQLiteConnection
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
|
||||
import com.vitorpamplona.quartz.nip40Expiration.expiration
|
||||
|
||||
class ExpirationModule : IModule {
|
||||
@@ -44,7 +45,7 @@ class ExpirationModule : IModule {
|
||||
FOR EACH ROW
|
||||
BEGIN
|
||||
-- Check for existing newer record
|
||||
SELECT RAISE(ABORT, 'blocked: this event is expired')
|
||||
SELECT RAISE(ABORT, '${RejectionReason.EXPIRED}')
|
||||
WHERE NEW.expiration <= unixepoch();
|
||||
END;
|
||||
""".trimIndent(),
|
||||
|
||||
+2
-1
@@ -21,6 +21,7 @@
|
||||
package com.vitorpamplona.quartz.nip01Core.store.sqlite
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Tag
|
||||
import com.vitorpamplona.quartz.nip01Core.tags.isIndexableTagName
|
||||
|
||||
interface IndexingStrategy {
|
||||
/**
|
||||
@@ -165,5 +166,5 @@ class DefaultIndexingStrategy(
|
||||
override fun shouldIndex(
|
||||
kind: Int,
|
||||
tag: Tag,
|
||||
) = tag.size >= 2 && tag[0].length == 1
|
||||
) = tag.size >= 2 && isIndexableTagName(tag[0])
|
||||
}
|
||||
|
||||
+32
-6
@@ -37,6 +37,7 @@ import com.vitorpamplona.quartz.nip01Core.store.FtsReindexProgress
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
|
||||
import com.vitorpamplona.quartz.nip01Core.store.RawEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.store.RejectionReason
|
||||
import com.vitorpamplona.quartz.nip09Deletions.DeletionEvent
|
||||
import com.vitorpamplona.quartz.nip40Expiration.isExpired
|
||||
import com.vitorpamplona.quartz.nip50Search.strippingSearchExtensions
|
||||
@@ -433,7 +434,7 @@ class SQLiteEventStore(
|
||||
}
|
||||
|
||||
suspend fun insertEvent(event: Event) {
|
||||
if (event.isExpired()) throw SQLiteException("blocked: Cannot insert an expired event")
|
||||
if (event.isExpired()) throw SQLiteException(RejectionReason.EXPIRED)
|
||||
if (event.kind.isEphemeral()) return
|
||||
|
||||
pool.useWriter { db ->
|
||||
@@ -463,8 +464,9 @@ class SQLiteEventStore(
|
||||
* stream still surfaces them; persistence is intentionally a
|
||||
* no-op per NIP-01.
|
||||
*
|
||||
* Outer-commit failure throws; the caller treats every entry as
|
||||
* `Rejected` (this is what the IEventStore contract documents).
|
||||
* Outer-commit failure throws; per the IEventStore contract a
|
||||
* throw means "nothing in this batch was written", and the
|
||||
* IngestQueue converts it to per-event `Failed`.
|
||||
*/
|
||||
suspend fun batchInsertEvents(events: List<Event>): List<IEventStore.InsertOutcome> {
|
||||
if (events.isEmpty()) return emptyList()
|
||||
@@ -489,7 +491,7 @@ class SQLiteEventStore(
|
||||
delta: LiveIndexDelta?,
|
||||
): IEventStore.InsertOutcome {
|
||||
if (event.isExpired()) {
|
||||
return IEventStore.InsertOutcome.Rejected("blocked: Cannot insert an expired event")
|
||||
return IEventStore.InsertOutcome.Rejected(RejectionReason.EXPIRED)
|
||||
}
|
||||
if (event.kind.isEphemeral()) return IEventStore.InsertOutcome.Accepted
|
||||
|
||||
@@ -509,7 +511,31 @@ class SQLiteEventStore(
|
||||
// ROLLBACK shouldn't mask the original cause.
|
||||
runCatching { db.execSQL("ROLLBACK TRANSACTION TO SAVEPOINT $sp") }
|
||||
runCatching { db.execSQL("RELEASE SAVEPOINT $sp") }
|
||||
IEventStore.InsertOutcome.Rejected(e.message ?: e::class.simpleName ?: "insert failed")
|
||||
classifyRowError(e)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Which side failed decides whether the caller may drop the event.
|
||||
* Policy refusals are recognizable — every schema trigger RAISEs with
|
||||
* a `blocked:` prefix, the immutability guards say "not allowed", and
|
||||
* a duplicate id is a constraint violation. Anything else (disk full,
|
||||
* I/O error, schema drift) is the store failing to write an acceptable
|
||||
* event: `Failed`, so a rising count is loud instead of blending into
|
||||
* the duplicate tally.
|
||||
*/
|
||||
private fun classifyRowError(e: Throwable): IEventStore.InsertOutcome {
|
||||
val message = e.message ?: e::class.simpleName ?: RejectionReason.INSERT_FAILED
|
||||
val refusal =
|
||||
message.contains("blocked:") ||
|
||||
message.contains("duplicate:") ||
|
||||
message.contains(RejectionReason.PREFIX_REPLACED) ||
|
||||
message.contains("not allowed") ||
|
||||
message.contains("constraint", ignoreCase = true)
|
||||
return if (refusal) {
|
||||
IEventStore.InsertOutcome.Rejected(message)
|
||||
} else {
|
||||
IEventStore.InsertOutcome.Failed(message)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -518,7 +544,7 @@ class SQLiteEventStore(
|
||||
private val delta: LiveIndexDelta?,
|
||||
) : IEventStore.ITransaction {
|
||||
override fun insert(event: Event) {
|
||||
if (event.isExpired()) throw SQLiteException("blocked: Cannot insert an expired event")
|
||||
if (event.isExpired()) throw SQLiteException(RejectionReason.EXPIRED)
|
||||
if (event.kind.isEphemeral()) return
|
||||
|
||||
innerInsertEvent(event, db, delta)
|
||||
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.tags
|
||||
|
||||
/**
|
||||
* Whether [name] is a tag name the NIP-01 `#x` filter space can address: a
|
||||
* single ASCII LETTER (`a`-`z` / `A`-`Z`), per NIP-01's "single-letter
|
||||
* (a-zA-Z) English-alphabet letters". Deliberately stricter than
|
||||
* `length == 1`: a `"5"` or `"#"` tag name is not filterable and indexing
|
||||
* it would let stores disagree about which tags `#x` filters reach.
|
||||
*/
|
||||
fun isIndexableTagName(name: String): Boolean = name.length == 1 && (name[0] in 'a'..'z' || name[0] in 'A'..'Z')
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip11RelayInfo
|
||||
|
||||
import androidx.collection.LruCache
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import kotlinx.coroutines.CancellationException
|
||||
|
||||
/**
|
||||
* TTL cache around any [Nip11Fetcher]. NIP-11 documents change rarely, so a
|
||||
* successful fetch is served from memory for [ttlSeconds]; a FAILED fetch is
|
||||
* also remembered — for the shorter [errorTtlSeconds] — so a mass census does
|
||||
* not hammer a host that just refused, while still retrying it soon.
|
||||
*
|
||||
* Concurrent first fetches of the same relay are not deduplicated: both hit the
|
||||
* network and the second result wins the cache slot. That is harmless (the
|
||||
* document is idempotent) and keeps this class lock-free.
|
||||
*/
|
||||
class CachedNip11Fetcher(
|
||||
private val delegate: Nip11Fetcher,
|
||||
private val ttlSeconds: Long = DEFAULT_TTL_SECONDS,
|
||||
private val errorTtlSeconds: Long = DEFAULT_ERROR_TTL_SECONDS,
|
||||
maxEntries: Int = 1000,
|
||||
private val now: () -> Long = { TimeUtils.now() },
|
||||
) : Nip11Fetcher {
|
||||
private sealed interface Cached {
|
||||
val at: Long
|
||||
}
|
||||
|
||||
private class Hit(
|
||||
val info: Nip11RelayInformation,
|
||||
override val at: Long,
|
||||
) : Cached
|
||||
|
||||
private class Miss(
|
||||
val message: String?,
|
||||
override val at: Long,
|
||||
) : Cached
|
||||
|
||||
private val cache = LruCache<NormalizedRelayUrl, Cached>(maxEntries)
|
||||
|
||||
/** The cached document if present and fresh; null otherwise. Never touches the network. */
|
||||
fun cachedOrNull(relay: NormalizedRelayUrl): Nip11RelayInformation? {
|
||||
val hit = cache[relay] as? Hit ?: return null
|
||||
return if (now() - hit.at < ttlSeconds) hit.info else null
|
||||
}
|
||||
|
||||
/** Drops the cache entry (success or failure) so the next [fetch] is fresh. */
|
||||
fun invalidate(relay: NormalizedRelayUrl) {
|
||||
cache.remove(relay)
|
||||
}
|
||||
|
||||
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation {
|
||||
when (val cached = cache[relay]) {
|
||||
is Hit -> if (now() - cached.at < ttlSeconds) return cached.info
|
||||
is Miss ->
|
||||
if (now() - cached.at < errorTtlSeconds) {
|
||||
throw Nip11FetchException(cached.message ?: "cached NIP-11 failure for ${relay.url}")
|
||||
}
|
||||
null -> {}
|
||||
}
|
||||
return try {
|
||||
delegate.fetch(relay).also { cache.put(relay, Hit(it, now())) }
|
||||
} catch (e: Exception) {
|
||||
if (e is CancellationException) throw e
|
||||
cache.put(relay, Miss(e.message, now()))
|
||||
throw e
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
/** Documents are near-static: trust a success for a day. */
|
||||
const val DEFAULT_TTL_SECONDS = 24L * 60 * 60
|
||||
|
||||
/** Failures are often transient: retry after five minutes. */
|
||||
const val DEFAULT_ERROR_TTL_SECONDS = 5L * 60
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip11RelayInfo
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
|
||||
/**
|
||||
* Fetches and parses a relay's NIP-11 information document (the
|
||||
* `application/nostr+json` answer on the relay's https url). Mirrors the
|
||||
* [com.vitorpamplona.quartz.nip05DnsIdentifiers.Nip05Fetcher] seam: the HTTP
|
||||
* transport lives in a platform implementation (OkHttpNip11Fetcher on
|
||||
* JVM/Android), so common code — probes, monitors, the CLI — depends only on
|
||||
* this interface. Wrap any implementation in [CachedNip11Fetcher] to add a TTL
|
||||
* cache.
|
||||
*
|
||||
* Throws [Nip11FetchException] (or a transport exception) when the document is
|
||||
* unavailable or unparseable.
|
||||
*/
|
||||
interface Nip11Fetcher {
|
||||
suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation
|
||||
}
|
||||
|
||||
/** The relay answered, but not with a usable NIP-11 document (bad status, not JSON). */
|
||||
class Nip11FetchException(
|
||||
message: String,
|
||||
) : Exception(message)
|
||||
@@ -0,0 +1,94 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip50Search
|
||||
|
||||
/**
|
||||
* An event's searchable text decomposed by ROLE, for full-text backends that
|
||||
* weight fields instead of indexing one concatenated blob
|
||||
* ([SearchableEvent.indexableContent]'s flat form). Produced per kind by
|
||||
* [SearchFieldExtractor].
|
||||
*
|
||||
* A kind is either PROFILE-shaped ([Profile] — kind-0-style identity: kind 0
|
||||
* itself, app handlers) or CONTENT-shaped ([Tiered] — title above summary
|
||||
* above body). The shape is part of the value, so a consumer discriminates on
|
||||
* the type instead of keeping its own list of profile kinds. Both shapes
|
||||
* carry a website role — a profile's homepage ([Profile.website]), or a
|
||||
* content kind's affiliation URLs ([Tiered.websites]: repo, trackers,
|
||||
* bookmarked page) — declared on each shape rather than the interface, so
|
||||
* [None] answers no question that doesn't apply to it.
|
||||
*
|
||||
* Multi-valued roles are carried UNJOINED, as lists: which separator to use —
|
||||
* or whether to index the values separately — is the backend's decision, and
|
||||
* once values are pre-joined a backend can't unmix them. Values produced by
|
||||
* [SearchFieldExtractor] are trimmed and non-empty; the types themselves do
|
||||
* not enforce it.
|
||||
*/
|
||||
sealed interface IndexableFields {
|
||||
fun isEmpty(): Boolean
|
||||
|
||||
/** No searchable text — the extraction result for non-searchable kinds. */
|
||||
data object None : IndexableFields {
|
||||
override fun isEmpty(): Boolean = true
|
||||
}
|
||||
|
||||
/**
|
||||
* Kind-0-shaped identity, each field in its own role. [SearchFieldExtractor]
|
||||
* never RETURNS an empty Profile — extract() normalizes every empty shape
|
||||
* to [None] — so an all-null value only exists mid-extraction or when
|
||||
* hand-built. Note the equality trap for hand-built values: an empty
|
||||
* shape isEmpty() but is not equal to [None].
|
||||
*/
|
||||
data class Profile(
|
||||
val name: String? = null,
|
||||
val displayName: String? = null,
|
||||
val about: String? = null,
|
||||
val nip05: String? = null,
|
||||
val lud16: String? = null,
|
||||
val website: String? = null,
|
||||
) : IndexableFields {
|
||||
override fun isEmpty(): Boolean = this == EMPTY
|
||||
|
||||
private companion object {
|
||||
val EMPTY = Profile()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Content decomposed by priority tier: [primary] (title-like values),
|
||||
* [secondary] (summary/description-like values), [text] (the body — the
|
||||
* one inherently single-valued role), plus the raw [hashtags] and
|
||||
* [locations] tag values.
|
||||
*/
|
||||
data class Tiered(
|
||||
val primary: List<String> = emptyList(),
|
||||
val secondary: List<String> = emptyList(),
|
||||
val text: String? = null,
|
||||
val hashtags: List<String> = emptyList(),
|
||||
val locations: List<String> = emptyList(),
|
||||
val websites: List<String> = emptyList(),
|
||||
) : IndexableFields {
|
||||
override fun isEmpty(): Boolean = this == EMPTY
|
||||
|
||||
private companion object {
|
||||
val EMPTY = Tiered()
|
||||
}
|
||||
}
|
||||
}
|
||||
+486
@@ -0,0 +1,486 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip50Search
|
||||
|
||||
import com.vitorpamplona.quartz.buzz.agentProfiles.AgentProfileEvent
|
||||
import com.vitorpamplona.quartz.buzz.apPersonas.PersonaEvent
|
||||
import com.vitorpamplona.quartz.buzz.managedAgents.ManagedAgentEvent
|
||||
import com.vitorpamplona.quartz.buzz.teams.TeamEvent
|
||||
import com.vitorpamplona.quartz.buzz.workflow.WorkflowDefEvent
|
||||
import com.vitorpamplona.quartz.experimental.agora.FundraiserEvent
|
||||
import com.vitorpamplona.quartz.experimental.audio.track.AudioTrackEvent
|
||||
import com.vitorpamplona.quartz.experimental.fitness.workout.ExerciseTemplateEvent
|
||||
import com.vitorpamplona.quartz.experimental.fitness.workout.WorkoutRecordEvent
|
||||
import com.vitorpamplona.quartz.experimental.interactiveStories.InteractiveStoryBaseEvent
|
||||
import com.vitorpamplona.quartz.experimental.music.playlist.MusicPlaylistEvent
|
||||
import com.vitorpamplona.quartz.experimental.music.track.MusicTrackEvent
|
||||
import com.vitorpamplona.quartz.experimental.nip82SoftwareApps.application.SoftwareApplicationEvent
|
||||
import com.vitorpamplona.quartz.experimental.nipsOnNostr.NipTextEvent
|
||||
import com.vitorpamplona.quartz.feedDefinition.FeedDefinitionEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.tags.hashtags.hashtags
|
||||
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import com.vitorpamplona.quartz.nip14Subject.subject
|
||||
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
|
||||
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelCreateEvent
|
||||
import com.vitorpamplona.quartz.nip28PublicChat.admin.ChannelMetadataEvent
|
||||
import com.vitorpamplona.quartz.nip29RelayGroups.metadata.GroupMetadataEvent
|
||||
import com.vitorpamplona.quartz.nip30CustomEmoji.pack.EmojiPackEvent
|
||||
import com.vitorpamplona.quartz.nip34Git.issue.GitIssueEvent
|
||||
import com.vitorpamplona.quartz.nip34Git.pr.GitPullRequestEvent
|
||||
import com.vitorpamplona.quartz.nip34Git.repository.GitRepositoryEvent
|
||||
import com.vitorpamplona.quartz.nip35Torrents.TorrentEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.appCurationSet.AppCurationSetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.articleCurationSet.ArticleCurationSetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.bookmarkList.BookmarkListEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.bookmarkList.OldBookmarkListEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.followList.FollowListEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.interestSet.InterestSetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.labeledBookmarkList.LabeledBookmarkListEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.mediaStarterPack.MediaStarterPackEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.peopleList.PeopleListEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.pictureCurationSet.PictureCurationSetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.relaySets.RelaySetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.releaseArtifactSet.ReleaseArtifactSetEvent
|
||||
import com.vitorpamplona.quartz.nip51Lists.videoCurationSet.VideoCurationSetEvent
|
||||
import com.vitorpamplona.quartz.nip52Calendar.appt.day.CalendarDateSlotEvent
|
||||
import com.vitorpamplona.quartz.nip52Calendar.appt.time.CalendarTimeSlotEvent
|
||||
import com.vitorpamplona.quartz.nip52Calendar.calendar.CalendarEvent
|
||||
import com.vitorpamplona.quartz.nip53LiveActivities.clip.LiveActivitiesClipEvent
|
||||
import com.vitorpamplona.quartz.nip53LiveActivities.meetingSpaces.MeetingRoomEvent
|
||||
import com.vitorpamplona.quartz.nip53LiveActivities.meetingSpaces.MeetingSpaceEvent
|
||||
import com.vitorpamplona.quartz.nip53LiveActivities.streaming.LiveActivitiesEvent
|
||||
import com.vitorpamplona.quartz.nip54Wiki.WikiNoteEvent
|
||||
import com.vitorpamplona.quartz.nip58Badges.definition.BadgeDefinitionEvent
|
||||
import com.vitorpamplona.quartz.nip5aStaticWebsites.NamedSiteEvent
|
||||
import com.vitorpamplona.quartz.nip5aStaticWebsites.RootSiteEvent
|
||||
import com.vitorpamplona.quartz.nip5dNapplets.NamedNappletEvent
|
||||
import com.vitorpamplona.quartz.nip5dNapplets.NappletSnapshotEvent
|
||||
import com.vitorpamplona.quartz.nip5dNapplets.RootNappletEvent
|
||||
import com.vitorpamplona.quartz.nip68Picture.PictureEvent
|
||||
import com.vitorpamplona.quartz.nip71Video.AddressableVideoEvent
|
||||
import com.vitorpamplona.quartz.nip71Video.RegularVideoEvent
|
||||
import com.vitorpamplona.quartz.nip72ModCommunities.definition.CommunityDefinitionEvent
|
||||
import com.vitorpamplona.quartz.nip75ZapGoals.GoalEvent
|
||||
import com.vitorpamplona.quartz.nip7DThreads.ThreadEvent
|
||||
import com.vitorpamplona.quartz.nip84Highlights.HighlightEvent
|
||||
import com.vitorpamplona.quartz.nip89AppHandlers.definition.AppDefinitionEvent
|
||||
import com.vitorpamplona.quartz.nip94FileMetadata.FileHeaderEvent
|
||||
import com.vitorpamplona.quartz.nip99Classifieds.ClassifiedsEvent
|
||||
import com.vitorpamplona.quartz.nipB0WebBookmarks.WebBookmarkEvent
|
||||
import com.vitorpamplona.quartz.nipC0CodeSnippets.CodeSnippetEvent
|
||||
import com.vitorpamplona.quartz.nipF4Podcasts.episode.PodcastEpisodeEvent
|
||||
import com.vitorpamplona.quartz.nipF4Podcasts.metadata.PodcastMetadataEvent
|
||||
|
||||
/**
|
||||
* Decomposes every [SearchableEvent] into [IndexableFields] by priority tier:
|
||||
* title-like accessors primary, summary/description secondary, body tertiary,
|
||||
* with hashtag and location tags carried raw beside the tiers. Each explicit
|
||||
* branch splits exactly the accessors that kind's `indexableContent()`
|
||||
* concatenates — keep the two in sync when a kind's parsing changes. Kinds
|
||||
* without an explicit branch fall back to the [SearchableEvent] branch (whole
|
||||
* `indexableContent()` in the tertiary tier), so EVERY searchable kind,
|
||||
* current or future, is extracted.
|
||||
*
|
||||
* A kind may also fill the profile roles when it carries that shape: kind
|
||||
* 31990 goes through the kind-0 fields wholesale, and any kind with a
|
||||
* homepage/site URL fills [IndexableFields.websites]. Hashtags and `location`
|
||||
* tags are filled SYSTEMICALLY by the [tiers] funnel every content branch
|
||||
* uses, so recall never depends on a branch remembering them.
|
||||
*
|
||||
* Non-searchable kinds return [IndexableFields.None]. The extraction is
|
||||
* derived data baked into the build: stores should re-derive after upgrades
|
||||
* (see [com.vitorpamplona.quartz.nip01Core.store.IEventStore.reindexFullTextSearch]).
|
||||
*/
|
||||
object SearchFieldExtractor {
|
||||
/** Empty extractions always come back as [IndexableFields.None], whatever shape produced them. */
|
||||
fun extract(event: Event): IndexableFields = base(event).let { if (it.isEmpty()) IndexableFields.None else it }
|
||||
|
||||
private fun base(event: Event): IndexableFields =
|
||||
when (event) {
|
||||
// kind 0 -> the profile fields, each in its own role.
|
||||
is MetadataEvent -> {
|
||||
val md = event.contactMetaData()
|
||||
if (md == null) {
|
||||
IndexableFields.Profile()
|
||||
} else {
|
||||
IndexableFields.Profile(
|
||||
name = clean(md.name),
|
||||
displayName = clean(md.displayName),
|
||||
about = clean(md.about),
|
||||
nip05 = clean(md.nip05),
|
||||
lud16 = clean(md.lud16),
|
||||
website = clean(md.website),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
is LongTextNoteEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is WikiNoteEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is ClassifiedsEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is GitRepositoryEvent -> {
|
||||
tiers(event, listOf(event.name()), listOf(event.description()), event.content, websites = event.webs())
|
||||
}
|
||||
|
||||
is GitIssueEvent -> {
|
||||
tiers(event, event.subject(), null, event.content)
|
||||
}
|
||||
|
||||
is GitPullRequestEvent -> {
|
||||
tiers(event, event.subject(), null, event.content)
|
||||
}
|
||||
|
||||
is CommunityDefinitionEvent -> {
|
||||
tiers(event, listOf(event.name()), listOf(event.description(), event.rules()), event.content)
|
||||
}
|
||||
|
||||
is EmojiPackEvent -> {
|
||||
tiers(event, event.titleOrName(), event.description(), event.content)
|
||||
}
|
||||
|
||||
is ChannelCreateEvent -> {
|
||||
event.channelInfo().let { tiers(event, it.name, it.about, null) }
|
||||
}
|
||||
|
||||
is ChannelMetadataEvent -> {
|
||||
event.channelInfo().let { tiers(event, it.name, it.about, null) }
|
||||
}
|
||||
|
||||
is PictureEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is RegularVideoEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is AddressableVideoEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
// Torrents are searched by FILE NAME above all — index the file
|
||||
// list into the secondary tier, trackers as the affiliation URL.
|
||||
is TorrentEvent -> {
|
||||
tiers(event, listOf(event.title()), event.files().map { it.fileName }, event.content, websites = event.trackers())
|
||||
}
|
||||
|
||||
is ThreadEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is FundraiserEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is NipTextEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is ExerciseTemplateEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is WorkoutRecordEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is CalendarEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is LiveActivitiesClipEvent -> {
|
||||
tiers(event, event.title(), null, event.content)
|
||||
}
|
||||
|
||||
is CalendarDateSlotEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is CalendarTimeSlotEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is LiveActivitiesEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content, website = event.streaming())
|
||||
}
|
||||
|
||||
is InteractiveStoryBaseEvent -> {
|
||||
tiers(event, event.title(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is MeetingSpaceEvent -> {
|
||||
tiers(event, event.room(), event.summary(), event.content)
|
||||
}
|
||||
|
||||
is MeetingRoomEvent -> {
|
||||
tiers(event, event.title(), event.summary(), null)
|
||||
}
|
||||
|
||||
// Code snippets are searched by language/runtime as much as name —
|
||||
// fold those keywords into the secondary tier, repo as affiliation.
|
||||
is CodeSnippetEvent -> {
|
||||
tiers(
|
||||
event,
|
||||
listOf(event.snippetName()),
|
||||
listOf(event.snippetDescription(), event.language(), event.extension(), event.runtime()),
|
||||
event.content,
|
||||
websites = listOf(event.repo()),
|
||||
)
|
||||
}
|
||||
|
||||
is BadgeDefinitionEvent -> {
|
||||
tiers(event, event.name(), event.description(), event.content)
|
||||
}
|
||||
|
||||
is MusicPlaylistEvent -> {
|
||||
tiers(event, event.title(), event.description(), event.content)
|
||||
}
|
||||
|
||||
is MusicTrackEvent -> {
|
||||
tiers(event, listOf(event.title()), listOf(event.artist(), event.album()), event.content)
|
||||
}
|
||||
|
||||
is SoftwareApplicationEvent -> {
|
||||
tiers(event, listOf(event.name()), listOf(event.summary()), event.content, websites = listOf(event.url(), event.repository()))
|
||||
}
|
||||
|
||||
is PodcastEpisodeEvent -> {
|
||||
tiers(event, event.title(), event.description(), event.content)
|
||||
}
|
||||
|
||||
is PodcastMetadataEvent -> {
|
||||
tiers(event, listOf(event.title()), listOf(event.description()), null, websites = event.websites())
|
||||
}
|
||||
|
||||
is GroupMetadataEvent -> {
|
||||
tiers(event, event.name(), event.about(), null)
|
||||
}
|
||||
|
||||
is InterestSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is FollowListEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is MediaStarterPackEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is PictureCurationSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is ArticleCurationSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is VideoCurationSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is ReleaseArtifactSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is AppCurationSetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is RelaySetEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
// A web bookmark IS its URL — route it to the affiliation website
|
||||
// field so the bookmark is findable by its domain.
|
||||
is WebBookmarkEvent -> {
|
||||
tiers(event, event.title(), event.description(), null, website = event.url())
|
||||
}
|
||||
|
||||
is NamedSiteEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is RootSiteEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is RootNappletEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is NappletSnapshotEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is NamedNappletEvent -> {
|
||||
tiers(event, event.title(), event.description(), null)
|
||||
}
|
||||
|
||||
is FeedDefinitionEvent -> {
|
||||
tiers(event, event.title(), null, null)
|
||||
}
|
||||
|
||||
is LabeledBookmarkListEvent -> {
|
||||
tiers(event, event.titleOrName(), event.description(), null)
|
||||
}
|
||||
|
||||
is PeopleListEvent -> {
|
||||
tiers(event, event.titleOrName(), event.description(), null)
|
||||
}
|
||||
|
||||
is BookmarkListEvent -> {
|
||||
tiers(event, event.title(), null, null)
|
||||
}
|
||||
|
||||
is OldBookmarkListEvent -> {
|
||||
tiers(event, event.title(), null, null)
|
||||
}
|
||||
|
||||
is GoalEvent -> {
|
||||
tiers(event, null, event.summary(), event.content)
|
||||
}
|
||||
|
||||
is HighlightEvent -> {
|
||||
tiers(event, emptyList(), listOf(event.comment(), event.context()), event.content)
|
||||
}
|
||||
|
||||
is FileHeaderEvent -> {
|
||||
tiers(event, null, event.summary(), event.content)
|
||||
}
|
||||
|
||||
is AudioTrackEvent -> {
|
||||
tiers(event, event.subject(), null, null)
|
||||
}
|
||||
|
||||
// Buzz agent/workspace kinds carry their metadata as JSON in `content`;
|
||||
// split each decoded object the way its indexableContent() concatenates it.
|
||||
is AgentProfileEvent -> {
|
||||
event.profileOrNull()?.let { tiers(event, listOf(it.name, it.displayName), emptyList(), null) } ?: tiers(event, null, null, null)
|
||||
}
|
||||
|
||||
is PersonaEvent -> {
|
||||
event.personaOrNull()?.let { tiers(event, it.displayName, null, it.systemPrompt) } ?: tiers(event, null, null, null)
|
||||
}
|
||||
|
||||
is ManagedAgentEvent -> {
|
||||
event.agentOrNull()?.let { tiers(event, it.name, null, it.systemPrompt) } ?: tiers(event, null, null, null)
|
||||
}
|
||||
|
||||
is TeamEvent -> {
|
||||
event.teamOrNull()?.let { tiers(event, it.name, it.description, it.instructions) } ?: tiers(event, null, null, null)
|
||||
}
|
||||
|
||||
is WorkflowDefEvent -> {
|
||||
tiers(event, event.name(), null, event.content)
|
||||
}
|
||||
|
||||
// kind 31990 — the app handler's metadata IS a UserMetadata clone,
|
||||
// so route it through the kind-0 profile fields: an app's
|
||||
// @-handle and site get the same treatment a person's do.
|
||||
is AppDefinitionEvent -> {
|
||||
val md = event.appMetaData()
|
||||
if (md == null) {
|
||||
IndexableFields.Profile()
|
||||
} else {
|
||||
IndexableFields.Profile(
|
||||
// Per NIP-24 the deprecated `username` folds into `name`.
|
||||
name = clean(md.name ?: md.username),
|
||||
displayName = clean(md.displayName),
|
||||
about = clean(md.about),
|
||||
nip05 = clean(md.nip05),
|
||||
lud16 = clean(md.lud16),
|
||||
website = clean(md.website),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// kind 1 LAST among the explicit branches, defensively: a future
|
||||
// kind extending the text-note base must hit its own branch first.
|
||||
is TextNoteEvent -> {
|
||||
tiers(event, event.subject(), null, event.content)
|
||||
}
|
||||
|
||||
// Everything else Quartz can search, current or future: the whole
|
||||
// indexableContent lands in the tertiary tier.
|
||||
is SearchableEvent -> {
|
||||
tiers(event, null, null, event.indexableContent())
|
||||
}
|
||||
|
||||
else -> {
|
||||
IndexableFields.None
|
||||
}
|
||||
}
|
||||
|
||||
/** Single-value convenience over the list funnel — most kinds carry one title, one summary, one body. */
|
||||
private fun tiers(
|
||||
event: Event,
|
||||
primary: String?,
|
||||
secondary: String?,
|
||||
text: String?,
|
||||
website: String? = null,
|
||||
) = tiers(event, listOf(primary), listOf(secondary), text, listOf(website))
|
||||
|
||||
/**
|
||||
* The one funnel every content branch uses — [IndexableFields.Tiered.hashtags]
|
||||
* and [IndexableFields.Tiered.locations] are filled here, so no branch can
|
||||
* forget them. Values stay UNJOINED: separator choices belong to the backend.
|
||||
*/
|
||||
private fun tiers(
|
||||
event: Event,
|
||||
primary: List<String?>,
|
||||
secondary: List<String?>,
|
||||
text: String?,
|
||||
websites: List<String?> = emptyList(),
|
||||
) = IndexableFields.Tiered(
|
||||
primary = cleanAll(primary),
|
||||
secondary = cleanAll(secondary),
|
||||
text = clean(text),
|
||||
hashtags = cleanAll(event.tags.hashtags()),
|
||||
locations = locationValues(event),
|
||||
websites = cleanAll(websites),
|
||||
)
|
||||
|
||||
/** Trim and drop empties at the single funnel every derived string passes through. */
|
||||
private fun clean(s: String?): String? = s?.trim()?.ifEmpty { null }
|
||||
|
||||
private fun cleanAll(parts: List<String?>): List<String> = parts.mapNotNull { clean(it) }
|
||||
|
||||
/**
|
||||
* Every `location` tag value, on ANY kind. Deliberately a raw scan, not a
|
||||
* typed accessor: Quartz's LocationTag classes are per-NIP (calendar,
|
||||
* picture, classifieds) and only those kinds expose locations(), while
|
||||
* this funnel must also catch location tags on kinds whose class doesn't
|
||||
* model them.
|
||||
*/
|
||||
private fun locationValues(event: Event): List<String> = event.tags.mapNotNull { tag -> if (tag.getOrNull(0) != "location") null else clean(tag.getOrNull(1)) }
|
||||
}
|
||||
@@ -28,43 +28,70 @@ import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
* NIP-50 defines the [com.vitorpamplona.quartz.nip01Core.relay.filters.Filter.search]
|
||||
* field as "a string describing a query in a human-readable form", optionally
|
||||
* carrying `key:value` extension tokens such as `domain:example.com` or
|
||||
* `language:en`. This class splits that raw string into the free-text [terms]
|
||||
* and the recognized [extensions], giving relays (and search redirectors) a
|
||||
* typed view of the query instead of forcing each one to re-parse the string.
|
||||
* `language:en`. This class splits that raw string into the free-text [terms],
|
||||
* the Google-style term syntax ([phrases], [notPhrases], [notTerms]), and the
|
||||
* recognized [extensions], giving relays (and search redirectors) a typed view
|
||||
* of the query instead of forcing each one to re-parse the string.
|
||||
*
|
||||
* Example:
|
||||
* ```
|
||||
* val q = SearchQuery.parse("best nostr apps domain:example.com language:en")
|
||||
* q.terms // "best nostr apps"
|
||||
* val q = SearchQuery.parse("best \"nostr apps\" -spam domain:example.com")
|
||||
* q.terms // "best"
|
||||
* q.phrases // ["nostr apps"]
|
||||
* q.notTerms // ["spam"]
|
||||
* q.domain // "example.com"
|
||||
* q.language // "en"
|
||||
* ```
|
||||
*
|
||||
* ## Tokenization
|
||||
* ## Parse order (load-bearing)
|
||||
*
|
||||
* The string is split on whitespace. A token is treated as an extension when:
|
||||
* - it contains a `:`,
|
||||
* - the part before the `:` is a non-empty run of lowercase ASCII letters
|
||||
* (`a`–`z`), and
|
||||
* - the part after the `:` is non-empty and does not start with `//` (so URLs
|
||||
* like `https://example.com` stay in [terms]).
|
||||
* Quoted spans are lifted off the RAW string FIRST, then the residual is
|
||||
* tokenized for extensions, then `-word` exclusions split off. The order
|
||||
* matters twice over: the extension pass is quote-blind, so a span ending in
|
||||
* an extension-shaped token (`"pizza sort:rank" -spam`) would otherwise lose
|
||||
* its closing quote and the unclosed quote would swallow the rest of the
|
||||
* query; and lifting first lets quotes protect extension-shaped tokens —
|
||||
* `"include:spam"` is the phrase [include, spam], not an extension.
|
||||
*
|
||||
* Everything else is free text. Per NIP-50 unknown extensions are kept (so they
|
||||
* can be forwarded to a backend) — relays "SHOULD ignore extensions they don't
|
||||
* support", which this models by simply not having a typed accessor for them;
|
||||
* they remain readable through [extensions] / [extension].
|
||||
* ## Term syntax
|
||||
*
|
||||
* Extension keys are matched case-sensitively against the lowercase forms
|
||||
* documented by NIP-50. Duplicate keys keep the last occurrence.
|
||||
* - A `"quoted span"` is an exact-phrase requirement ([phrases]); `-"…"` is a
|
||||
* phrase exclusion ([notPhrases]). A quote opens a span only at a token
|
||||
* boundary — mid-token quotes stay ordinary characters. An unclosed span
|
||||
* runs to the end of the string. Empty spans are dropped, but a positive
|
||||
* phrase keeps content a text index may not hold ("⚡"): it is an
|
||||
* unsatisfiable requirement the backend turns into provably-no-match —
|
||||
* dropping it here would silently flip that into match-all.
|
||||
* - A leading `-` on a 2+ character token makes it an exclusion ([notTerms],
|
||||
* all leading dashes stripped); a lone `-` stays an ordinary term. There is
|
||||
* no `-extension` syntax: extension keys are strictly `a`–`z`, so
|
||||
* `-include:spam` fails the key test and becomes the excluded literal.
|
||||
*
|
||||
* ## Extension tokenization
|
||||
*
|
||||
* The residual is split on whitespace. A token is treated as an extension when
|
||||
* it contains a `:`, the part before the `:` is a non-empty run of lowercase
|
||||
* ASCII letters (`a`–`z`), and the part after is non-empty and does not start
|
||||
* with `//` (so URLs like `https://example.com` stay in [terms]). Per NIP-50
|
||||
* unknown extensions are kept (readable through [extensions] / [extension]) so
|
||||
* they can be forwarded to a backend; relays "SHOULD ignore extensions they
|
||||
* don't support". Extension keys are matched case-sensitively against the
|
||||
* lowercase forms documented by NIP-50. Duplicate keys keep the last
|
||||
* occurrence.
|
||||
*/
|
||||
class SearchQuery(
|
||||
/** The human-readable search terms with all extension tokens removed. */
|
||||
/** The loose human-readable search terms: extensions, phrases, and exclusions all removed. */
|
||||
val terms: String,
|
||||
/**
|
||||
* All recognized `key:value` extension tokens, in the order they appeared.
|
||||
* Known keys: [INCLUDE], [DOMAIN], [LANGUAGE], [SENTIMENT], [NSFW].
|
||||
*/
|
||||
val extensions: Map<String, String>,
|
||||
/** Exact-phrase requirements (`"nostr apps"`), quotes removed, in order. */
|
||||
val phrases: List<String> = emptyList(),
|
||||
/** Exact-phrase exclusions (`-"nostr apps"`), quotes removed, in order. */
|
||||
val notPhrases: List<String> = emptyList(),
|
||||
/** Single-word exclusions (`-spam`), dashes removed, in order. */
|
||||
val notTerms: List<String> = emptyList(),
|
||||
) {
|
||||
/** `true` when the query carries the `include:spam` token (NIP-50: disable spam filtering). */
|
||||
val includeSpam: Boolean
|
||||
@@ -93,20 +120,41 @@ class SearchQuery(
|
||||
val nsfwIncluded: Boolean
|
||||
get() = nsfw ?: true
|
||||
|
||||
/**
|
||||
* Whether the query REQUIRES any text — loose terms or phrases. Exclusions
|
||||
* alone don't count: an exclusions-only query is plain recall minus the
|
||||
* excluded words, not a ranked text search.
|
||||
*/
|
||||
val hasText: Boolean
|
||||
get() = terms.isNotEmpty() || phrases.isNotEmpty()
|
||||
|
||||
/** Returns the raw value of an arbitrary extension key (including unknown ones), or null. */
|
||||
fun extension(key: String): String? = extensions[key]
|
||||
|
||||
/** Returns true when there are no free-text terms (the query is extensions-only or empty). */
|
||||
/** Returns true when there are no loose free-text terms. Phrases don't count — see [hasText] for "any required text". */
|
||||
fun isTermsEmpty(): Boolean = terms.isEmpty()
|
||||
|
||||
/**
|
||||
* Re-assembles a canonical NIP-50 search string: the free-text [terms]
|
||||
* followed by each `key:value` extension. Useful for a redirector that
|
||||
* normalizes the incoming query before forwarding it to a backend.
|
||||
* Re-assembles a canonical NIP-50 search string: the free-text [terms],
|
||||
* then each `"phrase"`, `-exclusion`, `-"phrase exclusion"`, and
|
||||
* `key:value` extension. Canonical, not order-preserving. Useful for a
|
||||
* redirector that normalizes the incoming query before forwarding it.
|
||||
*/
|
||||
fun toSearchString(): String =
|
||||
buildString {
|
||||
append(terms)
|
||||
for (phrase in phrases) {
|
||||
if (isNotEmpty()) append(' ')
|
||||
append('"').append(phrase).append('"')
|
||||
}
|
||||
for (word in notTerms) {
|
||||
if (isNotEmpty()) append(' ')
|
||||
append('-').append(word)
|
||||
}
|
||||
for (phrase in notPhrases) {
|
||||
if (isNotEmpty()) append(' ')
|
||||
append("-\"").append(phrase).append('"')
|
||||
}
|
||||
for ((key, value) in extensions) {
|
||||
if (isNotEmpty()) append(' ')
|
||||
append(key).append(':').append(value)
|
||||
@@ -134,20 +182,24 @@ class SearchQuery(
|
||||
|
||||
private val WHITESPACE = Regex("\\s+")
|
||||
|
||||
/** Empty query — no terms and no extensions. */
|
||||
/** Empty query — no terms, no syntax, no extensions. */
|
||||
val EMPTY = SearchQuery("", emptyMap())
|
||||
|
||||
/**
|
||||
* Parses a raw NIP-50 [search] string into a [SearchQuery]. A null or
|
||||
* blank input yields [EMPTY].
|
||||
* blank input yields [EMPTY]. See the class KDoc for the grammar and
|
||||
* why the quote pass runs before the extension pass.
|
||||
*/
|
||||
fun parse(search: String?): SearchQuery {
|
||||
if (search.isNullOrBlank()) return EMPTY
|
||||
|
||||
val quoted = liftQuotedSpans(search)
|
||||
val extensions = LinkedHashMap<String, String>()
|
||||
val terms = StringBuilder()
|
||||
val notTerms = ArrayList<String>()
|
||||
|
||||
for (token in search.trim().split(WHITESPACE)) {
|
||||
for (token in quoted.residual.trim().split(WHITESPACE)) {
|
||||
if (token.isEmpty()) continue
|
||||
val colon = token.indexOf(':')
|
||||
if (colon > 0 && colon < token.length - 1) {
|
||||
val key = token.substring(0, colon)
|
||||
@@ -157,18 +209,63 @@ class SearchQuery(
|
||||
continue
|
||||
}
|
||||
}
|
||||
if (token.length > 1 && token[0] == '-') {
|
||||
// All-dash tokens ("--") strip to nothing: an
|
||||
// exclude-nothing token is dropped, not surfaced — an
|
||||
// empty exclusion would round-trip into a required "-".
|
||||
token.trimStart('-').takeIf { it.isNotEmpty() }?.let { notTerms += it }
|
||||
continue
|
||||
}
|
||||
if (terms.isNotEmpty()) terms.append(' ')
|
||||
terms.append(token)
|
||||
}
|
||||
|
||||
return SearchQuery(terms.toString(), extensions)
|
||||
return SearchQuery(terms.toString(), extensions, quoted.phrases, quoted.notPhrases, notTerms)
|
||||
}
|
||||
|
||||
private fun isExtensionKey(key: String): Boolean = key.isNotEmpty() && key.all { it in 'a'..'z' }
|
||||
|
||||
/** The quoted spans lifted off the raw text, plus the residual for the extension and `-word` passes. */
|
||||
private class QuotedSpans(
|
||||
val phrases: List<String>,
|
||||
val notPhrases: List<String>,
|
||||
val residual: String,
|
||||
)
|
||||
|
||||
/** Stage one, over the RAW string: lift every `"…"` / `-"…"` span. See the class KDoc for the rules. */
|
||||
private fun liftQuotedSpans(text: String): QuotedSpans {
|
||||
val phrases = ArrayList<String>()
|
||||
val notPhrases = ArrayList<String>()
|
||||
val residual = StringBuilder()
|
||||
var i = 0
|
||||
var boundary = true
|
||||
while (i < text.length) {
|
||||
val c = text[i]
|
||||
val neg = c == '-' && i + 1 < text.length && text[i + 1] == '"'
|
||||
if (boundary && (c == '"' || neg)) {
|
||||
val start = i + if (neg) 2 else 1
|
||||
val close = text.indexOf('"', start)
|
||||
val end = if (close < 0) text.length else close
|
||||
val span = text.substring(start, end).trim()
|
||||
i = if (close < 0) text.length else close + 1
|
||||
if (span.isNotEmpty()) {
|
||||
if (neg) notPhrases += span else phrases += span
|
||||
}
|
||||
// The lifted span's place stays a token boundary for what follows.
|
||||
residual.append(' ')
|
||||
} else {
|
||||
residual.append(c)
|
||||
boundary = c.isWhitespace()
|
||||
i++
|
||||
}
|
||||
}
|
||||
return QuotedSpans(phrases, notPhrases, residual.toString())
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns [search] with every `key:value` extension token removed,
|
||||
* leaving only the free-text terms (tokenized as in [parse]).
|
||||
* leaving the free-text query (terms, phrases, and exclusions,
|
||||
* re-assembled as in [toSearchString]).
|
||||
*
|
||||
* Backends that hand the search string to an engine with its own
|
||||
* query syntax — e.g. SQLite FTS, where `:` is column-filter
|
||||
@@ -184,7 +281,8 @@ class SearchQuery(
|
||||
fun stripExtensions(search: String?): String? {
|
||||
if (search.isNullOrBlank()) return search
|
||||
val parsed = parse(search)
|
||||
return if (parsed.extensions.isEmpty()) search else parsed.terms
|
||||
if (parsed.extensions.isEmpty()) return search
|
||||
return SearchQuery(parsed.terms, emptyMap(), parsed.phrases, parsed.notPhrases, parsed.notTerms).toSearchString()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+139
@@ -0,0 +1,139 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentSet
|
||||
|
||||
/**
|
||||
* Which relays are worth dialling, within one fan-out cycle. A discovered
|
||||
* relay list is five figures of urls and most of them are corpses; without
|
||||
* this, every cycle re-dials all of them and the working relays queue behind
|
||||
* hosts that stopped existing years ago.
|
||||
*
|
||||
* Failures are counted per AUTHORITY (`host[:port]`), not per url: the outbox
|
||||
* model mints one url per user for a filtering relay, so a per-url counter
|
||||
* never reaches a threshold on any single one. The authority is host-only and
|
||||
* does NOT fold a subdomain into its parent — those are different servers.
|
||||
*
|
||||
* [produced] overrides a strike race: a host that has ever delivered is never
|
||||
* treated as dead for the rest of the cycle, whichever order the two events
|
||||
* land in. Cycle-local; nothing persists — see [RelayReachabilityStore] for
|
||||
* the part that survives a restart.
|
||||
*/
|
||||
class HostStrikes(
|
||||
private val strikeLimit: Int = DEFAULT_STRIKE_LIMIT,
|
||||
// Relays a previous run proved unreachable, and still within their TTL.
|
||||
private val knownDead: Set<NormalizedRelayUrl> = emptySet(),
|
||||
) {
|
||||
private val strikes = ConcurrentMap<String, Int>()
|
||||
private val deadHosts = ConcurrentSet<String>()
|
||||
private val producedHosts = ConcurrentSet<String>()
|
||||
|
||||
private val delivered = ConcurrentSet<NormalizedRelayUrl>()
|
||||
private val failed = ConcurrentSet<NormalizedRelayUrl>()
|
||||
|
||||
/** Relays this cycle actually got something from — worth remembering as live. */
|
||||
val reachable: Set<NormalizedRelayUrl> get() = delivered.snapshot()
|
||||
|
||||
/** Relays this cycle could not reach at all. A relay that later delivered is not in it. */
|
||||
val unreachable: Set<NormalizedRelayUrl> get() = failed.snapshot() - delivered.snapshot()
|
||||
|
||||
/**
|
||||
* Skip this relay? True when a previous run proved it dead (and no
|
||||
* [produced] since), or when its whole authority has been struck out here.
|
||||
*/
|
||||
fun isDead(url: NormalizedRelayUrl): Boolean {
|
||||
val authority = authorityOf(url.url)
|
||||
if (authority in producedHosts) return false
|
||||
return url in knownDead || authority in deadHosts
|
||||
}
|
||||
|
||||
/**
|
||||
* This relay connected but delivered nothing before giving up. Count it
|
||||
* against its authority and, at [strikeLimit], stop dialling the host.
|
||||
* Returns the eviction — for the caller to publish — exactly when this
|
||||
* strike is the one that took the host down: that is the only point where
|
||||
* the evidence exists, because every sibling url is skipped without being
|
||||
* dialled from here on.
|
||||
*/
|
||||
fun strike(url: NormalizedRelayUrl): Evicted? {
|
||||
failed.add(url)
|
||||
if (strikeLimit <= 0) return null
|
||||
val authority = authorityOf(url.url)
|
||||
if (authority in producedHosts || authority in deadHosts) return null
|
||||
if (strikes.merge(authority, 1) { old, new -> old + new } < strikeLimit) return null
|
||||
// Concurrent strikers can cross the threshold together; add() is the
|
||||
// atomic exactly-once gate on who publishes. Re-check produced after
|
||||
// winning it: a delivery that landed while this strike was in flight
|
||||
// outranks the verdict, and a verdict for a host that just answered
|
||||
// would be a false public record.
|
||||
if (!deadHosts.add(authority)) return null
|
||||
if (authority in producedHosts) return null
|
||||
return Evicted(authority, strikeLimit)
|
||||
}
|
||||
|
||||
/** An authority struck out, and the evidence for it. */
|
||||
class Evicted(
|
||||
val authority: String,
|
||||
val strikes: Int,
|
||||
)
|
||||
|
||||
/** This relay delivered. Its authority is alive, whatever else happened. */
|
||||
fun produced(url: NormalizedRelayUrl) {
|
||||
delivered.add(url)
|
||||
producedHosts.add(authorityOf(url.url))
|
||||
}
|
||||
|
||||
/** For a cycle's closing line: how many hosts were dropped. */
|
||||
fun evictedHosts(): Int = deadHosts.size()
|
||||
|
||||
fun summary(total: Int): String =
|
||||
"${reachable.size} live, ${unreachable.size} unreachable, " +
|
||||
"${deadHosts.size()} host(s) struck out, ${knownDead.size} skipped as known-dead of $total"
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* Three, because a single timeout is ordinary — a busy relay that
|
||||
* never answered one REQ is not a dead one — while three separate
|
||||
* urls on the same host all going silent is a server, not a
|
||||
* coincidence.
|
||||
*/
|
||||
const val DEFAULT_STRIKE_LIMIT = 3
|
||||
|
||||
/**
|
||||
* `host[:port]` — everything between the scheme and the first path
|
||||
* slash. The port is part of it: two ports on one machine are two
|
||||
* relays.
|
||||
*/
|
||||
fun authorityOf(url: String): String {
|
||||
val afterScheme =
|
||||
when {
|
||||
url.startsWith("wss://") -> url.substring(6)
|
||||
url.startsWith("ws://") -> url.substring(5)
|
||||
else -> url
|
||||
}
|
||||
val slash = afterScheme.indexOf('/')
|
||||
return if (slash >= 0) afterScheme.substring(0, slash) else afterScheme
|
||||
}
|
||||
}
|
||||
}
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.eventTemplate
|
||||
import com.vitorpamplona.quartz.nip40Expiration.ExpirationTag
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
|
||||
/**
|
||||
* The event a NIP-66 monitor publishes to measure a relay's WRITE path: publish
|
||||
* one of these (signed with the monitor key), time the `OK`, and read the
|
||||
* rejection prefix when refused (`auth-required:`/`restricted:`/`pow:` map to
|
||||
* the discovery record's `R` requirement tags; a timed acceptance is `rtt-write`).
|
||||
*
|
||||
* The kind is EPHEMERAL (20000–29999 per NIP-01), so a compliant relay serves it
|
||||
* to current subscribers and never stores it — the probe leaves nothing behind.
|
||||
* [KIND] 20166 is this library's convention (30166 discovery minus the
|
||||
* addressable range), not something NIP-66 standardizes; any ephemeral kind
|
||||
* works. Belt-and-braces, the template also carries a NIP-40 `expiration` tag
|
||||
* [EXPIRATION_SECONDS] out, so a relay that stores unknown ephemeral kinds
|
||||
* anyway purges it promptly.
|
||||
*
|
||||
* A rejection is still a MEASUREMENT: an `OK false` proves the write path works
|
||||
* and documents the relay's policy. Only silence is a failed write test.
|
||||
*/
|
||||
object RelayProbeWriteTest {
|
||||
const val KIND = 20166
|
||||
|
||||
/** Storage-window ceiling for non-compliant relays that store ephemeral events. */
|
||||
const val EXPIRATION_SECONDS = 60L
|
||||
|
||||
fun build(
|
||||
content: String = "NIP-66 write probe",
|
||||
createdAt: Long = TimeUtils.now(),
|
||||
): EventTemplate<Event> =
|
||||
eventTemplate(KIND, content, createdAt) {
|
||||
add(ExpirationTag.assemble(createdAt + EXPIRATION_SECONDS))
|
||||
}
|
||||
}
|
||||
+204
-23
@@ -22,6 +22,8 @@ package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.PublishResult
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.accessories.publishAndCollectResults
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient
|
||||
@@ -29,10 +31,20 @@ import com.vitorpamplona.quartz.nip01Core.relay.client.single.newSubId
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.NostrSigner
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IEventStore
|
||||
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.RelayDiscoveryEvent
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.networkType
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.requirement
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.rtt
|
||||
import com.vitorpamplona.quartz.nip66RelayMonitor.discovery.tags.RttType
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import com.vitorpamplona.quartz.utils.concurrent.ConcurrentMap
|
||||
import kotlinx.coroutines.channels.Channel
|
||||
import kotlinx.coroutines.flow.Flow
|
||||
import kotlinx.coroutines.flow.flow
|
||||
import kotlinx.coroutines.withTimeoutOrNull
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
@@ -74,6 +86,19 @@ class RelayProber(
|
||||
val error: String?,
|
||||
)
|
||||
|
||||
/**
|
||||
* One relay's read+write check outcome (see [readWriteCheck]). Latencies are
|
||||
* -1 when unobserved; [writeAccepted] is null when the relay never answered
|
||||
* the write with an OK (transport failure or silence).
|
||||
*/
|
||||
class ReadWriteVerdict(
|
||||
val relay: NormalizedRelayUrl,
|
||||
val rttReadMs: Long,
|
||||
val rttWriteMs: Long,
|
||||
val writeAccepted: Boolean?,
|
||||
val writeMessage: String?,
|
||||
)
|
||||
|
||||
class Result(
|
||||
val verdicts: List<Verdict>,
|
||||
val elapsedMs: Long,
|
||||
@@ -91,18 +116,24 @@ class RelayProber(
|
||||
/**
|
||||
* Probe every relay in [relays], [waveSize] at a time, giving each wave up to
|
||||
* [timeoutMs] to reach terminals. Returns one [Verdict] per input relay.
|
||||
*
|
||||
* [filters] is the REQ each relay is asked to answer. The default
|
||||
* [LIVENESS_FILTERS] matches nothing, so an EOSE proves liveness without
|
||||
* streaming a payload; pass [readTestFilter] to make [Verdict.rttEoseMs] a
|
||||
* real read test instead (the relay must query and stream an actual event).
|
||||
*/
|
||||
suspend fun probe(
|
||||
relays: Collection<NormalizedRelayUrl>,
|
||||
timeoutMs: Long = 15_000,
|
||||
waveSize: Int = 1000,
|
||||
filters: List<Filter> = LIVENESS_FILTERS,
|
||||
): Result {
|
||||
val mark = TimeSource.Monotonic.markNow()
|
||||
val all = ArrayList<Verdict>(relays.size)
|
||||
val distinct = relays.toSet()
|
||||
var done = 0
|
||||
for (wave in distinct.chunked(waveSize.coerceAtLeast(1))) {
|
||||
all += probeWave(wave, timeoutMs)
|
||||
probeWave(wave, timeoutMs, filters) { all += it }
|
||||
done += wave.size
|
||||
if (distinct.size > wave.size) {
|
||||
val liveSoFar = all.count { it.reachable }
|
||||
@@ -112,10 +143,94 @@ class RelayProber(
|
||||
return Result(all, mark.elapsedNow().inWholeMilliseconds)
|
||||
}
|
||||
|
||||
/**
|
||||
* Streaming variant of [probe]: a cold [Flow] that emits each relay's [Verdict]
|
||||
* the moment that relay resolves — an answering relay's verdict arrives as soon
|
||||
* as its EOSE/CLOSED/connect-failure lands, not when the whole census ends. Only
|
||||
* relays that stay silent wait for their wave's [timeoutMs] deadline.
|
||||
*
|
||||
* [filters] picks the check, as in [probe]: [LIVENESS_FILTERS] (default) or
|
||||
* [readTestFilter].
|
||||
*
|
||||
* Probing starts when the flow is collected, and emission is sequential — a slow
|
||||
* collector delays the next wave AND eats into the current wave's [timeoutMs]
|
||||
* window (the deadline is absolute; answers keep being recorded while the
|
||||
* collector runs, but silent relays get less listening time). Keep per-verdict
|
||||
* work light, or buffer, when precise deadlines matter. Pair each verdict with
|
||||
* [toDiscoveryEventTemplate] to turn the stream into signable NIP-66 kind:30166
|
||||
* records for another process to sign and publish.
|
||||
*/
|
||||
fun probeFlow(
|
||||
relays: Collection<NormalizedRelayUrl>,
|
||||
timeoutMs: Long = 15_000,
|
||||
waveSize: Int = 1000,
|
||||
filters: List<Filter> = LIVENESS_FILTERS,
|
||||
): Flow<Verdict> =
|
||||
flow {
|
||||
for (wave in relays.toSet().chunked(waveSize.coerceAtLeast(1))) {
|
||||
probeWave(wave, timeoutMs, filters) { emit(it) }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The deeper, still-honest check pair: READ (a real limit-[readLimit] REQ the
|
||||
* relay must query its store for) and WRITE (one ephemeral [RelayProbeWriteTest]
|
||||
* event signed by [signer], the monitor key, timed to its OK). Everything is a
|
||||
* direct observation — nothing is copied from the relay's NIP-11 self-claims.
|
||||
*
|
||||
* Run it against relays ALREADY PROVEN LIVE — typically [Result.reachable] of a
|
||||
* [probe] that just ran, while the pool's sockets are still open. On a warm
|
||||
* socket both numbers are honest NIP-66 rtts (`rtt-read`, `rtt-write`); against
|
||||
* a cold relay they silently include the dial, so don't.
|
||||
*
|
||||
* A write REJECTION is still a measurement: `OK false` proves the write path
|
||||
* works and documents policy ([ReadWriteVerdict.writeMessage] keeps the NIP-01
|
||||
* machine-readable reason; [toDiscoveryEventTemplate] maps `auth-required:` and
|
||||
* `pow:` to `R` tags). Only silence leaves [ReadWriteVerdict.writeAccepted] null.
|
||||
*/
|
||||
suspend fun readWriteCheck(
|
||||
relays: Collection<NormalizedRelayUrl>,
|
||||
signer: NostrSigner,
|
||||
timeoutMs: Long = 15_000,
|
||||
waveSize: Int = 1000,
|
||||
readLimit: Int = 1,
|
||||
readKinds: List<Int>? = listOf(0),
|
||||
): Map<NormalizedRelayUrl, ReadWriteVerdict> {
|
||||
val out = HashMap<NormalizedRelayUrl, ReadWriteVerdict>()
|
||||
val distinct = relays.toSet()
|
||||
for ((waveIndex, wave) in distinct.chunked(waveSize.coerceAtLeast(1)).withIndex()) {
|
||||
val reads = HashMap<NormalizedRelayUrl, Long>()
|
||||
probeWave(wave, timeoutMs, readTestFilter(readLimit, readKinds)) { reads[it.relay] = it.rttEoseMs }
|
||||
|
||||
// A distinct event id per wave (createdAt has second granularity, so the
|
||||
// content must vary) keeps a straggler OK from an earlier wave's relays
|
||||
// from ever matching this wave's confirmation window.
|
||||
val event = signer.sign(RelayProbeWriteTest.build(content = "NIP-66 write probe $waveIndex"))
|
||||
val writes = client.publishAndCollectResults(event, wave.toSet(), (timeoutMs / 1000).coerceAtLeast(1))
|
||||
|
||||
for (relay in wave) {
|
||||
// Only a real OK (true or false) counts as an answer; transport
|
||||
// failures and silence leave the write side unobserved.
|
||||
val answered = writes[relay]?.takeUnless { it.isTransportFailure || it.message == PublishResult.NO_RESPONSE }
|
||||
out[relay] =
|
||||
ReadWriteVerdict(
|
||||
relay = relay,
|
||||
rttReadMs = reads[relay] ?: -1,
|
||||
rttWriteMs = answered?.elapsedMs ?: -1,
|
||||
writeAccepted = answered?.accepted,
|
||||
writeMessage = answered?.message,
|
||||
)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
private suspend fun probeWave(
|
||||
wave: List<NormalizedRelayUrl>,
|
||||
timeoutMs: Long,
|
||||
): List<Verdict> {
|
||||
filters: List<Filter>,
|
||||
onVerdict: suspend (Verdict) -> Unit,
|
||||
) {
|
||||
val mark = TimeSource.Monotonic.markNow()
|
||||
val waveSet = wave.toHashSet()
|
||||
val openRtt = ConcurrentMap<NormalizedRelayUrl, Long>()
|
||||
@@ -178,22 +293,7 @@ class RelayProber(
|
||||
}
|
||||
}
|
||||
|
||||
client.addConnectionListener(connListener)
|
||||
try {
|
||||
client.subscribe(subId, wave.associateWith { PROBE_FILTERS }, subListener)
|
||||
val remaining = wave.toMutableSet()
|
||||
withTimeoutOrNull(timeoutMs) {
|
||||
while (remaining.isNotEmpty()) {
|
||||
remaining.remove(terminals.receive())
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
client.unsubscribe(subId)
|
||||
client.removeConnectionListener(connListener)
|
||||
terminals.close()
|
||||
}
|
||||
|
||||
return wave.map { relay ->
|
||||
fun verdictOf(relay: NormalizedRelayUrl): Verdict {
|
||||
val opened = openRtt[relay]
|
||||
val answered = eoseMs[relay]
|
||||
val error = errors[relay]
|
||||
@@ -202,7 +302,7 @@ class RelayProber(
|
||||
// Only a connect failure, or silence with no socket, is dead.
|
||||
val cannot = error?.startsWith("cannot:") == true
|
||||
val reachable = !cannot && (opened != null || answered != null || error != null)
|
||||
Verdict(
|
||||
return Verdict(
|
||||
relay = relay,
|
||||
reachable = reachable,
|
||||
rttOpenMs = opened ?: -1,
|
||||
@@ -210,13 +310,54 @@ class RelayProber(
|
||||
error = error,
|
||||
)
|
||||
}
|
||||
|
||||
client.addConnectionListener(connListener)
|
||||
try {
|
||||
client.subscribe(subId, wave.associateWith { filters }, subListener)
|
||||
val remaining = wave.toMutableSet()
|
||||
while (remaining.isNotEmpty()) {
|
||||
val left = timeoutMs - mark.elapsedNow().inWholeMilliseconds
|
||||
if (left <= 0) break
|
||||
val relay = withTimeoutOrNull(left) { terminals.receive() } ?: break
|
||||
// The emission happens OUTSIDE the timeout window so a collector that
|
||||
// suspends on a verdict can never be cancelled mid-emission and lose it.
|
||||
if (remaining.remove(relay)) onVerdict(verdictOf(relay))
|
||||
}
|
||||
// Whatever is left resolved nothing by the deadline: dead if the socket
|
||||
// never opened, reachable-but-slow if it did.
|
||||
for (relay in remaining) onVerdict(verdictOf(relay))
|
||||
} finally {
|
||||
client.unsubscribe(subId)
|
||||
client.removeConnectionListener(connListener)
|
||||
terminals.close()
|
||||
}
|
||||
}
|
||||
|
||||
companion object {
|
||||
// A filter no event can match (ids are 64-hex of a hash): the relay answers
|
||||
// with an immediate EOSE and never streams a payload. Same trick as the
|
||||
// crawler's warm pool.
|
||||
private val PROBE_FILTERS = listOf(Filter(ids = listOf("0".repeat(64))))
|
||||
/**
|
||||
* A filter no event can match (ids are 64-hex of a hash): the relay answers
|
||||
* with an immediate EOSE and never streams a payload. Same trick as the
|
||||
* crawler's warm pool. This is the default check — pure liveness.
|
||||
*/
|
||||
val LIVENESS_FILTERS = listOf(Filter(ids = listOf("0".repeat(64))))
|
||||
|
||||
/**
|
||||
* A REQ the relay must actually WORK for: query its store and stream up to
|
||||
* [limit] real events before the EOSE. Pass to [probe]/[probeFlow] as
|
||||
* [filters] to turn [Verdict.rttEoseMs] into a genuine read test rather
|
||||
* than a liveness ping — the time still counts from the wave start (dial
|
||||
* included), so compare it against [Verdict.rttOpenMs], not across waves.
|
||||
*
|
||||
* [kinds] defaults to kind 0: purpose relays (purplepag.es) reject any REQ
|
||||
* that names no kind with `blocked: filters must specify at least one kind`,
|
||||
* and practically every relay stores SOME profile — so a kind-0, limit-1
|
||||
* query works everywhere. Pass a different list to probe a specific shelf,
|
||||
* or null for a kind-less query on relays known to allow one.
|
||||
*/
|
||||
fun readTestFilter(
|
||||
limit: Int = 1,
|
||||
kinds: List<Int>? = listOf(0),
|
||||
) = listOf(Filter(kinds = kinds, limit = limit))
|
||||
|
||||
/**
|
||||
* The relay universe the local store knows: every read/write relay advertised
|
||||
@@ -263,3 +404,43 @@ class RelayProber(
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This verdict as an UNSIGNED NIP-66 kind:30166 Relay Discovery template — the d-tag
|
||||
* is the normalized relay url; sign it with the consumer's own monitor key (per
|
||||
* NIP-66 a monitor is its own identity, so the prober never signs on its own).
|
||||
*
|
||||
* Only facts a probe actually observed are tagged:
|
||||
* - `n` network type inferred from the url (clearnet/tor/i2p);
|
||||
* - `rtt-open` when the relay was reachable — the measured WS-upgrade round trip,
|
||||
* or 0 for "reachable, latency not observed" (liveness is the tag's PRESENCE);
|
||||
* - `rtt-read`/`rtt-write` when a [RelayProber.readWriteCheck] result is passed
|
||||
* as [readWrite] and actually measured that side;
|
||||
* - `R auth` when the relay answered a probe with a NIP-42 `auth-required`
|
||||
* (CLOSED on the REQ, or OK-false on the write) and `R pow` when the write
|
||||
* was refused with a `pow:` reason — observed walls, not NIP-11 claims.
|
||||
*
|
||||
* NIP-11-derived tags (`N` supported NIPs, `k` kinds, `T` type) are deliberately
|
||||
* absent: those are the relay's self-claims, and asserting them under a monitor
|
||||
* signature without a per-NIP compliance test would launder claims into
|
||||
* measurements. [Verdict.rttEoseMs] is likewise never written as `rtt-read` — it
|
||||
* is wave-relative (dial + TLS + queueing), so the honest read number only comes
|
||||
* from [readWrite] (or the [RelayObserver]/[RelayMonitor] real-traffic path).
|
||||
*/
|
||||
fun RelayProber.Verdict.toDiscoveryEventTemplate(
|
||||
createdAt: Long = TimeUtils.now(),
|
||||
readWrite: RelayProber.ReadWriteVerdict? = null,
|
||||
): EventTemplate<RelayDiscoveryEvent> =
|
||||
RelayDiscoveryEvent.build(relay, createdAt = createdAt) {
|
||||
networkType(RelayReachabilityStore.networkTypeOf(relay))
|
||||
if (reachable) rtt(RttType.OPEN, rttOpenMs.coerceAtLeast(0))
|
||||
if (readWrite != null) {
|
||||
if (readWrite.rttReadMs >= 0) rtt(RttType.READ, readWrite.rttReadMs)
|
||||
if (readWrite.rttWriteMs >= 0) rtt(RttType.WRITE, readWrite.rttWriteMs)
|
||||
}
|
||||
val authWalled =
|
||||
error?.startsWith("closed:auth-required") == true ||
|
||||
readWrite?.writeMessage?.startsWith("auth-required") == true
|
||||
if (authWalled) requirement("auth")
|
||||
if (readWrite?.writeMessage?.startsWith("pow:") == true) requirement("pow")
|
||||
}
|
||||
|
||||
+12
-11
@@ -22,8 +22,7 @@ package com.vitorpamplona.quartz.nip77Negentropy
|
||||
|
||||
import com.vitorpamplona.negentropy.storage.IStorage
|
||||
import com.vitorpamplona.quartz.nip01Core.store.IdAndTime
|
||||
import kotlin.concurrent.atomics.AtomicBoolean
|
||||
import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
import com.vitorpamplona.quartz.utils.concurrent.PlatformLock
|
||||
|
||||
/**
|
||||
* Always-current `(created_at, id)` index for NIP-77 negentropy — the
|
||||
@@ -54,13 +53,12 @@ import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
* sealed storage, so one snapshot backs any number of concurrent
|
||||
* sessions and stays valid even if the live index mutates after.
|
||||
*
|
||||
* Thread-safety: all operations take a short spin lock (same pattern as
|
||||
* `LiveEventStore`'s replay dedup). Mutations arrive from the store's
|
||||
* single writer; snapshots from any REQ coroutine.
|
||||
* Thread-safety: all operations take a short parking lock ([PlatformLock]).
|
||||
* Mutations arrive from the store's single writer; snapshots from any REQ
|
||||
* coroutine.
|
||||
*/
|
||||
@OptIn(ExperimentalAtomicApi::class)
|
||||
class LiveNegentropyIndex {
|
||||
private val lock = AtomicBoolean(false)
|
||||
private val lock = PlatformLock()
|
||||
|
||||
/** Sorted by (createdAt, id). Only touched under [locked]. */
|
||||
private var entries = ArrayList<IdAndTime>()
|
||||
@@ -74,14 +72,17 @@ class LiveNegentropyIndex {
|
||||
private var cachedSnapshot: IStorage? = null
|
||||
private var cachedGeneration = -1L
|
||||
|
||||
// Parks rather than busy-waits. This lock's critical sections are FAR longer than a
|
||||
// handful of map ops — [rebuild] sorts the whole entry list and snapshotting builds a
|
||||
// storage — so a spinning waiter would burn a core for the duration of a sort while
|
||||
// concurrent ingest threads pile up. Same defect that produced the client-side ANR
|
||||
// documented on PlatformLock; see quartz/.../prodbench/SpinLockConvoyBenchmark.kt.
|
||||
private inline fun <R> locked(block: () -> R): R {
|
||||
while (lock.exchange(true)) {
|
||||
while (lock.load()) { }
|
||||
}
|
||||
lock.lock()
|
||||
try {
|
||||
return block()
|
||||
} finally {
|
||||
lock.store(false)
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils
|
||||
|
||||
/**
|
||||
* Pure-Kotlin IPv4 literal parsing and range classification, the companion to [Ipv6].
|
||||
*
|
||||
* Exists because asking "is this host private?" with `url.contains("192.168.")` is wrong in
|
||||
* both directions: it misses `10.0.0.5` and `172.16.3.4` (so a LAN relay gets `wss://` and is
|
||||
* dialed through Tor) and it matches `192.168.evil.com`, a perfectly registrable domain (so a
|
||||
* hostile relay url can exempt itself from Tor). Parsing the host and testing the range is the
|
||||
* only form of the question that has a right answer.
|
||||
*/
|
||||
object Ipv4 {
|
||||
/** Parses a dotted quad in `[from, to)`, or null when the region is not one. */
|
||||
fun parse(
|
||||
text: String,
|
||||
from: Int,
|
||||
to: Int,
|
||||
): ByteArray? {
|
||||
// Cheapest possible rejection of a DNS host: a literal always starts with a digit.
|
||||
if (from >= to || text[from] !in '0'..'9') return null
|
||||
val out = ByteArray(4)
|
||||
return if (parseInto(text, from, to, out, 0)) out else null
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses `a.b.c.d` in `[from, to)` into four bytes at [at]. Leading zeros are rejected —
|
||||
* they invite the octal reading that makes `010.1.1.1` ambiguous across resolvers.
|
||||
*/
|
||||
fun parseInto(
|
||||
text: String,
|
||||
from: Int,
|
||||
to: Int,
|
||||
out: ByteArray,
|
||||
at: Int,
|
||||
): Boolean {
|
||||
var i = from
|
||||
for (octet in 0 until 4) {
|
||||
if (octet > 0) {
|
||||
if (i >= to || text[i] != '.') return false
|
||||
i++
|
||||
}
|
||||
var value = 0
|
||||
var digits = 0
|
||||
while (i < to && text[i] in '0'..'9') {
|
||||
if (digits == 3) return false
|
||||
if (digits == 1 && value == 0) return false // leading zero
|
||||
value = value * 10 + (text[i] - '0')
|
||||
digits++
|
||||
i++
|
||||
}
|
||||
if (digits == 0 || value > 255) return false
|
||||
out[at + octet] = value.toByte()
|
||||
}
|
||||
return i == to
|
||||
}
|
||||
|
||||
/** `127.0.0.0/8` — the whole loopback block, not just 127.0.0.1. */
|
||||
fun isLoopback(bytes: ByteArray): Boolean = octet(bytes, 0) == 127
|
||||
|
||||
/** RFC 1918: `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`. */
|
||||
fun isPrivate(bytes: ByteArray): Boolean {
|
||||
val first = octet(bytes, 0)
|
||||
val second = octet(bytes, 1)
|
||||
return first == 10 ||
|
||||
(first == 172 && second in 16..31) ||
|
||||
(first == 192 && second == 168)
|
||||
}
|
||||
|
||||
/** `169.254.0.0/16` — link-local / APIPA, reachable only on the local segment. */
|
||||
fun isLinkLocal(bytes: ByteArray): Boolean = octet(bytes, 0) == 169 && octet(bytes, 1) == 254
|
||||
|
||||
/** `0.0.0.0/8` — "this network"; never a routable relay. */
|
||||
fun isUnspecified(bytes: ByteArray): Boolean = octet(bytes, 0) == 0
|
||||
|
||||
private fun octet(
|
||||
bytes: ByteArray,
|
||||
at: Int,
|
||||
) = bytes[at].toInt() and 0xFF
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils
|
||||
|
||||
/**
|
||||
* Pure-Kotlin IPv6 literal parsing, RFC 5952 canonical formatting and address
|
||||
* classification. No `java.net`, so it works on every KMP target.
|
||||
*
|
||||
* Exists because relay identity is a *string*: [com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl]
|
||||
* is the key of the connection pool, the relay-list sets, the NIP-11 cache and every
|
||||
* per-relay stat map. RFC 4291 lets one address be written many ways
|
||||
* (`[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]` and `[201:d0e:9ba5:8bbc::1]` are the same
|
||||
* host), and without folding them the app treats one relay as two — two sockets, two REQ
|
||||
* sets, two rows in the UI. The canonical form here matches what OkHttp renders, so the
|
||||
* key the app stores is the host it actually dials.
|
||||
*/
|
||||
object Ipv6 {
|
||||
/** Longest legal literal is 45 chars (`::ffff:` + dotted quad is shorter than 8 full groups). */
|
||||
private const val MAX_LITERAL = 45
|
||||
|
||||
/**
|
||||
* Parses a bracket-less, zone-less IPv6 literal into its 16 bytes, or null when [address]
|
||||
* is not a valid literal. Accepts `::` compression and a trailing dotted quad
|
||||
* (`::ffff:192.168.1.1`).
|
||||
*/
|
||||
fun parse(address: String): ByteArray? {
|
||||
val len = address.length
|
||||
if (len < 2 || len > MAX_LITERAL) return null
|
||||
|
||||
val out = ByteArray(16)
|
||||
// Bytes written so far, counting from the left. When a `::` is present the bytes after
|
||||
// it are written contiguously here and shifted to the right end at the very end.
|
||||
var fill = 0
|
||||
var gapAt = -1
|
||||
var i = 0
|
||||
|
||||
if (address[0] == ':') {
|
||||
if (address[1] != ':') return null
|
||||
gapAt = 0
|
||||
i = 2
|
||||
if (i == len) return out
|
||||
}
|
||||
|
||||
while (true) {
|
||||
val groupStart = i
|
||||
var value = 0
|
||||
var digits = 0
|
||||
while (i < len) {
|
||||
val digit = hexDigit(address[i])
|
||||
if (digit < 0) break
|
||||
if (digits == 4) return null
|
||||
value = (value shl 4) or digit
|
||||
digits++
|
||||
i++
|
||||
}
|
||||
|
||||
if (i < len && address[i] == '.') {
|
||||
// Trailing dotted quad: occupies the last four bytes, so nothing may follow it.
|
||||
if (fill > 12) return null
|
||||
if (!Ipv4.parseInto(address, groupStart, len, out, fill)) return null
|
||||
fill += 4
|
||||
i = len
|
||||
break
|
||||
}
|
||||
|
||||
if (digits == 0) return null
|
||||
if (fill + 2 > 16) return null
|
||||
out[fill++] = (value ushr 8).toByte()
|
||||
out[fill++] = value.toByte()
|
||||
|
||||
if (i == len) break
|
||||
if (address[i] != ':') return null
|
||||
i++
|
||||
if (i == len) return null // a single trailing ':' is not a valid literal
|
||||
if (address[i] == ':') {
|
||||
if (gapAt >= 0) return null // only one `::` allowed
|
||||
gapAt = fill
|
||||
i++
|
||||
if (i == len) break
|
||||
}
|
||||
}
|
||||
|
||||
if (gapAt < 0) {
|
||||
if (fill != 16) return null
|
||||
} else {
|
||||
// `::` must stand for at least one omitted group.
|
||||
if (fill == 16) return null
|
||||
val tail = fill - gapAt
|
||||
for (k in tail - 1 downTo 0) {
|
||||
out[16 - tail + k] = out[gapAt + k]
|
||||
out[gapAt + k] = 0
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* RFC 5952 text form: lowercase hex, no leading zeros, and the longest run of two or more
|
||||
* zero groups replaced by `::` (leftmost run wins a tie). IPv4-mapped addresses keep their
|
||||
* dotted tail. This is byte-for-byte what OkHttp prints for the same address.
|
||||
*/
|
||||
fun format(bytes: ByteArray): String {
|
||||
require(bytes.size == 16) { "An IPv6 address is 16 bytes, got ${bytes.size}" }
|
||||
|
||||
var bestStart = -1
|
||||
var bestLen = 0
|
||||
var i = 0
|
||||
while (i < 16) {
|
||||
if (bytes[i] == ZERO && bytes[i + 1] == ZERO) {
|
||||
val runStart = i
|
||||
var j = i
|
||||
while (j < 16 && bytes[j] == ZERO && bytes[j + 1] == ZERO) j += 2
|
||||
if (j - runStart > bestLen) {
|
||||
bestLen = j - runStart
|
||||
bestStart = runStart
|
||||
}
|
||||
i = j
|
||||
} else {
|
||||
i += 2
|
||||
}
|
||||
}
|
||||
// A single zero group is written as `0`, never as `::`.
|
||||
if (bestLen < 4) {
|
||||
bestStart = -1
|
||||
bestLen = 0
|
||||
}
|
||||
|
||||
val out = StringBuilder(39)
|
||||
// ::ffff:a.b.c.d — IPv4-mapped addresses read as IPv4 everywhere else, so keep them that way.
|
||||
if (bestStart == 0 && bestLen == 10 && bytes[10] == ALL_ONES && bytes[11] == ALL_ONES) {
|
||||
out.append("::ffff:")
|
||||
appendIpv4(out, bytes, 12)
|
||||
return out.toString()
|
||||
}
|
||||
|
||||
i = 0
|
||||
while (i < 16) {
|
||||
if (i == bestStart) {
|
||||
out.append(':')
|
||||
i += bestLen
|
||||
if (i == 16) out.append(':')
|
||||
} else {
|
||||
if (i > 0) out.append(':')
|
||||
out.append(group(bytes, i).toString(16))
|
||||
i += 2
|
||||
}
|
||||
}
|
||||
return out.toString()
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonicalizes a bracket-less literal, preserving any `%zone` suffix verbatim (in URLs the
|
||||
* zone arrives percent-encoded, e.g. `fe80::1%25wlan0`). Returns null when [address] is not
|
||||
* a valid literal.
|
||||
*/
|
||||
fun canonicalizeOrNull(address: String): String? {
|
||||
val zoneAt = address.indexOf('%')
|
||||
if (zoneAt < 0) return parse(address)?.let(::format)
|
||||
val bytes = parse(address.substring(0, zoneAt)) ?: return null
|
||||
return format(bytes) + address.substring(zoneAt)
|
||||
}
|
||||
|
||||
/**
|
||||
* True when [address] is a valid bracket-less literal.
|
||||
*
|
||||
* The two-colon gate is what keeps this off the normalizer's hot path: every schemeless
|
||||
* `host:port` reaching [com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer]
|
||||
* has exactly one colon, and a literal needs at least two, so the common case answers
|
||||
* without entering [parse] and allocating its 16-byte buffer.
|
||||
*/
|
||||
fun isLiteral(address: String): Boolean {
|
||||
val firstColon = address.indexOf(':')
|
||||
if (firstColon < 0 || address.indexOf(':', firstColon + 1) < 0) return false
|
||||
// A zone id is part of the literal (`fe80::1%25eth0`), not a reason to reject it.
|
||||
val zoneAt = address.indexOf('%')
|
||||
return parse(if (zoneAt < 0) address else address.substring(0, zoneAt)) != null
|
||||
}
|
||||
|
||||
/** `::1` — the IPv6 loopback, twin of 127.0.0.1. */
|
||||
fun isLoopback(bytes: ByteArray): Boolean {
|
||||
for (i in 0 until 15) if (bytes[i] != ZERO) return false
|
||||
return bytes[15] == ONE
|
||||
}
|
||||
|
||||
/** `fe80::/10` — link-local, only meaningful on the interface it came from. */
|
||||
fun isLinkLocal(bytes: ByteArray): Boolean = bytes[0] == FE.toByte() && (bytes[1].toInt() and 0xC0) == 0x80
|
||||
|
||||
/** `fc00::/7` — unique local addresses, the IPv6 twin of 192.168.0.0/16. */
|
||||
fun isUniqueLocal(bytes: ByteArray): Boolean = (bytes[0].toInt() and 0xFE) == 0xFC
|
||||
|
||||
/**
|
||||
* `0200::/7` — the range Yggdrasil derives node addresses (`0200::/8`) and subnets
|
||||
* (`0300::/8`) from. Formally deprecated NSAP space, so nothing else routes here: an
|
||||
* address in this range is reachable only through a running mesh interface, is already
|
||||
* end-to-end encrypted by the overlay, and can never hold a CA-issued certificate.
|
||||
*/
|
||||
fun isOverlayMesh(bytes: ByteArray): Boolean = (bytes[0].toInt() and 0xFE) == 0x02
|
||||
|
||||
private fun group(
|
||||
bytes: ByteArray,
|
||||
at: Int,
|
||||
) = ((bytes[at].toInt() and 0xFF) shl 8) or (bytes[at + 1].toInt() and 0xFF)
|
||||
|
||||
private fun appendIpv4(
|
||||
out: StringBuilder,
|
||||
bytes: ByteArray,
|
||||
from: Int,
|
||||
) {
|
||||
for (k in 0 until 4) {
|
||||
if (k > 0) out.append('.')
|
||||
out.append(bytes[from + k].toInt() and 0xFF)
|
||||
}
|
||||
}
|
||||
|
||||
private fun hexDigit(c: Char): Int =
|
||||
when (c) {
|
||||
in '0'..'9' -> c - '0'
|
||||
in 'a'..'f' -> c - 'a' + 10
|
||||
in 'A'..'F' -> c - 'A' + 10
|
||||
else -> -1
|
||||
}
|
||||
|
||||
private const val FE = 0xFE
|
||||
private const val ZERO = 0.toByte()
|
||||
private const val ONE = 1.toByte()
|
||||
private const val ALL_ONES = 0xFF.toByte()
|
||||
}
|
||||
+11
@@ -61,6 +61,17 @@ expect class ConcurrentMap<K : Any, V : Any>() {
|
||||
remap: (old: V, new: V) -> V,
|
||||
): V
|
||||
|
||||
/**
|
||||
* Removes [key] and returns the value it held, or null when absent.
|
||||
*
|
||||
* CAUTION: if the removed value owns a lock that callers acquire, removal
|
||||
* breaks mutual exclusion — a thread holding the old value's lock and a
|
||||
* thread that re-created the entry are no longer excluding each other. Keep
|
||||
* such locks in a structure whose identity is stable (see
|
||||
* `quartz/plans/2026-08-03-poolrequests-lock-contention.md`).
|
||||
*/
|
||||
fun remove(key: K): V?
|
||||
|
||||
fun size(): Int
|
||||
|
||||
/** A point-in-time copy of the entries — safe to iterate without holding a lock. */
|
||||
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils.concurrent
|
||||
|
||||
/**
|
||||
* A blocking mutual-exclusion lock whose waiters **park** (yield the core to the
|
||||
* scheduler) instead of busy-waiting.
|
||||
*
|
||||
* Why this exists: `commonMain` has no `java.util.concurrent.locks.Lock`, so the
|
||||
* relay client previously hand-rolled a spin lock over an `AtomicBoolean`. A
|
||||
* busy-wait is only ever correct when the holder cannot be descheduled while
|
||||
* holding the lock — and on Android that assumption is false. A production ANR on
|
||||
* a Pixel 8 (`anr_2026-08-03-12-55-26-256`, Amethyst 1.13.1) caught the exact
|
||||
* failure: the thread holding the lock was parked in `WaitingForGcToComplete`
|
||||
* while **51 of 52** runnable relay-dispatch threads sat in the inlined spin loop,
|
||||
* burning 596% CPU (6 of the phone's 9 cores) waiting for a holder that could not
|
||||
* be scheduled to release it. The UI thread, needing to allocate, then waited on
|
||||
* the same GC for over 5s and Android killed the frame with
|
||||
* "Input dispatching timed out".
|
||||
*
|
||||
* Measured on `SpinLockConvoyBenchmark` (12-core dev machine — a phone is worse):
|
||||
* the spin lock delivered 138M critical sections/s uncontended but only 1.2M/s
|
||||
* with 52 contenders (0.86%, a 116x collapse), and an *unrelated* allocating
|
||||
* thread's p90 latency went from 22µs to 10ms. Parking removes the CPU burn: a
|
||||
* waiter costs one context switch instead of a whole core.
|
||||
*
|
||||
* Splits the same way [ConcurrentMap] does:
|
||||
* - JVM / Android → `ReentrantLock` (real parking via `AbstractQueuedSynchronizer`).
|
||||
* - Apple → `NSRecursiveLock`, which also parks. The relay client genuinely runs
|
||||
* on iOS, so this must not spin — same choice commons' `KmpLock` already made.
|
||||
* - Linux → a spin, since Kotlin/Native ships no parking lock and there is no
|
||||
* Foundation; linuxX64 is a build/CI target, not a host for the many-relay
|
||||
* workload. Swap in a pthread mutex if that changes.
|
||||
*
|
||||
* (`commons` has an equivalent `KmpLock`, but `commons` depends on `quartz` and not
|
||||
* the reverse, so quartz cannot use it — keep the two in sync by hand.)
|
||||
*
|
||||
* Unlike the primitive it replaces, the JVM/Android actual is **reentrant**, so an
|
||||
* accidental re-entry degrades into a no-op rather than a self-deadlock. Callers
|
||||
* should still keep I/O and listener callbacks outside the critical section — that
|
||||
* discipline is about holding the lock briefly, not about avoiding a hang.
|
||||
*/
|
||||
expect class PlatformLock() {
|
||||
fun lock()
|
||||
|
||||
fun unlock()
|
||||
}
|
||||
|
||||
/** Runs [block] holding [this]. Inline so hot paths allocate no closure. */
|
||||
inline fun <R> PlatformLock.withLock(block: () -> R): R {
|
||||
lock()
|
||||
try {
|
||||
return block()
|
||||
} finally {
|
||||
unlock()
|
||||
}
|
||||
}
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrl
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* [RelayUrlNormalizer.isLocalHost] and [RelayUrlNormalizer.isOnion] decide whether a relay is
|
||||
* exempt from Tor, so they must read the authority and nothing else. Relay urls arrive from
|
||||
* other people — NIP-65 lists, relay hints, `r` tags — so a url whose *path* can flip those
|
||||
* answers is a url that can drop a Tor user's protection.
|
||||
*/
|
||||
class RelayUrlAuthorityAnchoringTest {
|
||||
@Test
|
||||
fun aPathCannotMakeAForeignHostLookLocal() {
|
||||
listOf(
|
||||
"wss://evil.example.com/127.0.0.1",
|
||||
"wss://evil.example.com/localhost",
|
||||
"wss://evil.example.com/192.168.1.1",
|
||||
"wss://evil.example.com/umbrel",
|
||||
"wss://evil.example.com/x.local/y",
|
||||
"wss://evil.example.com/[fd00::1]",
|
||||
"wss://evil.example.com/[::1]",
|
||||
"wss://evil.example.com/?q=127.0.0.1",
|
||||
"wss://evil.example.com/?q=[::1]",
|
||||
).forEach {
|
||||
assertFalse(RelayUrlNormalizer.isLocalHost(it), "$it must not read as localhost")
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aPathCannotMakeAForeignHostLookLikeAnOverlayOrOnion() {
|
||||
assertFalse(RelayUrlNormalizer.isOverlayNetwork("wss://evil.example.com/[201:d0e:9ba5:8bbc::1]"))
|
||||
assertFalse(RelayUrlNormalizer.isOnion("wss://evil.example.com/?u=http://nos.lol/.onion/"))
|
||||
assertFalse(RelayUrlNormalizer.isOnion("wss://evil.example.com/abc.onion"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun realLocalAndOnionHostsStillMatch() {
|
||||
listOf(
|
||||
"ws://127.0.0.1:8080/",
|
||||
"ws://localhost:4869/",
|
||||
"ws://umbrel:4848/",
|
||||
"ws://192.168.1.100:8080/",
|
||||
"ws://myrelay.local:8080/",
|
||||
"ws://myrelay.local/",
|
||||
"ws://foo.localhost:8080/",
|
||||
"ws://[::1]:4869/",
|
||||
"ws://[fd12:3456::1]:8080/",
|
||||
"ws://[fe80::1]/",
|
||||
).forEach {
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost(it), "$it must read as localhost")
|
||||
}
|
||||
// schemeless, as fix() sees it before choosing ws:// vs wss://
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("127.0.0.1:8080"))
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("umbrel:4848"))
|
||||
}
|
||||
|
||||
/**
|
||||
* `.onion:8080` never matched the old `.onion/` test, so an onion relay on an explicit port
|
||||
* was not recognized as onion — it skipped the forced-Tor branch and its hostname went to
|
||||
* the clearnet DNS resolver.
|
||||
*/
|
||||
@Test
|
||||
fun onionRelaysOnAnExplicitPortAreRecognized() {
|
||||
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion:8080/"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion/"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("abc123.onion:8080"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("abc123.onion"))
|
||||
// and it now gets ws:// like any other onion relay
|
||||
assertEquals("ws://abc123.onion:8080/", "abc123.onion:8080".normalizeRelayUrl().url)
|
||||
assertFalse(RelayUrlNormalizer.isOnion("wss://notonion.example.com/"))
|
||||
}
|
||||
|
||||
/** Canonicalization must never rewrite anything outside the authority. */
|
||||
@Test
|
||||
fun canonicalizationLeavesPathsAndQueriesAlone() {
|
||||
assertEquals(
|
||||
"wss://evil.example.com/x[0:0:0:0:0:0:0:1]y",
|
||||
"wss://evil.example.com/x[0:0:0:0:0:0:0:1]y".normalizeRelayUrl().url,
|
||||
)
|
||||
// ...while still folding a real IPv6 authority
|
||||
assertEquals(
|
||||
"wss://[::1]/x[0:0:0:0:0:0:0:1]y",
|
||||
"wss://[0:0:0:0:0:0:0:1]/x[0:0:0:0:0:0:0:1]y".normalizeRelayUrl().url,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Private IPv4 was substring-matched, which was wrong in both directions.
|
||||
*/
|
||||
@Test
|
||||
fun allPrivateIpv4RangesCountAsLocal() {
|
||||
listOf(
|
||||
"ws://127.0.0.1:8080/",
|
||||
"ws://127.1.2.3:8080/",
|
||||
"ws://10.0.0.5:4869/",
|
||||
"ws://172.16.3.4:4869/",
|
||||
"ws://172.31.255.1/",
|
||||
"ws://192.168.1.5:4869/",
|
||||
"ws://169.254.1.1:4869/",
|
||||
"ws://0.0.0.0:4869/",
|
||||
).forEach {
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost(it), "$it must read as localhost")
|
||||
}
|
||||
// a LAN relay therefore gets ws://, not a wss:// that can never hold a certificate
|
||||
assertEquals("ws://10.0.0.5:4869/", "10.0.0.5:4869".normalizeRelayUrl().url)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun publicIpv4AndPrivateLookalikeDomainsAreNotLocal() {
|
||||
listOf(
|
||||
"wss://127.0.0.1.evil.com/",
|
||||
"wss://192.168.evil.com/",
|
||||
"wss://10.0.0.5.evil.com/",
|
||||
"wss://8.8.8.8:4869/",
|
||||
"wss://172.32.0.1/",
|
||||
"wss://193.168.1.5/",
|
||||
"wss://relay.damus.io/",
|
||||
"wss://notlocalhost.example.com/",
|
||||
"wss://mylocalhost.io/",
|
||||
).forEach {
|
||||
assertFalse(RelayUrlNormalizer.isLocalHost(it), "$it must not read as localhost")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Host names are case-insensitive (RFC 4343), and `fix()` asks these questions before the
|
||||
* RFC 3986 pass folds the case — so a case-sensitive test handed `LOCALHOST:8080` and
|
||||
* `ABC.ONION:8080` a `wss://` scheme neither host can serve.
|
||||
*/
|
||||
@Test
|
||||
fun hostTestsAreCaseInsensitive() {
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("wss://LocalHost:8080/"))
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("LOCALHOST:8080"))
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("wss://MyRelay.LOCAL/"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("wss://ABC123.ONION/"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("ABC.ONION:8080"))
|
||||
assertEquals("ws://localhost:8080/", "LOCALHOST:8080".normalizeRelayUrl().url)
|
||||
assertEquals("ws://abc.onion:8080/", "ABC.ONION:8080".normalizeRelayUrl().url)
|
||||
}
|
||||
|
||||
/** RFC 1034's fully-qualified form ends in a dot; it names the same host. */
|
||||
@Test
|
||||
fun trailingDotFqdnIsTheSameHost() {
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("wss://localhost./"))
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("wss://myrelay.local./"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion./"))
|
||||
assertTrue(RelayUrlNormalizer.isOnion("wss://abc123.onion.:8080/"))
|
||||
}
|
||||
|
||||
/**
|
||||
* A `://` inside a path is not a scheme separator; only a real RFC 3986 scheme starts the
|
||||
* authority. Otherwise the path gets read as the host.
|
||||
*/
|
||||
@Test
|
||||
fun aColonSlashSlashInThePathIsNotASchemeSeparator() {
|
||||
assertFalse(RelayUrlNormalizer.isLocalHost("relay.example.com/x://127.0.0.1"))
|
||||
assertFalse(RelayUrlNormalizer.isOnion("nos.lol/?u=x://abc.onion"))
|
||||
// a real scheme still starts the authority
|
||||
assertTrue(RelayUrlNormalizer.isLocalHost("wss://127.0.0.1/x://evil.com"))
|
||||
}
|
||||
}
|
||||
+89
@@ -52,4 +52,93 @@ class RelayUrlFormatterTest {
|
||||
fun weirdRelay() {
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://relay%20list%20to%20discover%20the%20user's%20content"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun trailingPercentTwentyIsTrimmedButBareTrailingZeroIsNot() {
|
||||
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom%20")?.url)
|
||||
// urls merely ending in '%', '2' or '0' (every port ending in zero) must pass untouched
|
||||
assertEquals("wss://nostr.mom:3030/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom:3030")?.url)
|
||||
assertEquals("wss://nostr.mom:8020/", RelayUrlNormalizer.normalizeOrNull("wss://nostr.mom:8020")?.url)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun httpWithPathIsNotARelay() {
|
||||
// Mastodon/bridge actor urls from `proxy` tags: web resources, not relays
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("https://mastodon.social/users/amanita_muscaria"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("https://fosstodon.org/ap/users/115532410310000993"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("http://example.com/relay"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/?author=0"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/#section"))
|
||||
|
||||
// but bare hosts still convert, with or without port and trailing slash
|
||||
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom")?.url)
|
||||
assertEquals("wss://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom/")?.url)
|
||||
assertEquals("wss://nostr.mom:4443/", RelayUrlNormalizer.normalizeOrNull("https://nostr.mom:4443/")?.url)
|
||||
assertEquals("ws://nostr.mom/", RelayUrlNormalizer.normalizeOrNull("http://nostr.mom")?.url)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun wsWithPathIsStillARelay() {
|
||||
assertEquals("wss://relay.nostr.band/all", RelayUrlNormalizer.normalizeOrNull("wss://relay.nostr.band/all")?.url)
|
||||
assertEquals(
|
||||
"wss://bostr.lecturify.net/?accept=0,1",
|
||||
RelayUrlNormalizer.normalizeOrNull("wss://bostr.lecturify.net/?accept=0,1")?.url,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun brokenSchemeGarbage() {
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss:"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://https//nostr.watch/relay/nostr.21crypto.ch"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://https://lockbox.fiatjaf.com"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("ws://http//nos.lol"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://://plebstr.com"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun nostrUriIsNotARelay() {
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr://nrelay1qqxhwumn8ghj77tpvf6jumt9e2ckgn/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr://npub1dwy079xmpz7mk02kvz6wan49h02635umk32aa4ufek8t8mjxv58qy2nr22/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("nostr:nrelay1qq8k2cnfwejhyum99eek7cmfv9kqsm7sdm"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun addressablePointerIsNotARelay() {
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun authorityGarbage() {
|
||||
// userinfo, percent-encoding and commas never appear in a real relay host
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://catuaba@plebs.place/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://africa.nostr.joburg%0A/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://bitcoiner,social/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://#web3/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("name@domain.com"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun interiorWhitespaceAndBackslashes() {
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://nos lol"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://nos.lol/ wss:/nostr.land/ avatar wss:/nostr.wine/"))
|
||||
assertNull(RelayUrlNormalizer.normalizeOrNull("wss://\\\\relay.damus.io/"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun invisibleCharactersAreStripped() {
|
||||
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("wss://\u200Bnos.lol")?.url)
|
||||
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("\uFEFFwss://nos.lol")?.url)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun protocolRelativeUrls() {
|
||||
assertEquals("wss://relay.most.pub/", RelayUrlNormalizer.normalizeOrNull("//relay.most.pub/")?.url)
|
||||
assertEquals("wss://nos.lol/", RelayUrlNormalizer.normalizeOrNull("//nos.lol/")?.url)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ipv6AndLanHostsStillWork() {
|
||||
assertEquals("ws://[31b:6f20:c7f2:3ddf::3221]/", RelayUrlNormalizer.normalizeOrNull("ws://[31b:6f20:c7f2:3ddf::3221]/")?.url)
|
||||
assertEquals("ws://geyser-relay:7777/", RelayUrlNormalizer.normalizeOrNull("ws://geyser-relay:7777/")?.url)
|
||||
}
|
||||
}
|
||||
|
||||
+385
@@ -0,0 +1,385 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.accessories
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.utils.TimeUtils
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertSame
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* A paged relay has no memory of what it already sent, so without a band every
|
||||
* restart re-downloads its whole corpus. These pin the band arithmetic and, more
|
||||
* importantly, the cases where a band must NOT be used — a stale band silently
|
||||
* skips events, which is a worse failure than re-reading them.
|
||||
*/
|
||||
class SyncCoverageTest {
|
||||
private val relay = RelayUrlNormalizer.normalize("wss://relay.example")
|
||||
private val other = RelayUrlNormalizer.normalize("wss://other.example")
|
||||
private val profiles = Filter(kinds = listOf(0))
|
||||
|
||||
private fun now(): Long = TimeUtils.now()
|
||||
|
||||
// ---- the band arithmetic ----------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `with nothing recorded the whole filter is fetched`() {
|
||||
val c = SyncCoverage()
|
||||
assertEquals(listOf(profiles), c.legs(relay, profiles))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a recorded band is fetched around rather than through`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, observedMin = 1_700_001_000L, observedMax = 1_700_002_000L, paged = true)
|
||||
|
||||
val legs = c.legs(relay, profiles)
|
||||
assertEquals(2, legs.size, "one leg older than the band and one newer")
|
||||
assertEquals(1_700_001_000L, legs[0].until, "older leg stops AT the band floor")
|
||||
assertNull(legs[0].since, "and reaches as far back as the filter allows")
|
||||
assertEquals(1_700_002_000L, legs[1].since, "newer leg starts AT its ceiling")
|
||||
assertNull(legs[1].until)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an event sharing the band boundary second is still reachable`() {
|
||||
// A paged relay cuts pages by count, so a boundary can fall inside a run
|
||||
// of events sharing one created_at. Excluding the edge would strand the
|
||||
// rest of that second in no leg at all, while the band called it covered.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
|
||||
val legs = c.legs(relay, profiles)
|
||||
|
||||
fun reachable(t: Long) = legs.any { (it.since ?: Long.MIN_VALUE) <= t && t <= (it.until ?: Long.MAX_VALUE) }
|
||||
|
||||
assertTrue(reachable(1_700_001_000L), "the band floor second must be re-read")
|
||||
assertTrue(reachable(1_700_002_000L), "and its ceiling second")
|
||||
assertTrue(reachable(1_700_000_999L), "below the band")
|
||||
assertTrue(reachable(1_700_002_001L), "above it")
|
||||
// Only the interior is skipped, which is the entire point.
|
||||
assertTrue(!reachable(1_700_001_500L), "the covered interior is not re-read")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `successive runs widen the band rather than replacing it`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
// A later run reaches further back and picks up newer events.
|
||||
c.record(relay, profiles, 1_700_000_500L, 1_700_002_500L, paged = true)
|
||||
|
||||
val band = c.band(relay, profiles)!!
|
||||
assertEquals(1_700_000_500L, band.minCreatedAt)
|
||||
assertEquals(1_700_002_500L, band.maxCreatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a capped relay walks further back on each run`() {
|
||||
// The case that makes this worth having: a relay that only ever answers
|
||||
// with its newest N events. Each run starts below the last one's floor.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_009_000L, 1_700_010_000L, paged = true)
|
||||
assertEquals(1_700_009_000L, c.legs(relay, profiles)[0].until)
|
||||
|
||||
c.record(relay, profiles, 1_700_008_000L, 1_700_008_999L, paged = true)
|
||||
assertEquals(1_700_008_000L, c.legs(relay, profiles)[0].until)
|
||||
}
|
||||
|
||||
// ---- when a band must not be used --------------------------------------
|
||||
|
||||
@Test
|
||||
fun `a negentropy sync that reported no outcome records nothing`() {
|
||||
// Only a sync that says how far it reconciled earns a band; a bare
|
||||
// paged=false call carries no claim to record.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = false)
|
||||
assertNull(c.band(relay, profiles))
|
||||
assertEquals(listOf(profiles), c.legs(relay, profiles))
|
||||
}
|
||||
|
||||
// ---- coverage: what a finished reconcile earns -------------------------
|
||||
|
||||
@Test
|
||||
fun `a finished reconcile is in sync through the instant it started`() {
|
||||
// Not through the newest event it happened to see: "the relay had nothing
|
||||
// newer" and "we never asked" must not record the same thing.
|
||||
val c = SyncCoverage()
|
||||
val startedAt = now() - 60
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = false, reconciledThrough = startedAt)
|
||||
|
||||
val band = c.band(relay, profiles)!!
|
||||
assertTrue(band.complete)
|
||||
assertEquals(startedAt, band.maxCreatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a reconcile that downloaded nothing still records coverage`() {
|
||||
// The empty case is the WHOLE point: nothing came back because we already
|
||||
// have it, and that is exactly when the next run should ask for a sliver.
|
||||
val c = SyncCoverage()
|
||||
val startedAt = now() - 60
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = startedAt)
|
||||
|
||||
val leg = c.legs(relay, profiles).single()
|
||||
assertEquals(startedAt, leg.since)
|
||||
assertNull(leg.until)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a complete band drops its older leg while a paged one keeps it`() {
|
||||
val reconciled = SyncCoverage()
|
||||
reconciled.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_002_000L)
|
||||
val only = reconciled.legs(relay, profiles).single()
|
||||
assertEquals(1_700_002_000L, only.since)
|
||||
|
||||
val walked = SyncCoverage()
|
||||
walked.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
assertEquals(2, walked.legs(relay, profiles).size, "a paged walk says nothing about what it never asked for")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a deeper floor re-opens history below a complete band`() {
|
||||
// A reconcile only compared down to the window it ran against. When
|
||||
// the operator raises the backfill window, the span below the band's
|
||||
// recorded floor is ground nobody ever asked for.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_002_000L)
|
||||
|
||||
assertEquals(1, c.legs(relay, profiles, floor = 1_700_000_000L).size, "same floor: nothing older to ask")
|
||||
|
||||
val legs = c.legs(relay, profiles, floor = 1_600_000_000L)
|
||||
assertEquals(2, legs.size, "a deeper floor re-opens the older span")
|
||||
assertEquals(1_700_000_000L, legs[0].until, "up to the floor the reconcile actually compared")
|
||||
assertEquals(1_700_002_000L, legs[1].since)
|
||||
}
|
||||
|
||||
// ---- the periodic full re-walk -----------------------------------------
|
||||
|
||||
@Test
|
||||
fun `a band stops narrowing once it is older than the resync period`() {
|
||||
val c = SyncCoverage(fullResyncSeconds = 60)
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = now() - 3600)
|
||||
// Recorded 'now' whatever the created_at claim, so age it by rewriting.
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = now())
|
||||
assertEquals(1, c.legs(relay, profiles).size, "fresh band still narrows")
|
||||
|
||||
val stale = SyncCoverage(fullResyncSeconds = 0)
|
||||
stale.record(relay, profiles, null, null, paged = false, reconciledThrough = now())
|
||||
assertSame(profiles, stale.legs(relay, profiles).single(), "a band past its period re-walks everything")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `the re-walk replaces the old claim instead of widening it`() {
|
||||
// Widening would carry the stale band's floor forward forever and the
|
||||
// periodic pass would never actually reset anything.
|
||||
val c = SyncCoverage(fullResyncSeconds = 0)
|
||||
c.record(relay, profiles, 1_700_000_000L, 1_700_001_000L, paged = true)
|
||||
c.record(relay, profiles, 1_700_005_000L, 1_700_006_000L, paged = true)
|
||||
|
||||
val band = c.band(relay, profiles)!!
|
||||
assertEquals(1_700_005_000L, band.minCreatedAt, "the second pass walked everything; its span is the whole picture")
|
||||
}
|
||||
|
||||
// ---- the shared snapshot window ----------------------------------------
|
||||
|
||||
@Test
|
||||
fun `covering window collapses to the oldest ceiling once everyone is caught up`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
|
||||
c.record(other, profiles, null, null, paged = false, reconciledThrough = 1_700_003_000L)
|
||||
|
||||
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other), profiles).since)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `one relay that has never synced puts the window back to the whole filter`() {
|
||||
// It genuinely needs everything — narrowing the shared snapshot would
|
||||
// reconcile it against ids we never looked up.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
|
||||
|
||||
// The filter itself, unnarrowed — identity, since Filter has no equals.
|
||||
assertSame(profiles, c.coveringWindow(listOf(relay, other), profiles))
|
||||
assertSame(profiles, c.coveringWindow(emptyList(), profiles))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `one shared window serves a whole stream of relays`() {
|
||||
// Every url in a stream shares that stream's filter, so a backfill can
|
||||
// take ONE snapshot for all of them instead of walking the identical
|
||||
// range once per relay for byte-identical answers.
|
||||
val c = SyncCoverage()
|
||||
val third = RelayUrlNormalizer.normalize("wss://third.example")
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
|
||||
c.record(other, profiles, null, null, paged = false, reconciledThrough = 1_700_003_000L)
|
||||
c.record(third, profiles, null, null, paged = false, reconciledThrough = 1_700_007_000L)
|
||||
|
||||
// The hungriest of them sets the floor; the other two re-read a little.
|
||||
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other, third), profiles).since)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a fully covered relay does not widen the shared window`() {
|
||||
// A complete band past a bounded filter's ceiling needs no legs at
|
||||
// all. The best case must not force the snapshot back to the whole
|
||||
// filter — that would make full coverage cost the most.
|
||||
val window = Filter(kinds = listOf(0), since = 1_700_000_000L, until = 1_700_005_000L)
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, window, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_009_000L)
|
||||
c.record(other, window, 1_700_000_000L, null, paged = false, reconciledThrough = 1_700_003_000L)
|
||||
|
||||
assertEquals(0, c.legs(relay, window).size, "covered past the ceiling: nothing to ask")
|
||||
assertEquals(1_700_003_000L, c.coveringWindow(listOf(relay, other), window).since)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a relay with an older gap also widens the shared window`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, null, null, paged = false, reconciledThrough = 1_700_009_000L)
|
||||
c.record(other, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
|
||||
assertSame(profiles, c.coveringWindow(listOf(relay, other), profiles))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an empty fetch records nothing`() {
|
||||
// No events says nothing about what the relay holds, only that this
|
||||
// window was empty — recording it would fabricate coverage.
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, null, null, paged = true)
|
||||
assertNull(c.band(relay, profiles))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `one misdated event does not cost a relay its whole band`() {
|
||||
// A single future-dated stamp among hundreds of thousands must not fail
|
||||
// a check applied to the aggregate. Screening per event keeps the rest.
|
||||
val c = SyncCoverage()
|
||||
val far = now() + 400L * 86_400
|
||||
val observed = listOf(1_700_001_000L, far, 1_700_002_000L, 0L)
|
||||
|
||||
val plausible = observed.filter { SyncCoverage.isPlausible(it) }
|
||||
c.record(relay, profiles, plausible.min(), plausible.max(), paged = true)
|
||||
|
||||
val band = c.band(relay, profiles)!!
|
||||
assertEquals(1_700_001_000L, band.minCreatedAt)
|
||||
assertEquals(1_700_002_000L, band.maxCreatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `changing the filter starts over`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
|
||||
// Widening the kinds means the old band skipped events it never fetched.
|
||||
val wider = Filter(kinds = listOf(0, 10002))
|
||||
assertEquals(listOf(wider), c.legs(relay, wider), "a new filter has no band")
|
||||
assertNull(c.band(relay, wider))
|
||||
// ...and the original is untouched, so reverting resumes where it was.
|
||||
assertEquals(1_700_001_000L, c.band(relay, profiles)!!.minCreatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `each relay keeps its own band`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
assertEquals(listOf(profiles), c.legs(other, profiles))
|
||||
}
|
||||
|
||||
// ---- the filter's own bounds still win ---------------------------------
|
||||
|
||||
@Test
|
||||
fun `a bounded filter never widens past its own since and until`() {
|
||||
val bounded = Filter(kinds = listOf(0), since = 1_700_001_000L, until = 1_700_005_000L)
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, bounded, 1_700_002_000L, 1_700_003_000L, paged = true)
|
||||
|
||||
val legs = c.legs(relay, bounded)
|
||||
assertEquals(2, legs.size)
|
||||
assertEquals(1_700_001_000L, legs[0].since, "the older leg keeps the configured floor")
|
||||
assertEquals(1_700_002_000L, legs[0].until)
|
||||
assertEquals(1_700_003_000L, legs[1].since)
|
||||
assertEquals(1_700_005_000L, legs[1].until, "the newer leg keeps the configured ceiling")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a fully covered bounded filter re-reads only its two edge seconds`() {
|
||||
// Inclusive edges mean "covered" can never quite mean "ask for nothing":
|
||||
// the two boundary seconds are always re-read, because that is the only
|
||||
// way to catch a run of same-second events a page boundary cut in half.
|
||||
val bounded = Filter(kinds = listOf(0), since = 1_700_001_000L, until = 1_700_005_000L)
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, bounded, 1_700_001_000L, 1_700_005_000L, paged = true)
|
||||
|
||||
val legs = c.legs(relay, bounded)
|
||||
assertEquals(2, legs.size)
|
||||
assertEquals(1_700_001_000L to 1_700_001_000L, legs[0].since to legs[0].until, "the floor second only")
|
||||
assertEquals(1_700_005_000L to 1_700_005_000L, legs[1].since to legs[1].until, "the ceiling second only")
|
||||
}
|
||||
|
||||
// ---- persistence hooks --------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `export and restore round-trip the bands`() {
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
|
||||
val reopened = SyncCoverage()
|
||||
reopened.restore(c.export())
|
||||
|
||||
val band = reopened.band(relay, profiles)!!
|
||||
assertEquals(1_700_001_000L, band.minCreatedAt)
|
||||
assertEquals(1_700_002_000L, band.maxCreatedAt)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `onChange fires when a band changes so persistence can mark dirty`() {
|
||||
var changes = 0
|
||||
val c = SyncCoverage(onChange = { changes++ })
|
||||
|
||||
c.record(relay, profiles, null, null, paged = true)
|
||||
assertEquals(0, changes, "an empty fetch records nothing and must not dirty the store")
|
||||
|
||||
c.record(relay, profiles, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
assertEquals(1, changes)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `the same filter instance is fingerprinted once`() {
|
||||
// Filter.toJson() runs to tens of thousands of characters for an
|
||||
// author-scoped filter, and a fan-out keys once per relay per cycle.
|
||||
val big = Filter(kinds = listOf(30382), authors = (1..500).map { it.toString(16).padStart(64, '0') })
|
||||
val c = SyncCoverage()
|
||||
c.record(relay, big, 1_700_001_000L, 1_700_002_000L, paged = true)
|
||||
|
||||
// Same instance, many lookups: still one band, and cheap.
|
||||
repeat(50) { c.legs(relay, big) }
|
||||
assertEquals(1_700_001_000L, c.band(relay, big)!!.minCreatedAt)
|
||||
|
||||
// An equal-but-distinct instance keys the same way; it just misses the cache.
|
||||
val copy = Filter(kinds = listOf(30382), authors = (1..500).map { it.toString(16).padStart(64, '0') })
|
||||
assertEquals(1_700_001_000L, c.band(relay, copy)?.minCreatedAt, "identity caching must not change the key")
|
||||
}
|
||||
}
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.paging
|
||||
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
class PagingWindowProgressTest {
|
||||
private fun assertClose(
|
||||
expected: Double,
|
||||
actual: Double?,
|
||||
message: String? = null,
|
||||
) {
|
||||
assertTrue(actual != null && kotlin.math.abs(expected - actual) < 0.001, "${message ?: ""} expected $expected got $actual")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `progress is the paged share of the time window`() {
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("a", top = 1_000L, bottom = 0L)
|
||||
|
||||
assertClose(0.0, p.fraction(), "nothing paged yet")
|
||||
|
||||
p.mark("a", 750L)
|
||||
assertClose(0.25, p.fraction())
|
||||
|
||||
p.mark("a", 100L)
|
||||
assertClose(0.90, p.fraction())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a page that jumps backwards cannot un-advance the pagination`() {
|
||||
// Pages arrive from one relay in order, but nothing in the protocol
|
||||
// guarantees it, and a percentage that goes DOWN is worse than one that
|
||||
// is slightly wrong — it reads as the sync having lost ground.
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("a", top = 1_000L, bottom = 0L)
|
||||
|
||||
p.mark("a", 200L)
|
||||
p.mark("a", 900L)
|
||||
|
||||
assertClose(0.80, p.fraction(), "the later higher until is ignored")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `windows average rather than sum`() {
|
||||
// Two relays each paging their own window: one done and one untouched
|
||||
// is half way — not 100% as summing would give.
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("a", top = 1_000L, bottom = 0L)
|
||||
p.begin("b", top = 500L, bottom = 0L)
|
||||
|
||||
p.mark("a", 0L)
|
||||
|
||||
assertClose(0.5, p.fraction())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a finished window leaves the average`() {
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("a", top = 1_000L, bottom = 0L)
|
||||
p.begin("b", top = 1_000L, bottom = 0L)
|
||||
p.mark("b", 500L)
|
||||
|
||||
p.finish("a")
|
||||
|
||||
assertClose(0.5, p.fraction(), "only b is still paging")
|
||||
p.finish("b")
|
||||
assertNull(p.fraction(), "nothing paging means no number to report")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a group prefix scopes the numbers to its own paginations`() {
|
||||
// One instance serves many concurrent paginations; without the scope two
|
||||
// streams would print each other's percentages.
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("streamA|wss://r1", top = 1_000L, bottom = 0L)
|
||||
p.begin("streamB|wss://r2", top = 1_000L, bottom = 0L)
|
||||
p.mark("streamA|wss://r1", 0L)
|
||||
|
||||
assertClose(1.0, p.fraction("streamA"))
|
||||
assertClose(0.0, p.fraction("streamB"))
|
||||
assertClose(0.5, p.fraction())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an inverted window is not a pagination`() {
|
||||
// A leg whose since is above its until asks for a range nothing can be
|
||||
// in. Dividing by that span would produce infinities on the status line.
|
||||
val p = PagingWindowProgress()
|
||||
|
||||
p.begin("a", top = 100L, bottom = 900L)
|
||||
|
||||
assertNull(p.fraction())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a single-second window is a pagination`() {
|
||||
// Coverage legs re-read a band's edge second: since == until is a
|
||||
// real, one-second range, not an inverted one.
|
||||
val p = PagingWindowProgress()
|
||||
|
||||
p.begin("a", top = 500L, bottom = 500L)
|
||||
|
||||
assertClose(0.0, p.fraction(), "tracked from its start")
|
||||
p.mark("a", 500L)
|
||||
assertClose(0.0, p.fraction(), "still at its only second")
|
||||
p.finish("a")
|
||||
assertNull(p.fraction())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `no ETA before the estimate means anything`() {
|
||||
val p = PagingWindowProgress()
|
||||
p.begin("a", top = 1_000_000L, bottom = 0L)
|
||||
|
||||
p.mark("a", 999_000L)
|
||||
|
||||
// 0.1% in: extrapolating here yields days-long ETAs from connect
|
||||
// latency alone, which is worse than printing nothing.
|
||||
assertNull(p.etaMs(), "too early to extrapolate")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `ETA extrapolates from the rate achieved so far`() {
|
||||
var clock = 1_000_000L
|
||||
val p = PagingWindowProgress(nowMillis = { clock })
|
||||
p.begin("a", top = 1_000L, bottom = 0L)
|
||||
p.mark("a", 500L)
|
||||
|
||||
// Half way after six seconds: whatever has elapsed is also what remains.
|
||||
clock += 6_000
|
||||
assertEquals(6_000L, p.etaMs())
|
||||
}
|
||||
}
|
||||
+93
@@ -0,0 +1,93 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.client.pool
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
class FiltersChangedTest {
|
||||
private fun authors(vararg a: String) = Filter(authors = a.toList())
|
||||
|
||||
@Test
|
||||
fun emptyListsAreUnchanged() {
|
||||
assertFalse(FiltersChanged.needsToResendRequest(emptyList(), emptyList()))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun differentSizesNeedResend() {
|
||||
assertTrue(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("a"), authors("b"))))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun identicalSingleFilterDoesNotNeedResend() {
|
||||
assertFalse(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("a"))))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun changedFirstFilterNeedsResend() {
|
||||
assertTrue(FiltersChanged.needsToResendRequest(listOf(authors("a")), listOf(authors("b"))))
|
||||
}
|
||||
|
||||
/**
|
||||
* Regression: the loop used a non-local `return` on the first iteration, so only
|
||||
* filters[0] was ever compared. A subscription whose first filter was unchanged
|
||||
* reported "no resend needed" no matter what happened to the rest, and silently
|
||||
* went stale — the relay kept serving the old filter set.
|
||||
*/
|
||||
@Test
|
||||
fun changedSecondFilterNeedsResend() {
|
||||
val old = listOf(authors("a"), authors("b"))
|
||||
val new = listOf(authors("a"), authors("CHANGED"))
|
||||
assertTrue(FiltersChanged.needsToResendRequest(old, new))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun changedLastOfManyNeedsResend() {
|
||||
val old = listOf(authors("a"), authors("b"), authors("c"), authors("d"))
|
||||
val new = listOf(authors("a"), authors("b"), authors("c"), authors("CHANGED"))
|
||||
assertTrue(FiltersChanged.needsToResendRequest(old, new))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun allIdenticalOfManyDoesNotNeedResend() {
|
||||
val old = listOf(authors("a"), authors("b"), authors("c"))
|
||||
val new = listOf(authors("a"), authors("b"), authors("c"))
|
||||
assertFalse(FiltersChanged.needsToResendRequest(old, new))
|
||||
}
|
||||
|
||||
/** `since` moving forward is deliberately NOT a resend trigger, on any index. */
|
||||
@Test
|
||||
fun sinceMovingForwardOnLaterFilterDoesNotNeedResend() {
|
||||
val old = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 100))
|
||||
val new = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 200))
|
||||
assertFalse(FiltersChanged.needsToResendRequest(old, new))
|
||||
}
|
||||
|
||||
/** ...but moving backwards in time is, including on a later filter. */
|
||||
@Test
|
||||
fun sinceMovingBackwardsOnLaterFilterNeedsResend() {
|
||||
val old = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 200))
|
||||
val new = listOf(Filter(authors = listOf("a"), since = 100), Filter(authors = listOf("b"), since = 100))
|
||||
assertTrue(FiltersChanged.needsToResendRequest(old, new))
|
||||
}
|
||||
}
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip11RelayInfo
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFailsWith
|
||||
import kotlin.test.assertNull
|
||||
|
||||
class CachedNip11FetcherTest {
|
||||
private val relay = RelayUrlNormalizer.normalize("wss://nostr.example.com")
|
||||
|
||||
/** Counts network hits; serves [doc] or throws when [failing]. */
|
||||
private class FakeFetcher : Nip11Fetcher {
|
||||
var calls = 0
|
||||
var failing = false
|
||||
var doc = Nip11RelayInformation(name = "v1")
|
||||
|
||||
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation {
|
||||
calls++
|
||||
if (failing) throw Nip11FetchException("boom")
|
||||
return doc
|
||||
}
|
||||
}
|
||||
|
||||
private fun cached(
|
||||
delegate: FakeFetcher,
|
||||
clock: () -> Long,
|
||||
) = CachedNip11Fetcher(delegate, ttlSeconds = 100, errorTtlSeconds = 10, now = clock)
|
||||
|
||||
@Test
|
||||
fun freshSuccessIsServedFromCache() =
|
||||
runBlocking {
|
||||
val net = FakeFetcher()
|
||||
var now = 0L
|
||||
val fetcher = cached(net) { now }
|
||||
|
||||
assertEquals("v1", fetcher.fetch(relay).name)
|
||||
now = 99
|
||||
assertEquals("v1", fetcher.fetch(relay).name)
|
||||
assertEquals(1, net.calls, "second fetch inside the TTL must not touch the network")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun successExpiresAfterTtl() =
|
||||
runBlocking {
|
||||
val net = FakeFetcher()
|
||||
var now = 0L
|
||||
val fetcher = cached(net) { now }
|
||||
|
||||
fetcher.fetch(relay)
|
||||
net.doc = Nip11RelayInformation(name = "v2")
|
||||
now = 100
|
||||
assertEquals("v2", fetcher.fetch(relay).name)
|
||||
assertEquals(2, net.calls)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun failureIsCachedForItsOwnShorterTtl() =
|
||||
runBlocking {
|
||||
val net = FakeFetcher().apply { failing = true }
|
||||
var now = 0L
|
||||
val fetcher = cached(net) { now }
|
||||
|
||||
assertFailsWith<Nip11FetchException> { fetcher.fetch(relay) }
|
||||
now = 9
|
||||
assertFailsWith<Nip11FetchException> { fetcher.fetch(relay) }
|
||||
assertEquals(1, net.calls, "a fresh failure must be served from cache, not re-fetched")
|
||||
|
||||
now = 10
|
||||
net.failing = false
|
||||
assertEquals("v1", fetcher.fetch(relay).name)
|
||||
assertEquals(2, net.calls, "an expired failure must be retried")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun invalidateForcesAFreshFetch() =
|
||||
runBlocking {
|
||||
val net = FakeFetcher()
|
||||
val fetcher = cached(net) { 0 }
|
||||
|
||||
fetcher.fetch(relay)
|
||||
fetcher.invalidate(relay)
|
||||
fetcher.fetch(relay)
|
||||
assertEquals(2, net.calls)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun cachedOrNullNeverTouchesTheNetwork() =
|
||||
runBlocking {
|
||||
val net = FakeFetcher()
|
||||
var now = 0L
|
||||
val fetcher = cached(net) { now }
|
||||
|
||||
assertNull(fetcher.cachedOrNull(relay))
|
||||
assertEquals(0, net.calls)
|
||||
|
||||
fetcher.fetch(relay)
|
||||
assertEquals("v1", fetcher.cachedOrNull(relay)?.name)
|
||||
now = 100
|
||||
assertNull(fetcher.cachedOrNull(relay), "a stale hit must not be served")
|
||||
assertEquals(1, net.calls)
|
||||
}
|
||||
}
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip50Search
|
||||
|
||||
import com.vitorpamplona.quartz.buzz.agentProfiles.AgentProfileEvent
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
|
||||
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
|
||||
import com.vitorpamplona.quartz.nip17Dm.messages.ChatMessageEvent
|
||||
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
|
||||
import com.vitorpamplona.quartz.nip35Torrents.TorrentEvent
|
||||
import com.vitorpamplona.quartz.nip89AppHandlers.definition.AppDefinitionEvent
|
||||
import com.vitorpamplona.quartz.nipB0WebBookmarks.WebBookmarkEvent
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
|
||||
class SearchFieldExtractorTest {
|
||||
private val alice = "a1".repeat(32)
|
||||
|
||||
@Test
|
||||
fun kind0DecomposesIntoTheProfileRoles() {
|
||||
val content = """{"name":"vitor","display_name":"Vitor P","about":"builds nostr","nip05":"vitor@vitorpamplona.com","lud16":"me@wallet.com","website":"https://vitorpamplona.com","picture":"https://x/y.jpg"}"""
|
||||
val fields = SearchFieldExtractor.extract(MetadataEvent("1".repeat(64), alice, 1L, emptyArray(), content, ""))
|
||||
assertEquals(
|
||||
IndexableFields.Profile(
|
||||
name = "vitor",
|
||||
displayName = "Vitor P",
|
||||
about = "builds nostr",
|
||||
nip05 = "vitor@vitorpamplona.com",
|
||||
lud16 = "me@wallet.com",
|
||||
website = "https://vitorpamplona.com",
|
||||
),
|
||||
fields,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun longFormDecomposesIntoTitleSummaryHashtagsContent() {
|
||||
val tags = arrayOf(arrayOf("d", "post"), arrayOf("title", "My Post"), arrayOf("summary", "tl;dr"), arrayOf("t", "nostr"), arrayOf("t", "search"))
|
||||
val fields = SearchFieldExtractor.extract(LongTextNoteEvent("2".repeat(64), alice, 1L, tags, "the whole article", ""))
|
||||
assertEquals(IndexableFields.Tiered(primary = listOf("My Post"), secondary = listOf("tl;dr"), text = "the whole article", hashtags = listOf("nostr", "search")), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun notesUseTheSubjectAndHashtags() {
|
||||
val tags = arrayOf(arrayOf("subject", "meetup"), arrayOf("t", "brazil"))
|
||||
val fields = SearchFieldExtractor.extract(TextNoteEvent("3".repeat(64), alice, 1L, tags, "see you there", ""))
|
||||
assertEquals(IndexableFields.Tiered(primary = listOf("meetup"), text = "see you there", hashtags = listOf("brazil")), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun locationTagsAreCarriedRawLikeHashtags() {
|
||||
val tags = arrayOf(arrayOf("location", "Rio de Janeiro"))
|
||||
val fields = SearchFieldExtractor.extract(TextNoteEvent("4".repeat(64), alice, 1L, tags, "gm", ""))
|
||||
assertEquals(IndexableFields.Tiered(text = "gm", locations = listOf("Rio de Janeiro")), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun torrentsIndexFileNamesAndTrackers() {
|
||||
val tags =
|
||||
arrayOf(
|
||||
arrayOf("title", "Great Torrent"),
|
||||
arrayOf("file", "episode1.mkv"),
|
||||
arrayOf("file", "episode2.mkv"),
|
||||
arrayOf("tracker", "https://tracker.example.com"),
|
||||
)
|
||||
val fields = SearchFieldExtractor.extract(TorrentEvent("5".repeat(64), alice, 1L, tags, "a series", ""))
|
||||
assertEquals(
|
||||
IndexableFields.Tiered(
|
||||
primary = listOf("Great Torrent"),
|
||||
// Unjoined: the backend decides how file names are indexed.
|
||||
secondary = listOf("episode1.mkv", "episode2.mkv"),
|
||||
text = "a series",
|
||||
websites = listOf("https://tracker.example.com"),
|
||||
),
|
||||
fields,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun webBookmarksAreFindableByTheirUrl() {
|
||||
// NIP-B0: the d tag carries the URL scheme-less; url() re-adds https://.
|
||||
val tags = arrayOf(arrayOf("d", "vitorpamplona.com/post"), arrayOf("title", "A Post"))
|
||||
val fields = SearchFieldExtractor.extract(WebBookmarkEvent("6".repeat(64), alice, 1L, tags, "", ""))
|
||||
assertEquals(IndexableFields.Tiered(primary = listOf("A Post"), websites = listOf("https://vitorpamplona.com/post")), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun appHandlerMetadataReusesTheProfileRoles() {
|
||||
val content = """{"name":"CoolApp","about":"an app","website":"https://coolapp.example"}"""
|
||||
val fields = SearchFieldExtractor.extract(AppDefinitionEvent("7".repeat(64), alice, 1L, arrayOf(arrayOf("d", "x")), content, ""))
|
||||
assertEquals(IndexableFields.Profile(name = "CoolApp", about = "an app", website = "https://coolapp.example"), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun nonSearchableKindsExtractNothing() {
|
||||
// Kind 7 reactions are not SearchableEvent.
|
||||
val reaction = Event("8".repeat(64), alice, 1L, 7, emptyArray(), "+", "")
|
||||
assertEquals(IndexableFields.None, SearchFieldExtractor.extract(reaction))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun unmappedSearchableKindsFallBackToTheTextTier() {
|
||||
val fields = SearchFieldExtractor.extract(ChatMessageEvent("a".repeat(64), alice, 1L, emptyArray(), "hello group", ""))
|
||||
assertEquals(IndexableFields.Tiered(text = "hello group"), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun blankContentKindsNormalizeToNone() {
|
||||
val fields = SearchFieldExtractor.extract(TextNoteEvent("b".repeat(64), alice, 1L, emptyArray(), " ", ""))
|
||||
assertEquals(IndexableFields.None, fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun unparseableBuzzContentStillIndexesItsHashtags() {
|
||||
// The branch finds no text, but the tiers() funnel still carries the
|
||||
// event's raw tags — the one subtle reachability seam of the port.
|
||||
val tags = arrayOf(arrayOf("t", "agents"))
|
||||
val fields = SearchFieldExtractor.extract(AgentProfileEvent("c".repeat(64), alice, 1L, tags, "not json", ""))
|
||||
assertEquals(IndexableFields.Tiered(hashtags = listOf("agents")), fields)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun profileShapesNeverGetHashtagFolding() {
|
||||
// Hashtags/locations are filled only by the tiers() funnel, which
|
||||
// profile branches never use: a kind-0 with t-tags stays a pure
|
||||
// profile (and an empty one normalizes to None).
|
||||
val tags = arrayOf(arrayOf("t", "nostr"))
|
||||
val fields = SearchFieldExtractor.extract(MetadataEvent("9".repeat(64), alice, 1L, tags, "{}", ""))
|
||||
assertEquals(IndexableFields.None, fields)
|
||||
}
|
||||
}
|
||||
@@ -199,4 +199,148 @@ class SearchQueryTest {
|
||||
assertSame(plain, mixed[0])
|
||||
assertEquals("bitcoin", mixed[1].search)
|
||||
}
|
||||
|
||||
// ---- quoted phrases and -word exclusions --------------------------------
|
||||
|
||||
@Test
|
||||
fun quotedSpanBecomesPhrase() {
|
||||
val q = SearchQuery.parse("best \"nostr apps\" today")
|
||||
assertEquals("best today", q.terms)
|
||||
assertEquals(listOf("nostr apps"), q.phrases)
|
||||
assertTrue(q.notPhrases.isEmpty())
|
||||
assertTrue(q.hasText)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun negatedQuotedSpanBecomesPhraseExclusion() {
|
||||
val q = SearchQuery.parse("pizza -\"pineapple pizza\"")
|
||||
assertEquals("pizza", q.terms)
|
||||
assertEquals(listOf("pineapple pizza"), q.notPhrases)
|
||||
assertTrue(q.phrases.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun minusWordBecomesExclusion() {
|
||||
val q = SearchQuery.parse("pizza -pineapple")
|
||||
assertEquals("pizza", q.terms)
|
||||
assertEquals(listOf("pineapple"), q.notTerms)
|
||||
// Exclusions alone are not required text.
|
||||
assertFalse(SearchQuery.parse("-pineapple").hasText)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun loneMinusStaysATerm() {
|
||||
val q = SearchQuery.parse("a - b")
|
||||
assertEquals("a - b", q.terms)
|
||||
assertTrue(q.notTerms.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun allLeadingDashesAreStripped() {
|
||||
assertEquals(listOf("word"), SearchQuery.parse("--word").notTerms)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun quotesProtectExtensionShapedTokens() {
|
||||
// The quote pass runs BEFORE the extension pass, so a quoted
|
||||
// extension-shaped token is a phrase, not an extension.
|
||||
val q = SearchQuery.parse("\"include:spam\"")
|
||||
assertEquals(listOf("include:spam"), q.phrases)
|
||||
assertFalse(q.includeSpam)
|
||||
assertTrue(q.extensions.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun spanEndingInExtensionKeepsTrailingExclusion() {
|
||||
// Quote-blind extension parsing would eat the closing quote of
|
||||
// "pizza include:spam" and swallow the trailing -word; the quote-first
|
||||
// order keeps the exclusion an exclusion.
|
||||
val q = SearchQuery.parse("\"pizza include:spam\" -pineapple")
|
||||
assertEquals(listOf("pizza include:spam"), q.phrases)
|
||||
assertEquals(listOf("pineapple"), q.notTerms)
|
||||
assertTrue(q.extensions.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun minusOnExtensionShapedTokenExcludesTheLiteral() {
|
||||
// There is no `-extension` syntax: keys are strictly a-z, so the `-`
|
||||
// makes the whole token an excluded literal.
|
||||
val q = SearchQuery.parse("pizza -include:spam")
|
||||
assertEquals(listOf("include:spam"), q.notTerms)
|
||||
assertFalse(q.includeSpam)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun unclosedQuoteRunsToEnd() {
|
||||
val q = SearchQuery.parse("\"nostr apps today")
|
||||
assertEquals(listOf("nostr apps today"), q.phrases)
|
||||
assertEquals("", q.terms)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun midTokenQuoteStaysOrdinary() {
|
||||
val q = SearchQuery.parse("don\"t panic")
|
||||
assertEquals("don\"t panic", q.terms)
|
||||
assertTrue(q.phrases.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun emptySpansAreDropped() {
|
||||
val q = SearchQuery.parse("a \"\" b -\"\"")
|
||||
assertEquals("a b", q.terms)
|
||||
assertTrue(q.phrases.isEmpty())
|
||||
assertTrue(q.notPhrases.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun phrasesComposeWithExtensions() {
|
||||
val q = SearchQuery.parse("\"nostr apps\" best domain:example.com -spam")
|
||||
assertEquals("best", q.terms)
|
||||
assertEquals(listOf("nostr apps"), q.phrases)
|
||||
assertEquals(listOf("spam"), q.notTerms)
|
||||
assertEquals("example.com", q.domain)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun toSearchStringRoundTripsFullGrammar() {
|
||||
val q = SearchQuery.parse("best \"nostr apps\" -spam -\"bad phrase\" domain:example.com")
|
||||
assertEquals("best \"nostr apps\" -spam -\"bad phrase\" domain:example.com", q.toSearchString())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun allDashTokensAreDroppedNotEmptied() {
|
||||
// "--" strips to nothing; surfacing an empty exclusion would
|
||||
// round-trip into a required "-" term.
|
||||
val q = SearchQuery.parse("a --")
|
||||
assertEquals("a", q.terms)
|
||||
assertTrue(q.notTerms.isEmpty())
|
||||
assertEquals("a", SearchQuery.parse(q.toSearchString()).terms)
|
||||
assertEquals("", SearchQuery.stripExtensions("-- include:spam"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun consecutiveSpansEachLift() {
|
||||
val q = SearchQuery.parse("\"a\"\"b\" -\"c\"")
|
||||
assertEquals(listOf("a", "b"), q.phrases)
|
||||
assertEquals(listOf("c"), q.notPhrases)
|
||||
assertEquals("", q.terms)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun textAfterClosingQuoteIsItsOwnTerm() {
|
||||
// The lifted span's place stays a token boundary for what follows.
|
||||
val q = SearchQuery.parse("\"a b\"c")
|
||||
assertEquals(listOf("a b"), q.phrases)
|
||||
assertEquals("c", q.terms)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun stripExtensionsKeepsPhrasesAndExclusions() {
|
||||
assertEquals(
|
||||
"best \"nostr apps\" -spam",
|
||||
SearchQuery.stripExtensions("best \"nostr apps\" -spam language:en"),
|
||||
)
|
||||
// No extensions -> the original string comes back untouched.
|
||||
assertEquals("best \"nostr apps\" -spam", SearchQuery.stripExtensions("best \"nostr apps\" -spam"))
|
||||
}
|
||||
}
|
||||
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertNotNull
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* An outbox list is five figures of urls and mostly corpses. These pin the two
|
||||
* rules that make the difference: failures count per HOST (or a filtering
|
||||
* relay's hundreds of per-user urls never add up to anything), and a host that
|
||||
* has ever delivered is never dropped (or a fan-out this wide sheds working
|
||||
* relays on a race).
|
||||
*/
|
||||
class HostStrikesTest {
|
||||
private fun url(u: String) = RelayUrlNormalizer.normalize(u)
|
||||
|
||||
// ---- the authority key --------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `per-user path urls on one host share an authority`() {
|
||||
val a = HostStrikes.authorityOf("wss://filter.nostr.wine/npub1aaaa?broadcast=true")
|
||||
val b = HostStrikes.authorityOf("wss://filter.nostr.wine/npub1bbbb")
|
||||
assertEquals("filter.nostr.wine", a)
|
||||
assertEquals(a, b)
|
||||
assertEquals(a, HostStrikes.authorityOf("wss://filter.nostr.wine"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a subdomain is not folded into its parent`() {
|
||||
// Different servers. Shedding the filtering one must never take out the
|
||||
// bare host, which may be perfectly open.
|
||||
assertTrue(
|
||||
HostStrikes.authorityOf("wss://filter.nostr.wine/npub1x") !=
|
||||
HostStrikes.authorityOf("wss://nostr.wine"),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `the port is part of the authority`() {
|
||||
assertEquals("relay.example.com:443", HostStrikes.authorityOf("wss://relay.example.com:443/npub1z"))
|
||||
assertTrue(
|
||||
HostStrikes.authorityOf("wss://example.com:443") != HostStrikes.authorityOf("wss://example.com:8080"),
|
||||
)
|
||||
}
|
||||
|
||||
// ---- striking -----------------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `one host is struck out by failures spread across its many urls`() {
|
||||
// The whole point: no single url would ever reach the threshold alone.
|
||||
val h = HostStrikes()
|
||||
h.strike(url("wss://filter.example/npub1aaa"))
|
||||
h.strike(url("wss://filter.example/npub1bbb"))
|
||||
assertFalse(h.isDead(url("wss://filter.example/npub1ccc")), "two strikes is not yet a verdict")
|
||||
|
||||
h.strike(url("wss://filter.example/npub1ccc"))
|
||||
assertTrue(h.isDead(url("wss://filter.example/npub1ddd")), "the host is out — including urls never tried")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `eviction returns a verdict exactly once for publishing`() {
|
||||
// The eviction is the only finding that will ever exist about the sibling
|
||||
// urls under this host: from here they are skipped without being dialled,
|
||||
// so nothing observes them again. It must surface exactly once — silent
|
||||
// would publish nothing, repeated would rewrite the record every strike.
|
||||
val h = HostStrikes()
|
||||
assertNull(h.strike(url("wss://filter.example/npub1")), "one strike is not a verdict")
|
||||
assertNull(h.strike(url("wss://filter.example/npub2")), "two is not either")
|
||||
|
||||
val evicted = h.strike(url("wss://filter.example/npub3"))
|
||||
assertNotNull(evicted, "the third strike is the finding")
|
||||
assertEquals("filter.example", evicted.authority)
|
||||
assertEquals(3, evicted.strikes)
|
||||
|
||||
assertNull(h.strike(url("wss://filter.example/npub4")), "already evicted — do not report it again")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a host that has delivered is never evicted so nothing is published`() {
|
||||
val h = HostStrikes()
|
||||
h.produced(url("wss://busy.example/npubY"))
|
||||
repeat(5) { assertNull(h.strike(url("wss://busy.example/npub$it")), "ever-produced outranks any strike") }
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `striking one host leaves every other alone`() {
|
||||
val h = HostStrikes()
|
||||
repeat(5) { h.strike(url("wss://filter.example/npub$it")) }
|
||||
assertTrue(h.isDead(url("wss://filter.example/npub1")))
|
||||
assertFalse(h.isDead(url("wss://example.com")))
|
||||
assertFalse(h.isDead(url("wss://other.example")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a host that ever delivered is never dead whichever way the race lands`() {
|
||||
// At a hundred relays in flight one worker can strike an authority out at
|
||||
// the same instant another is receiving from it. Ever-produced must win in
|
||||
// both orders, which is why it overrides rather than clearing strikes.
|
||||
val strikeFirst = HostStrikes()
|
||||
repeat(3) { strikeFirst.strike(url("wss://busy.example/npub$it")) }
|
||||
assertTrue(strikeFirst.isDead(url("wss://busy.example/npubX")))
|
||||
strikeFirst.produced(url("wss://busy.example/npubY"))
|
||||
assertFalse(strikeFirst.isDead(url("wss://busy.example/npubX")), "a delivery revives the whole host")
|
||||
|
||||
val produceFirst = HostStrikes()
|
||||
produceFirst.produced(url("wss://busy.example/npubY"))
|
||||
repeat(5) { produceFirst.strike(url("wss://busy.example/npub$it")) }
|
||||
assertFalse(produceFirst.isDead(url("wss://busy.example/npubX")), "later strikes cannot bury it")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a zero strike limit disables eviction entirely`() {
|
||||
val h = HostStrikes(strikeLimit = 0)
|
||||
repeat(50) { h.strike(url("wss://filter.example/npub$it")) }
|
||||
assertFalse(h.isDead(url("wss://filter.example/npub1")))
|
||||
}
|
||||
|
||||
// ---- what a previous run already learned ---------------------------------
|
||||
|
||||
@Test
|
||||
fun `a relay a previous run proved dead is skipped without dialling`() {
|
||||
val gone = url("wss://gone.example")
|
||||
val h = HostStrikes(knownDead = setOf(gone))
|
||||
assertTrue(h.isDead(gone))
|
||||
assertFalse(h.isDead(url("wss://alive.example")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a known-dead relay that answers anyway is believed over the record`() {
|
||||
// Relays come back. A TTL'd record is "not now" and never "never again":
|
||||
// a delivery this cycle must beat what an earlier one wrote down.
|
||||
val back = url("wss://back.example")
|
||||
val h = HostStrikes(knownDead = setOf(back))
|
||||
h.produced(back)
|
||||
assertFalse(h.isDead(back))
|
||||
}
|
||||
|
||||
// ---- what gets written back ---------------------------------------------
|
||||
|
||||
@Test
|
||||
fun `only relays actually dialled are reported and delivery clears a failure`() {
|
||||
val h = HostStrikes()
|
||||
val good = url("wss://good.example")
|
||||
val bad = url("wss://bad.example")
|
||||
h.strike(bad)
|
||||
h.strike(good)
|
||||
h.produced(good)
|
||||
|
||||
assertEquals(setOf(good), h.reachable)
|
||||
assertEquals(setOf(bad), h.unreachable, "a relay that later delivered is not reported dead")
|
||||
}
|
||||
}
|
||||
+465
@@ -0,0 +1,465 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.EmptyNostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.INostrClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.listeners.RelayConnectionListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.SubscriptionListener
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.single.IRelayClient
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toClient.OkMessage
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.commands.toRelay.Command
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
|
||||
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerInternal
|
||||
import kotlinx.coroutines.ExperimentalCoroutinesApi
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.test.currentTime
|
||||
import kotlinx.coroutines.test.runTest
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Pins [RelayProber.probeFlow]'s streaming contract: an answering relay's verdict
|
||||
* is emitted the moment its terminal arrives, while silent relays only resolve at
|
||||
* the wave deadline — and pins the observed-facts-only tag set of
|
||||
* [toDiscoveryEventTemplate].
|
||||
*/
|
||||
@OptIn(ExperimentalCoroutinesApi::class)
|
||||
class RelayProberFlowTest {
|
||||
/** Captures the probe subscription and publish so the test can play the relays. */
|
||||
private class ScriptedClient : INostrClient by EmptyNostrClient() {
|
||||
var listener: SubscriptionListener? = null
|
||||
var sentFilters: Map<NormalizedRelayUrl, List<Filter>>? = null
|
||||
var published: Event? = null
|
||||
val connListeners = mutableListOf<RelayConnectionListener>()
|
||||
|
||||
override fun subscribe(
|
||||
subId: String,
|
||||
filters: Map<NormalizedRelayUrl, List<Filter>>,
|
||||
listener: SubscriptionListener?,
|
||||
) {
|
||||
this.listener = listener
|
||||
this.sentFilters = filters
|
||||
}
|
||||
|
||||
override fun publish(
|
||||
event: Event,
|
||||
relayList: Set<NormalizedRelayUrl>,
|
||||
) {
|
||||
published = event
|
||||
}
|
||||
|
||||
override fun addConnectionListener(listener: RelayConnectionListener) {
|
||||
connListeners += listener
|
||||
}
|
||||
|
||||
override fun removeConnectionListener(listener: RelayConnectionListener) {
|
||||
connListeners -= listener
|
||||
}
|
||||
|
||||
/** Plays a relay's OK answer for the published event to every armed listener. */
|
||||
fun answerOk(
|
||||
relay: NormalizedRelayUrl,
|
||||
success: Boolean,
|
||||
message: String,
|
||||
) {
|
||||
val ok = OkMessage(published!!.id, success, message)
|
||||
connListeners.toList().forEach { it.onIncomingMessage(FakeRelayClient(relay), "", ok) }
|
||||
}
|
||||
}
|
||||
|
||||
private class FakeRelayClient(
|
||||
override val url: NormalizedRelayUrl,
|
||||
) : IRelayClient {
|
||||
override fun connect() = Unit
|
||||
|
||||
override fun needsToReconnect() = false
|
||||
|
||||
override fun connectAndSyncFiltersIfDisconnected(ignoreRetryDelays: Boolean) = Unit
|
||||
|
||||
override fun isConnected() = true
|
||||
|
||||
override fun sendOrConnectAndSync(cmd: Command) = Unit
|
||||
|
||||
override fun sendIfConnected(cmd: Command) = Unit
|
||||
|
||||
override fun disconnect() = Unit
|
||||
}
|
||||
|
||||
private val fast = RelayUrlNormalizer.normalize("wss://fast.example.com")
|
||||
private val silent = RelayUrlNormalizer.normalize("wss://silent.example.com")
|
||||
private val walled = RelayUrlNormalizer.normalize("wss://walled.example.com")
|
||||
|
||||
@Test
|
||||
fun answeringRelayStreamsBeforeTheWaveDeadline() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val arrivals = mutableListOf<Pair<RelayProber.Verdict, Long>>()
|
||||
|
||||
val collector =
|
||||
launch {
|
||||
RelayProber(client)
|
||||
.probeFlow(listOf(fast, silent), timeoutMs = 10_000)
|
||||
.collect { arrivals += it to currentTime }
|
||||
}
|
||||
launch {
|
||||
delay(200)
|
||||
client.listener!!.onEose(fast, null)
|
||||
}
|
||||
collector.join()
|
||||
|
||||
assertEquals(listOf(fast, silent), arrivals.map { it.first.relay })
|
||||
// The EOSE'd relay resolved when it answered, not at the deadline …
|
||||
val (fastVerdict, fastAt) = arrivals[0]
|
||||
assertTrue(fastVerdict.reachable)
|
||||
assertTrue(fastAt < 1_000, "verdict should stream at answer time, arrived at ${fastAt}ms")
|
||||
// … while the silent one waited out the wave and is dead (socket never opened).
|
||||
val (silentVerdict, silentAt) = arrivals[1]
|
||||
assertFalse(silentVerdict.reachable)
|
||||
assertTrue(silentAt >= 10_000, "silent relay must wait for the deadline, arrived at ${silentAt}ms")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun authWalledRelayIsReachableWithTheWallRecorded() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val arrivals = mutableListOf<RelayProber.Verdict>()
|
||||
|
||||
val collector =
|
||||
launch {
|
||||
RelayProber(client)
|
||||
.probeFlow(listOf(walled), timeoutMs = 10_000)
|
||||
.collect { arrivals += it }
|
||||
}
|
||||
launch {
|
||||
delay(100)
|
||||
client.listener!!.onClosed("auth-required: sign in first", walled, null)
|
||||
}
|
||||
collector.join()
|
||||
|
||||
assertEquals(1, arrivals.size)
|
||||
assertTrue(arrivals[0].reachable, "an auth wall is an app-level answer: the relay works")
|
||||
assertEquals("closed:auth-required: sign in first", arrivals[0].error)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun connectFailureStreamsImmediatelyAsDead() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val arrivals = mutableListOf<Pair<RelayProber.Verdict, Long>>()
|
||||
|
||||
val collector =
|
||||
launch {
|
||||
RelayProber(client)
|
||||
.probeFlow(listOf(fast), timeoutMs = 10_000)
|
||||
.collect { arrivals += it to currentTime }
|
||||
}
|
||||
launch {
|
||||
delay(50)
|
||||
client.listener!!.onCannotConnect(fast, "dns failure", null)
|
||||
}
|
||||
collector.join()
|
||||
|
||||
val (verdict, at) = arrivals.single()
|
||||
assertFalse(verdict.reachable)
|
||||
assertEquals("cannot:dns failure", verdict.error)
|
||||
assertTrue(at < 1_000, "a failed dial must not wait for the deadline, arrived at ${at}ms")
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Check options — liveness default, read-test override, write-test event
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
@Test
|
||||
fun livenessFilterIsTheDefaultCheck() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val collector =
|
||||
launch {
|
||||
RelayProber(client).probeFlow(listOf(fast), timeoutMs = 1_000).collect {}
|
||||
}
|
||||
launch {
|
||||
delay(10)
|
||||
assertEquals(RelayProber.LIVENESS_FILTERS, client.sentFilters!![fast])
|
||||
client.listener!!.onEose(fast, null)
|
||||
}
|
||||
collector.join()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun readTestFilterIsSentWhenChosen() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val collector =
|
||||
launch {
|
||||
RelayProber(client)
|
||||
.probeFlow(listOf(fast), timeoutMs = 1_000, filters = RelayProber.readTestFilter())
|
||||
.collect {}
|
||||
}
|
||||
launch {
|
||||
delay(10)
|
||||
val sent = client.sentFilters!![fast]!!.single()
|
||||
assertEquals(1, sent.limit, "read test defaults to limit 1")
|
||||
assertEquals(listOf(0), sent.kinds, "read test defaults to kind 0 — accepted by purpose relays too")
|
||||
assertNull(sent.ids, "read test must query real events, not the impossible id")
|
||||
client.listener!!.onEose(fast, null)
|
||||
}
|
||||
collector.join()
|
||||
}
|
||||
|
||||
@Test
|
||||
fun writeTestEventIsEphemeralAndSelfExpiring() {
|
||||
val template = RelayProbeWriteTest.build(createdAt = 5000)
|
||||
|
||||
assertEquals(20166, template.kind)
|
||||
assertTrue(template.kind in 20000..29999, "the write probe must be an ephemeral kind")
|
||||
assertTrue(listOf("expiration", "5060") in template.tags.map { it.toList() })
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// readWriteCheck — honest read + write measurements, nothing claimed
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
private suspend fun ScriptedClient.playReadThenWrite(
|
||||
relay: NormalizedRelayUrl,
|
||||
ok: Boolean?,
|
||||
okMessage: String = "",
|
||||
) {
|
||||
delay(50)
|
||||
listener!!.onEose(relay, null) // read phase answers
|
||||
while (published == null) delay(10) // write phase begins
|
||||
if (ok != null) answerOk(relay, ok, okMessage)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun readWriteCheckMeasuresBothSides() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val signer = NostrSignerInternal(KeyPair())
|
||||
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
|
||||
|
||||
val check =
|
||||
launch {
|
||||
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 5_000)
|
||||
}
|
||||
launch { client.playReadThenWrite(fast, ok = true) }
|
||||
check.join()
|
||||
|
||||
val verdict = result!![fast]!!
|
||||
assertTrue(verdict.rttReadMs >= 0, "an answered read must be measured")
|
||||
assertTrue(verdict.rttWriteMs >= 0, "an answered write must be measured")
|
||||
assertEquals(true, verdict.writeAccepted)
|
||||
assertEquals(20166, client.published!!.kind, "the write test must use the ephemeral probe event")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun writeRejectionIsAnAnswerNotAFailure() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val signer = NostrSignerInternal(KeyPair())
|
||||
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
|
||||
|
||||
val check =
|
||||
launch {
|
||||
result = RelayProber(client).readWriteCheck(listOf(walled), signer, timeoutMs = 5_000)
|
||||
}
|
||||
launch { client.playReadThenWrite(walled, ok = false, okMessage = "pow: 28 bits needed") }
|
||||
check.join()
|
||||
|
||||
val verdict = result!![walled]!!
|
||||
assertEquals(false, verdict.writeAccepted, "OK false is a measured policy answer")
|
||||
assertEquals("pow: 28 bits needed", verdict.writeMessage)
|
||||
assertTrue(verdict.rttWriteMs >= 0, "a rejection is still a round trip")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun foreignRelayOkDoesNotEndTheWriteConfirmationEarly() =
|
||||
runTest {
|
||||
// A relay OUTSIDE the checked set answering with the same event id (a
|
||||
// straggler from an earlier wave that got the same probe event) must not
|
||||
// count toward the confirmation window — before the relayList guard in
|
||||
// publishAndCollectResults, it ended the wait early and misreported the
|
||||
// real relay as silent.
|
||||
val client = ScriptedClient()
|
||||
val signer = NostrSignerInternal(KeyPair())
|
||||
val foreign = RelayUrlNormalizer.normalize("wss://foreign.example.com")
|
||||
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
|
||||
|
||||
val check =
|
||||
launch {
|
||||
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 5_000)
|
||||
}
|
||||
launch {
|
||||
delay(50)
|
||||
client.listener!!.onEose(fast, null)
|
||||
while (client.published == null) delay(10)
|
||||
client.answerOk(foreign, true, "")
|
||||
delay(100)
|
||||
client.answerOk(fast, true, "")
|
||||
}
|
||||
check.join()
|
||||
|
||||
val verdict = result!![fast]!!
|
||||
assertEquals(true, verdict.writeAccepted, "the listed relay's OK must still be awaited and recorded")
|
||||
assertNull(result!![foreign], "the foreign relay must not appear in the result")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun silentWriteLeavesTheWriteSideUnobserved() =
|
||||
runTest {
|
||||
val client = ScriptedClient()
|
||||
val signer = NostrSignerInternal(KeyPair())
|
||||
var result: Map<NormalizedRelayUrl, RelayProber.ReadWriteVerdict>? = null
|
||||
|
||||
val check =
|
||||
launch {
|
||||
result = RelayProber(client).readWriteCheck(listOf(fast), signer, timeoutMs = 2_000)
|
||||
}
|
||||
launch { client.playReadThenWrite(fast, ok = null) }
|
||||
check.join()
|
||||
|
||||
val verdict = result!![fast]!!
|
||||
assertTrue(verdict.rttReadMs >= 0)
|
||||
assertNull(verdict.writeAccepted, "silence is not evidence about the write path")
|
||||
assertEquals(-1, verdict.rttWriteMs)
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// toDiscoveryEventTemplate — only observed facts become tags
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
private fun tagsOf(template: EventTemplate<*>) = template.tags.map { it.toList() }
|
||||
|
||||
@Test
|
||||
fun reachableVerdictTemplateCarriesLivenessAndNetwork() {
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(fast, reachable = true, rttOpenMs = 150, rttEoseMs = 480, error = null)
|
||||
.toDiscoveryEventTemplate(createdAt = 1000)
|
||||
|
||||
val tags = tagsOf(template)
|
||||
assertEquals(30166, template.kind)
|
||||
assertEquals(1000, template.createdAt)
|
||||
assertTrue(listOf("d", fast.url) in tags)
|
||||
assertTrue(listOf("n", "clearnet") in tags)
|
||||
assertTrue(listOf("rtt-open", "150") in tags)
|
||||
// rtt-eose is wave-relative (dial + queue + read) — never published as rtt-read.
|
||||
assertNull(tags.firstOrNull { it[0] == "rtt-read" })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun deadVerdictTemplateHasNoRttOpen() {
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(silent, reachable = false, rttOpenMs = -1, rttEoseMs = -1, error = "cannot:timeout")
|
||||
.toDiscoveryEventTemplate()
|
||||
|
||||
val tags = tagsOf(template)
|
||||
assertTrue(listOf("d", silent.url) in tags)
|
||||
// Liveness is the PRESENCE of rtt-open; a dead record must not carry one.
|
||||
assertNull(tags.firstOrNull { it[0] == "rtt-open" })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun reachableWithoutMeasuredLatencyWritesZeroFlag() {
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(fast, reachable = true, rttOpenMs = -1, rttEoseMs = 300, error = null)
|
||||
.toDiscoveryEventTemplate()
|
||||
|
||||
// 0 = "reachable, latency not observed": the flag form, never an invented number.
|
||||
assertTrue(listOf("rtt-open", "0") in tagsOf(template))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun observedAuthWallBecomesARequirementTag() {
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(walled, reachable = true, rttOpenMs = 90, rttEoseMs = -1, error = "closed:auth-required: sign in")
|
||||
.toDiscoveryEventTemplate()
|
||||
|
||||
assertTrue(listOf("R", "auth") in tagsOf(template))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun policyClosedIsNotAnAuthRequirement() {
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(walled, reachable = true, rttOpenMs = 90, rttEoseMs = -1, error = "closed:blocked: not welcome")
|
||||
.toDiscoveryEventTemplate()
|
||||
|
||||
assertNull(tagsOf(template).firstOrNull { it[0] == "R" })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun readWriteResultsBecomeRttTags() {
|
||||
val verdict = RelayProber.Verdict(fast, reachable = true, rttOpenMs = 100, rttEoseMs = 300, error = null)
|
||||
val readWrite = RelayProber.ReadWriteVerdict(fast, rttReadMs = 40, rttWriteMs = 55, writeAccepted = true, writeMessage = "")
|
||||
|
||||
val tags = tagsOf(verdict.toDiscoveryEventTemplate(readWrite = readWrite))
|
||||
assertTrue(listOf("rtt-read", "40") in tags)
|
||||
assertTrue(listOf("rtt-write", "55") in tags)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun unobservedReadWriteSidesStayUntagged() {
|
||||
val verdict = RelayProber.Verdict(fast, reachable = true, rttOpenMs = 100, rttEoseMs = -1, error = null)
|
||||
val readWrite = RelayProber.ReadWriteVerdict(fast, rttReadMs = -1, rttWriteMs = -1, writeAccepted = null, writeMessage = null)
|
||||
|
||||
val tags = tagsOf(verdict.toDiscoveryEventTemplate(readWrite = readWrite))
|
||||
assertNull(tags.firstOrNull { it[0] == "rtt-read" })
|
||||
assertNull(tags.firstOrNull { it[0] == "rtt-write" })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun writeRejectionReasonsBecomeRequirementTags() {
|
||||
val verdict = RelayProber.Verdict(walled, reachable = true, rttOpenMs = 100, rttEoseMs = -1, error = null)
|
||||
|
||||
val pow = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "pow: 28 bits needed")
|
||||
assertTrue(listOf("R", "pow") in tagsOf(verdict.toDiscoveryEventTemplate(readWrite = pow)))
|
||||
|
||||
val auth = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "auth-required: sign in")
|
||||
assertTrue(listOf("R", "auth") in tagsOf(verdict.toDiscoveryEventTemplate(readWrite = auth)))
|
||||
|
||||
val blocked = RelayProber.ReadWriteVerdict(walled, -1, 30, writeAccepted = false, writeMessage = "blocked: not welcome")
|
||||
assertNull(tagsOf(verdict.toDiscoveryEventTemplate(readWrite = blocked)).firstOrNull { it[0] == "R" })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun onionRelayIsTaggedTor() {
|
||||
val onion = RelayUrlNormalizer.normalize("ws://someonionaddressabcdefghijklmnop.onion")
|
||||
val template =
|
||||
RelayProber
|
||||
.Verdict(onion, reachable = true, rttOpenMs = 900, rttEoseMs = -1, error = null)
|
||||
.toDiscoveryEventTemplate()
|
||||
|
||||
assertTrue(listOf("n", "tor") in tagsOf(template))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils
|
||||
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
class Ipv6Test {
|
||||
private fun canonical(address: String) = Ipv6.canonicalizeOrNull(address)
|
||||
|
||||
@Test
|
||||
fun rfc5952CanonicalForm() {
|
||||
// leading zeros suppressed, hex lowercased
|
||||
assertEquals("201:d0e:9ba5:8bbc::1", canonical("201:0d0e:9ba5:8bbc:0000:0000:0000:0001"))
|
||||
assertEquals("201:d0e:9ba5:8bbc::1", canonical("201:D0E:9BA5:8BBC::1"))
|
||||
assertEquals("2001:db8::1", canonical("2001:0DB8:0000:0000:0000:0000:0000:0001"))
|
||||
// already canonical stays put
|
||||
assertEquals("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5", canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5"))
|
||||
assertEquals("::", canonical("::"))
|
||||
assertEquals("::1", canonical("0:0:0:0:0:0:0:1"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun singleZeroGroupIsNotCompressed() {
|
||||
// RFC 5952 §4.2.2: `::` must not stand for a single group.
|
||||
assertEquals("2001:db8:0:1:1:1:1:1", canonical("2001:db8:0:1:1:1:1:1"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun longestZeroRunWinsAndTiesGoLeft() {
|
||||
assertEquals("2001:0:0:1::1", canonical("2001:0:0:1:0:0:0:1"))
|
||||
// equal runs of two groups: the leftmost is the one compressed
|
||||
assertEquals("2001::1:1:0:0:1", canonical("2001:0:0:1:1:0:0:1"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ipv4MappedKeepsDottedTail() {
|
||||
assertEquals("::ffff:192.168.1.1", canonical("::ffff:192.168.1.1"))
|
||||
assertEquals("::ffff:127.0.0.1", canonical("::FFFF:127.0.0.1"))
|
||||
// an embedded quad that is not ipv4-mapped collapses to plain hex
|
||||
assertEquals("::c0a8:101", canonical("::192.168.1.1"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun zoneIdIsPreservedVerbatim() {
|
||||
// In URLs the zone arrives percent-encoded.
|
||||
assertEquals("fe80::1%25wlan0", canonical("fe80:0000:0000:0000:0000:0000:0000:0001%25wlan0"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun rejectsMalformedLiterals() {
|
||||
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2")) // too few groups
|
||||
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5:1234")) // too many
|
||||
assertNull(canonical("201::9ba5::1")) // two `::`
|
||||
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5:")) // trailing colon
|
||||
assertNull(canonical("201:d0e:9ba5:8bbc:f4a1:d34:1c2:gggg")) // non-hex
|
||||
assertNull(canonical("201:00d0e:9ba5:8bbc::1")) // five-digit group
|
||||
assertNull(canonical("192.168.1.1")) // ipv4
|
||||
assertNull(canonical("localhost"))
|
||||
assertNull(canonical("::ffff:192.168.1")) // short quad
|
||||
assertNull(canonical("::ffff:010.1.1.1")) // leading zero in quad
|
||||
assertNull(canonical("0:0:0:0:0:0:0:0:0"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun compressionMustCoverAtLeastOneGroup() {
|
||||
// A `::` that stands for nothing is not a legal literal.
|
||||
assertNull(canonical("1:2:3:4:5:6:7::8"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun classifiesYggdrasilAndPrivateRanges() {
|
||||
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("201:d0e:9ba5:8bbc::1")!!), "0200::/8 node address")
|
||||
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("300:1b5d:d0e9:ba58::1")!!), "0300::/8 subnet address")
|
||||
assertTrue(Ipv6.isOverlayMesh(Ipv6.parse("2ff::1")!!))
|
||||
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("2001:db8::1")!!), "documentation range is clearnet")
|
||||
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("400::1")!!), "just past 0200::/7")
|
||||
assertFalse(Ipv6.isOverlayMesh(Ipv6.parse("::1")!!))
|
||||
|
||||
assertTrue(Ipv6.isLoopback(Ipv6.parse("::1")!!))
|
||||
assertFalse(Ipv6.isLoopback(Ipv6.parse("::2")!!))
|
||||
assertFalse(Ipv6.isLoopback(Ipv6.parse("::")!!))
|
||||
|
||||
assertTrue(Ipv6.isLinkLocal(Ipv6.parse("fe80::1")!!))
|
||||
assertTrue(Ipv6.isLinkLocal(Ipv6.parse("febf::1")!!))
|
||||
assertFalse(Ipv6.isLinkLocal(Ipv6.parse("fec0::1")!!))
|
||||
|
||||
assertTrue(Ipv6.isUniqueLocal(Ipv6.parse("fd00::1")!!))
|
||||
assertTrue(Ipv6.isUniqueLocal(Ipv6.parse("fc00::1")!!))
|
||||
assertFalse(Ipv6.isUniqueLocal(Ipv6.parse("fe00::1")!!))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun isLiteralDiscriminatesAgainstNonAddresses() {
|
||||
assertTrue(Ipv6.isLiteral("201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5"))
|
||||
assertTrue(Ipv6.isLiteral("201:d0e:9ba5:8bbc::1"))
|
||||
// Things a relay-url field realistically receives, none of which may pass as an address.
|
||||
assertFalse(Ipv6.isLiteral("relay.example.com:8080"))
|
||||
assertFalse(Ipv6.isLiteral("wss:"))
|
||||
assertFalse(Ipv6.isLiteral("localhost:4869"))
|
||||
assertFalse(Ipv6.isLiteral("31990:abcdef:mydtag"), "addressable event pointer")
|
||||
assertFalse(Ipv6.isLiteral("abcd:1234"))
|
||||
assertFalse(Ipv6.isLiteral("nos.lol"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun roundTripsEveryFormOfTheSameAddress() {
|
||||
val forms =
|
||||
listOf(
|
||||
"201:d0e:9ba5:8bbc:0:0:0:1",
|
||||
"201:0d0e:9ba5:8bbc:0000:0000:0000:0001",
|
||||
"201:d0e:9ba5:8bbc::1",
|
||||
"201:D0E:9BA5:8BBC::0001",
|
||||
)
|
||||
val canonicalForms = forms.map { canonical(it) }.toSet()
|
||||
assertEquals(setOf("201:d0e:9ba5:8bbc::1"), canonicalForms)
|
||||
}
|
||||
}
|
||||
+34
@@ -71,6 +71,40 @@ class ConcurrentCollectionsTest {
|
||||
assertEquals(4, m["k"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun mapRemoveReturnsOldValueAndDeletes() {
|
||||
val map = ConcurrentMap<String, Int>()
|
||||
map["a"] = 1
|
||||
map["b"] = 2
|
||||
|
||||
assertEquals(1, map.remove("a"))
|
||||
assertNull(map["a"])
|
||||
assertEquals(1, map.size())
|
||||
assertEquals(2, map["b"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun mapRemoveAbsentKeyIsNullAndNoOp() {
|
||||
val map = ConcurrentMap<String, Int>()
|
||||
map["b"] = 2
|
||||
|
||||
assertNull(map.remove("missing"))
|
||||
assertEquals(1, map.size())
|
||||
assertEquals(2, map["b"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun mapRemoveThenGetOrPutRecreates() {
|
||||
// The lifecycle PoolRequests needs: connecting()/disconnected() drop a relay's
|
||||
// wire state, and the next REQ re-creates it. A stale value must never survive.
|
||||
val map = ConcurrentMap<String, Int>()
|
||||
map.getOrPut("relay") { 1 }
|
||||
map.remove("relay")
|
||||
|
||||
assertEquals(9, map.getOrPut("relay") { 9 })
|
||||
assertEquals(9, map["relay"])
|
||||
}
|
||||
|
||||
@Test
|
||||
fun mapSnapshotIsDetached() {
|
||||
val m = ConcurrentMap<String, Int>()
|
||||
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip11RelayInfo
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.withContext
|
||||
import okhttp3.OkHttpClient
|
||||
import okhttp3.Request
|
||||
import okhttp3.coroutines.executeAsync
|
||||
|
||||
/**
|
||||
* OkHttp-backed [Nip11Fetcher]: GETs the relay's https url with
|
||||
* `Accept: application/nostr+json` and parses the document. The client is
|
||||
* resolved per relay so callers can route Tor/proxy relays through a different
|
||||
* OkHttp instance (the same seam Amethyst's Nip11Retriever uses).
|
||||
*/
|
||||
class OkHttpNip11Fetcher(
|
||||
private val okHttpClient: (NormalizedRelayUrl) -> OkHttpClient,
|
||||
) : Nip11Fetcher {
|
||||
override suspend fun fetch(relay: NormalizedRelayUrl): Nip11RelayInformation =
|
||||
withContext(Dispatchers.IO) {
|
||||
val request =
|
||||
Request
|
||||
.Builder()
|
||||
.header("Accept", "application/nostr+json")
|
||||
.url(relay.toHttp())
|
||||
.build()
|
||||
|
||||
okHttpClient(relay).newCall(request).executeAsync().use { response ->
|
||||
if (!response.isSuccessful) {
|
||||
throw Nip11FetchException("HTTP ${response.code} fetching NIP-11 from ${relay.url}")
|
||||
}
|
||||
val body = response.body.string()
|
||||
if (!body.startsWith("{")) {
|
||||
throw Nip11FetchException("Not a NIP-11 document from ${relay.url}")
|
||||
}
|
||||
Nip11RelayInformation.fromJson(body)
|
||||
}
|
||||
}
|
||||
}
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
/**
|
||||
* Whether a failure may be published as "this relay is unreachable".
|
||||
*
|
||||
* The distinction matters because the answer is PUBLISHED: a negative NIP-66
|
||||
* record is a signed, public statement about someone else's server. A relay
|
||||
* that completes a handshake and then hangs up mid-page is emphatically
|
||||
* reachable, and an exception thrown by the caller's own code says nothing
|
||||
* about the relay at all. So this asks only about the connection itself —
|
||||
* name resolution, routing, refusal, TLS.
|
||||
*
|
||||
* Unknown failures stay quiet: the cost of silence is one retry next cycle,
|
||||
* the cost of being wrong is a false record carrying the monitor's signature.
|
||||
*/
|
||||
object Unreachability {
|
||||
fun proves(e: Exception): Boolean =
|
||||
when (e) {
|
||||
is java.net.UnknownHostException,
|
||||
is java.net.ConnectException,
|
||||
is java.net.NoRouteToHostException,
|
||||
is java.net.PortUnreachableException,
|
||||
is javax.net.ssl.SSLHandshakeException,
|
||||
-> true
|
||||
|
||||
else -> false
|
||||
}
|
||||
}
|
||||
+2
@@ -49,6 +49,8 @@ actual class ConcurrentMap<K : Any, V : Any> {
|
||||
remap: (old: V, new: V) -> V,
|
||||
): V = map.merge(key, value) { old, new -> remap(old, new) }!!
|
||||
|
||||
actual fun remove(key: K): V? = map.remove(key)
|
||||
|
||||
actual fun size(): Int = map.size
|
||||
|
||||
actual fun snapshot(): Map<K, V> = HashMap(map)
|
||||
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils.concurrent
|
||||
|
||||
import java.util.concurrent.locks.ReentrantLock
|
||||
|
||||
// ReentrantLock parks contended waiters through AbstractQueuedSynchronizer, so a
|
||||
// thread waiting on a holder that lost its core (GC, preemption) costs one context
|
||||
// switch rather than a spinning core. Non-fair on purpose: fairness would add a
|
||||
// handoff per acquisition and the critical sections here are microseconds long.
|
||||
actual class PlatformLock {
|
||||
private val lock = ReentrantLock()
|
||||
|
||||
actual fun lock() = lock.lock()
|
||||
|
||||
actual fun unlock() = lock.unlock()
|
||||
}
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.RelayUrlNormalizer
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isLocalHost
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.isOverlayNetwork
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.normalizeRelayUrl
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.toHttp
|
||||
import okhttp3.Request
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertNotNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Relay URLs on an Yggdrasil overlay, end to end through the normalizer.
|
||||
*
|
||||
* Yggdrasil gives every node an IPv6 address inside `0200::/7` (nodes in `0200::/8`, subnets in
|
||||
* `0300::/8`), with no DNS and no CA-issuable certificate. A relay on the mesh is therefore
|
||||
* always a **bracketed IPv6 literal over plain `ws://`**.
|
||||
*
|
||||
* The differential assertions against OkHttp are the point of this file living in
|
||||
* `jvmAndroidTest`: OkHttp is what actually dials the socket, so a normalized url that
|
||||
* disagrees with OkHttp's own canonical host is a relay the app tracks under a name it does
|
||||
* not connect to.
|
||||
*/
|
||||
class YggdrasilCompatCharacterizationTest {
|
||||
// Same node, several legal RFC 4291 spellings of one address.
|
||||
private val canonical = "ws://[201:d0e:9ba5:8bbc::1]:8080"
|
||||
private val expanded = "ws://[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]:8080"
|
||||
private val uppercase = "ws://[201:D0E:9BA5:8BBC::1]:8080"
|
||||
|
||||
private fun host(url: String) =
|
||||
Request
|
||||
.Builder()
|
||||
.url(url)
|
||||
.build()
|
||||
.url.host
|
||||
|
||||
@Test
|
||||
fun bracketedLiteralsSurviveNormalizationAndReachOkHttp() {
|
||||
val n = canonical.normalizeRelayUrl()
|
||||
assertEquals("ws://[201:d0e:9ba5:8bbc::1]:8080/", n.url)
|
||||
assertEquals("201:d0e:9ba5:8bbc::1", host(n.url))
|
||||
// NIP-11 / relay-icon fetches derive their http url from the same string.
|
||||
assertEquals("http://[201:d0e:9ba5:8bbc::1]:8080/", n.toHttp())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun yggdrasilSubnetAddressesWork() {
|
||||
assertEquals("ws://[300:1b5d:d0e9:ba58::1]:4848/", "ws://[300:1b5d:d0e9:ba58::1]:4848".normalizeRelayUrl().url)
|
||||
}
|
||||
|
||||
/**
|
||||
* Every legal spelling of one address collapses to one [NormalizedRelayUrl] — the key of the
|
||||
* connection pool, the relay-list sets, the NIP-11 cache and the per-relay stat maps. Without
|
||||
* this the app dials one relay twice and shows it twice.
|
||||
*/
|
||||
@Test
|
||||
fun everySpellingOfOneAddressIsOneRelay() {
|
||||
val identities = listOf(canonical, expanded, uppercase).map { it.normalizeRelayUrl() }.toSet()
|
||||
assertEquals(setOf("ws://[201:d0e:9ba5:8bbc::1]:8080/"), identities.map { it.url }.toSet())
|
||||
// ...and that one identity is the host OkHttp dials for all of them.
|
||||
assertEquals(setOf("201:d0e:9ba5:8bbc::1"), listOf(canonical, expanded, uppercase).map { host(it) }.toSet())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun normalizedIdentityAlwaysMatchesTheHostOkHttpDials() {
|
||||
listOf(
|
||||
"ws://[201:0d0e:9ba5:8bbc:0000:0000:0000:0001]:8080",
|
||||
"ws://[300:1b5d:d0e9:ba58:0:0:0:1]:4848",
|
||||
"ws://[2001:0DB8:0000:0000:0000:0000:0000:0001]:7777",
|
||||
"ws://[::1]:4869",
|
||||
).forEach { raw ->
|
||||
val normalized = raw.normalizeRelayUrl().url
|
||||
assertEquals(host(normalized), host(raw), "identity for $raw disagrees with the dialed host")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A schemeless overlay address defaults to `ws://`: no CA issues certificates for
|
||||
* `0200::/7`, so `wss://` could only ever fail its handshake. The mesh already encrypts
|
||||
* end to end, so this is not a downgrade.
|
||||
*/
|
||||
@Test
|
||||
fun schemelessOverlayAddressDefaultsToWs() {
|
||||
assertEquals("ws://[201:d0e:9ba5:8bbc::1]:8080/", "[201:d0e:9ba5:8bbc::1]:8080".normalizeRelayUrl().url)
|
||||
assertTrue("ws://[201:d0e:9ba5:8bbc::1]:8080/".normalizeRelayUrl().isOverlayNetwork())
|
||||
// A clearnet IPv6 relay keeps requiring TLS.
|
||||
assertEquals("wss://[2001:db8::1]:8080/", "[2001:db8::1]:8080".normalizeRelayUrl().url)
|
||||
assertFalse("wss://[2001:db8::1]:8080/".normalizeRelayUrl().isOverlayNetwork())
|
||||
}
|
||||
|
||||
/**
|
||||
* `::1`, `fc00::/7` and `fe80::/10` are the IPv6 twins of 127.0.0.1 and 192.168., so they
|
||||
* answer [isLocalHost] the same way — no TLS, no Tor, never advertised to the network.
|
||||
*/
|
||||
@Test
|
||||
fun ipv6LoopbackAndPrivateRangesCountAsLocalHost() {
|
||||
assertEquals("ws://[::1]:4869/", "[::1]:4869".normalizeRelayUrl().url)
|
||||
assertTrue("ws://[::1]:4869/".normalizeRelayUrl().isLocalHost())
|
||||
assertTrue("ws://[fd12:3456::1]:8080/".normalizeRelayUrl().isLocalHost(), "unique local address")
|
||||
assertTrue("ws://[fe80::1]:8080/".normalizeRelayUrl().isLocalHost(), "link local address")
|
||||
assertFalse("wss://[2001:db8::1]:8080/".normalizeRelayUrl().isLocalHost(), "clearnet ipv6")
|
||||
// An overlay relay is reachable across the mesh, so it is NOT localhost.
|
||||
assertFalse("ws://[201:d0e:9ba5:8bbc::1]:8080/".normalizeRelayUrl().isLocalHost())
|
||||
}
|
||||
|
||||
/**
|
||||
* `yggdrasilctl getSelf` prints the address unbracketed, which is what a user pastes into
|
||||
* the "add a relay" field. It is bracketed automatically rather than rejected.
|
||||
*/
|
||||
@Test
|
||||
fun bareUnbracketedLiteralIsBracketedAutomatically() {
|
||||
assertEquals(
|
||||
"ws://[201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5]/",
|
||||
"201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5".normalizeRelayUrl().url,
|
||||
)
|
||||
assertEquals("ws://[201:d0e:9ba5:8bbc::1]/", "201:d0e:9ba5:8bbc::1".normalizeRelayUrl().url)
|
||||
assertNotNull(RelayUrlNormalizer.normalizeOrNull("[201:d0e:9ba5:8bbc:f4a1:d34:1c2:eae5]:8080"))
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto-bracketing must not swallow the other colon-bearing strings that reach the
|
||||
* normalizer. Only a string that parses as a whole IPv6 address is bracketed.
|
||||
*/
|
||||
@Test
|
||||
fun autoBracketingDoesNotCaptureNonAddresses() {
|
||||
assertEquals("wss://relay.example.com:8080/", "relay.example.com:8080".normalizeRelayUrl().url)
|
||||
assertEquals("ws://localhost:4869/", "localhost:4869".normalizeRelayUrl().url)
|
||||
// addressable-event pointer, not a relay
|
||||
assertEquals(null, RelayUrlNormalizer.normalizeOrNull("31990:abcdef:mydtag"))
|
||||
// two hex-looking groups are a host and a port, not an address
|
||||
assertEquals("wss://abcd:1234/", "abcd:1234".normalizeRelayUrl().url)
|
||||
}
|
||||
}
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils
|
||||
|
||||
import okhttp3.HttpUrl.Companion.toHttpUrl
|
||||
import java.net.InetAddress
|
||||
import kotlin.random.Random
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Differential tests for [Ipv6] against the two parsers that actually matter at runtime: the
|
||||
* JDK's (what `InetAddress` will do with the host) and OkHttp's (what dials the socket).
|
||||
*
|
||||
* A hand-written address parser is exactly the kind of code that passes its own examples and
|
||||
* then disagrees with the real world on the hundredth input, so this pins it against
|
||||
* references over a deterministic random corpus rather than against more of my own examples.
|
||||
*/
|
||||
class Ipv6DifferentialTest {
|
||||
/**
|
||||
* Parses through the JDK. IPv4-mapped literals come back as an `Inet4Address` of 4 bytes,
|
||||
* so they are widened back to the 16-byte mapped form — [Ipv6] keeps them at 16 bytes,
|
||||
* which is also what OkHttp does.
|
||||
*/
|
||||
private fun jdkBytes(literal: String): ByteArray {
|
||||
val raw = InetAddress.getByName("[$literal]").address
|
||||
if (raw.size == 16) return raw
|
||||
return ByteArray(16).also {
|
||||
it[10] = 0xFF.toByte()
|
||||
it[11] = 0xFF.toByte()
|
||||
raw.copyInto(it, 12)
|
||||
}
|
||||
}
|
||||
|
||||
private fun randomAddresses(count: Int): List<ByteArray> {
|
||||
val rnd = Random(20260805)
|
||||
return List(count) {
|
||||
ByteArray(16) { rnd.nextInt(256).toByte() }.also { bytes ->
|
||||
// Sprinkle zero runs so every `::` compression path gets exercised.
|
||||
val runStart = rnd.nextInt(8) * 2
|
||||
val runLen = rnd.nextInt(1, 5) * 2
|
||||
for (k in runStart until minOf(16, runStart + runLen)) bytes[k] = 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ourTextParsesToTheSameBytesInTheJdk() {
|
||||
randomAddresses(4000).forEach { bytes ->
|
||||
val text = Ipv6.format(bytes)
|
||||
assertTrue(Ipv6.parse(text)!!.contentEquals(bytes), "our own round trip failed for $text")
|
||||
assertTrue(jdkBytes(text).contentEquals(bytes), "the JDK reads $text as a different address")
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theJdksTextParsesBackThroughUs() {
|
||||
randomAddresses(2000).forEach { bytes ->
|
||||
val jdkText = InetAddress.getByAddress(bytes).hostAddress!!
|
||||
assertTrue(Ipv6.parse(jdkText)?.contentEquals(bytes) == true, "we cannot read the JDK's own rendering: $jdkText")
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The canonical form is the app's relay identity, so it has to equal the host OkHttp shows
|
||||
* for the same address — otherwise the app keys a relay under a name it does not dial.
|
||||
*/
|
||||
@Test
|
||||
fun ourCanonicalFormMatchesOkHttp() {
|
||||
randomAddresses(2000).forEach { bytes ->
|
||||
val text = Ipv6.format(bytes)
|
||||
assertEquals("http://[$text]/".toHttpUrl().host, text)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun expandedSpellingsCollapseOntoOkHttpsHost() {
|
||||
randomAddresses(500).forEach { bytes ->
|
||||
// The fully expanded, zero-padded, uppercase spelling of the same address.
|
||||
val expanded =
|
||||
(0 until 8).joinToString(":") { g ->
|
||||
val value = ((bytes[g * 2].toInt() and 0xFF) shl 8) or (bytes[g * 2 + 1].toInt() and 0xFF)
|
||||
value.toString(16).padStart(4, '0').uppercase()
|
||||
}
|
||||
assertEquals(Ipv6.format(bytes), Ipv6.canonicalizeOrNull(expanded))
|
||||
assertEquals("http://[$expanded]/".toHttpUrl().host, Ipv6.canonicalizeOrNull(expanded))
|
||||
}
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -120,6 +120,7 @@ class ConcurrentIngestLossTest {
|
||||
|
||||
val accepted = ConcurrentHashMap.newKeySet<String>()
|
||||
val rejected = AtomicInteger()
|
||||
val failed = AtomicInteger()
|
||||
val window = Semaphore(200)
|
||||
val done = CompletableDeferred<Unit>()
|
||||
val remaining = AtomicInteger(events.size)
|
||||
@@ -130,6 +131,7 @@ class ConcurrentIngestLossTest {
|
||||
when (outcome) {
|
||||
is IEventStore.InsertOutcome.Accepted -> accepted.add(e.id)
|
||||
is IEventStore.InsertOutcome.Rejected -> rejected.incrementAndGet()
|
||||
is IEventStore.InsertOutcome.Failed -> failed.incrementAndGet()
|
||||
}
|
||||
window.release()
|
||||
if (remaining.decrementAndGet() == 0) done.complete(Unit)
|
||||
@@ -154,7 +156,7 @@ class ConcurrentIngestLossTest {
|
||||
}
|
||||
}
|
||||
val stored = store.count(Filter())
|
||||
println(" submitted=${events.size} accepted=${accepted.size} rejected=${rejected.get()} stored=$stored lostAcceptedRegular=$lost")
|
||||
println(" submitted=${events.size} accepted=${accepted.size} rejected=${rejected.get()} failed=${failed.get()} stored=$stored lostAcceptedRegular=$lost")
|
||||
|
||||
ingest.close()
|
||||
queueJob.cancel()
|
||||
|
||||
+316
@@ -0,0 +1,316 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.RequestSubscriptionState
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import kotlinx.coroutines.Dispatchers
|
||||
import kotlinx.coroutines.launch
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import kotlinx.coroutines.sync.Mutex
|
||||
import kotlinx.coroutines.sync.withLock
|
||||
import org.junit.Test
|
||||
import java.util.concurrent.ConcurrentHashMap
|
||||
import java.util.concurrent.atomic.AtomicBoolean
|
||||
import java.util.concurrent.atomic.AtomicLong
|
||||
import java.util.concurrent.locks.ReentrantLock
|
||||
|
||||
/**
|
||||
* Compares the three candidate designs for `PoolRequests`' subscription-state lock,
|
||||
* under the real production topology: R relay-consumer coroutines on a
|
||||
* limited-parallelism dispatcher (`Dispatchers.IO` = 64) all delivering EVENTs for a
|
||||
* small number of subscription ids.
|
||||
*
|
||||
* - **PER_SUB_BLOCKING** — today: one blocking lock per subId, shared by every relay
|
||||
* that sub runs on. Waiters park, which fixed the CPU burn, but a parked waiter
|
||||
* still OCCUPIES its dispatcher thread.
|
||||
* - **PER_SUB_MUTEX** — kotlinx `Mutex`: a waiter *suspends* and releases its thread.
|
||||
* Requires making the whole listener chain `suspend` (262 overrides, 110 call sites).
|
||||
* - **STRIPED** — one lock per (subId, relay). Every critical section in PoolRequests
|
||||
* is already scoped to a single relay, so this removes the contention outright and
|
||||
* needs no API change.
|
||||
*
|
||||
* The metric that matters is NOT lock throughput — it is whether unrelated work can
|
||||
* still get a thread while the lock is contended. `bystanderLatency` models that: a
|
||||
* task that never touches the lock, submitted to the same dispatcher.
|
||||
*/
|
||||
class LockDesignComparisonBenchmark {
|
||||
private val relays = 191
|
||||
private val dispatcherThreads = 64
|
||||
private val durationMs = 3000L
|
||||
|
||||
/** One relay's slice of a subscription's state — all real fields are relay-keyed. */
|
||||
private class PerRelay {
|
||||
val lock = ReentrantLock()
|
||||
var status: Int = 0
|
||||
var filters: List<String>? = null
|
||||
var lastKnown: List<String>? = null
|
||||
}
|
||||
|
||||
private class Striped {
|
||||
val perRelay = ConcurrentHashMap<Int, PerRelay>()
|
||||
|
||||
inline fun <R> withLock(
|
||||
relay: Int,
|
||||
block: (PerRelay) -> R,
|
||||
): R {
|
||||
val s = perRelay.computeIfAbsent(relay) { PerRelay() }
|
||||
s.lock.lock()
|
||||
try {
|
||||
return block(s)
|
||||
} finally {
|
||||
s.lock.unlock()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private class PerSubBlocking {
|
||||
val lock = ReentrantLock()
|
||||
val status = HashMap<Int, Int>()
|
||||
val filters = HashMap<Int, List<String>>()
|
||||
}
|
||||
|
||||
private class PerSubMutex {
|
||||
val mutex = Mutex()
|
||||
val status = HashMap<Int, Int>()
|
||||
val filters = HashMap<Int, List<String>>()
|
||||
}
|
||||
|
||||
private fun report(
|
||||
name: String,
|
||||
subs: Int,
|
||||
ops: Long,
|
||||
bystanderSamples: List<Long>,
|
||||
) {
|
||||
val sorted = bystanderSamples.sorted()
|
||||
val p50 = sorted[sorted.size / 2] / 1000.0
|
||||
val p99 = sorted[(sorted.size * 99) / 100] / 1000.0
|
||||
val max = sorted.last() / 1000.0
|
||||
println(
|
||||
"%-18s subs=%-3d ops/s=%,10d bystander p50=%8.1fus p99=%9.1fus max=%9.1fus (n=%d)".format(
|
||||
name,
|
||||
subs,
|
||||
ops * 1000 / durationMs,
|
||||
p50,
|
||||
p99,
|
||||
max,
|
||||
sorted.size,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun compareDesigns() {
|
||||
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
|
||||
println("compareDesigns skipped. Run with -PprodRelayBench=1 to enable.")
|
||||
return
|
||||
}
|
||||
println("relays=$relays dispatcherThreads=$dispatcherThreads window=${durationMs}ms")
|
||||
println("bystander = a task that NEVER touches the lock, on the same dispatcher.")
|
||||
println("Lower bystander latency = the lock is not stealing dispatcher threads.\n")
|
||||
for (subs in listOf(1, 4, 16)) {
|
||||
runPerSubBlocking(subs)
|
||||
runPerSubMutex(subs)
|
||||
runStriped(subs)
|
||||
runRealStriped(subs)
|
||||
println()
|
||||
}
|
||||
}
|
||||
|
||||
private fun runPerSubBlocking(subs: Int) =
|
||||
runBlocking {
|
||||
@Suppress("DEPRECATION")
|
||||
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
|
||||
val states = Array(subs) { PerSubBlocking() }
|
||||
val stop = AtomicBoolean(false)
|
||||
val ops = AtomicLong(0)
|
||||
val bystander = ArrayList<Long>()
|
||||
val jobs =
|
||||
(0 until relays).map { relay ->
|
||||
launch(dispatcher) {
|
||||
var n = 0L
|
||||
while (!stop.get()) {
|
||||
val s = states[relay % subs]
|
||||
s.lock.lock()
|
||||
try {
|
||||
s.status[relay] = 1
|
||||
s.filters[relay] = SAMPLE
|
||||
s.status[relay]
|
||||
} finally {
|
||||
s.lock.unlock()
|
||||
}
|
||||
n++
|
||||
// Models `for (message in incomingMessages)`: every real
|
||||
// iteration suspends, letting the dispatcher multiplex.
|
||||
kotlinx.coroutines.yield()
|
||||
}
|
||||
ops.addAndGet(n)
|
||||
}
|
||||
}
|
||||
val by =
|
||||
launch(dispatcher) {
|
||||
while (!stop.get()) {
|
||||
val t = System.nanoTime()
|
||||
kotlinx.coroutines.yield()
|
||||
bystander.add(System.nanoTime() - t)
|
||||
}
|
||||
}
|
||||
Thread.sleep(durationMs)
|
||||
stop.set(true)
|
||||
jobs.forEach { it.join() }
|
||||
by.join()
|
||||
report("PER_SUB_BLOCKING", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
|
||||
}
|
||||
|
||||
private fun runPerSubMutex(subs: Int) =
|
||||
runBlocking {
|
||||
@Suppress("DEPRECATION")
|
||||
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
|
||||
val states = Array(subs) { PerSubMutex() }
|
||||
val stop = AtomicBoolean(false)
|
||||
val ops = AtomicLong(0)
|
||||
val bystander = ArrayList<Long>()
|
||||
val jobs =
|
||||
(0 until relays).map { relay ->
|
||||
launch(dispatcher) {
|
||||
var n = 0L
|
||||
while (!stop.get()) {
|
||||
val s = states[relay % subs]
|
||||
s.mutex.withLock {
|
||||
s.status[relay] = 1
|
||||
s.filters[relay] = SAMPLE
|
||||
s.status[relay]
|
||||
}
|
||||
n++
|
||||
// Models `for (message in incomingMessages)`: every real
|
||||
// iteration suspends, letting the dispatcher multiplex.
|
||||
kotlinx.coroutines.yield()
|
||||
}
|
||||
ops.addAndGet(n)
|
||||
}
|
||||
}
|
||||
val by =
|
||||
launch(dispatcher) {
|
||||
while (!stop.get()) {
|
||||
val t = System.nanoTime()
|
||||
kotlinx.coroutines.yield()
|
||||
bystander.add(System.nanoTime() - t)
|
||||
}
|
||||
}
|
||||
Thread.sleep(durationMs)
|
||||
stop.set(true)
|
||||
jobs.forEach { it.join() }
|
||||
by.join()
|
||||
report("PER_SUB_MUTEX", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
|
||||
}
|
||||
|
||||
private fun runStriped(subs: Int) =
|
||||
runBlocking {
|
||||
@Suppress("DEPRECATION")
|
||||
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
|
||||
val states = Array(subs) { Striped() }
|
||||
val stop = AtomicBoolean(false)
|
||||
val ops = AtomicLong(0)
|
||||
val bystander = ArrayList<Long>()
|
||||
val jobs =
|
||||
(0 until relays).map { relay ->
|
||||
launch(dispatcher) {
|
||||
var n = 0L
|
||||
while (!stop.get()) {
|
||||
states[relay % subs].withLock(relay) { s ->
|
||||
s.status = 1
|
||||
s.filters = SAMPLE
|
||||
s.lastKnown = SAMPLE
|
||||
}
|
||||
n++
|
||||
// Models `for (message in incomingMessages)`: every real
|
||||
// iteration suspends, letting the dispatcher multiplex.
|
||||
kotlinx.coroutines.yield()
|
||||
}
|
||||
ops.addAndGet(n)
|
||||
}
|
||||
}
|
||||
val by =
|
||||
launch(dispatcher) {
|
||||
while (!stop.get()) {
|
||||
val t = System.nanoTime()
|
||||
kotlinx.coroutines.yield()
|
||||
bystander.add(System.nanoTime() - t)
|
||||
}
|
||||
}
|
||||
Thread.sleep(durationMs)
|
||||
stop.set(true)
|
||||
jobs.forEach { it.join() }
|
||||
by.join()
|
||||
report("STRIPED", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
|
||||
}
|
||||
|
||||
/**
|
||||
* Same topology, but driving the REAL shipped [RequestSubscriptionState] (striped per
|
||||
* relay) instead of a prototype — so the measured win is a property of the code that
|
||||
* ships, not of this file.
|
||||
*/
|
||||
private fun runRealStriped(subs: Int) =
|
||||
runBlocking {
|
||||
@Suppress("DEPRECATION")
|
||||
val dispatcher = Dispatchers.IO.limitedParallelism(dispatcherThreads)
|
||||
val states = Array(subs) { RequestSubscriptionState<Int>() }
|
||||
val stop = AtomicBoolean(false)
|
||||
val ops = AtomicLong(0)
|
||||
val bystander = ArrayList<Long>()
|
||||
val filters = listOf(Filter(kinds = listOf(1)))
|
||||
val jobs =
|
||||
(0 until relays).map { relay ->
|
||||
launch(dispatcher) {
|
||||
var n = 0L
|
||||
val state = states[relay % subs]
|
||||
while (!stop.get()) {
|
||||
state.withLock(relay) {
|
||||
state.onNewEvent(relay)
|
||||
state.currentState(relay)
|
||||
state.onOpenReq(relay, filters)
|
||||
state.lastKnownFilterStates(relay)
|
||||
}
|
||||
n++
|
||||
kotlinx.coroutines.yield()
|
||||
}
|
||||
ops.addAndGet(n)
|
||||
}
|
||||
}
|
||||
val by =
|
||||
launch(dispatcher) {
|
||||
while (!stop.get()) {
|
||||
val t = System.nanoTime()
|
||||
kotlinx.coroutines.yield()
|
||||
bystander.add(System.nanoTime() - t)
|
||||
}
|
||||
}
|
||||
Thread.sleep(durationMs)
|
||||
stop.set(true)
|
||||
jobs.forEach { it.join() }
|
||||
by.join()
|
||||
report("REAL_STRIPED", subs, ops.get(), bystander.ifEmpty { listOf(0L) })
|
||||
}
|
||||
|
||||
companion object {
|
||||
private val SAMPLE = listOf("kinds:1", "authors:abc")
|
||||
}
|
||||
}
|
||||
+242
@@ -0,0 +1,242 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.client.reqs.RequestSubscriptionState
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.Test
|
||||
import java.util.concurrent.CountDownLatch
|
||||
import java.util.concurrent.atomic.AtomicBoolean
|
||||
import java.util.concurrent.atomic.AtomicLong
|
||||
import kotlin.concurrent.thread
|
||||
|
||||
/**
|
||||
* Reproduces the production ANR seen on a Pixel 8 (2026-08-03, anr_2026-08-03-12-55-26-256):
|
||||
* 191 live relay sockets feed EVENT frames for the same handful of subscription ids, so
|
||||
* dozens of DefaultDispatcher workers pile onto ONE [RequestSubscriptionState] busy-wait
|
||||
* lock. In that trace 51 of 52 runnable workers sat in the inlined spin loop while the
|
||||
* single lock holder was parked in `WaitingForGcToComplete`.
|
||||
*
|
||||
* Two things are measured:
|
||||
* - [contendedThroughput]: aggregate critical sections/s as the contender count grows past
|
||||
* the core count (the "negative scaling" the 2026-07-02 plan measured with 4 feeders,
|
||||
* re-run at production fan-out).
|
||||
* - [victimLatencyUnderSpin]: what an UNRELATED thread (stand-in for the UI thread) sees
|
||||
* while the spinners run — this is the ANR mechanism, not the lock throughput.
|
||||
*/
|
||||
class SpinLockConvoyBenchmark {
|
||||
private val cores = Runtime.getRuntime().availableProcessors()
|
||||
|
||||
/**
|
||||
* Every waiter targets ONE reference on purpose. The lock is striped per relay, so
|
||||
* spreading threads over distinct relays would put them on different stripes and
|
||||
* this benchmark would measure nothing — the point here is the primitive's behaviour
|
||||
* when contention DOES land on a single stripe.
|
||||
*/
|
||||
private val hotRelay = 0
|
||||
|
||||
/** Mirrors the real critical section: a handful of map reads/writes, no I/O. */
|
||||
private fun criticalSection(
|
||||
state: RequestSubscriptionState<Int>,
|
||||
relay: Int,
|
||||
) {
|
||||
state.onNewEvent(relay)
|
||||
state.currentState(relay)
|
||||
state.lastKnownFilterStates(relay)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun contendedThroughput() {
|
||||
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
|
||||
println("contendedThroughput skipped. Run with -PprodRelayBench=1 to enable.")
|
||||
return
|
||||
}
|
||||
println("cores = $cores")
|
||||
println("threads | ops/s | vs 1 thread")
|
||||
var baseline = 0.0
|
||||
for (threads in listOf(1, 2, 4, 8, 16, 32, 52)) {
|
||||
val state = RequestSubscriptionState<Int>()
|
||||
val stop = AtomicBoolean(false)
|
||||
val ops = AtomicLong(0)
|
||||
val start = CountDownLatch(1)
|
||||
val workers =
|
||||
(0 until threads).map { id ->
|
||||
thread {
|
||||
start.await()
|
||||
var local = 0L
|
||||
while (!stop.get()) {
|
||||
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
|
||||
local++
|
||||
}
|
||||
ops.addAndGet(local)
|
||||
}
|
||||
}
|
||||
val t0 = System.nanoTime()
|
||||
start.countDown()
|
||||
Thread.sleep(2000)
|
||||
stop.set(true)
|
||||
workers.forEach { it.join() }
|
||||
val secs = (System.nanoTime() - t0) / 1e9
|
||||
val rate = ops.get() / secs
|
||||
if (threads == 1) baseline = rate
|
||||
println("%7d | %9.0f | %.2fx".format(threads, rate, rate / baseline))
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* REGRESSION GUARD: contended waiters on [RequestSubscriptionState.withLock] must PARK,
|
||||
* never busy-wait. Fails if the lock is ever turned back into a spin lock.
|
||||
*
|
||||
* This is the signature that identified the production ANR: in
|
||||
* `anr_2026-08-03-12-55-26-256`, 37 of 52 runnable workers sat at one obfuscated line of
|
||||
* `PoolRequests.onIncomingMessage` and 12 more at one line of `syncState$lambda$0` — all
|
||||
* `state=R`, `sCount=0`, burning 596% CPU while the lock holder was stuck in
|
||||
* `WaitingForGcToComplete`. With a spin lock this test observes ~40/40 waiters RUNNABLE;
|
||||
* with a parking lock it observes 0.
|
||||
*/
|
||||
@Test
|
||||
fun contendedWaitersParkInsteadOfSpinning() {
|
||||
val spinners = 40
|
||||
val state = RequestSubscriptionState<Int>()
|
||||
val stop = AtomicBoolean(false)
|
||||
val start = CountDownLatch(1)
|
||||
|
||||
val holder =
|
||||
thread(name = "holder") {
|
||||
start.await()
|
||||
while (!stop.get()) {
|
||||
state.withLock(hotRelay) {
|
||||
val t = System.nanoTime()
|
||||
while (System.nanoTime() - t < 5_000_000) { /* hold 5ms */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
val threads =
|
||||
(0 until spinners).map { id ->
|
||||
thread(name = "spinner-$id") {
|
||||
start.await()
|
||||
while (!stop.get()) {
|
||||
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
|
||||
}
|
||||
}
|
||||
}
|
||||
start.countDown()
|
||||
Thread.sleep(300)
|
||||
|
||||
// Sample every spinner's stack, exactly like an ANR dump would.
|
||||
val points = HashMap<String, Int>()
|
||||
var runnable = 0
|
||||
threads.forEach { t ->
|
||||
if (t.state == Thread.State.RUNNABLE) runnable++
|
||||
val top = t.stackTrace.firstOrNull { it.className.contains("SpinLockConvoyBenchmark") || it.className.contains("RequestSubscriptionState") }
|
||||
if (top != null) {
|
||||
val key = "${top.className.substringAfterLast('.')}.${top.methodName}:${top.lineNumber}"
|
||||
points[key] = (points[key] ?: 0) + 1
|
||||
}
|
||||
}
|
||||
stop.set(true)
|
||||
threads.forEach { it.join() }
|
||||
holder.join()
|
||||
|
||||
println("sampled $spinners waiters: RUNNABLE=$runnable, distinct program points=${points.size}")
|
||||
points.entries.sortedByDescending { it.value }.forEach { println(" ${it.value}x ${it.key}") }
|
||||
|
||||
// A parked waiter reports WAITING, so the parking lock measures ~0 here while a
|
||||
// spin lock measures ~100%. The line sits at 50% deliberately: it separates the
|
||||
// two cases by a mile and leaves headroom on a loaded CI box, where a few waiters
|
||||
// can legitimately be mid-acquire when we sample.
|
||||
assertTrue(
|
||||
"Expected contended waiters to park, but $runnable/$spinners were RUNNABLE — " +
|
||||
"RequestSubscriptionState.withLock looks like it is busy-waiting again. " +
|
||||
"See PlatformLock's kdoc: this is what caused anr_2026-08-03-12-55-26-256.",
|
||||
runnable <= spinners / 2,
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun victimLatencyUnderSpin() {
|
||||
if (System.getenv("PROD_RELAY_BENCH") == null && System.getProperty("prodRelayBench") == null) {
|
||||
println("victimLatencyUnderSpin skipped. Run with -PprodRelayBench=1 to enable.")
|
||||
return
|
||||
}
|
||||
// One holder that briefly stalls inside the critical section (in production: a GC
|
||||
// pause, or simply being descheduled). Everyone else spins.
|
||||
val spinners = 52
|
||||
val state = RequestSubscriptionState<Int>()
|
||||
val stop = AtomicBoolean(false)
|
||||
|
||||
fun measureVictim(label: String) {
|
||||
// The "UI thread": allocates and measures its own scheduling latency.
|
||||
val samples = ArrayList<Long>()
|
||||
repeat(200) {
|
||||
val t = System.nanoTime()
|
||||
// trivial allocation work, like a Compose semantics traversal step
|
||||
val junk = ArrayList<String>(64)
|
||||
repeat(64) { i -> junk.add("node$i") }
|
||||
Thread.yield()
|
||||
samples.add(System.nanoTime() - t)
|
||||
}
|
||||
samples.sort()
|
||||
println(
|
||||
"%-22s p50=%6.0fus p90=%7.0fus p99=%8.0fus max=%8.0fus".format(
|
||||
label,
|
||||
samples[samples.size / 2] / 1000.0,
|
||||
samples[(samples.size * 90) / 100] / 1000.0,
|
||||
samples[(samples.size * 99) / 100] / 1000.0,
|
||||
samples.last() / 1000.0,
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
measureVictim("idle (no spinners)")
|
||||
|
||||
val start = CountDownLatch(1)
|
||||
val holder =
|
||||
thread {
|
||||
start.await()
|
||||
while (!stop.get()) {
|
||||
state.withLock(hotRelay) {
|
||||
// Simulate the holder losing its core / waiting on GC mid-section.
|
||||
val t = System.nanoTime()
|
||||
while (System.nanoTime() - t < 2_000_000) { /* 2ms stall */ }
|
||||
}
|
||||
Thread.sleep(1)
|
||||
}
|
||||
}
|
||||
val threads =
|
||||
(0 until spinners).map { id ->
|
||||
thread {
|
||||
start.await()
|
||||
while (!stop.get()) {
|
||||
state.withLock(hotRelay) { criticalSection(state, hotRelay) }
|
||||
}
|
||||
}
|
||||
}
|
||||
start.countDown()
|
||||
Thread.sleep(500)
|
||||
measureVictim("$spinners spinners")
|
||||
stop.set(true)
|
||||
threads.forEach { it.join() }
|
||||
holder.join()
|
||||
|
||||
measureVictim("after (no spinners)")
|
||||
}
|
||||
}
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.nip66RelayMonitor.reachability
|
||||
|
||||
import java.io.EOFException
|
||||
import java.net.ConnectException
|
||||
import java.net.SocketTimeoutException
|
||||
import java.net.UnknownHostException
|
||||
import javax.net.ssl.SSLHandshakeException
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertFalse
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* What a monitor is willing to SAY about someone else's relay. A negative
|
||||
* NIP-66 record is signed and public, so only failures of the connection
|
||||
* itself may be published as "unreachable".
|
||||
*/
|
||||
class UnreachabilityTest {
|
||||
private fun proves(e: Exception) = Unreachability.proves(e)
|
||||
|
||||
@Test
|
||||
fun `a connection that never opened is unreachable`() {
|
||||
assertTrue(proves(UnknownHostException("no such host")))
|
||||
assertTrue(proves(ConnectException("connection refused")))
|
||||
assertTrue(proves(SSLHandshakeException("cert expired")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `a relay that hung up mid-transfer is not unreachable`() {
|
||||
// A relay that answers the handshake in 50ms and then sends EOFException
|
||||
// part-way through a large page is reachable; it declined to finish a
|
||||
// query. Publishing "unreachable" would be a false statement about a
|
||||
// working server.
|
||||
assertFalse(proves(EOFException("stream closed")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `our own bug is never the relay's fault`() {
|
||||
assertFalse(proves(ConcurrentModificationException()))
|
||||
assertFalse(proves(NullPointerException()))
|
||||
assertFalse(proves(ClassCastException("HashMap\$Node cannot be cast")))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `an unrecognised failure stays quiet`() {
|
||||
// Conservative on purpose: staying quiet costs one retry next cycle,
|
||||
// being wrong costs a false record carrying the monitor's signature.
|
||||
assertFalse(proves(SocketTimeoutException("read timed out")))
|
||||
assertFalse(proves(RuntimeException("something new")))
|
||||
}
|
||||
}
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.quartz.utils.concurrent
|
||||
|
||||
import kotlin.concurrent.atomics.AtomicBoolean
|
||||
import kotlin.concurrent.atomics.ExperimentalAtomicApi
|
||||
|
||||
// Linux/JVM-less target: Kotlin/Native's stdlib ships no parking lock and there is
|
||||
// no Foundation here, so this keeps a test-and-test-and-set spin. Correct but not
|
||||
// scalable. Acceptable ONLY because linuxX64 is a build/CI target for quartz, not a
|
||||
// host for the many-relay client workload whose contention motivated the parking
|
||||
// actuals on jvmAndroid and Apple. If that ever changes, swap in a pthread mutex.
|
||||
@OptIn(ExperimentalAtomicApi::class)
|
||||
actual class PlatformLock {
|
||||
private val held = AtomicBoolean(false)
|
||||
|
||||
actual fun lock() {
|
||||
while (held.exchange(true)) {
|
||||
while (held.load()) { }
|
||||
}
|
||||
}
|
||||
|
||||
actual fun unlock() {
|
||||
held.store(false)
|
||||
}
|
||||
}
|
||||
+10
@@ -74,6 +74,16 @@ actual class ConcurrentMap<K : Any, V : Any> {
|
||||
}
|
||||
}
|
||||
|
||||
actual fun remove(key: K): V? {
|
||||
while (true) {
|
||||
val cur = ref.load()
|
||||
val old = cur[key] ?: return null
|
||||
val copy = HashMap(cur)
|
||||
copy.remove(key)
|
||||
if (ref.compareAndSet(cur, copy)) return old
|
||||
}
|
||||
}
|
||||
|
||||
actual fun size(): Int = ref.load().size
|
||||
|
||||
actual fun snapshot(): Map<K, V> = HashMap(ref.load())
|
||||
|
||||
@@ -29,15 +29,26 @@
|
||||
# amethyst-desktop-1.08.0-linux-x64.AppImage
|
||||
# amethyst-desktop-1.08.0-linux-x64.flatpak
|
||||
# amethyst-desktop-1.08.0-linux-x64.tar.gz
|
||||
# amethyst-desktop-1.08.0-linux-arm64.deb
|
||||
# amethyst-desktop-1.08.0-linux-arm64.rpm
|
||||
# amethyst-desktop-1.08.0-linux-arm64.AppImage
|
||||
# amethyst-desktop-1.08.0-linux-arm64.flatpak
|
||||
# amethyst-desktop-1.08.0-linux-arm64.tar.gz
|
||||
# amy-1.08.0-macos-arm64.tar.gz
|
||||
# amy-1.08.0-macos-x64.tar.gz
|
||||
# amy-1.08.0-linux-x64.tar.gz
|
||||
# amy-1.08.0-linux-x64.deb
|
||||
# amy-1.08.0-linux-x64.rpm
|
||||
# amy-1.08.0-linux-arm64.tar.gz
|
||||
# amy-1.08.0-linux-arm64.deb
|
||||
# amy-1.08.0-linux-arm64.rpm
|
||||
# geode-1.08.0-macos-arm64.tar.gz
|
||||
# geode-1.08.0-linux-x64.tar.gz
|
||||
# geode-1.08.0-linux-x64.deb
|
||||
# geode-1.08.0-linux-x64.rpm
|
||||
# geode-1.08.0-linux-arm64.tar.gz
|
||||
# geode-1.08.0-linux-arm64.deb
|
||||
# geode-1.08.0-linux-arm64.rpm
|
||||
#
|
||||
# Two assets break the family/arch shape on purpose: the no-JRE jar bundles for
|
||||
# Homebrew-core are pure JVM bytecode (no bundled runtime), so a single
|
||||
|
||||
Reference in New Issue
Block a user