laantungir/amethyst
1
0
Fork 0
mirror of https://github.com/vitorpamplona/amethyst.git synced 2026-08-09 08:04:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into claude/buzz-repo-analysis-7k54ga

Browse Source
This commit is contained in:
Claude
2026-07-22 18:28:40 +00:00
parent 7205d62de2 1e874239b8
commit f8bb53848b
83 changed files with 5139 additions and 634 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/find-missing-translations/SKILL.md
+26 -7
View File
@@ -212,9 +212,25 @@ Before presenting results, **scan the missing English strings** for two red-flag
Also **audit existing `<plurals>` resources** for two anti-patterns:
1. **`quantity="one"` items that hardcode the literal `1`** (instead of using a `%d` / `%1$d` placeholder) — broken for languages where the `one` CLDR category covers more than just `n=1` (Russian, Ukrainian, Croatian, etc.).
2. **`quantity="zero"` items in any locale that doesn't natively use the `zero` CLDR category** — i.e. **everything except Arabic (`ar`) and Welsh (`cy`)**. ICU/CLDR maps `count=0` to `other` for English and all the locales we ship to (cs, de, pt-BR, sv, etc.), so `<item quantity="zero">` is **dead code** there: `getQuantityString(id, 0)` will pick `other`, never the zero entry, and the visible runtime string ends up `"…0 items"` instead of the intended `"…no items"`.
2. **`quantity="zero"` items in any locale that doesn't natively use the `zero` CLDR category** — i.e. everything except **Arabic (`ar`)**, **Latvian (`lv`)** and **Welsh (`cy`)**. ICU/CLDR maps `count=0` to `other` for English and most of the locales we ship to (cs, de, pt-BR, sv, etc.), so `<item quantity="zero">` is **dead code** there: `getQuantityString(id, 0)` will pick `other`, never the zero entry, and the visible runtime string ends up `"…0 items"` instead of the intended `"…no items"`.
If a UX genuinely wants special "no items" wording at count=0, that has to be a call-site `if (count == 0)` branch to a separate `<string>`, **not** a `quantity="zero"` plural item.
> ⚠️ **Latvian is the trap here — do NOT strip its `zero` items** (we nearly did, 2026-07-22). `lv` has an integer-bearing `zero` category that covers far more than 0: `select(0)`, `select(10)` and `select(11)` all return `zero` (the rule is `n % 10 = 0` or `n % 100 = 11..19`). So a Latvian `<item quantity="zero">` is *live code on the majority of counts*, and it must read as a normal plural form ("%1$d minūšu"), **not** as "no items" wording. An earlier version of this skill claimed only `ar` and `cy` had `zero`, which flagged all ~40 correct Latvian entries as dead and would have deleted working translations.
If a UX genuinely wants special "no items" wording at count=0, that has to be a call-site `if (count == 0)` branch to a separate `<string>`, **not** a `quantity="zero"` plural item. (This is why `zero` is the wrong tool even where it exists: in `lv` it does not mean "zero".)
**Verify, don't recall.** Before asserting any locale's category set, check it against CLDR rather than memory:
```bash
python3 -m venv /tmp/cldr && /tmp/cldr/bin/pip -q install babel
/tmp/cldr/bin/python -c "
from babel import Locale
for c in ['en','lv','ar','cy','cs','de','sv','pt_BR','ru','pl']:
r = Locale.parse(c).plural_form
print(c, sorted({r(n) for n in range(0,10001)}), 'select(0)=', r(0), 'select(10)=', r(10))
"
```
Across the 56 locale dirs this repo ships, **only `ar-rSA` and `lv-rLV`** have an integer-bearing `zero`.
Flag and offer to fix:
@@ -240,15 +256,16 @@ for f in amethyst/src/main/res/values/strings.xml amethyst/src/main/res/values-*
done
```
Then scan for dead `quantity="zero"` entries. CLDR's `zero` category is integer-bearing only in **Arabic (`ar`)** and **Welsh (`cy`)**. In every other locale, count=0 falls through to `other`, so a `<item quantity="zero">` entry is dead and likely a translator/author bug (or it silently never fires):
Then scan for dead `quantity="zero"` entries. CLDR's `zero` category is integer-bearing only in **Arabic (`ar`)**, **Latvian (`lv`)** and **Welsh (`cy`)** — those three are skipped below, so a hit is a genuine bug. In every other locale, count=0 falls through to `other`, so a `<item quantity="zero">` entry is dead and likely a translator/author bug (or it silently never fires):
```bash
for f in amethyst/src/main/res/values/strings.xml amethyst/src/main/res/values-*/strings.xml \
commons/src/commonMain/composeResources/values/strings.xml \
commons/src/commonMain/composeResources/values-*/strings.xml; do
# Skip Arabic and Welsh — they natively use the zero category.
# Skip Arabic, Latvian and Welsh — they natively use the zero category.
# (Latvian's zero covers 0, 10, 11-19, 20, 30, … — stripping it breaks most counts.)
case "$f" in
*values-ar*|*values-cy*) continue ;;
*values-ar*|*values-cy*|*values-lv*) continue ;;
esac
awk -v file="$f" '
/<plurals/ { in_plurals = 1; name = $0; sub(/.*name="/, "", name); sub(/".*/, "", name) }
@@ -312,9 +329,10 @@ When adding or proposing **`<plurals>`** entries, follow these rules:
- Polish (`pl`): `one`, `few`, `many`, `other`
- Russian (`ru`): `one`, `few`, `many`, `other`
- Arabic (`ar`): `zero`, `one`, `two`, `few`, `many`, `other`
- Latvian (`lv`): `zero`, `one`, `other` — its `zero` is **not** "no items"; it covers 0, 10, 11–19, 20, 30, …
- German / Swedish / Brazilian Portuguese: `one`, `other`
- When a missing string contains a count placeholder and is conceptually a singular/plural pair, **flag it before translating** — it may belong as a `<plurals>` resource rather than a single `<string>`. Surface this to the user before proposing translations.
- **Do not use `quantity="zero"` outside Arabic (`ar`) and Welsh (`cy`).** CLDR's `zero` category is integer-bearing only in those two languages. Android calls `PluralRules.select(0)` for the device locale; in English/German/Czech/Polish/Russian/Swedish/Portuguese/etc. it returns `other`, so the explicit `<item quantity="zero">` is never picked at runtime and the user sees `"…0 items"` instead of the intended wording. If the design calls for "no items" at count=0, model it as a separate `<string>` and an `if (count == 0)` branch at the call site:
- **Do not use `quantity="zero"` outside Arabic (`ar`), Latvian (`lv`) and Welsh (`cy`).** CLDR's `zero` category is integer-bearing only in those three languages. Android calls `PluralRules.select(0)` for the device locale; in English/German/Czech/Polish/Russian/Swedish/Portuguese/etc. it returns `other`, so the explicit `<item quantity="zero">` is never picked at runtime and the user sees `"…0 items"` instead of the intended wording. Conversely, **never delete an existing `zero` item from `ar`/`lv`/`cy`** — there it is live. If the design calls for "no items" at count=0, model it as a separate `<string>` and an `if (count == 0)` branch at the call site:
```kotlin
val label = if (count == 0) {
stringRes(R.string.foo_no_items, dateLabel)
@@ -377,4 +395,5 @@ When adding translated strings to locale files:
- **Inserting strings in a specific position** — always append at the bottom; ordering is handled separately
- **Hardcoding `"1"` in a `<plurals>` `quantity="one"` item** — always use the count placeholder; otherwise non-English `one` categories produce wrong text
- **Copying English's `one`/`other` set into every locale** — each language must include all CLDR plural categories it uses (e.g. Czech needs `one`, `few`, `many`, `other`)
- **Using `<item quantity="zero">` to special-case count=0** — outside Arabic and Welsh, this entry is unreachable: ICU/CLDR maps 0 → `other`, so the runtime never picks the zero item and the user sees `"…0 items"`. Special-case at the call site with a separate `<string>` instead.
- **Using `<item quantity="zero">` to special-case count=0** — outside Arabic, Latvian and Welsh, this entry is unreachable: ICU/CLDR maps 0 → `other`, so the runtime never picks the zero item and the user sees `"…0 items"`. Special-case at the call site with a separate `<string>` instead.
- **Reporting Latvian `quantity="zero"` entries as dead code** — `lv` has a real, integer-bearing `zero` category covering 0, 10, 11–19, 20, 30, … so those entries fire on *most* counts. An earlier version of this skill excluded only `ar`/`cy` from the zero audit and flagged all ~40 correct `values-lv-rLV` entries; acting on that would have deleted working translations. Confirm any locale's category set against CLDR (the babel snippet in Step 4) before calling a `zero` item dead.
.github/workflows/build.yml
+15 -1
View File
@@ -231,9 +231,23 @@ jobs:
name: Android Lint Reports
path: amethyst/build/reports/lint-results-*.html
# Publishes the JUnit XML produced by the unit-test tasks above as inline
# annotations plus a job summary. Replaces asadmansr/android-test-report-action,
# which was abandoned (last release 2020) and rebuilt an EOL Ubuntu 18.04 +
# Python 2 Docker image on every run — bionic's apt archives have since gone
# unreliable and broke this job. Pinned to a commit SHA (not the movable
# v6.4.2 tag) to close the supply-chain hole. annotate_only avoids needing
# `checks: write`, so it keeps working on pull requests from forks (where the
# GITHUB_TOKEN is read-only). fail_on_failure preserves the old step's
# behavior of marking the job red when a test fails.
- name: Android Test Report
uses: asadmansr/android-test-report-action@v1.2.0
uses: mikepenz/action-junit-report@d9f48fc87bc235f7e214acf696ca5abc0a986f16 # v6.4.2
if: always()
with:
report_paths: '**/build/test-results/**/TEST-*.xml'
annotate_only: true
detailed_summary: true
fail_on_failure: true
- name: Upload Test Results
uses: actions/upload-artifact@v7
.idea/kotlinc.xml
Generated
+1 -1
View File
@@ -8,6 +8,6 @@
</component>
<component name="KotlinJpsPluginSettings">
<option name="externalSystemId" value="Gradle" />
<option name="version" value="2.4.0" />
<option name="version" value="2.4.10" />
</component>
</project>
README.md
+1 -1
View File
@@ -17,7 +17,7 @@ Join the social network you control.
[![Maven Central](https://img.shields.io/maven-central/v/com.vitorpamplona.quartz/quartz?label=Quartz%20%28Maven%20Central%29&labelColor=27303D&color=0877d2)](https://central.sonatype.com/artifact/com.vitorpamplona.quartz/quartz)
[![JitPack snapshots](https://img.shields.io/badge/Quartz%20snapshots-JitPack-27303D?labelColor=27303D&color=0877d2)](https://jitpack.io/#vitorpamplona/amethyst)
[![CI](https://img.shields.io/github/actions/workflow/status/vitorpamplona/amethyst/build.yml?labelColor=27303D)](https://github.com/vitorpamplona/amethyst/actions/workflows/build.yml)
[![License: Apache-2.0](https://img.shields.io/github/license/vitorpamplona/amethyst?labelColor=27303D&color=0877d2)](/LICENSE)
[![License: MIT](https://img.shields.io/github/license/vitorpamplona/amethyst?labelColor=27303D&color=0877d2)](/LICENSE)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/vitorpamplona/amethyst)
## Download and Install
SKILL.md
+43 -43
View File
@@ -24,7 +24,9 @@ Build customized Amethyst Nostr clients for Android. Fork, rebrand, customize, a
2. **Android SDK**
- Command-line tools from https://developer.android.com/studio#command-line-tools-only
- Required components: build-tools, platform-tools, platforms;android-35
- Required components: build-tools, platform-tools, platforms;android-37
- The exact SDK level is `android-compileSdk` in `gradle/libs.versions.toml` —
check there if this number has drifted.
3. **Git** for cloning the repository
@@ -66,48 +68,54 @@ keyPassword=your-password
### 3. Configure Signing
Add to `amethyst/build.gradle` inside the `android {}` block:
Add to `amethyst/build.gradle.kts` inside the `android {}` block:
```gradle
def keystorePropertiesFile = rootProject.file("keystore.properties")
def keystoreProperties = new Properties()
```kotlin
val keystorePropertiesFile = rootProject.file("keystore.properties")
val keystoreProperties = Properties()
if (keystorePropertiesFile.exists()) {
keystoreProperties.load(new FileInputStream(keystorePropertiesFile))
keystorePropertiesFile.inputStream().use { keystoreProperties.load(it) }
}
signingConfigs {
release {
create("release") {
if (keystorePropertiesFile.exists()) {
storeFile rootProject.file(keystoreProperties['storeFile'])
storePassword keystoreProperties['storePassword']
keyAlias keystoreProperties['keyAlias']
keyPassword keystoreProperties['keyPassword']
storeFile = rootProject.file(keystoreProperties["storeFile"] as String)
storePassword = keystoreProperties["storePassword"] as String
keyAlias = keystoreProperties["keyAlias"] as String
keyPassword = keystoreProperties["keyPassword"] as String
}
}
}
```
This needs `import java.util.Properties` at the top of the file.
Update the release buildType to use the signing config:
```gradle
```kotlin
buildTypes {
release {
signingConfig signingConfigs.release
getByName("release") {
signingConfig = signingConfigs.getByName("release")
// ... existing config
}
}
```
Verify with `./gradlew :amethyst:signingReport` — the release variants should
report your keystore rather than `~/.android/debug.keystore`.
### 4. Disable Google Services (Required for F-Droid)
**⚠️ CRITICAL:** The Google Services plugin fails when you change the package name. For F-Droid builds, disable it.
Edit `amethyst/build.gradle`, comment out the plugin:
```gradle
Edit `amethyst/build.gradle.kts`, comment out the plugin:
```kotlin
plugins {
alias(libs.plugins.androidApplication)
alias(libs.plugins.jetbrainsKotlinAndroid)
// alias(libs.plugins.googleServices) // DISABLED for F-Droid
alias(libs.plugins.jetbrainsComposeCompiler)
alias(libs.plugins.serialization)
alias(libs.plugins.googleKsp)
}
```
@@ -141,8 +149,8 @@ Edit `amethyst/src/main/res/values/strings.xml`:
### Change Package ID
Edit `amethyst/build.gradle`:
```gradle
Edit `amethyst/build.gradle.kts`:
```kotlin
android {
defaultConfig {
applicationId = "com.yourcompany.yourapp"
@@ -152,8 +160,8 @@ android {
### Change Project Name
Edit `settings.gradle`:
```gradle
Edit `settings.gradle.kts`:
```kotlin
rootProject.name = "YourAppName"
```
@@ -167,36 +175,28 @@ Replace icon files in:
Make your app identify itself on posts with `["client", "YourAppName"]`.
**1. Create tag builder extension:**
You do **not** need to add the tag per event type. The client tag is applied
centrally by `NostrSignerWithClientTag`, a signer decorator that appends the tag
to everything it signs (and respects the user's "add client tag" privacy
setting). Changing the name is a one-constant edit:
Create `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/tags/clientTag/TagArrayBuilderExt.kt`:
Edit `amethyst/src/main/java/com/vitorpamplona/amethyst/model/accountsCache/AccountCacheState.kt`:
```kotlin
package com.vitorpamplona.quartz.nip01Core.tags.clientTag
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.TagArrayBuilder
fun <T : Event> TagArrayBuilder<T>.client(clientName: String) =
addUnique(arrayOf(ClientTag.TAG_NAME, clientName))
const val CLIENT_TAG_NAME = "YourAppName"
```
**2. Add to TextNoteEvent:**
That constant is passed to `NostrSignerWithClientTag` when the account's signer
is built, so every signed event carries your name.
Edit `quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip10Notes/TextNoteEvent.kt`:
Add import:
```kotlin
import com.vitorpamplona.quartz.nip01Core.tags.clientTag.client
```
In both `build()` functions, add after `alt(...)`:
```kotlin
client("YourAppName")
```
The tag itself lives in
`quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip89AppHandlers/clientTag/`
(`ClientTag`, `TagArrayBuilderExt`, `NostrSignerWithClientTag`) — you only need to
touch it if you want the optional NIP-89 handler address / relay hint variants.
### Modify Default Relays
Edit relay configuration in `quartz/src/main/java/com/vitorpamplona/quartz/nip01Core/relay/` or the UI settings files.
Edit `commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/defaults/Constants.kt`
(see also `AmethystDefaults.kt` and `DefaultDmIndexerRelays.kt` in the same folder).
## Troubleshooting
amethyst/src/main/java/com/vitorpamplona/amethyst/LocalPreferences.kt
+21
View File
@@ -38,8 +38,10 @@ import com.vitorpamplona.amethyst.model.UiSettings
import com.vitorpamplona.amethyst.service.checkNotInMainThread
import com.vitorpamplona.amethyst.ui.actions.mediaServers.DEFAULT_MEDIA_SERVERS
import com.vitorpamplona.amethyst.ui.actions.mediaServers.ServerName
import com.vitorpamplona.quartz.concord.cord02Community.ConcordCommunityListEvent
import com.vitorpamplona.quartz.experimental.ephemChat.list.EphemeralChatListEvent
import com.vitorpamplona.quartz.experimental.nipA3.PaymentTargetsEvent
import com.vitorpamplona.quartz.marmot.mip00KeyPackages.KeyPackageRelayListEvent
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.core.JsonMapper
@@ -55,6 +57,7 @@ import com.vitorpamplona.quartz.nip28PublicChat.list.ChannelListEvent
import com.vitorpamplona.quartz.nip37Drafts.privateOutbox.PrivateOutboxRelayListEvent
import com.vitorpamplona.quartz.nip47WalletConnect.Nip47WalletConnect
import com.vitorpamplona.quartz.nip50Search.SearchRelayListEvent
import com.vitorpamplona.quartz.nip51Lists.favoriteAlgoFeedsList.FavoriteAlgoFeedsListEvent
import com.vitorpamplona.quartz.nip51Lists.geohashList.GeohashListEvent
import com.vitorpamplona.quartz.nip51Lists.hashtagList.HashtagListEvent
import com.vitorpamplona.quartz.nip51Lists.muteList.MuteListEvent
@@ -168,7 +171,10 @@ private object PrefKeys {
const val LATEST_GEOHASH_LIST = "latestGeohashList"
const val LATEST_EPHEMERAL_LIST = "latestEphemeralChatList"
const val LATEST_RELAY_GROUP_LIST = "latestRelayGroupList"
const val LATEST_CONCORD_LIST = "latestConcordList"
const val LATEST_TRUST_PROVIDER_LIST = "latestTrustProviderList"
const val LATEST_KEY_PACKAGE_RELAY_LIST = "latestKeyPackageRelayList"
const val LATEST_FAVORITE_ALGO_FEEDS_LIST = "latestFavoriteAlgoFeedsList"
const val CALLS_ENABLED = "calls_enabled"
const val HIDE_DELETE_REQUEST_DIALOG = "hide_delete_request_dialog"
const val HIDE_BLOCK_ALERT_DIALOG = "hide_block_alert_dialog"
@@ -577,7 +583,10 @@ object LocalPreferences {
putOrRemove(PrefKeys.LATEST_GEOHASH_LIST, settings.backupGeohashList)
putOrRemove(PrefKeys.LATEST_EPHEMERAL_LIST, settings.backupEphemeralChatList)
putOrRemove(PrefKeys.LATEST_RELAY_GROUP_LIST, settings.backupRelayGroupList)
putOrRemove(PrefKeys.LATEST_CONCORD_LIST, settings.backupConcordList)
putOrRemove(PrefKeys.LATEST_TRUST_PROVIDER_LIST, settings.backupTrustProviderList)
putOrRemove(PrefKeys.LATEST_KEY_PACKAGE_RELAY_LIST, settings.backupKeyPackageRelayList)
putOrRemove(PrefKeys.LATEST_FAVORITE_ALGO_FEEDS_LIST, settings.backupFavoriteAlgoFeedsList)
putOrRemove(PrefKeys.LATEST_PAYMENT_TARGETS, settings.backupNipA3PaymentTargets)
putOrRemove(PrefKeys.LATEST_CASHU_WALLET, settings.backupCashuWallet)
putOrRemove(PrefKeys.LATEST_NUTZAP_INFO, settings.backupNutzapInfo)
@@ -763,7 +772,10 @@ object LocalPreferences {
val latestGeohashListStr = getString(PrefKeys.LATEST_GEOHASH_LIST, null)
val latestEphemeralListStr = getString(PrefKeys.LATEST_EPHEMERAL_LIST, null)
val latestRelayGroupListStr = getString(PrefKeys.LATEST_RELAY_GROUP_LIST, null)
val latestConcordListStr = getString(PrefKeys.LATEST_CONCORD_LIST, null)
val latestTrustProviderListStr = getString(PrefKeys.LATEST_TRUST_PROVIDER_LIST, null)
val latestKeyPackageRelayListStr = getString(PrefKeys.LATEST_KEY_PACKAGE_RELAY_LIST, null)
val latestFavoriteAlgoFeedsListStr = getString(PrefKeys.LATEST_FAVORITE_ALGO_FEEDS_LIST, null)
val latestPaymentTargetsStr = getString(PrefKeys.LATEST_PAYMENT_TARGETS, null)
val latestCashuWalletStr = getString(PrefKeys.LATEST_CASHU_WALLET, null)
val latestNutzapInfoStr = getString(PrefKeys.LATEST_NUTZAP_INFO, null)
@@ -823,7 +835,10 @@ object LocalPreferences {
val latestGeohashList = async { parseEventOrNull<GeohashListEvent>(latestGeohashListStr) }
val latestEphemeralList = async { parseEventOrNull<EphemeralChatListEvent>(latestEphemeralListStr) }
val latestRelayGroupList = async { parseEventOrNull<SimpleGroupListEvent>(latestRelayGroupListStr) }
val latestConcordList = async { parseEventOrNull<ConcordCommunityListEvent>(latestConcordListStr) }
val latestTrustProviderList = async { parseEventOrNull<TrustProviderListEvent>(latestTrustProviderListStr) }
val latestKeyPackageRelayList = async { parseEventOrNull<KeyPackageRelayListEvent>(latestKeyPackageRelayListStr) }
val latestFavoriteAlgoFeedsList = async { parseEventOrNull<FavoriteAlgoFeedsListEvent>(latestFavoriteAlgoFeedsListStr) }
val latestPaymentTargets = async { parseEventOrNull<PaymentTargetsEvent>(latestPaymentTargetsStr) }
val latestCashuWallet =
async {
@@ -875,7 +890,10 @@ object LocalPreferences {
val latestGeohashListResolved = latestGeohashList.await()
val latestEphemeralListResolved = latestEphemeralList.await()
val latestRelayGroupListResolved = latestRelayGroupList.await()
val latestConcordListResolved = latestConcordList.await()
val latestTrustProviderListResolved = latestTrustProviderList.await()
val latestKeyPackageRelayListResolved = latestKeyPackageRelayList.await()
val latestFavoriteAlgoFeedsListResolved = latestFavoriteAlgoFeedsList.await()
val latestPaymentTargetsResolved = latestPaymentTargets.await()
val latestCashuWalletResolved = latestCashuWallet.await()
val latestNutzapInfoResolved = latestNutzapInfo.await()
@@ -969,7 +987,10 @@ object LocalPreferences {
backupGeohashList = latestGeohashListResolved,
backupEphemeralChatList = latestEphemeralListResolved,
backupRelayGroupList = latestRelayGroupListResolved,
backupConcordList = latestConcordListResolved,
backupTrustProviderList = latestTrustProviderListResolved,
backupKeyPackageRelayList = latestKeyPackageRelayListResolved,
backupFavoriteAlgoFeedsList = latestFavoriteAlgoFeedsListResolved,
lastReadPerRoute = MutableStateFlow(lastReadPerRouteResolved),
hasDonatedInVersion = MutableStateFlow(hasDonatedInVersion),
dismissedPollNoteIds = MutableStateFlow(dismissedPollNoteIds),
amethyst/src/main/java/com/vitorpamplona/amethyst/model/Account.kt
+5 -1
View File
@@ -332,6 +332,7 @@ import com.vitorpamplona.quartz.nip7DThreads.ThreadEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import com.vitorpamplona.quartz.nip89AppHandlers.clientTag.NostrSignerWithClientTag
import com.vitorpamplona.quartz.nip89AppHandlers.clientTag.withoutClientTag
import com.vitorpamplona.quartz.nip90Dvms.contentDiscoveryRequest.NIP90ContentDiscoveryRequestEvent
import com.vitorpamplona.quartz.nip92IMeta.IMetaTag
import com.vitorpamplona.quartz.nip92IMeta.imetas
@@ -474,7 +475,10 @@ class Account(
*/
val nip46Signer =
Nip46SignerState(
signer = signer,
// Acting as someone else's bunker: the templates arriving here were composed by the
// connected client, so they are signed exactly as received — our client tag would both
// misattribute the event and change the id the client expects back.
signer = signer.withoutClientTag(),
client = client,
ledger = signerPermissionLedger,
clientStore = nip46ClientStore,
amethyst/src/main/java/com/vitorpamplona/amethyst/model/MediaAspectRatioCache.kt
+22 -3
View File
@@ -21,6 +21,8 @@
package com.vitorpamplona.amethyst.model
import androidx.collection.LruCache
import androidx.compose.runtime.MutableState
import androidx.compose.runtime.mutableStateOf
interface MutableMediaAspectRatioCache {
fun get(url: String): Float?
@@ -32,10 +34,27 @@ interface MutableMediaAspectRatioCache {
)
}
/**
* Aspect ratios keyed by media URL, learned from imeta `dim` tags up front or from the decoder once
* a first frame lands.
*
* Entries are snapshot state, so a composable that calls [get] **during composition** recomposes
* when the real dimensions arrive later. That matters because players and image loaders only report
* size after the first frame decodes: a caller that sized itself off a plain cache miss would stay
* wrong for the whole visit and only look right the *next* time the media is opened. Note this only
* works for reads made in composition — a read from inside `remember { }` is cached by `remember`
* itself and won't pick the update up.
*/
object MediaAspectRatioCache : MutableMediaAspectRatioCache {
val mediaAspectRatioCacheByUrl = LruCache<String, Float>(1000)
private val cache = LruCache<String, MutableState<Float?>>(1000)
override fun get(url: String): Float? = mediaAspectRatioCacheByUrl.get(url)
// get-then-put has to be atomic, so the compound op is guarded even though LruCache is itself
// thread-safe. A miss still stores a slot: that empty slot is what the caller observes until
// add() fills it in.
@Synchronized
private fun entry(url: String): MutableState<Float?> = cache.get(url) ?: mutableStateOf<Float?>(null).also { cache.put(url, it) }
override fun get(url: String): Float? = entry(url).value
override fun add(
url: String,
@@ -43,7 +62,7 @@ object MediaAspectRatioCache : MutableMediaAspectRatioCache {
height: Int,
) {
if (height > 1) {
mediaAspectRatioCacheByUrl.put(url, width.toFloat() / height.toFloat())
entry(url).value = width.toFloat() / height.toFloat()
}
}
}
amethyst/src/main/java/com/vitorpamplona/amethyst/napplet/gateways/AccountNappletGateways.kt
+5 -1
View File
@@ -60,6 +60,7 @@ import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip47WalletConnect.rpc.NwcErrorResponse
import com.vitorpamplona.quartz.nip47WalletConnect.rpc.PayInvoiceErrorResponse
import com.vitorpamplona.quartz.nip47WalletConnect.rpc.PayInvoiceSuccessResponse
import com.vitorpamplona.quartz.nip89AppHandlers.clientTag.withoutClientTag
import com.vitorpamplona.quartz.utils.sha256.sha256
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.withTimeout
@@ -155,7 +156,10 @@ class AccountNappletGateways(
)
}
return NappletBroker(account.signer, ledger, consent, signerLedger = signerLedger, nostrConnectPrompt = connectPrompt, signerConsentPrompt = signerConsent, relay = relay, storage = storage, wallet = wallet, resource = resource, upload = upload, identityReads = identityReads, theme = theme, notify = notify)
// Everything the broker signs belongs to the guest — a napplet, an nSite, or a web app
// calling NIP-07 — never to Amethyst, so our client tag has no business on it. It would also
// corrupt the template a NIP-07 caller re-checks the returned event against.
return NappletBroker(account.signer.withoutClientTag(), ledger, consent, signerLedger = signerLedger, nostrConnectPrompt = connectPrompt, signerConsentPrompt = signerConsent, relay = relay, storage = storage, wallet = wallet, resource = resource, upload = upload, identityReads = identityReads, theme = theme, notify = notify)
}
/**
amethyst/src/main/java/com/vitorpamplona/amethyst/service/playback/composable/VideoView.kt
+5
View File
@@ -124,6 +124,11 @@ fun VideoView(
// DimensionTag uses reference equality, not structural.
val dimW = dimensions?.width
val dimH = dimensions?.height
// Deliberately snapshotted in a remember rather than observing MediaAspectRatioCache: when the
// ratio flips null -> known mid-playback this branch both adds an aspectRatio and emits an
// extra Spacer, and restructuring the children around a live AndroidView leaves the player's
// TextureView on a stale surface (the video redraws at native size in the corner). The
// enclosing box in ZoomableContentView is what sizes the player, and that one does observe.
val ratio =
remember(videoUri, dimW, dimH) {
if (dimW != null && dimH != null && dimW > 0 && dimH > 0) {
amethyst/src/main/java/com/vitorpamplona/amethyst/service/playback/playerPool/ExoPlayerPool.kt
+61
View File
@@ -37,6 +37,7 @@ import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.yield
import java.util.concurrent.ConcurrentLinkedQueue
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicInteger
@OptIn(UnstableApi::class)
class ExoPlayerPool(
@@ -71,6 +72,10 @@ class ExoPlayerPool(
private val warmPool = ArrayDeque<WarmPlayer>(warmSlotsCap.coerceAtLeast(1))
private val warmPoolLock = Any()
init {
livePools.add(this)
}
// Exists to avoid exceptions stopping the coroutine
val exceptionHandler =
CoroutineExceptionHandler { _, throwable ->
@@ -127,15 +132,53 @@ class ExoPlayerPool(
Log.d("PlaybackService") { "ExoPlayerPool discarding errored warm player: $preferredMediaId (${error.errorCodeName})" }
PcmTapRegistry.unregisterPlayer(warm)
warm.release()
liveDecoders.decrementAndGet()
} else {
Log.d("PlaybackService") { "ExoPlayerPool warm hit: $preferredMediaId" }
// Already counted against the decoder budget for as long as it sat warm.
return warm
}
}
}
ensureDecoderHeadroom()
liveDecoders.incrementAndGet()
return coldPool.poll() ?: builder.build(context)
}
/**
* Frees decoder headroom before a cold or freshly built player is handed out.
*
* Every player that still holds a prepared MediaItem — checked out or merely warm — owns a
* MediaCodec instance, and devices advertise a hard ceiling on those (the emulator's
* c2.goldfish.h264.decoder declares `concurrent-instances max="4"`). Past that ceiling
* MediaCodec.start() fails with NO_MEMORY and the video surfaces as "can't load", so the
* budget has to be enforced at acquisition rather than only at retention.
*
* Warm players are a scroll-back cache, so they are what gives way: demoting one to cold
* stop()s it and releases its codec. This pool's own entries go first, then any other pool's
* — [PlaybackService] keeps a separate pool for direct and for Tor-proxied traffic, and both
* draw on the one per-process pile of decoders.
*/
private fun ensureDecoderHeadroom() {
while (liveDecoders.get() >= poolSize) {
if (!evictOldestWarm() && !evictOldestWarmElsewhere()) return
}
}
private fun evictOldestWarm(): Boolean {
val oldest = synchronized(warmPoolLock) { warmPool.removeFirstOrNull() } ?: return false
Log.d("PlaybackService") { "ExoPlayerPool decoder-budget evict: ${oldest.mediaId}" }
demoteToCold(oldest.player)
return true
}
private fun evictOldestWarmElsewhere(): Boolean {
livePools.forEach { pool ->
if (pool !== this && pool.evictOldestWarm()) return true
}
return false
}
private fun takeWarm(mediaId: String): ExoPlayer? =
synchronized(warmPoolLock) {
// Iterate from the newest end so a duplicated URI returns the freshest player.
@@ -170,6 +213,7 @@ class ExoPlayerPool(
Log.d("PlaybackService") { "ExoPlayerPool dropping errored player: ${player.currentMediaItem?.mediaId} (${error.errorCodeName})" }
PcmTapRegistry.unregisterPlayer(player)
player.release()
liveDecoders.decrementAndGet()
return@withLock
}
@@ -214,7 +258,10 @@ class ExoPlayerPool(
private fun demoteToCold(player: ExoPlayer) {
if (player.isReleased) return
player.pause()
// stop() tears the renderers down, which is what actually hands the MediaCodec instance
// back to the system — so this is the point where the player stops costing budget.
player.stop()
liveDecoders.decrementAndGet()
player.clearVideoSurface()
player.clearMediaItems()
@@ -260,6 +307,7 @@ class ExoPlayerPool(
}
fun destroy() {
livePools.remove(this)
scope
.launch {
mutex.withLock {
@@ -272,6 +320,7 @@ class ExoPlayerPool(
warmSnapshot.forEach {
PcmTapRegistry.unregisterPlayer(it.player)
it.player.release()
liveDecoders.decrementAndGet()
}
coldPool.forEach {
PcmTapRegistry.unregisterPlayer(it)
@@ -286,5 +335,17 @@ class ExoPlayerPool(
companion object {
private const val DEFAULT_WARM_SLOTS = 3
// MediaCodec instances are a per-process resource, but PlaybackService builds one pool for
// direct traffic and another for Tor-proxied traffic, so a per-pool budget would let the
// app hold twice the device's decoder ceiling. Both counters below are therefore global.
// Players currently holding a decoder: checked out, or warm (paused but still prepared).
// Cold players have been stop()'d and own none.
private val liveDecoders = AtomicInteger(0)
// Every pool that hasn't been destroy()'d, so a pool starved of headroom can reclaim a
// warm player from a sibling instead of overshooting the shared ceiling.
private val livePools = ConcurrentLinkedQueue<ExoPlayerPool>()
}
}
amethyst/src/main/java/com/vitorpamplona/amethyst/service/playback/playerPool/MediaSessionPool.kt
+9 -1
View File
@@ -64,6 +64,10 @@ class MediaSessionPool(
val exoPlayerPool: ExoPlayerPool,
val dataSourceFactory: DataSource.Factory,
val appContext: Context,
// Ceiling on cached sessions. Each one holds a checked-out ExoPlayer, so on a device whose
// decoder ceiling is lower than [MAX_CACHED_SESSIONS] this is what keeps the session cache
// from pinning more MediaCodec instances than the hardware will grant.
maxSessions: Int = MAX_CACHED_SESSIONS,
val reset: (MediaSession, Boolean) -> Unit,
) {
private val exceptionHandler =
@@ -123,7 +127,7 @@ class MediaSessionPool(
private val playingMap = mutableMapOf<String, SessionListener>()
private val cache =
object : LruCache<String, SessionListener>(10) { // up to 10 videos in the screen at the same time
object : LruCache<String, SessionListener>(maxSessions.coerceIn(1, MAX_CACHED_SESSIONS)) {
override fun entryRemoved(
evicted: Boolean,
key: String?,
@@ -296,6 +300,10 @@ class MediaSessionPool(
companion object {
private val CLEANUP_INTERVAL_NS = TimeUnit.MINUTES.toNanos(1)
// Roughly how many videos can share a screen at once. Acts as the upper bound only —
// a device that advertises fewer concurrent decoders than this caps lower.
const val MAX_CACHED_SESSIONS = 10
// AOSP default for config_mediaMetadataBitmapMaxSize, used when the framework resource
// can't be resolved by name on a given ROM.
private const val DEFAULT_METADATA_BITMAP_DP = 320
amethyst/src/main/java/com/vitorpamplona/amethyst/service/playback/service/PlaybackService.kt
+7 -1
View File
@@ -80,14 +80,20 @@ class PlaybackService : MediaSessionService() {
},
)
// The device's concurrent-decoder ceiling bounds both how many players may be checked out
// at once (the session cache) and how many the pool may retain, since a session and a warm
// pool entry each pin one MediaCodec instance.
val decoderBudget = SimultaneousPlaybackCalculator.max(applicationContext)
return MediaSessionPool(
exoPlayerPool =
ExoPlayerPool(
ExoPlayerBuilder(videoCache, resolvingDataSourceFactory),
poolSize = SimultaneousPlaybackCalculator.max(applicationContext),
poolSize = decoderBudget,
),
dataSourceFactory = resolvingDataSourceFactory,
appContext = applicationContext,
maxSessions = decoderBudget,
reset = { session, keepPlaying ->
(session.player as ExoPlayer).apply {
repeatMode = if (keepPlaying) Player.REPEAT_MODE_ONE else Player.REPEAT_MODE_OFF
amethyst/src/main/java/com/vitorpamplona/amethyst/service/relayClient/reqCommand/event/loaders/AddressableAuthorRelayLoaderSubAssembler.kt
+48 -9
View File
@@ -21,11 +21,14 @@
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.loaders
import com.vitorpamplona.amethyst.commons.relayClient.eoseManagers.IEoseManager
import com.vitorpamplona.amethyst.commons.service.BundledUpdate
import com.vitorpamplona.amethyst.model.AddressableNote
import com.vitorpamplona.amethyst.model.LocalCache
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.EventFinderQueryState
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.user.UserFinderFilterAssembler
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.user.UserFinderQueryState
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.IO
/**
* Bridges missing-addressable-note authors into [UserFinderFilterAssembler].
@@ -42,9 +45,24 @@ class AddressableAuthorRelayLoaderSubAssembler(
val allKeys: () -> Set<EventFinderQueryState>,
val userFinder: UserFinderFilterAssembler,
) : IEoseManager {
private val activeSubscriptions = mutableSetOf<UserFinderQueryState>()
// Private monitor: @Synchronized locks on `this`, which leaves the instance's monitor
// reachable to anything holding a reference to this assembler.
private val lock = Any()
// Only ever touched while holding [lock]. See commit() and destroy().
private var activeSubscriptions: Set<UserFinderQueryState> = emptySet()
private var destroyed = false
// Keeps the scan off the caller's thread. invalidateFilters() is reached synchronously from
// ComposeSubscriptionManager.subscribe/unsubscribe on every note composable mount/unmount,
// and those are documented "called by main. Keep it really fast."
private val bundler = BundledUpdate(500, Dispatchers.IO)
override fun invalidateFilters(ignoreIfDoing: Boolean) {
bundler.invalidate(ignoreIfDoing, ::forceInvalidate)
}
private fun forceInvalidate() {
val needed = mutableSetOf<UserFinderQueryState>()
allKeys().forEach { key ->
@@ -57,18 +75,39 @@ class AddressableAuthorRelayLoaderSubAssembler(
}
}
val toAdd = needed - activeSubscriptions
val toRemove = activeSubscriptions - needed
commit(needed)
}
userFinder.subscribe(toAdd.toList())
userFinder.unsubscribe(toRemove.toList())
/**
* Serializes against [destroy] — the one caller the bundler cannot order, because
* `bundler.cancel()` cannot stop a body that is already running (it has no suspension points).
*
* The scan in [forceInvalidate] stays outside [lock], so [destroy] never waits on a
* [LocalCache] sweep. It can still wait on the two calls below, which are bounded: a pair of
* map updates inside [UserFinderFilterAssembler] plus the coroutine launches its
* `invalidateKeys()` fans out to.
*
* Calling [userFinder] while holding [lock] relies on subscribe/unsubscribe only taking
* ComposeSubscriptionManager's own lock and deferring real work to bundled coroutines — they
* never call back into this class. Revisit if that changes.
*/
private fun commit(needed: Set<UserFinderQueryState>) {
synchronized(lock) {
if (destroyed) return
activeSubscriptions.clear()
activeSubscriptions.addAll(needed)
userFinder.subscribe((needed - activeSubscriptions).toList())
userFinder.unsubscribe((activeSubscriptions - needed).toList())
activeSubscriptions = needed
}
}
override fun destroy() {
userFinder.unsubscribe(activeSubscriptions.toList())
activeSubscriptions.clear()
synchronized(lock) {
destroyed = true
bundler.cancel()
userFinder.unsubscribe(activeSubscriptions.toList())
activeSubscriptions = emptySet()
}
}
}
amethyst/src/main/java/com/vitorpamplona/amethyst/ui/components/ZoomableContentView.kt
+16 -2
View File
@@ -136,6 +136,14 @@ import java.io.IOException
// Allows time for receiving app to copy the file after user confirms share.
private const val SHARED_VIDEO_CLEANUP_DELAY_MS = 120_000L
// Assumed shape of a video whose dimensions nobody has reported yet — no imeta `dim` and nothing
// cached, which is the norm for a NIP-53 live stream on its first play. Without a ratio the sizing
// modifier leaves height unconstrained, so the player stretches to whatever ceiling encloses it
// (300.dp on the live-stream screen) and letterboxes the real frame inside, leaving black bars top
// and bottom. Guessing the overwhelmingly common video shape puts the first layout in the right
// place; [MediaAspectRatioCache] then corrects anything unusual once the decoder reports its size.
private const val DEFAULT_VIDEO_ASPECT_RATIO = 16f / 9f
@Composable
fun ZoomableContentView(
content: BaseMediaContent,
@@ -195,7 +203,7 @@ fun ZoomableContentView(
}
is MediaUrlVideo -> {
val ratio = content.dim?.aspectRatio() ?: MediaAspectRatioCache.get(content.url)
val ratio = content.dim?.aspectRatio() ?: MediaAspectRatioCache.get(content.url) ?: DEFAULT_VIDEO_ASPECT_RATIO
val bridgedUrl =
remember(content.url, useLocalBlossomBridge) {
content.toCoilModel(useLocalBlossomBridge)
@@ -209,7 +217,13 @@ fun ZoomableContentView(
backdrop = (content.thumbhash ?: content.blurhash)?.let { { BlurhashBackdrop(content.blurhash, content.description, content.thumbhash) } },
) {
Box(
modifier = Modifier.fillMaxWidth().then(boundsTrackingModifier),
// The sizing modifier is repeated here because ContentWarningGate only applies
// the one it is handed when the content is actually sensitive — the common
// non-sensitive path emits content() bare. Without a height constraint of its
// own this box stretches to whatever ceiling encloses it and the player
// letterboxes the frame inside, which is what put black bars above and below
// live streams (their enclosure is StreamingHeaderModifier's 300.dp cap).
modifier = mediaSizingModifier(ratio, contentScale).then(boundsTrackingModifier),
contentAlignment = Alignment.Center,
) {
VideoView(
amethyst/src/main/java/com/vitorpamplona/amethyst/ui/screen/loggedIn/AccountFeedContentStates.kt
+22
View File
@@ -77,7 +77,9 @@ import com.vitorpamplona.amethyst.ui.screen.loggedIn.webBookmarks.dal.WebBookmar
import com.vitorpamplona.amethyst.ui.screen.loggedIn.workouts.dal.WorkoutFeedFilter
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.flow.drop
import kotlinx.coroutines.flow.sample
import kotlinx.coroutines.launch
class AccountFeedContentStates(
@@ -201,6 +203,26 @@ class AccountFeedContentStates(
}
}
// A Concord control-plane fold is what first reveals a community's channels (and what makes
// ConcordCommunitySession.state non-null, without which ChatroomListKnownFeedFilter emits
// nothing at all for that community). None of it flows through LocalCache.newEventBundles,
// so the additive path can't see it: a folded channel reaches the Messages tab only if a
// message for it happens to arrive afterwards. Cold boot therefore shows a *subset* of a
// community's channels, or omits a quiet community entirely, until some unrelated
// invalidation fires. Rebuild on every structural change instead. `revision` bumps only on
// fold/membership/rekey (never a plain message), and sample() coalesces the burst of folds
// that lands as each control plane catches up — the same pairing Account.kt uses to drive
// refreshConcordChannelIndex off this flow.
scope.launch(Dispatchers.IO) {
@OptIn(FlowPreview::class)
account.concordSessions.revision
.drop(1)
.sample(500)
.collect {
dmKnown.invalidateData()
}
}
// Same for the Concord view mode (inline channels vs one row per community).
scope.launch(Dispatchers.IO) {
account.settings.concordViewMode
amethyst/src/main/res/CLAUDE.md
+3
View File
@@ -11,8 +11,11 @@ Always consider Slavic / Baltic / Semitic / Celtic languages when a string conta
- English / German / Swedish / Brazilian Portuguese / Hungarian: `one`, `other`
- Czech / Polish / Russian / Ukrainian / Croatian: `one`, `few`, `many`, `other`
- Arabic: `zero`, `one`, `two`, `few`, `many`, `other`
- Latvian: `zero`, `one`, `other`
- Chinese / Japanese: `other` only
Latvian's `zero` does **not** mean "no items" — it covers 0, 10, 11–19, 20, 30, … (`n % 10 = 0` or `n % 100 = 11..19`), so it fires on most counts and must read as a normal plural form. Never strip `<item quantity="zero">` from `values-lv-rLV`; among the locales we ship, only Arabic and Latvian have an integer-bearing `zero`.
## Anti-patterns to flag
When adding or reviewing strings, flag these:
amethyst/src/main/res/values-cs/strings.xml
+1
View File
@@ -2125,6 +2125,7 @@
<string name="default_relays_longer">Obnovit výchozí nastavení</string>
<string name="geohash_title">Zveřejnit polohu jako </string>
<string name="geohash_explainer">Přidá Geohash vaší polohy do příspěvku. Veřejnost bude vědět, že se nacházíte do 5 km od aktuální polohy</string>
<string name="geohash_teleport_title">Teleportace</string>
<string name="geohash_teleport_action">✈ Teleportovat sem</string>
<string name="location_pick_on_map">Vyberte místo na mapě</string>
<string name="location_change_place">Změnit místo na mapě</string>
amethyst/src/main/res/values-hi-rIN/strings.xml
+9
View File
@@ -503,6 +503,15 @@
<string name="share_as_image_url">चित्र योजक के रूप में बाँटें</string>
<string name="share_as_image_generating">पूर्वीक्षण चित्र उत्पादन चालू…</string>
<string name="share_as_image_watermark">अमेथिस्ट द्वारा बाँटा गया</string>
<string name="share_as_qr">क्यूआर॰ चित्र के रूप में बाँटें</string>
<string name="share_as_qr_mode_web">जाल योजक</string>
<string name="share_as_qr_mode_nostr">नोस्टर योजक</string>
<string name="share_as_qr_hint_web">किसी भी संचारयन्त्र चित्रग्राहक के साथ परखें</string>
<string name="share_as_qr_hint_nostr">नोस्टर क्रमक के साथ परखें</string>
<string name="share_as_qr_kind_picture">चित्र</string>
<string name="share_as_qr_code_description_web">क्यूआर॰ चित्र जिसमें इस टीका का एक जाल योजक समाविष्ट है</string>
<string name="share_as_qr_code_description_nostr">क्यूआर॰ चित्र जिसमें इस टीका का एक नोस्टर योजक समाविष्ट है</string>
<string name="share_as_qr_thumbnail_hidden_sensitive">अंगुलचित्र छिपाया गया संवेदनशिल विषयवस्तु के कारण</string>
<string name="quick_action_copy_user_id">लेखक विभेदक</string>
<string name="quick_action_copy_note_id">टीका विभेदक</string>
<string name="quick_action_copy_text">लेख की अनुकृति करें</string>
amethyst/src/main/res/values-pl-rPL/strings.xml
+45
View File
@@ -32,6 +32,7 @@
<string name="referenced_event_not_found">Przywołane zdarzenie nie zostało znalezione</string>
<string name="could_not_decrypt_the_message">Nie można odszyfrować wiadomości</string>
<!-- Placeholder shown in a chat row while an encrypted message is still being decrypted -->
<string name="chat_preview_decrypting">Rozszyfrowywanie…</string>
<string name="group_picture">Zdjęcie grupy</string>
<string name="explicit_content">Niedozwolona zawartość</string>
<string name="relay_notice">Uwagi transmitera</string>
@@ -308,6 +309,10 @@
<string name="concord_invite_failed_invalid">Ten link do zaproszenia jest nieprawidłowy lub nie można go otworzyć przy użyciu tego konta.</string>
<string name="concord_invite_failed_incompatible">Nie można otworzyć tego linku do zaproszenia. Być może jest on nieaktualny, został już zastąpiony nowszym lub został utworzony w nowszej wersji aplikacji. Poproś o nowy link do zaproszenia.</string>
<string name="concord_invite_failed_revoked">Ten link do zaproszenia został unieważniony i nie można już z niego korzystać. Poproś o nowy.</string>
<string name="concord_invite_failed_expired">Ten link do zaproszenia wygasł i nie może być już używany. Poproś o nowy link.</string>
<string name="concord_invite_preview_unknown_name">Nazwa społeczności zostanie ujawniona dopiero po dołączeniu</string>
<string name="concord_invite_preview_explainer">Dołączenie do transmiterów tego zaproszenia, publikuje ogłoszenie o dołączeniu podpisane przez twoje konto i dodaje społeczność do listy. Nic nie zostanie wysłane, dopóki nie klikniesz Dołącz.</string>
<string name="concord_invite_preview_relays">Transmiter tego zaproszenia skontaktuje się: %1$s</string>
<string name="concord_home_title">Kanały Concord</string>
<string name="concord_home_empty">Nie dołączyłeś(aś) jeszcze do kanału Concord. Utwórz kanał lub otwórz link z zaproszeniem.</string>
<string name="concord_channels_empty">Brak kanałów.</string>
@@ -323,6 +328,10 @@
<string name="concord_channel_delete_title">Usunąć kanał?</string>
<string name="concord_channel_delete_message">Usunąć #%1$s? Tej operacji nie można cofnąć i nie będzie można odtworzyć kanału z tym samym identyfikatorem.</string>
<string name="concord_channel_delete_confirm">Usuń</string>
<string name="concord_leave_community">Opuść społeczność</string>
<string name="concord_leave_title">Opuścić społeczność?</string>
<string name="concord_leave_message">Chcesz opuścić %1$s? Zostaniesz usunięty z listy członków tej społeczności, a synchronizacja z Twoimi urządzeniami zostanie wstrzymana. Społeczność nie zostanie o tym powiadomiona, a Ty nie zostaniesz usunięty z listy jej członków. Wiadomości, których nie będziesz już mógł odszyfrować, mogą okazać się nie do odzyskania, a powrót do społeczności będzie możliwy wyłącznie po otrzymaniu nowego zaproszenia.</string>
<string name="concord_leave_owner_warning">To Ty stworzyłeś tę społeczność. Odejście nie powoduje jej usunięcia ani przekazania komukolwiek innemu, ale powoduje usunięcie klucza właściciela przechowywanego na Twojej liście — nie będziesz mógł już nią zarządzać.</string>
<string name="concord_edit_relays_desc">Miejsce, w którym publikowane i czytane są zaszyfrowane plany tej społeczności.</string>
<string name="concord_typing_one">%1$s pisze…</string>
<string name="concord_typing_two">%1$s i %2$s piszą…</string>
@@ -373,6 +382,13 @@
<string name="concord_members_remove_title">Usunąć członka?</string>
<string name="concord_members_remove_message">Spowoduje to zmianę klucza szyfrującego społeczności, przez co ten użytkownik nie będzie już mógł odczytać żadnych wiadomości wysłanych po tej zmianie. Klucze pozostałych użytkowników zostaną automatycznie zaktualizowane. Czynności tej nie można cofnąć.</string>
<string name="concord_members_remove_confirm">Usuń</string>
<string name="concord_members_roles">Role…</string>
<string name="concord_members_roles_title">Przypisz rolę</string>
<string name="concord_members_roles_message">Wybierz każdą rolę, jaką powinien pełnić ten członek. Odznaczenie roli usuwa ją.</string>
<string name="concord_members_roles_save">Zapisz</string>
<string name="concord_members_roles_out_of_reach">Nie masz wyższej rangi od tego członka</string>
<string name="concord_members_roles_none_assignable">Brak ról, które można przydzielić</string>
<string name="concord_members_roles_failed">Nie udało się zaktualizować ról tego członka.</string>
<string name="concord_role_owner">Właściciel</string>
<string name="concord_role_admin">Admin</string>
<string name="concord_role_banned">Zbanowany</string>
@@ -498,6 +514,15 @@
<string name="share_as_image_url">Udostępnij jako adres Url obrazu</string>
<string name="share_as_image_generating">Generowanie podglądu…</string>
<string name="share_as_image_watermark">Udostępnione przez Ametyst</string>
<string name="share_as_qr">Udostępnij jako QR</string>
<string name="share_as_qr_mode_web">Link do strony internetowej</string>
<string name="share_as_qr_mode_nostr">Nostr link</string>
<string name="share_as_qr_hint_web">Skanuj za pomocą dowolnej kamery telefonu</string>
<string name="share_as_qr_hint_nostr">Zeskanuj za pomocą aplikacji Nostr</string>
<string name="share_as_qr_kind_picture">Zdjęcie</string>
<string name="share_as_qr_code_description_web">Kod QR zawierający link do tej notatki</string>
<string name="share_as_qr_code_description_nostr">Kod QR zawierający link Nostr do tej notatki</string>
<string name="share_as_qr_thumbnail_hidden_sensitive">Miniaturka ukryta ze względu na wrażliwą treść</string>
<string name="quick_action_copy_user_id">ID autora</string>
<string name="quick_action_copy_note_id">ID wpisu</string>
<string name="quick_action_copy_text">Skopiuj tekst</string>
@@ -834,6 +859,7 @@ Zaplanowane posty z innych kont nie zostaną opublikowane, dopóki to konto jest
<string name="napplet_consent_storage">Ta aplikacja nApplet chce korzystać ze swojego prywatnego schowka.</string>
<string name="napplet_consent_pay">Ta aplikacja nApplet chce zapłacić fakturę w systemie Lightning.</string>
<!-- An amountless BOLT11: the payee decides how much. Never render this as "0 sats". -->
<string name="napplet_consent_pay_no_amount">⚠ Ta aplikacja nApplet ma na celu opłacenie faktury Lightning, w której NIE podano kwoty — to odbiorca płatności decyduje, jaka kwota zostanie pobrana. Zezwól na to tylko wtedy, gdy ufasz tej aplikacji.</string>
<string name="napplet_consent_resource">Ten nApplet chce pobrać zasób internetowy.</string>
<string name="napplet_consent_upload">Ten nApplet chce przesłać plik na Twój serwer multimedialny.</string>
<string name="napplet_consent_notify">Ten nApplet chce pokazywać Ci powiadomienia.</string>
@@ -848,6 +874,18 @@ Zaplanowane posty z innych kont nie zostaną opublikowane, dopóki to konto jest
kind-only summary would hide what is actually being signed. These replaceable lists are
already cached on the account, so the dialog diffs the proposed list against the current
one and reports what actually changes rather than a raw total. -->
<plurals name="napplet_consent_diff_follow_added">
<item quantity="one">obserwuje %1$d nowe konto</item>
<item quantity="few">obserwuje %1$d nowych kont</item>
<item quantity="many">obserwuje %1$d nowych kont</item>
<item quantity="other">obserwuje %1$d nowe konta</item>
</plurals>
<plurals name="napplet_consent_diff_follow_removed">
<item quantity="one">Przestaje obserwować %1$d konto</item>
<item quantity="few">Przestaje obserwować %1$d kont</item>
<item quantity="many">Przestaje obserwować %1$d kont</item>
<item quantity="other">Przestaje obserwować %1$d konta</item>
</plurals>
<!-- Single-account edits, by far the common case: name who it is instead of counting. %1$s is
the display name, shown next to their avatar. -->
<!-- %1$s is the joined change list, e.g. "follows 2 new accounts and UNFOLLOWS 130 accounts". -->
@@ -889,9 +927,13 @@ Zaplanowane posty z innych kont nie zostaną opublikowane, dopóki to konto jest
<!-- Decrypt: the message is already decrypted by Amethyst; the permission is to expose it to the app -->
<string name="napplet_op_decrypt">odczytaj prywatne wiadomości</string>
<!-- Decrypt scoped to one counterparty. %1$s is that person's name (or a shortened npub) -->
<string name="napplet_op_decrypt_from">przeczytaj swoje prywatne wiadomości za pomocą %1$s</string>
<!-- Narrower "remember" choice offered next to "Always allow". %1$s is the counterparty's name -->
<string name="nip46_signer_allow_always_for">Zawsze zezwalaj na %1$s</string>
<!-- Shown as the preview when Amethyst itself cannot decrypt the message the app asked to read -->
<string name="nip46_signer_decrypt_failed">Amethyst nie zdołała odszyfrować tej wiadomości. Być może nie jest ona adresowana do tego konta.</string>
<!-- Label above the counterparty avatar in the decrypt consent dialog -->
<string name="nip46_signer_messages_with">Wiadomości z</string>
<!-- Permissions management screen -->
<string name="napplet_permissions_title">Podłączone aplikacje</string>
<string name="napplet_permissions_revoke_all">Cofnij wszystkie uprawnienia</string>
@@ -1559,6 +1601,7 @@ Zaplanowane posty z innych kont nie zostaną opublikowane, dopóki to konto jest
<string name="blossom_payment_required">Ten serwer wymaga płatności do wysłania: %1$s</string>
<string name="blossom_payment_title">Wymagana płatność</string>
<string name="blossom_payment_message">%1$s pobiera opłatę w systemie Lightning za zapisanie tego pliku. Aby kontynuować, dokonaj płatności z podłączonego portfela.</string>
<string name="blossom_payment_server_says">%1$s powiedział: „%2$s”</string>
<string name="blossom_pay">Zapłać</string>
<plurals name="blossom_pay_sats">
<item quantity="one">Zapłać %1$d sat</item>
@@ -2392,6 +2435,8 @@ Zaplanowane posty z innych kont nie zostaną opublikowane, dopóki to konto jest
<string name="error_dialog_pay_withdraw_error">Nie można wypłacić</string>
<string name="cashu_failed_redemption">Nie można wykupić Cashu</string>
<string name="cashu_failed_redemption_explainer_error_msg">Mint dostarczył następujący komunikat błędu: %1$s</string>
<string name="cashu_unsafe_mint_url">Niebezpieczny adres Cashu mint</string>
<string name="cashu_unsafe_mint_url_explainer">Amethyst nie skontaktował się z emitentem tego tokena. %1$s</string>
<string name="cashu_successful_redemption">Cashu odebrano</string>
<string name="cashu_successful_redemption_explainer">%1$s satsy zostały wysłane do Twojego portfela. (opłata: %2$s satoszy)</string>
<string name="cashu_no_wallet_found">W systemie nie znaleziono kompatybilnego portfela Cashu</string>
amethyst/src/test/java/com/vitorpamplona/amethyst/service/relayClient/reqCommand/event/loaders/AddressableAuthorRelayLoaderSubAssemblerTest.kt
+170
View File
@@ -0,0 +1,170 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.loaders
import com.vitorpamplona.amethyst.model.Account
import com.vitorpamplona.amethyst.model.LocalCache
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.event.EventFinderQueryState
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.user.UserFinderFilterAssembler
import com.vitorpamplona.amethyst.service.relayClient.reqCommand.user.UserFinderQueryState
import com.vitorpamplona.quartz.nip01Core.core.Address
import io.mockk.every
import io.mockk.mockk
import io.mockk.verify
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
import java.util.concurrent.CountDownLatch
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicInteger
import kotlin.concurrent.thread
class AddressableAuthorRelayLoaderSubAssemblerTest {
/**
* Unique per-test-class identities so the shared [LocalCache] singleton isn't polluted with
* notes another test class also claims.
*/
private fun stubKeys(count: Int): Set<EventFinderQueryState> {
val account = mockk<Account>()
return (1..count).mapTo(mutableSetOf()) { i ->
val address = Address(30023, "ad04%060x".format(i), "d$i")
EventFinderQueryState(LocalCache.getOrCreateAddressableNoteInternal(address), account)
}
}
/**
* Regression test for the `ConcurrentModificationException` in `SetsKt.minus` reported from
* `DefaultDispatcher-worker-70`: this manager is genuinely re-entered from several threads at
* once, because `ComposeSubscriptionManager.subscribe`/`unsubscribe` call `invalidateKeys()`
* *after* releasing their own lock, and `LifecycleAwareSubscription`'s 30s grace-period
* unsubscribe fires on a `Dispatchers.Default` worker.
*
* Overlap is detected through the injected `allKeys()` lambda rather than by catching — once
* the body is bundled its throwables are swallowed by `BundledUpdate`'s
* `CoroutineExceptionHandler` and would never reach the test thread.
*/
@Test
fun concurrentInvalidateFiltersNeverOverlap() {
val userFinder = mockk<UserFinderFilterAssembler>(relaxed = true)
val keys = stubKeys(50)
val inFlight = AtomicInteger(0)
val overlaps = AtomicInteger(0)
val assembler =
AddressableAuthorRelayLoaderSubAssembler(
LocalCache,
{
if (inFlight.incrementAndGet() > 1) overlaps.incrementAndGet()
try {
keys
} finally {
inFlight.decrementAndGet()
}
},
userFinder,
)
val start = CountDownLatch(1)
try {
val threads =
(1..8).map {
thread(start = false) {
start.await()
repeat(500) { assembler.invalidateFilters() }
}
}
threads.forEach { it.start() }
start.countDown()
threads.forEach { it.join() }
} finally {
assembler.destroy()
}
assertEquals("forceInvalidate bodies overlapped", 0, overlaps.get())
}
/** The manager still does its job: unresolved stub authors reach the user finder. */
@Test
fun bridgesMissingAuthorsIntoUserFinder() {
val userFinder = mockk<UserFinderFilterAssembler>(relaxed = true)
val keys = stubKeys(3)
val assembler = AddressableAuthorRelayLoaderSubAssembler(LocalCache, { keys }, userFinder)
try {
assembler.invalidateFilters()
verify(timeout = 3000) {
userFinder.subscribe(
match<List<UserFinderQueryState>> { it.size == 3 },
)
}
} finally {
assembler.destroy()
}
}
/**
* `destroy()` must win against a body that is already past its `allKeys()` scan: otherwise the
* body re-subscribes authors `destroy()` just released, leaving live kind-0/10002 REQs (and
* retained `User`/`Account` references) for a dead account after logout.
*
* The body is gated inside the injected `allKeys()` lambda so `destroy()` provably runs
* underneath an in-flight run rather than racing it by luck.
*/
@Test
fun destroyDuringInFlightInvalidateDoesNotLeakSubscriptions() {
val userFinder = mockk<UserFinderFilterAssembler>(relaxed = true)
val subscribeHappened = CountDownLatch(1)
every { userFinder.subscribe(any<List<UserFinderQueryState>>()) } answers {
subscribeHappened.countDown()
}
val keys = stubKeys(3)
val invalidateEntered = CountDownLatch(1)
val destroyFinished = CountDownLatch(1)
val assembler =
AddressableAuthorRelayLoaderSubAssembler(
LocalCache,
{
invalidateEntered.countDown()
destroyFinished.await(5, TimeUnit.SECONDS)
keys
},
userFinder,
)
try {
assembler.invalidateFilters()
assertTrue("bundled run never started", invalidateEntered.await(5, TimeUnit.SECONDS))
} finally {
assembler.destroy()
destroyFinished.countDown()
}
// The gated body resumes the instant destroyFinished counts down, so a leak shows up
// immediately; the wait only has to outlast that hand-off.
assertFalse(
"in-flight body subscribed after destroy()",
subscribeHappened.await(500, TimeUnit.MILLISECONDS),
)
}
}
commons/src/commonMain/composeResources/values-pt-rBR/strings.xml
+1
View File
@@ -65,6 +65,7 @@
<!-- Notes & Replies -->
<string name="replying_to">respondendo para </string>
<!-- Static sites (NIP-5A) & napplets (NIP-5D) feed card -->
<string name="nsite_title">Site Estático: %1$s</string>
<string name="napplet_card_permissions">O que ele pode acessar</string>
<string name="nsite_root_site">Site raiz</string>
<string name="nsite_source">Origem:</string>
commons/src/commonMain/composeResources/values-sv-rSE/strings.xml
+1
View File
@@ -69,6 +69,7 @@
<!-- Notes & Replies -->
<string name="replying_to">Svarar till </string>
<!-- Static sites (NIP-5A) & napplets (NIP-5D) feed card -->
<string name="nsite_title">Statisk webbplats: %1$s</string>
<string name="napplet_card_permissions">Vad det har åtkomst till</string>
<string name="nsite_root_site">Rotplats</string>
<string name="nsite_source">Källa:</string>
commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/model/concord/ConcordCommunitySession.kt
+50 -7
View File
@@ -62,8 +62,17 @@ enum class ConcordIngestOutcome {
* a chat/reaction/reply/delete message landing, or a duplicate wrap. Must NOT bump the revision. */
NON_STRUCTURAL,
/** Ours and changed structure: a Control-Plane fold (metadata/channels/membership/authority), a
* guestbook membership change, or a buffered base-rekey. Bumps the revision. */
/** Ours and re-folded the Control Plane. The fold republishes [ConcordCommunitySession.state],
* so the session's own state watcher is what bumps the revision — and, because the folded state
* compares by value, only when the fold actually *changed* something. A control wrap that folds
* to an identical state (a prior-epoch wrap that doesn't move the anti-rollback floor, a role
* edition that touches nothing we subscribe on) therefore costs no bump at all. The manager must
* NOT bump on this outcome as well, or every control wrap counts twice. */
STRUCTURAL_FOLD,
/** Ours and changed structure *without* touching [ConcordCommunitySession.state]: a guestbook
* membership change (which republishes `members`) or a buffered base-rekey. No state watcher
* covers these, so the manager bumps the revision directly. */
STRUCTURAL,
;
@@ -145,6 +154,22 @@ class ConcordCommunitySession(
// Deduped inbound wraps.
private val controlWraps = LinkedHashMap<HexKey, Event>()
/**
* Decrypted control editions memoized by wrap id.
*
* Both [refold] and [controlFloorsLocked] fold their WHOLE buffer on every inbound control
* wrap, and turning a wrap into an edition is a NIP-44 open + parse. Re-deriving them each
* time made a cold-boot backfill quadratic in decryptions — one measured boot did ~8.6k opens
* to ingest 93 control wraps for a single community. Memoizing makes it one open per wrap.
*
* Wrap ids are unique and a wrap only ever belongs to one plane (it is routed by `pubKey`), so
* a single id-keyed map is safe across the current and prior-epoch Control Planes even though
* they open under different keys. A wrap that fails to open caches `null` so it is not retried
* on every subsequent fold. The wrap buffers are only ever added to, so this tracks their
* lifetime exactly and needs no separate eviction.
*/
private val editionByWrapId = HashMap<HexKey, ControlEdition?>()
// Prior-epoch Control Plane address -> (wrapId -> wrap). Kept apart from [controlWraps]: these
// never join the live fold, they only produce the anti-rollback floor.
private val historicalControlWraps = HashMap<HexKey, LinkedHashMap<HexKey, Event>>()
@@ -280,7 +305,7 @@ class ConcordCommunitySession(
fun auxStreamKeys(): List<GroupKey> = listOf(guestbookKey, nextBaseRekeyKey)
/** The community's current Control Plane editions — the input a moderation edition chains onto. */
fun controlEditions(): List<ControlEdition> = lock.withLock { ConcordActions.controlEditions(controlWraps.values.toList(), controlPlaneKey) }
fun controlEditions(): List<ControlEdition> = lock.withLock { editionsLocked(controlWraps.values.toList(), controlPlaneKey) }
/** The raw Control Plane wraps buffered so far — the input a Refounding compacts (CORD-06 §3). */
fun controlPlaneWraps(): List<Event> = lock.withLock { controlWraps.values.toList() }
@@ -314,7 +339,7 @@ class ConcordCommunitySession(
if (controlWraps.put(wrap.id, wrap) != null) return ConcordIngestOutcome.NON_STRUCTURAL // dup
}
refold()
return ConcordIngestOutcome.STRUCTURAL
return ConcordIngestOutcome.STRUCTURAL_FOLD
}
guestbookAddress -> {
lock.withLock {
@@ -341,7 +366,7 @@ class ConcordCommunitySession(
if (buffer.put(wrap.id, wrap) != null) return ConcordIngestOutcome.NON_STRUCTURAL // dup
}
refold()
return ConcordIngestOutcome.STRUCTURAL
return ConcordIngestOutcome.STRUCTURAL_FOLD
}
val current = lock.withLock { channelKeysByAddress[wrap.pubKey] }
if (current != null) {
@@ -422,7 +447,7 @@ class ConcordCommunitySession(
val wraps = controlWraps.values.toList()
val folded =
ConcordCommunityState.fold(
ConcordActions.controlEditions(wraps, controlPlaneKey),
editionsLocked(wraps, controlPlaneKey),
entry.owner,
controlFloorsLocked(),
)
@@ -455,6 +480,24 @@ class ConcordCommunitySession(
for (channelIdHex in newChannels) reprojectChannel(channelIdHex)
}
/**
* [wraps] opened into editions through [editionByWrapId], so a wrap is only ever decrypted
* once no matter how many folds it participates in. Caller must hold [lock].
*/
private fun editionsLocked(
wraps: Collection<Event>,
planeKey: GroupKey,
): List<ControlEdition> =
wraps.mapNotNull { wrap ->
if (editionByWrapId.containsKey(wrap.id)) {
editionByWrapId[wrap.id]
} else {
val edition = ConcordStreamEnvelope.openOrNull(wrap, planeKey)?.let { ControlEdition.fromRumor(it.rumor) }
editionByWrapId[wrap.id] = edition
edition
}
}
/**
* The per-entity anti-rollback floor: the authority-gated heads of every prior epoch's
* Control Plane we still hold a root for, folded **oldest epoch first** so each epoch is
@@ -472,7 +515,7 @@ class ConcordCommunitySession(
var floors = emptyMap<String, EntityFloor>()
for ((address, keyAtEpoch) in historicalControlKeys.entries.sortedBy { it.value.second }) {
val wraps = historicalControlWraps[address]?.values?.toList() ?: continue
val editions = ConcordActions.controlEditions(wraps, keyAtEpoch.first)
val editions = editionsLocked(wraps, keyAtEpoch.first)
if (editions.isEmpty()) continue
floors = ConcordCommunityState.authorizedHeads(editions, entry.owner, floors)
}
commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/model/concord/ConcordSessionManager.kt
+3
View File
@@ -150,6 +150,9 @@ class ConcordSessionManager(
seenOnRelays: Set<NormalizedRelayUrl> = emptySet(),
): Boolean {
val outcome = registry.ingest(wrap, seenOnRelays)
// Only the planes that change structure *without* republishing `state`. A control-plane
// fold returns STRUCTURAL_FOLD and is bumped by the per-session state watcher instead, which
// (since the folded state compares by value) fires only when the fold genuinely changed.
if (outcome == ConcordIngestOutcome.STRUCTURAL) bumpRevision()
return outcome.claimed
}
commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/relayClient/eoseManagers/IEoseManager.kt
+6
View File
@@ -21,6 +21,12 @@
package com.vitorpamplona.amethyst.commons.relayClient.eoseManagers
interface IEoseManager {
/**
* May be called from any thread, concurrently with itself and with [destroy], and is reached
* synchronously from main on every composable mount/unmount. Implementations must return fast
* and must serialize their own state — see [BaseEoseManager], which does both by routing the
* work through a [com.vitorpamplona.amethyst.commons.service.BundledUpdate].
*/
fun invalidateFilters(ignoreIfDoing: Boolean = false)
fun destroy()
commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/search/KindRegistry.kt
+3
View File
@@ -32,6 +32,7 @@ import com.vitorpamplona.quartz.nip53LiveActivities.streaming.LiveActivitiesEven
import com.vitorpamplona.quartz.nip54Wiki.WikiNoteEvent
import com.vitorpamplona.quartz.nip72ModCommunities.definition.CommunityDefinitionEvent
import com.vitorpamplona.quartz.nip84Highlights.HighlightEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip99Classifieds.ClassifiedsEvent
data class ContentPreset(
@@ -63,6 +64,7 @@ object KindRegistry {
"wiki" to listOf(WikiNoteEvent.KIND),
"classified" to listOf(ClassifiedsEvent.KIND),
"highlight" to listOf(HighlightEvent.KIND),
"poll" to listOf(PollEvent.KIND),
)
val pseudoKinds: Set<String> = setOf("reply", "media")
@@ -75,6 +77,7 @@ object KindRegistry {
"Channels" to ContentPreset(kinds = listOf(ChannelCreateEvent.KIND, ChannelMetadataEvent.KIND)),
"Communities" to ContentPreset(kinds = listOf(CommunityDefinitionEvent.KIND)),
"Wiki" to ContentPreset(kinds = listOf(WikiNoteEvent.KIND)),
"Polls" to ContentPreset(kinds = listOf(PollEvent.KIND)),
)
fun resolve(alias: String): List<Int>? = aliases[alias.lowercase()]
commons/src/commonMain/kotlin/com/vitorpamplona/amethyst/commons/ui/feeds/FeedContentState.kt
+8 -1
View File
@@ -161,7 +161,14 @@ class FeedContentState(
if (noteEvent != null) {
!cacheProvider.hasBeenDeleted(noteEvent)
} else {
false
// An event-less row is a placeholder the filter synthesized for a room
// that has no message yet — a just-joined Concord channel, NIP-29 group,
// Marmot group or geohash cell. It carries no event, so it cannot have
// been deleted, and dropping it here deleted every such row from the
// Messages list the moment ANY kind-5 landed in an unrelated batch. The
// row then stayed gone until the next full rebuild, which is why a quiet
// community looked like it had never loaded at all.
true
}
}.toImmutableList()
}
commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/model/concord/ConcordCommunitySessionTest.kt
+1 -1
View File
@@ -108,7 +108,7 @@ class ConcordCommunitySessionTest {
// Feed the genesis control wraps → state folds, channels + membership resolve. A fold is
// STRUCTURAL (it moves the subscription set), so it's allowed to bump the revision.
community.genesisWraps.forEach { assertEquals(ConcordIngestOutcome.STRUCTURAL, session.ingest(it)) }
community.genesisWraps.forEach { assertEquals(ConcordIngestOutcome.STRUCTURAL_FOLD, session.ingest(it)) }
val state = session.state.value
assertEquals("Nostrichs", state?.metadata?.name)
assertTrue(state!!.channels.containsKey(community.generalChannelIdHex))
commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/model/concord/ConcordSessionRegistryTest.kt
+1 -1
View File
@@ -70,7 +70,7 @@ class ConcordSessionRegistryTest {
assertTrue(registry.subscribeAddresses().contains(beta.controlPlane.publicKeyHex))
// A genesis control wrap routes to Alpha's session and folds it (STRUCTURAL).
alpha.genesisWraps.forEach { assertEquals(ConcordIngestOutcome.STRUCTURAL, registry.ingest(it)) }
alpha.genesisWraps.forEach { assertEquals(ConcordIngestOutcome.STRUCTURAL_FOLD, registry.ingest(it)) }
val alphaState = registry.sessionFor(alpha.communityIdHex)!!.state.value
assertEquals("Alpha", alphaState?.metadata?.name)
commons/src/commonTest/kotlin/com/vitorpamplona/amethyst/commons/model/nip88Polls/PollResponsesCacheTest.kt
+145
View File
@@ -0,0 +1,145 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.commons.model.nip88Polls
import com.vitorpamplona.amethyst.commons.model.Note
import com.vitorpamplona.amethyst.commons.model.User
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
class PollResponsesCacheTest {
private val pollId = "a".repeat(64)
// The tally keys votes by `User` identity, and the real cache returns one `User`
// instance per pubkey (getOrCreateUser). Mirror that here so same-pubkey re-votes
// and hasPubKeyVoted() lookups resolve to the same object.
private val userCache = mutableMapOf<HexKey, User>()
private fun user(pubKey: HexKey): User = userCache.getOrPut(pubKey) { User(pubKey) { addr -> Note(addr.toValue()) } }
/** Builds a kind-1018 response Note authored by [pubKey] choosing [option] at [createdAt]. */
private fun responseNote(
id: HexKey,
pubKey: HexKey,
option: String,
createdAt: Long,
): Note {
val event =
PollResponseEvent(
id = id,
pubKey = pubKey,
createdAt = createdAt,
tags =
arrayOf(
arrayOf("e", pollId),
arrayOf("response", option),
),
content = "",
sig = "0".repeat(128),
)
val note = Note(id)
note.loadEvent(event, user(pubKey), emptyList())
return note
}
@Test
fun latestVoteWinsDedup() {
val cache = PollResponsesCache()
val voter = "b".repeat(64)
// Same voter votes twice; the later-timestamp response must win.
cache.addResponse(responseNote("1".repeat(64), voter, option = "yes", createdAt = 100))
cache.addResponse(responseNote("2".repeat(64), voter, option = "no", createdAt = 200))
val tally = cache.responses.value
// Exactly one vote counted for this user.
assertEquals(1, tally.totalVotes())
// The winning option is the newer one.
assertEquals("no", tally.winning())
// Old option carries no voters.
assertTrue(tally.tally["yes"].isNullOrEmpty())
}
@Test
fun tallyPercentReflectsVoteShare() {
val cache = PollResponsesCache()
val forKey = "0".repeat(64)
cache.addResponse(responseNote("1".repeat(64), "b".repeat(64), option = "yes", createdAt = 10))
cache.addResponse(responseNote("2".repeat(64), "c".repeat(64), option = "yes", createdAt = 10))
cache.addResponse(responseNote("3".repeat(64), "d".repeat(64), option = "no", createdAt = 10))
val yes = cache.currentTally("yes", forKey, emptySet())
val no = cache.currentTally("no", forKey, emptySet())
assertEquals(2f / 3f, yes.percent)
assertEquals(1f / 3f, no.percent)
assertTrue(yes.isWinning)
assertFalse(no.isWinning)
}
@Test
fun wotPrioritySortOrdersUsers() {
val cache = PollResponsesCache()
val forKey = "f".repeat(64) // the logged-in user
val followed = "e".repeat(64)
val stranger = "d".repeat(64)
// Three voters all pick "yes": self, a followed user, and a stranger.
cache.addResponse(responseNote("1".repeat(64), forKey, option = "yes", createdAt = 10))
cache.addResponse(responseNote("2".repeat(64), stranger, option = "yes", createdAt = 10))
cache.addResponse(responseNote("3".repeat(64), followed, option = "yes", createdAt = 10))
val tally = cache.currentTally("yes", forKey, priorityAccounts = setOf(followed))
val order = tally.users.map { it.pubkeyHex }
// Self first, then followed (WoT priority), then the stranger.
assertEquals(listOf(forKey, followed, stranger), order)
}
@Test
fun hasPubKeyVotedTracksVoter() {
val cache = PollResponsesCache()
val voter = "b".repeat(64)
val other = "c".repeat(64)
cache.addResponse(responseNote("1".repeat(64), voter, option = "yes", createdAt = 10))
assertTrue(cache.hasPubKeyVoted(user(voter)))
assertFalse(cache.hasPubKeyVoted(user(other)))
}
@Test
fun addResponseIsIdempotentForSameNote() {
val cache = PollResponsesCache()
val note = responseNote("1".repeat(64), "b".repeat(64), option = "yes", createdAt = 10)
cache.addResponse(note)
cache.addResponse(note) // relay echo of the same note must not double-count
assertEquals(1, cache.responses.value.totalVotes())
}
}
desktopApp/plans/2026-07-16-desktop-polls-manual-testing-sheet.md
+106
View File
@@ -0,0 +1,106 @@
# Desktop Polls (NIP-88) — Manual Testing Sheet
**Feature:** render + vote + create polls on Amethyst Desktop
**Branch:** `worktree-feat+desktop-polls` (worktree `.claude/worktrees/feat+desktop-polls`)
**Plan:** `docs/plans/2026-07-16-feat-desktop-polls-nip88-plan.md`
**Status when written:** code-complete; compile + unit tests + spotless GREEN; **manual run not yet done.**
## Automated gates already passing
- [x] `./gradlew :commons:compileKotlinJvm :desktopApp:compileKotlin` — clean
- [x] `./gradlew :commons:jvmTest --tests "*nip88Polls*"` — 5 pass (dedup, tally %, WoT sort, hasVoted, idempotency)
- [x] `./gradlew :desktopApp:test --tests "*Poll*"` — 2 pass (response links into tally; relay-echo dedup)
- [x] `./gradlew :commons:spotlessKotlinCheck :desktopApp:spotlessKotlinCheck` — clean
## How to run
```bash
cd .claude/worktrees/feat+desktop-polls
./gradlew :desktopApp:run
```
Log in (existing account, or NIP-46 bunker). Use a relay set that carries polls — good sources: relays where clients post NIP-88 polls, or create one yourself (Test C) and read it back. A **second client** (Amethyst Android, or `amy`) is useful to cross-verify events on the wire.
---
## A. Rendering (feed + thread)
- [ ] A1. A kind-1068 poll appears in the **Home/Global feed as a poll card** (description + options), NOT as plain text. *(If polls never show: verify `DesktopFeedFilters.isFeedNote` includes `PollEvent` and `FEED_KINDS` has 1068.)*
- [ ] A2. Open the poll in a **thread column** → renders as a poll card there too.
- [ ] A3. Single-choice poll shows **radio**-style option rows; multi-choice shows **checkbox**-style rows with a **Submit** button.
- [ ] A4. Before voting, **no percentages/tally are shown** — only actionable options + a **"View results"** button.
- [ ] A5. Tapping **"View results"** reveals the tally without casting a vote; a way back to voting exists (unless ended/author).
- [ ] A6. A poll authored by **you**, seen in your own feed, shows **results-only** (you cannot vote).
- [ ] A7. An **ended** poll (deadline in the past) shows results-only, no vote controls.
- [ ] A8. Media/description of the poll render via the normal note card (links, images in the description behave as usual).
## B. Voting
- [ ] B1. Cast a **single-choice** vote → card immediately flips to results (optimistic), your option marked as your vote.
- [ ] B2. Results show a **% bar per option**, a **winning** highlight, and **voter avatars** (up to ~4) + "+N".
- [ ] B3. Voter avatars are **ordered with people you follow first** (WoT). Verify by having a followed account vote — their avatar should sort ahead of strangers.
- [ ] B4. The bar does **not** do a distracting 0→N sweep when opening an already-tallied poll (first-frame animation guard).
- [ ] B5. **Multi-choice**: select 2 options → Submit → both recorded; results reflect both.
- [ ] B6. **Multi-choice empty submit is rejected** — with nothing selected, Submit does nothing / is disabled (no empty response event sent).
- [ ] B7. **Change vote**: after voting, use **"Change vote"** → re-open options → pick a different option → tally updates so the **new** choice wins for you (newest response wins).
- [ ] B8. Cross-check on a second client (Android/amy): your vote is a **kind-1018** event referencing the poll via a lowercase `e` tag.
- [ ] B9. **Scroll-away during send** (stress the scope fix): cast a vote and immediately scroll the poll out of view. Re-find it / check a second client — the vote should have **broadcast to relays**, not just shown locally. *(This validates `voteOnPoll` runs on the long-lived `appScope`, not the card scope.)*
## C. Creating a poll
- [ ] C1. Open the composer; toggle **Poll** on → poll option editor appears; image attachment is disabled while Poll is on.
- [ ] C2. Add/remove options; **minimum 2 non-blank** options enforced before send is allowed.
- [ ] C3. Toggle **Single vs Multiple** choice.
- [ ] C4. Set a **duration** (Never / 1d / 3d / 7d). "Never" = open-ended (no deadline).
- [ ] C5. Send → a **kind-1068 PollEvent** is published (verify on a second client): correct options, `polltype`, and `endsAt` (absent for "Never").
- [ ] C6. The poll you created appears in your feed and is votable from **another** account/client; its tally updates as votes arrive.
## D. Edge cases
- [ ] D1. Poll with an unusually **long option label** wraps/renders without breaking layout.
- [ ] D2. A poll received with **0 options** (malformed) does not crash the feed (renders degraded / skipped).
- [ ] D3. Receiving **many responses from multiple relays** converges to a stable, non-inflated tally (no double counting of the same response).
- [ ] D4. Late votes arriving **after** a poll's deadline: they may still count in the tally, but the card stays results-only (no re-vote UI).
## E. Regression (nothing else broke)
- [ ] E1. Normal text notes, reposts, and reactions still render + behave in the feed.
- [ ] E2. Composing a normal note (Poll toggle OFF) works exactly as before, including image attachment.
- [ ] E3. Thread view still loads reactions/zaps/reposts for non-poll notes.
---
---
## F. Search "Polls" content-type filter (added 2026-07-20)
*Feature: filter search to only polls + interact with them. Plan: `docs/plans/2026-07-20-feat-desktop-search-polls-facet-plan.md`.*
- [ ] F1. Open the **Search** column → advanced filter panel shows a **"Polls"** chip alongside Notes/Articles/Media/Channels/Communities/Wiki.
- [ ] F2. Enter a query, select **Polls** → results contain **only** poll notes (kind 1068); other content types are excluded.
- [ ] F3. Results appear under a dedicated **"Polls" section** (poll icon) and render as **interactive `DesktopPollCard`** — options visible, not plain text.
- [ ] F4. **Vote from search** (dedicated Search screen): cast a vote on a poll in results → flips to results/optimistic tally, and a kind-1018 event is published (cross-check on a 2nd client).
- [ ] F5. Deselect the Polls chip → results return to mixed content; other facets still work.
- [ ] F6. Section collapse/expand + "Show all N more" work like the other search sections.
- [ ] F7. **Feed header quick-search** (the search box in the feed header): polls render as cards **and are now votable** (account threaded 2026-07-20).
## G. Cross-context consistency fixes (2026-07-20)
*Fixes for reported bugs: "can't always tap depending on how it's opened" + "only see my own answer, no other tallies".*
- [ ] G1. **Thread view:** open a poll into a thread → you can vote, and **after voting the card correctly shows your choice** as selected (previously your vote-state didn't register — missing `myPubKeyHex`).
- [ ] G2. **Thread tallies:** a poll opened in a thread shows **other people's votes**, not just yours.
- [ ] G3. **Profile tabs (Notes/Replies):** polls on a user's profile are votable and reflect your vote correctly.
- [ ] G4. **Dedicated Search tallies:** filter to Polls → results now show **existing tallies from others** (search fetches kind-1018 responses via `requestInteractions`), not just your own vote.
- [ ] G5. **Feed header quick-search:** polls there are now **votable** (account threaded).
- [ ] G6. **Consistency:** the SAME poll shows consistent vote-state + tallies whether opened in feed, thread, profile, or search.
**Remaining known gaps (expected):**
- **Notifications tab** renders polls as the compact notification card (not interactive) — out of scope.
- **Poll posted as a thread *reply*** (not root) renders via the thread's custom reply card (not interactive) — edge case, deferred.
- Feed-header quick-search fetches tallies only after the poll is also seen in a context that requests interactions; the **dedicated Search** column always fetches them.
## Known caveats (expected, not bugs)
- **Same-second re-vote:** if you change your vote **within the same 1-second** as the first, the tally may not flip until a later-second vote (tie-break on `createdAt` uses strict `>` with no id fallback). Wait ~1s between re-votes to see B7 flip reliably.
- **Option labels are plain text (v1):** links/custom-emoji inside an option label are shown literally, not hyperlinked/rendered (no desktop rich-text path for option labels yet).
- **Deadline = preset chips (v1):** Never/1d/3d/7d instead of a full date/time picker.
- **Not wired this PR (deferred):** poll rendering in profile/bookmarks/search/notifications tabs (still show as plain notes there); poll-draft round-trip; zap-weighted polls.
## If something fails — where to look
| Symptom | Check |
|---|---|
| Polls never appear in feed | `feeds/DesktopFeedFilters.kt` `isFeedNote` (PollEvent), `subscriptions/FilterBuilders.kt` `FEED_KINDS`=…,1068 |
| Poll shows but tally always empty | kind-1018 sub: `ui/FeedScreen.kt` fetch-interactions filter + `DesktopRelaySubscriptionsCoordinator.requestInteractions` (`e` tag) |
| Vote shows locally but never reaches relays | `DesktopPollCard.castVote` must launch on `localCache.appScope`; `voteOnPoll` in `ui/NoteActions.kt` |
| Double-counted votes | `DesktopLocalCache.consumePollResponse` new-event gate (line ~385) |
| Created poll malformed | `ComposeNoteDialog.publishPoll` → `PollEvent.build` options/type/endsAt |
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/cache/DesktopLocalCache.kt
Vendored
+67 -2
View File
@@ -58,13 +58,16 @@ import com.vitorpamplona.quartz.nip51Lists.followList.FollowListEvent
import com.vitorpamplona.quartz.nip57Zaps.LnZapEvent
import com.vitorpamplona.quartz.nip57Zaps.LnZapRequestEvent
import com.vitorpamplona.quartz.nip65RelayList.AdvertisedRelayListEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import com.vitorpamplona.quartz.nipB7Blossom.BlossomServersEvent
import com.vitorpamplona.quartz.utils.DualCase
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.DelicateCoroutinesApi
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.GlobalScope
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.channels.BufferOverflow
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.MutableStateFlow
@@ -129,6 +132,17 @@ class DesktopLocalCache : ICacheProvider {
val paymentTracker = NwcPaymentTracker()
/**
* Long-lived, cache-scoped coroutine scope for fire-and-forget work that must
* outlive any single composition — e.g. the optimistic-consume → relay-broadcast
* pair of a poll vote (see [com.vitorpamplona.amethyst.desktop.ui.voteOnPoll]).
* Using a card's [androidx.compose.runtime.rememberCoroutineScope] there would let
* scrolling the card out of composition cancel the broadcast after the local consume,
* leaving the vote visible locally but never sent. Uses a [SupervisorJob] so one
* failed job doesn't tear down the rest.
*/
val appScope = CoroutineScope(SupervisorJob() + Dispatchers.IO)
private fun trackNoteAuthor(
note: Note,
authorPubkey: HexKey,
@@ -337,6 +351,14 @@ class DesktopLocalCache : ICacheProvider {
consumeBlossomServerList(event, relay)
}
is PollEvent -> {
consumePoll(event, relay)
}
is PollResponseEvent -> {
consumePollResponse(event, relay)
}
else -> {
false
}
@@ -455,6 +477,49 @@ class DesktopLocalCache : ICacheProvider {
return true
}
/**
* Consumes a kind 1068 poll event (NIP-88).
* Creates a Note in the cache like a text note, minus reply-linking — a poll is
* always a root post. The [Note.pollState] tally is populated by the responses.
*/
private fun consumePoll(
event: PollEvent,
relay: NormalizedRelayUrl?,
): Boolean {
val note = getOrCreateNote(event.id)
if (note.event != null) return false
val author = getOrCreateUser(event.pubKey)
note.loadEvent(event, author, emptyList())
trackNoteAuthor(note, event.pubKey)
relay?.let { note.addRelay(it) }
return true
}
/**
* Consumes a kind 1018 poll response event (NIP-88).
* Resolves the referenced poll, loads the response note, and links it into the
* poll's tally. Mirrors Android `LocalCache.consume(PollResponseEvent)`: the
* [com.vitorpamplona.amethyst.commons.model.nip88Polls.PollResponsesCache.addResponse]
* call and the `true` return happen only on a genuinely new event, so a relay echo
* of the user's own optimistically-consumed vote can't double-count (id-dedup here
* plus `addResponse`'s own containment guard).
*/
private fun consumePollResponse(
event: PollResponseEvent,
relay: NormalizedRelayUrl?,
): Boolean {
val pollId = event.poll()?.eventId ?: return false
val pollNote = getOrCreateNote(pollId)
val responseNote = getOrCreateNote(event.id)
if (responseNote.event != null) return false
val author = getOrCreateUser(event.pubKey)
responseNote.loadEvent(event, author, emptyList())
trackNoteAuthor(responseNote, event.pubKey)
relay?.let { responseNote.addRelay(it) }
pollNote.pollState().addResponse(responseNote)
return true
}
/**
* NIP-18 quote reposts: a note carrying a `q` tag is a quote-repost of the quoted
* note, so it counts as a boost in the quoted note's repost counter alongside
@@ -821,7 +886,7 @@ class DesktopLocalCache : ICacheProvider {
requestNote?.let { req -> pending.zappedNote?.addZapPayment(req, note) }
// Invoke callback on IO dispatcher
GlobalScope.launch(Dispatchers.IO) {
appScope.launch {
pending.onResponse(event)
}
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/feeds/DesktopFeedFilters.kt
+2
View File
@@ -37,9 +37,11 @@ import com.vitorpamplona.quartz.nip22Comments.CommentEvent
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
import com.vitorpamplona.quartz.nip25Reactions.ReactionEvent
import com.vitorpamplona.quartz.nip57Zaps.LnZapEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
private fun isFeedNote(event: Event?): Boolean =
event is TextNoteEvent ||
event is PollEvent ||
event.isRenderableRepost()
private fun List<Note>.deduplicateReposts(): List<Note> =
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/relay/LocalRelayStore.kt
+15 -2
View File
@@ -26,6 +26,7 @@ import com.vitorpamplona.amethyst.desktop.cache.DesktopLocalCache
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.store.sqlite.DefaultIndexingStrategy
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.utils.Log
import kotlinx.coroutines.CoroutineScope
@@ -42,6 +43,18 @@ class LocalRelayStore(
) : AutoCloseable {
companion object {
val LOCAL_RELAY_URL: NormalizedRelayUrl = NormalizedRelayUrl("ws://localhost/amethyst-local/")
/**
* Client defaults plus the authors-without-kinds index: shared
* ViewModels (`Nip65RelayListViewModel`, `PrivateOutboxRelayListViewModel`,
* `VanishRequestsState`) replay `authors`-only filters against this
* store, which full-scan without `(pubkey, created_at)` — the
* `(kind, pubkey, …)` index can't serve them, pubkey is its second
* column. A personal store is small, so the extra insert cost is
* negligible; existing DBs get the index built on next open via
* `ensureOptionalIndexes`.
*/
val INDEX_STRATEGY = DefaultIndexingStrategy(indexEventsByPubkeyAlone = true)
}
private fun dbDir(pubKeyHex: String): File = File(homeDir, ".amethyst/accounts/${pubKeyHex.take(8)}")
@@ -87,14 +100,14 @@ class LocalRelayStore(
dir.mkdirs()
val path = File(dir, "events.db").absolutePath
try {
store = EventStore(dbName = path, relay = LOCAL_RELAY_URL)
store = EventStore(dbName = path, relay = LOCAL_RELAY_URL, indexStrategy = INDEX_STRATEGY)
_lastError.value = null
refreshStats()
} catch (e: Exception) {
Log.w("LocalRelayStore") { "DB open failed, recreating: ${e.message}" }
try {
deleteDbFiles(path)
store = EventStore(dbName = path, relay = LOCAL_RELAY_URL)
store = EventStore(dbName = path, relay = LOCAL_RELAY_URL, indexStrategy = INDEX_STRATEGY)
_lastError.value = "Database was recreated: ${e.message}"
} catch (e2: Exception) {
_lastError.value = "Cannot open local store: ${e2.message}"
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/subscriptions/DesktopRelaySubscriptionsCoordinator.kt
+5
View File
@@ -294,6 +294,11 @@ class DesktopRelaySubscriptionsCoordinator(
kinds = listOf(com.vitorpamplona.quartz.nip10Notes.TextNoteEvent.KIND),
tags = mapOf("e" to noteIds),
),
// Poll responses (kind 1018) targeting these notes (NIP-88, lowercase `e`)
Filter(
kinds = listOf(com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent.KIND),
tags = mapOf("e" to noteIds),
),
)
val listener =
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/subscriptions/FilterBuilders.kt
+1 -1
View File
@@ -29,7 +29,7 @@ import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent
* Provides convenience functions for creating relay subscription filters.
*/
object FilterBuilders {
private val FEED_KINDS = listOf(1, 6, 16) // TextNoteEvent, RepostEvent, GenericRepostEvent
private val FEED_KINDS = listOf(1, 6, 16, 1068) // TextNoteEvent, RepostEvent, GenericRepostEvent, PollEvent
/**
* Creates a filter for text notes (kind 1) from all authors.
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/ComposeNoteDialog.kt
+446 -241
View File
@@ -43,6 +43,9 @@ import androidx.compose.material3.Button
import androidx.compose.material3.Card
import androidx.compose.material3.CardDefaults
import androidx.compose.material3.Checkbox
import androidx.compose.material3.FilterChip
import androidx.compose.material3.FilterChipDefaults
import androidx.compose.material3.IconButton
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedButton
import androidx.compose.material3.OutlinedTextField
@@ -56,6 +59,7 @@ import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.runtime.snapshots.SnapshotStateList
import androidx.compose.ui.Alignment
import androidx.compose.ui.ExperimentalComposeUiApi
import androidx.compose.ui.Modifier
@@ -66,6 +70,8 @@ import androidx.compose.ui.text.input.TextFieldValue
import androidx.compose.ui.unit.dp
import androidx.compose.ui.window.Dialog
import androidx.compose.ui.window.DialogProperties
import com.vitorpamplona.amethyst.commons.icons.symbols.Icon
import com.vitorpamplona.amethyst.commons.icons.symbols.MaterialSymbols
import com.vitorpamplona.amethyst.commons.model.User
import com.vitorpamplona.amethyst.commons.scheduledposts.ScheduledPost
import com.vitorpamplona.amethyst.commons.scheduledposts.ScheduledPostStore
@@ -114,6 +120,9 @@ import com.vitorpamplona.quartz.nip18Reposts.quotes.quote
import com.vitorpamplona.quartz.nip19Bech32.entities.NEvent
import com.vitorpamplona.quartz.nip22Comments.CommentEvent
import com.vitorpamplona.quartz.nip37Drafts.DraftWrapEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.poll.tags.OptionTag
import com.vitorpamplona.quartz.nip88Polls.poll.tags.PollType
import com.vitorpamplona.quartz.nip89AppHandlers.clientTag.isClient
import com.vitorpamplona.quartz.nip92IMeta.IMetaTag
import com.vitorpamplona.quartz.utils.TimeUtils
@@ -228,6 +237,14 @@ fun ComposeNoteDialog(
var syncDraft by remember { mutableStateOf(false) }
var isSavingDraft by remember { mutableStateOf(false) }
// Poll (NIP-88) composer state. `wantsPoll` gates the poll UI; options start with
// two blank fields (a poll needs ≥2 non-blank options to publish).
var wantsPoll by remember { mutableStateOf(false) }
val pollOptions = remember { mutableStateListOf("", "") }
var pollType by remember { mutableStateOf(PollType.SINGLE_CHOICE) }
// Optional poll deadline, expressed as seconds-from-now (null = open-ended).
var pollDurationDays by remember { mutableStateOf<Int?>(null) }
// Image compression: global default + optional per-post override.
// Override resets after every successful send so the next post
// starts from the saved default again.
@@ -377,7 +394,21 @@ fun ComposeNoteDialog(
}
val scheduleAt = scheduledForSec
if (postAsPicture) {
if (wantsPoll) {
val endsAt =
pollDurationDays?.let { days ->
TimeUtils.now() + days * 24L * 60L * 60L
}
publishPoll(
description = content,
options = pollOptions.map { it.trim() }.filter { it.isNotEmpty() },
pollType = pollType,
endsAt = endsAt,
account = account,
relayManager = relayManager,
relays = selectedRelays,
)
} else if (postAsPicture) {
val pictureMetas = buildPictureMetas(uploadResults)
publishPicture(
description = content,
@@ -434,8 +465,9 @@ fun ComposeNoteDialog(
Modifier
.width(780.dp)
// Cap the dialog height so a tall composer (e.g. the schedule
// picker expanded) can't push the Cancel/Schedule buttons off
// screen — the body scrolls instead (see the content Column).
// picker expanded, or the poll composer with many options) can't
// push the Cancel/Publish buttons off screen — the body scrolls
// instead (see the content Column).
.heightIn(max = 760.dp)
.padding(16.dp)
.dragAndDropTarget(shouldStartDragAndDrop = { true }, target = dropTarget)
@@ -458,206 +490,246 @@ fun ComposeNoteDialog(
color = MaterialTheme.colorScheme.onSurface,
)
replyTo?.let { reply ->
Spacer(Modifier.height(8.dp))
Text(
"Replying to: ${reply.content.take(50)}${if (reply.content.length > 50) "..." else ""}",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
// Scrollable content area so the pinned Cancel/Publish row below stays
// reachable even when the poll section grows with many options.
Column(
modifier =
Modifier
.weight(1f, fill = false)
.verticalScroll(rememberScrollState()),
) {
replyTo?.let { reply ->
Spacer(Modifier.height(8.dp))
Text(
"Replying to: ${reply.content.take(50)}${if (reply.content.length > 50) "..." else ""}",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
quoteOf?.let { quoted ->
Spacer(Modifier.height(8.dp))
Text(
"Quoting: ${quoted.content.take(50)}${if (quoted.content.length > 50) "..." else ""}",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
quoteOf?.let { quoted ->
Spacer(Modifier.height(8.dp))
Text(
"Quoting: ${quoted.content.take(50)}${if (quoted.content.length > 50) "..." else ""}",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
Spacer(Modifier.height(16.dp))
Spacer(Modifier.height(16.dp))
Box {
OutlinedTextField(
value = if (postAsPicture) TextFieldValue("") else contentField,
onValueChange = {
contentField = it
errorMessage = null
},
modifier = Modifier.fillMaxWidth().height(if (postAsPicture) 60.dp else 200.dp),
label = {
Text(
if (postAsPicture) "Text disabled for picture posts" else "What's on your mind?",
)
},
placeholder = { Text(if (postAsPicture) "" else "Write your note... (type @ to mention)") },
enabled = !isPosting && !postAsPicture,
maxLines = if (postAsPicture) 1 else 10,
)
Box {
OutlinedTextField(
value = if (postAsPicture) TextFieldValue("") else contentField,
onValueChange = {
contentField = it
errorMessage = null
},
modifier = Modifier.fillMaxWidth().height(if (postAsPicture) 60.dp else 200.dp),
label = {
Text(
if (postAsPicture) "Text disabled for picture posts" else "What's on your mind?",
)
},
placeholder = { Text(if (postAsPicture) "" else "Write your note... (type @ to mention)") },
enabled = !isPosting && !postAsPicture,
maxLines = if (postAsPicture) 1 else 10,
)
// Mention autocomplete dropdown
if (mentionSuggestions.isNotEmpty()) {
Card(
modifier = Modifier.fillMaxWidth().padding(top = 4.dp),
elevation = CardDefaults.cardElevation(defaultElevation = 4.dp),
) {
LazyColumn(modifier = Modifier.heightIn(max = 200.dp)) {
items(mentionSuggestions, key = { it.pubkeyHex }) { user ->
MentionSuggestionRow(
user = user,
onClick = {
val npub = user.pubkeyNpub()
val replacement = "nostr:$npub "
val cursorEnd = contentField.selection.end
val newText =
contentField.text.replaceRange(
mentionWordStart,
cursorEnd,
replacement,
)
val newCursor = mentionWordStart + replacement.length
contentField = TextFieldValue(newText, TextRange(newCursor))
mentionSuggestions = emptyList()
mentionQuery = null
},
)
// Mention autocomplete dropdown
if (mentionSuggestions.isNotEmpty()) {
Card(
modifier = Modifier.fillMaxWidth().padding(top = 4.dp),
elevation = CardDefaults.cardElevation(defaultElevation = 4.dp),
) {
LazyColumn(modifier = Modifier.heightIn(max = 200.dp)) {
items(mentionSuggestions, key = { it.pubkeyHex }) { user ->
MentionSuggestionRow(
user = user,
onClick = {
val npub = user.pubkeyNpub()
val replacement = "nostr:$npub "
val cursorEnd = contentField.selection.end
val newText =
contentField.text.replaceRange(
mentionWordStart,
cursorEnd,
replacement,
)
val newCursor = mentionWordStart + replacement.length
contentField = TextFieldValue(newText, TextRange(newCursor))
mentionSuggestions = emptyList()
mentionQuery = null
},
)
}
}
}
}
}
}
Spacer(Modifier.height(8.dp))
Row(
modifier = Modifier.fillMaxWidth(),
verticalAlignment = Alignment.CenterVertically,
) {
// MediaAttachmentRow fills its width, so give it a weighted slot;
// otherwise it consumes the whole Row and pushes the schedule
// button off the right edge (making it invisible).
Box(Modifier.weight(1f)) {
MediaAttachmentRow(
attachedFiles = attachedFiles,
isUploading = uploadState.isUploading,
onAttach = {
val files = DesktopFilePicker.pickMediaFiles()
attachedFiles.addAll(files)
},
onPaste = {
val files = ClipboardPasteHandler.getClipboardFiles()
attachedFiles.addAll(files)
},
onRemove = { attachedFiles.remove(it) },
)
}
// Picture posts aren't schedulable in v1 — hide the toggle then.
if (!postAsPicture) {
DesktopScheduleAtButton(
isActive = scheduledForSec != null,
onClick = {
scheduledForSec =
if (scheduledForSec != null) {
null
} else {
sanitizeScheduleTime(presetInOneHour())
}
},
)
}
}
if (scheduledForSec != null && !postAsPicture) {
Spacer(Modifier.height(8.dp))
DesktopScheduleAtPicker(
scheduledForSec = scheduledForSec ?: 0L,
onChanged = { scheduledForSec = it },
// Poll toggle — mutually exclusive with image posting (a poll carries no
// media attachments). Disabled while there are attached files.
Row(verticalAlignment = Alignment.CenterVertically) {
FilterChip(
selected = wantsPoll,
onClick = { wantsPoll = !wantsPoll },
enabled = attachedFiles.isEmpty(),
label = { Text("Poll") },
leadingIcon =
if (wantsPoll) {
{ Icon(MaterialSymbols.Check, contentDescription = null, modifier = Modifier.size(18.dp)) }
} else {
null
},
)
}
if (wantsPoll) {
PollComposerSection(
options = pollOptions,
pollType = pollType,
onPollTypeChange = { pollType = it },
pollDurationDays = pollDurationDays,
onDurationChange = { pollDurationDays = it },
)
}
// Media attachment + scheduling — hidden for polls (a poll carries no
// media attachments and isn't schedulable in v1).
if (!wantsPoll) {
Row(
modifier = Modifier.fillMaxWidth(),
verticalAlignment = Alignment.CenterVertically,
) {
// MediaAttachmentRow fills its width, so give it a weighted slot;
// otherwise it consumes the whole Row and pushes the schedule
// button off the right edge (making it invisible).
Box(Modifier.weight(1f)) {
MediaAttachmentRow(
attachedFiles = attachedFiles,
isUploading = uploadState.isUploading,
onAttach = {
val files = DesktopFilePicker.pickMediaFiles()
attachedFiles.addAll(files)
},
onPaste = {
val files = ClipboardPasteHandler.getClipboardFiles()
attachedFiles.addAll(files)
},
onRemove = { attachedFiles.remove(it) },
)
}
// Picture posts aren't schedulable in v1 — hide the toggle then.
if (!postAsPicture) {
DesktopScheduleAtButton(
isActive = scheduledForSec != null,
onClick = {
scheduledForSec =
if (scheduledForSec != null) {
null
} else {
sanitizeScheduleTime(presetInOneHour())
}
},
)
}
}
if (scheduledForSec != null && !postAsPicture) {
Spacer(Modifier.height(8.dp))
DesktopScheduleAtPicker(
scheduledForSec = scheduledForSec ?: 0L,
onChanged = { scheduledForSec = it },
)
}
}
// Server selector + per-post quality + post type — shown when files are attached
if (attachedFiles.isNotEmpty()) {
Spacer(Modifier.height(4.dp))
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.SpaceBetween,
verticalAlignment = androidx.compose.ui.Alignment.CenterVertically,
) {
ServerSelector(
servers = effectiveServers,
selectedServer = selectedServer,
onServerSelected = { selectedServer = it },
)
// Quality override chip — only when images are attached
// (no point picking a JPEG preset for a video upload).
if (hasImages) {
QualitySelectorChip(
activeQuality = activeQuality,
isOverride = perPostQualityOverride != null,
onSelect = { perPostQualityOverride = it },
onReset = { perPostQualityOverride = null },
)
}
// Post type toggle — only when images are attached
if (hasImages) {
PostTypeSelector(
isPicture = postAsPicture,
onToggle = { postAsPicture = it },
)
}
}
}
Spacer(Modifier.height(4.dp))
// Character count
Text(
"${content.length} characters",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
errorMessage?.let { error ->
Spacer(Modifier.height(8.dp))
SelectionContainer {
Text(
error,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
}
}
uploadState.error?.let { error ->
Spacer(Modifier.height(4.dp))
SelectionContainer {
Text(
"Upload error: $error",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
}
}
Spacer(Modifier.height(8.dp))
ComposeRelayPicker(
pickerState = pickerState,
selectedRelays = selectedRelays,
onToggleRelay = { url ->
selectedRelays =
if (url in selectedRelays) {
selectedRelays - url
} else {
selectedRelays + url
}
},
)
}
// Server selector + per-post quality + post type — shown when files are attached
if (attachedFiles.isNotEmpty()) {
Spacer(Modifier.height(4.dp))
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.SpaceBetween,
verticalAlignment = androidx.compose.ui.Alignment.CenterVertically,
) {
ServerSelector(
servers = effectiveServers,
selectedServer = selectedServer,
onServerSelected = { selectedServer = it },
)
// Quality override chip — only when images are attached
// (no point picking a JPEG preset for a video upload).
if (hasImages) {
QualitySelectorChip(
activeQuality = activeQuality,
isOverride = perPostQualityOverride != null,
onSelect = { perPostQualityOverride = it },
onReset = { perPostQualityOverride = null },
)
}
// Post type toggle — only when images are attached
if (hasImages) {
PostTypeSelector(
isPicture = postAsPicture,
onToggle = { postAsPicture = it },
)
}
}
}
Spacer(Modifier.height(4.dp))
// Character count
Text(
"${content.length} characters",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
errorMessage?.let { error ->
Spacer(Modifier.height(8.dp))
SelectionContainer {
Text(
error,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
}
}
uploadState.error?.let { error ->
Spacer(Modifier.height(4.dp))
SelectionContainer {
Text(
"Upload error: $error",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.error,
)
}
}
Spacer(Modifier.height(8.dp))
ComposeRelayPicker(
pickerState = pickerState,
selectedRelays = selectedRelays,
onToggleRelay = { url ->
selectedRelays =
if (url in selectedRelays) {
selectedRelays - url
} else {
selectedRelays + url
}
},
)
Spacer(Modifier.height(8.dp))
// NIP-37 draft-sync opt-in — only meaningful for plain notes.
@@ -704,61 +776,74 @@ fun ComposeNoteDialog(
// Save-as-draft: always writes a local row; optionally publishes a
// NIP-37 encrypted event. Local save still succeeds if sync fails.
OutlinedButton(
onClick = {
if (content.isBlank()) {
errorMessage = "Draft cannot be empty"
return@OutlinedButton
}
scope.launch {
isSavingDraft = true
errorMessage = null
var syncError: String? = null
try {
if (syncDraft) {
syncError =
syncDraftToRelays(
content = content,
dTag = draftTag,
account = account,
relayManager = relayManager,
replyTo = replyTo,
quoteOf = quoteOf,
relays = selectedRelays,
)
}
noteDraftStore.save(
NoteDraft(
dTag = draftTag,
content = content,
updatedAt = TimeUtils.now(),
synced = syncDraft && syncError == null,
accountPubkey = account.pubKeyHex,
),
)
if (syncError != null) {
errorMessage = "Draft saved locally, but sync failed: $syncError"
} else {
onDismiss()
}
} catch (e: Exception) {
errorMessage = "Failed to save draft: ${e.message}"
} finally {
isSavingDraft = false
// Not shown while composing a poll — polls aren't draftable in v1.
if (!wantsPoll) {
OutlinedButton(
onClick = {
if (content.isBlank()) {
errorMessage = "Draft cannot be empty"
return@OutlinedButton
}
}
},
enabled = !isPosting && !isSavingDraft && content.isNotBlank(),
) {
Text(if (isSavingDraft) "Saving..." else "Save as draft")
scope.launch {
isSavingDraft = true
errorMessage = null
var syncError: String? = null
try {
if (syncDraft) {
syncError =
syncDraftToRelays(
content = content,
dTag = draftTag,
account = account,
relayManager = relayManager,
replyTo = replyTo,
quoteOf = quoteOf,
relays = selectedRelays,
)
}
noteDraftStore.save(
NoteDraft(
dTag = draftTag,
content = content,
updatedAt = TimeUtils.now(),
synced = syncDraft && syncError == null,
accountPubkey = account.pubKeyHex,
),
)
if (syncError != null) {
errorMessage = "Draft saved locally, but sync failed: $syncError"
} else {
onDismiss()
}
} catch (e: Exception) {
errorMessage = "Failed to save draft: ${e.message}"
} finally {
isSavingDraft = false
}
}
},
enabled = !isPosting && !isSavingDraft && content.isNotBlank(),
) {
Text(if (isSavingDraft) "Saving..." else "Save as draft")
}
Spacer(Modifier.width(8.dp))
}
Spacer(Modifier.width(8.dp))
// A poll needs a question (description) and at least two options.
val pollValid = content.isNotBlank() && pollOptions.count { it.trim().isNotEmpty() } >= 2
Button(
onClick = {
if (wantsPoll) {
if (!pollValid) {
errorMessage = "A poll needs a question and at least two options"
return@Button
}
runPublish(null, emptySet())
return@Button
}
if (content.isBlank() && attachedFiles.isEmpty()) {
errorMessage = "Note cannot be empty"
return@Button
@@ -786,7 +871,9 @@ fun ComposeNoteDialog(
}
runPublish(null, emptySet())
},
enabled = !isPosting && !isSavingDraft && (content.isNotBlank() || attachedFiles.isNotEmpty()),
enabled =
!isPosting && !isSavingDraft &&
if (wantsPoll) pollValid else (content.isNotBlank() || attachedFiles.isNotEmpty()),
) {
Text(
when {
@@ -982,6 +1069,40 @@ private suspend fun publishPicture(
}
}
private suspend fun publishPoll(
description: String,
options: List<String>,
pollType: PollType,
endsAt: Long?,
account: AccountState.LoggedIn,
relayManager: DesktopRelayConnectionManager,
relays: Set<NormalizedRelayUrl>,
) {
withContext(Dispatchers.IO) {
if (account.isReadOnly) {
throw IllegalStateException("Cannot post in read-only mode")
}
require(options.size >= 2) { "A poll needs at least two options" }
// Deterministic per-position codes; labels come straight from the fields.
val optionTags = options.mapIndexed { index, label -> OptionTag(index.toString(), label) }
val template =
PollEvent.build(
description = description,
options = optionTags,
endsAt = endsAt,
relays = relays.toList(),
pollType = pollType,
) {
hashtags(findHashtags(description))
}
val signedEvent = account.signer.sign(template)
relayManager.publish(signedEvent, relays)
}
}
private suspend fun publishNote(
content: String,
account: AccountState.LoggedIn,
@@ -1174,6 +1295,90 @@ private suspend fun syncDraftToRelays(
}
}
/**
* Poll composer body: N option fields (add/remove, ≥2), a single/multi choice chip pair,
* and an optional duration (deadline) chip row. The description is the main note text field.
*/
@Composable
private fun PollComposerSection(
options: SnapshotStateList<String>,
pollType: PollType,
onPollTypeChange: (PollType) -> Unit,
pollDurationDays: Int?,
onDurationChange: (Int?) -> Unit,
) {
Column(
modifier = Modifier.fillMaxWidth().padding(vertical = 8.dp),
verticalArrangement = Arrangement.spacedBy(8.dp),
) {
options.forEachIndexed { index, value ->
Row(
modifier = Modifier.fillMaxWidth(),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(4.dp),
) {
OutlinedTextField(
value = value,
onValueChange = { options[index] = it },
modifier = Modifier.weight(1f),
singleLine = true,
placeholder = { Text("Option ${index + 1}") },
)
IconButton(
onClick = { if (options.size > 2) options.removeAt(index) },
enabled = options.size > 2,
) {
Icon(MaterialSymbols.Close, contentDescription = "Remove option", modifier = Modifier.size(20.dp))
}
}
}
OutlinedButton(onClick = { options.add("") }) {
Icon(MaterialSymbols.Add, contentDescription = null, modifier = Modifier.size(18.dp))
Spacer(Modifier.width(4.dp))
Text("Add option")
}
Row(horizontalArrangement = Arrangement.spacedBy(8.dp)) {
FilterChip(
selected = pollType == PollType.SINGLE_CHOICE,
onClick = { onPollTypeChange(PollType.SINGLE_CHOICE) },
label = { Text("Single choice") },
)
FilterChip(
selected = pollType == PollType.MULTI_CHOICE,
onClick = { onPollTypeChange(PollType.MULTI_CHOICE) },
label = { Text("Multiple choice") },
)
}
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
Text("Ends:", style = MaterialTheme.typography.bodySmall)
listOf<Pair<String, Int?>>(
"Never" to null,
"1d" to 1,
"3d" to 3,
"7d" to 7,
).forEach { (label, days) ->
FilterChip(
selected = pollDurationDays == days,
onClick = { onDurationChange(days) },
label = { Text(label) },
leadingIcon =
if (pollDurationDays == days) {
{ Icon(MaterialSymbols.Check, contentDescription = null, modifier = Modifier.size(FilterChipDefaults.IconSize)) }
} else {
null
},
)
}
}
}
}
@Composable
private fun MentionSuggestionRow(
user: User,
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/FeedScreen.kt
+44
View File
@@ -132,6 +132,7 @@ import com.vitorpamplona.amethyst.desktop.subscriptions.createThreadRepliesSubsc
import com.vitorpamplona.amethyst.desktop.subscriptions.generateSubId
import com.vitorpamplona.amethyst.desktop.subscriptions.rememberSubscription
import com.vitorpamplona.amethyst.desktop.ui.media.LightboxOverlay
import com.vitorpamplona.amethyst.desktop.ui.note.DesktopPollCard
import com.vitorpamplona.amethyst.desktop.ui.note.NoteCard
import com.vitorpamplona.amethyst.desktop.ui.note.SpamCheckedNoteRender
import com.vitorpamplona.amethyst.desktop.ui.note.WoTBadgedAvatar
@@ -159,6 +160,8 @@ import com.vitorpamplona.quartz.nip19Bech32.Nip19Parser
import com.vitorpamplona.quartz.nip19Bech32.entities.NEvent
import com.vitorpamplona.quartz.nip19Bech32.entities.NNote
import com.vitorpamplona.quartz.nip22Comments.CommentEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.FlowPreview
import kotlinx.coroutines.GlobalScope
@@ -247,6 +250,21 @@ private fun FeedNoteCardBody(
myPubKeyHex: String? = null,
onFollow: ((String) -> Unit)? = null,
) {
if (event is PollEvent) {
DesktopPollCard(
note = note,
event = event,
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = myPubKeyHex,
onNavigateToThread = onNavigateToThread,
onNavigateToProfile = onNavigateToProfile,
onHashtagClick = onHashtagClick,
)
return
}
val isRepost = event is RepostEvent || event is GenericRepostEvent
if (isRepost) {
@@ -273,6 +291,22 @@ private fun FeedNoteCardBody(
return
}
// A boosted poll must still render as an interactive poll card, not a plain note.
if (originalEvent is PollEvent) {
DesktopPollCard(
note = originalNote,
event = originalEvent,
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = myPubKeyHex,
onNavigateToThread = onNavigateToThread,
onNavigateToProfile = onNavigateToProfile,
onHashtagClick = onHashtagClick,
)
return
}
val reactionCount = remember(reactionsState) { originalNote.countReactions() }
val replyCount = remember(repliesState) { originalNote.replies.size }
val repostCount = remember(metadataState) { originalNote.boosts.size }
@@ -859,6 +893,11 @@ fun FeedScreen(
kinds = listOf(com.vitorpamplona.quartz.nip10Notes.TextNoteEvent.KIND),
tags = mapOf("e" to interactionNoteIds),
),
// Poll responses (kind 1018) referencing these notes (NIP-88, lowercase `e`).
Filter(
kinds = listOf(PollResponseEvent.KIND),
tags = mapOf("e" to interactionNoteIds),
),
),
relays = allRelayUrls,
onEvent = { event, _, relay, _ ->
@@ -1117,6 +1156,7 @@ fun FeedScreen(
onSearchClick = openFullSearch,
relayManager = relayManager,
localCache = localCache,
account = account,
onNavigateToProfile = onNavigateToProfile,
onNavigateToThread = onNavigateToThread,
)
@@ -1156,6 +1196,7 @@ private fun FeedTabsHeader(
onSearchClick: () -> Unit = {},
relayManager: DesktopRelayConnectionManager? = null,
localCache: DesktopLocalCache? = null,
account: AccountState.LoggedIn? = null,
onNavigateToProfile: (String) -> Unit = {},
onNavigateToThread: (String) -> Unit = {},
) {
@@ -1508,6 +1549,9 @@ private fun FeedTabsHeader(
onNavigateToThread(noteId)
},
localCache = localCache,
relayManager = relayManager,
account = account,
myPubKeyHex = account?.pubKeyHex,
modifier = Modifier.heightIn(max = 400.dp).fillMaxWidth(),
)
} else if (isSearching) {
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/NoteActions.kt
+37
View File
@@ -103,6 +103,8 @@ import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip47WalletConnect.Nip47WalletConnect
import com.vitorpamplona.quartz.nip51Lists.bookmarkList.BookmarkListEvent
import com.vitorpamplona.quartz.nip57Zaps.LnZapEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import kotlinx.coroutines.DelicateCoroutinesApi
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.launch
@@ -1396,6 +1398,41 @@ private suspend fun reactToNote(
}
}
/**
* Casts a NIP-88 poll vote: builds a kind-1018 [PollResponseEvent] referencing [poll],
* signs it, optimistically consumes it locally (so the tally + hasVoted gate flip
* immediately), then broadcasts to all relays. The relay echo of the same signed event
* is deduped by id, so no double count.
*
* MUST be launched on a long-lived scope (e.g. `localCache.appScope`) — never a card's
* `rememberCoroutineScope()` — so scrolling the poll out of composition between the
* local consume and the broadcast can't cancel the send.
*/
suspend fun voteOnPoll(
poll: PollEvent,
responses: Set<String>,
account: AccountState.LoggedIn,
relayManager: DesktopRelayConnectionManager,
localCache: DesktopLocalCache,
) {
if (responses.isEmpty()) return
withContext(Dispatchers.IO) {
val template = PollResponseEvent.build(EventHintBundle(poll), responses)
val signed = account.signer.sign(template)
localCache.consume(signed, null, wasVerified = true)
// Publish to the poll's OWN declared relays (NIP-88 `relay` tags) as well as our
// connected relays — the poll author and other viewers read votes from the poll's
// relays, which we may not be connected to. broadcastToAll alone would lose the vote
// for everyone but us (mirrors the read path in DesktopPollCard.responseRelays).
val targetRelays = (poll.relays() + relayManager.connectedRelays.value).toSet()
if (targetRelays.isNotEmpty()) {
relayManager.publish(signed, targetRelays)
} else {
relayManager.broadcastToAll(signed)
}
}
}
/**
* Adds an event to bookmarks (public or private).
* Returns the new bookmark list event, or null if operation failed.
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/SearchScreen.kt
+22
View File
@@ -51,6 +51,7 @@ import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
@@ -109,6 +110,7 @@ import com.vitorpamplona.quartz.nip01Core.metadata.MetadataEvent
import com.vitorpamplona.quartz.nip05DnsIdentifiers.namecoin.NamecoinNameResolver
import com.vitorpamplona.quartz.nip05DnsIdentifiers.namecoin.NamecoinResolveOutcome
import com.vitorpamplona.quartz.nip19Bech32.decodePublicKeyAsHexOrNull
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import kotlinx.coroutines.launch
@Composable
@@ -361,6 +363,23 @@ fun SearchScreen(
}
}
// Fetch interactions (incl. kind-1018 poll responses) for poll results so their
// tallies populate — NIP-50 search returns the polls but not their responses.
val pollResultIds =
remember(noteResults) {
noteResults.filter { it.kind == PollEvent.KIND }.map { it.id }
}
DisposableEffect(pollResultIds, subscriptionsCoordinator, searchRelays) {
val coordinator = subscriptionsCoordinator
val subId =
if (coordinator != null && pollResultIds.isNotEmpty() && searchRelays.isNotEmpty()) {
coordinator.requestInteractions(pollResultIds, searchRelays)
} else {
null
}
onDispose { subId?.let { coordinator?.releaseInteractions(it) } }
}
// History state
val historyItems by SearchHistoryStore.history.collectAsState()
val savedSearches by SearchHistoryStore.savedSearches.collectAsState()
@@ -720,6 +739,9 @@ fun SearchScreen(
onNavigateToProfile = onNavigateToProfile,
onNavigateToThread = onNavigateToThread,
localCache = localCache,
relayManager = relayManager,
account = account,
myPubKeyHex = account?.pubKeyHex,
modifier = Modifier.padding(horizontal = sidePadding),
)
} else if (!debouncedQuery.isEmpty && !isSearching) {
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/ThreadScreen.kt
+1
View File
@@ -304,6 +304,7 @@ fun ThreadScreen(
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = account?.pubKeyHex,
nwcConnection = nwcConnection,
onReply = { rootNote.event?.let { onReply(it) } },
onZapFeedback = onZapFeedback,
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/UserProfileScreen.kt
+2
View File
@@ -944,6 +944,7 @@ fun UserProfileScreen(
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = account?.pubKeyHex,
nwcConnection = nwcConnection,
onReply = onCompose,
onZapFeedback = onZapFeedback,
@@ -1034,6 +1035,7 @@ fun UserProfileScreen(
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = account?.pubKeyHex,
nwcConnection = nwcConnection,
onReply = onCompose,
onZapFeedback = onZapFeedback,
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/note/DesktopPollCard.kt
+650
View File
@@ -0,0 +1,650 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.desktop.ui.note
import androidx.compose.animation.core.Animatable
import androidx.compose.foundation.background
import androidx.compose.foundation.border
import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.heightIn
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.layout.width
import androidx.compose.foundation.layout.widthIn
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.Button
import androidx.compose.material3.ElevatedCard
import androidx.compose.material3.HorizontalDivider
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.LaunchedEffect
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.key
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.alpha
import androidx.compose.ui.draw.clip
import androidx.compose.ui.draw.drawWithContent
import androidx.compose.ui.graphics.drawscope.clipRect
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.IntOffset
import androidx.compose.ui.unit.dp
import androidx.compose.ui.unit.sp
import androidx.compose.ui.window.Popup
import androidx.compose.ui.window.PopupProperties
import androidx.compose.ui.zIndex
import com.vitorpamplona.amethyst.commons.icons.symbols.Icon
import com.vitorpamplona.amethyst.commons.icons.symbols.MaterialSymbols
import com.vitorpamplona.amethyst.commons.model.Note
import com.vitorpamplona.amethyst.commons.model.User
import com.vitorpamplona.amethyst.commons.model.nip88Polls.TallyResults
import com.vitorpamplona.amethyst.commons.ui.components.UserAvatar
import com.vitorpamplona.amethyst.desktop.account.AccountState
import com.vitorpamplona.amethyst.desktop.cache.DesktopLocalCache
import com.vitorpamplona.amethyst.desktop.network.DesktopRelayConnectionManager
import com.vitorpamplona.amethyst.desktop.subscriptions.SubscriptionConfig
import com.vitorpamplona.amethyst.desktop.subscriptions.generateSubId
import com.vitorpamplona.amethyst.desktop.subscriptions.rememberSubscription
import com.vitorpamplona.amethyst.desktop.ui.toNoteDisplayData
import com.vitorpamplona.amethyst.desktop.ui.voteOnPoll
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.poll.tags.OptionTag
import com.vitorpamplona.quartz.nip88Polls.poll.tags.PollType
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.launch
import java.time.Instant
import java.time.ZoneId
import java.time.format.DateTimeFormatter
/**
* A single poll option paired with its (stable, per-event) tally flow so each option
* row is its own leaf collector — no combining all options into one flow (which would
* cause a recomposition storm).
*/
private class PollOptionFlow(
val option: OptionTag,
val results: Flow<TallyResults>,
val currentResults: () -> TallyResults,
)
/**
* Desktop NIP-88 poll card. Reuses [NoteCard] for the author/description/media header
* (via [bottomContent] slot for the interactive options) and renders the tally itself.
*
* Hide-until-voted (decision #2): controls are shown unless the viewer is the author,
* has voted, the poll ended, or opted into "View results". Re-vote allowed (decision #6):
* results view offers "Change vote".
*/
@Composable
fun DesktopPollCard(
note: Note,
event: PollEvent,
relayManager: DesktopRelayConnectionManager,
localCache: DesktopLocalCache,
account: AccountState.LoggedIn?,
myPubKeyHex: String?,
onNavigateToThread: (String) -> Unit = {},
onNavigateToProfile: (String) -> Unit = {},
onHashtagClick: ((String) -> Unit)? = null,
) {
val options = remember(event) { event.options() }
if (options.isEmpty()) return
val pollState = remember(note) { note.pollState() }
val forKey = myPubKeyHex ?: ""
// Load this poll's responses from the poll's OWN declared relays (NIP-88 `relay` tags)
// unioned with the viewer's connected relays. Votes are published to the poll's relays,
// which the viewer usually isn't subscribed to — so the feed/thread/search interaction
// fetches (which only query the viewer's relays) miss them and the tally shows just the
// viewer's own vote. Querying the poll's declared relays makes the full tally load in
// any context that renders this card.
val connectedRelays by relayManager.connectedRelays.collectAsState()
val responseRelays =
remember(event, connectedRelays) {
(event.relays() + connectedRelays).toSet()
}
rememberSubscription(responseRelays, relayManager = relayManager) {
if (responseRelays.isEmpty()) return@rememberSubscription null
SubscriptionConfig(
subId = generateSubId("poll-resp-${event.id.take(8)}"),
filters =
listOf(
Filter(
kinds = listOf(PollResponseEvent.KIND),
tags = mapOf("e" to listOf(event.id)),
),
),
relays = responseRelays,
onEvent = { ev, _, relay, _ -> localCache.consume(ev, relay, wasVerified = false) },
)
}
// One stable flow per option, built once per event (Delta #6).
val optionFlows =
remember(event) {
options.map { option ->
PollOptionFlow(
option = option,
results = pollState.tallyFlow(option.code, forKey, localCache.followedUsers),
currentResults = { pollState.currentTally(option.code, forKey, localCache.followedUsers.value) },
)
}
}
val pollType = remember(event) { event.pollType() }
val hasEnded = remember(event) { event.hasEnded() }
val isMyPoll = myPubKeyHex != null && event.pubKey == myPubKeyHex
// A read-only (watch-only) account can't sign — show results instead of dead controls.
val canVote = account != null && !account.isReadOnly
// Seed the voted-gate synchronously to avoid a first-frame flash (Delta #7).
val myUser = remember(note, myPubKeyHex) { myPubKeyHex?.let { localCache.getOrCreateUser(it) } }
val hasVotedSeed = remember(pollState, myUser) { myUser?.let { pollState.hasPubKeyVoted(it) } ?: false }
val hasVoted by
remember(pollState, myUser) {
myUser?.let { pollState.hasPubKeyVotedFlow(it) } ?: flowOf(false)
}.collectAsState(hasVotedSeed)
// Local UI state keyed by note id so LazyColumn slot recycling can't leak one
// poll's selection into another (Delta #9). `viewingResults` = opted into results
// before voting; `revoting` = tapped "Change vote" to reopen controls after voting.
var viewingResults by remember(note.idHex) { mutableStateOf(false) }
var revoting by remember(note.idHex) { mutableStateOf(false) }
// Tap a result row to see who voted for that option.
var voterPopup by remember(note.idHex) { mutableStateOf<Pair<String, List<User>>?>(null) }
// Total votes + deadline label for the footer.
val tallyState by pollState.responses.collectAsState()
// Distinct voters (not total selections) so a multi-choice voter counts once.
val totalVotes = tallyState.votes.size
// Pre-seed a multi-choice re-vote with the viewer's existing selection.
val myCurrentVote =
remember(tallyState, myUser) {
myUser?.let { tallyState.votes[it]?.responses()?.toSet() } ?: emptySet()
}
val endsAtSec = remember(event) { event.endsAt() }
val deadlineLabel =
remember(endsAtSec, hasEnded) {
endsAtSec?.let { (if (hasEnded) "Ended " else "Ends ") + formatPollTimestamp(it) }
}
val displayData = remember(event) { event.toNoteDisplayData(localCache) }
NoteCard(
note = displayData,
modifier = Modifier.fillMaxWidth(),
localCache = localCache,
onClick = { onNavigateToThread(event.id) },
onAuthorClick = onNavigateToProfile,
onMentionClick = onNavigateToProfile,
onHashtagClick = onHashtagClick,
onNavigateToThread = onNavigateToThread,
bottomContent = {
Column(
modifier = Modifier.fillMaxWidth().padding(horizontal = 16.dp, vertical = 4.dp),
verticalArrangement = Arrangement.spacedBy(6.dp),
) {
// Results gate (decision #2): author / ended / already-voted / opted-in
// see results — unless the viewer explicitly reopened controls to re-vote.
val showResults = !revoting && (isMyPoll || hasVoted || hasEnded || viewingResults || !canVote)
if (showResults) {
optionFlows.forEach { of ->
key(of.option.code) {
PollResultRow(of, forKey) { label, voters ->
voterPopup = label to voters
}
}
}
if (!hasEnded && !isMyPoll && canVote && hasVoted) {
Text(
text = "Change vote",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.primary,
modifier =
Modifier
.clip(RoundedCornerShape(4.dp))
.clickable { revoting = true }
.padding(horizontal = 6.dp, vertical = 4.dp),
)
}
} else {
when (pollType) {
PollType.SINGLE_CHOICE ->
SingleChoiceOptions(options, account) { code ->
revoting = false
castVote(event, setOf(code), account, relayManager, localCache)
}
PollType.MULTI_CHOICE ->
MultiChoiceOptions(note, options, account, myCurrentVote) { codes ->
revoting = false
castVote(event, codes, account, relayManager, localCache)
}
}
Text(
text = if (revoting) "Back to results" else "View results",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.primary,
modifier =
Modifier
.clip(RoundedCornerShape(4.dp))
.clickable {
if (revoting) revoting = false else viewingResults = true
}.padding(horizontal = 6.dp, vertical = 4.dp),
)
}
if (totalVotes > 0 || deadlineLabel != null) {
Row(
modifier = Modifier.fillMaxWidth().padding(top = 2.dp),
horizontalArrangement = Arrangement.SpaceBetween,
) {
Text(
text = "$totalVotes ${if (totalVotes == 1) "vote" else "votes"}",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
deadlineLabel?.let {
Text(
text = it,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
}
}
}
}
},
)
voterPopup?.let { (label, voters) ->
VoterListPopup(
optionLabel = label,
voters = voters,
forKey = forKey,
onDismiss = { voterPopup = null },
onNavigateToProfile = onNavigateToProfile,
)
}
}
private fun castVote(
event: PollEvent,
codes: Set<String>,
account: AccountState.LoggedIn?,
relayManager: DesktopRelayConnectionManager,
localCache: DesktopLocalCache,
) {
if (account == null || account.isReadOnly || codes.isEmpty()) return
// Launch on the cache-scoped scope, NOT the card's scope, so the consume→broadcast
// pair can't be half-cancelled when the card leaves composition (Delta #2).
localCache.appScope.launch {
voteOnPoll(event, codes, account, relayManager, localCache)
}
}
@Composable
private fun SingleChoiceOptions(
options: List<OptionTag>,
account: AccountState.LoggedIn?,
onRespond: (String) -> Unit,
) {
options.forEach { option ->
Box(
modifier =
Modifier
.fillMaxWidth()
.clip(RoundedCornerShape(8.dp))
.border(1.dp, MaterialTheme.colorScheme.outline, RoundedCornerShape(8.dp))
.then(
if (account != null) {
Modifier.clickable { onRespond(option.code) }
} else {
Modifier
},
),
) {
Row(
modifier = Modifier.fillMaxWidth().padding(12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
Icon(
symbol = MaterialSymbols.RadioButtonUnchecked,
contentDescription = null,
modifier = Modifier.size(20.dp),
tint = MaterialTheme.colorScheme.primary,
)
Text(text = option.label, style = MaterialTheme.typography.bodyMedium)
}
}
}
}
@Composable
private fun MultiChoiceOptions(
note: Note,
options: List<OptionTag>,
account: AccountState.LoggedIn?,
initialSelection: Set<String>,
onRespond: (Set<String>) -> Unit,
) {
// Keyed by note id so recycling doesn't leak selection across polls (Delta #9);
// seeded with the viewer's existing vote so a re-vote starts from prior choices.
var selected by remember(note.idHex) { mutableStateOf(initialSelection) }
options.forEach { option ->
val isChecked = option.code in selected
Box(
modifier =
Modifier
.fillMaxWidth()
.clip(RoundedCornerShape(8.dp))
.border(1.dp, MaterialTheme.colorScheme.outline, RoundedCornerShape(8.dp))
.clickable {
selected = if (isChecked) selected - option.code else selected + option.code
},
) {
Row(
modifier = Modifier.fillMaxWidth().padding(12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
// No CheckBox glyph in the subset font — a bordered box with a Check
// glyph when selected (avoids a new codepoint / font regen).
Box(
modifier =
Modifier
.size(20.dp)
.clip(RoundedCornerShape(4.dp))
.then(
if (isChecked) {
Modifier.background(MaterialTheme.colorScheme.primary)
} else {
Modifier.border(
1.dp,
MaterialTheme.colorScheme.outline,
RoundedCornerShape(4.dp),
)
},
),
contentAlignment = Alignment.Center,
) {
if (isChecked) {
Icon(
symbol = MaterialSymbols.Check,
contentDescription = null,
modifier = Modifier.size(16.dp),
tint = MaterialTheme.colorScheme.onPrimary,
)
}
}
Text(text = option.label, style = MaterialTheme.typography.bodyMedium)
}
}
}
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.End,
) {
Button(
onClick = { onRespond(selected) },
enabled = account != null && selected.isNotEmpty(),
) {
Text("Submit")
}
}
}
@Composable
private fun PollResultRow(
of: PollOptionFlow,
forKey: String,
onShowVoters: (String, List<User>) -> Unit,
) {
val tally by of.results.collectAsState(of.currentResults())
// First-frame bar guard: snap on first emission, animate afterwards (Delta #10).
val animated = remember { Animatable(tally.percent) }
LaunchedEffect(tally.percent) {
animated.animateTo(tally.percent)
}
val isMyVote = forKey.isNotEmpty() && tally.users.any { it.pubkeyHex == forKey }
val winning = tally.isWinning
val barColor = if (winning) MaterialTheme.colorScheme.tertiary else MaterialTheme.colorScheme.primary
// Border marks YOUR choice (primary); the winner is conveyed by the bar fill color.
val borderColor =
when {
isMyVote -> MaterialTheme.colorScheme.primary
winning -> MaterialTheme.colorScheme.tertiary
else -> MaterialTheme.colorScheme.outline
}
val borderWidth = if (isMyVote) 2.dp else 1.dp
Box(
modifier =
Modifier
.fillMaxWidth()
.clip(RoundedCornerShape(8.dp))
.border(borderWidth, borderColor, RoundedCornerShape(8.dp))
.background(MaterialTheme.colorScheme.surfaceVariant.copy(alpha = 0.4f))
.clickable { onShowVoters(of.option.label, tally.users) },
) {
Box(
modifier =
Modifier
.matchParentSize()
.alpha(0.32f)
.drawWithContent {
clipRect(right = size.width * animated.value) {
drawRect(barColor)
}
drawContent()
},
)
Row(
modifier = Modifier.fillMaxWidth().padding(12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween,
) {
Row(
modifier = Modifier.weight(1f),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(6.dp),
) {
if (isMyVote) {
Icon(
symbol = MaterialSymbols.Check,
contentDescription = "Your vote",
modifier = Modifier.size(16.dp),
tint = MaterialTheme.colorScheme.primary,
)
}
Text(
text = of.option.label,
style = MaterialTheme.typography.bodyMedium,
fontWeight = if (isMyVote) FontWeight.SemiBold else FontWeight.Normal,
)
}
Spacer(Modifier.width(12.dp))
Row(verticalAlignment = Alignment.CenterVertically) {
VoterGallery(tally.users, forKey)
Spacer(Modifier.width(8.dp))
Text(
text = "${(tally.percent * 100).toInt()}%",
style = MaterialTheme.typography.bodyMedium,
fontWeight = FontWeight.Bold,
textAlign = TextAlign.End,
)
}
}
}
}
@Composable
private fun VoterGallery(
users: List<User>,
forKey: String,
) {
if (users.isEmpty()) return
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy((-10).dp),
) {
users.take(4).forEachIndexed { index, user ->
key(user.pubkeyHex) {
val isMe = forKey.isNotEmpty() && user.pubkeyHex == forKey
UserAvatar(
userHex = user.pubkeyHex,
pictureUrl = user.profilePicture(),
size = 24.dp,
// Earlier avatars draw on top so the leftmost (you, sorted first) is
// front-most instead of buried under the next ones; ring your own.
modifier =
Modifier
.zIndex((users.size - index).toFloat())
.then(
if (isMe) {
Modifier.border(2.dp, MaterialTheme.colorScheme.primary, CircleShape)
} else {
Modifier
},
),
)
}
}
if (users.size > 4) {
Box(
contentAlignment = Alignment.Center,
modifier =
Modifier
.size(24.dp)
.clip(CircleShape)
.background(MaterialTheme.colorScheme.secondaryContainer),
) {
Text(
text = "+${users.size - 4}",
fontSize = 10.sp,
color = MaterialTheme.colorScheme.onSecondaryContainer,
)
}
}
}
}
@Composable
private fun VoterListPopup(
optionLabel: String,
voters: List<User>,
forKey: String,
onDismiss: () -> Unit,
onNavigateToProfile: (String) -> Unit,
) {
Popup(
alignment = Alignment.Center,
offset = IntOffset(0, 0),
onDismissRequest = onDismiss,
properties = PopupProperties(focusable = true),
) {
ElevatedCard(modifier = Modifier.widthIn(max = 320.dp)) {
Column(
modifier =
Modifier
.verticalScroll(rememberScrollState())
.heightIn(max = 360.dp)
.padding(12.dp),
verticalArrangement = Arrangement.spacedBy(8.dp),
) {
Text(
text = "${voters.size} ${if (voters.size == 1) "vote" else "votes"} · $optionLabel",
style = MaterialTheme.typography.titleSmall,
fontWeight = FontWeight.SemiBold,
)
HorizontalDivider()
if (voters.isEmpty()) {
Text(
text = "No votes yet",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
)
} else {
voters.forEach { user ->
key(user.pubkeyHex) {
val isMe = forKey.isNotEmpty() && user.pubkeyHex == forKey
Row(
modifier =
Modifier
.fillMaxWidth()
.clip(RoundedCornerShape(6.dp))
.clickable {
onNavigateToProfile(user.pubkeyHex)
onDismiss()
}.padding(vertical = 4.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp),
) {
UserAvatar(
userHex = user.pubkeyHex,
pictureUrl = user.profilePicture(),
size = 28.dp,
)
Text(
text = user.toBestDisplayName() + if (isMe) " (you)" else "",
style = MaterialTheme.typography.bodyMedium,
)
}
}
}
}
}
}
}
}
private val POLL_TIME_FORMAT: DateTimeFormatter = DateTimeFormatter.ofPattern("MMM d, HH:mm")
private fun formatPollTimestamp(epochSeconds: Long): String =
Instant
.ofEpochSecond(epochSeconds)
.atZone(ZoneId.systemDefault())
.format(POLL_TIME_FORMAT)
desktopApp/src/jvmMain/kotlin/com/vitorpamplona/amethyst/desktop/ui/search/SearchResultsList.kt
+103 -1
View File
@@ -61,12 +61,16 @@ import com.vitorpamplona.amethyst.commons.search.SearchSortOrder
import com.vitorpamplona.amethyst.commons.ui.components.UserSearchCard
import com.vitorpamplona.amethyst.commons.wot.LocalWoTReady
import com.vitorpamplona.amethyst.commons.wot.LocalWoTService
import com.vitorpamplona.amethyst.desktop.account.AccountState
import com.vitorpamplona.amethyst.desktop.cache.DesktopLocalCache
import com.vitorpamplona.amethyst.desktop.network.DesktopRelayConnectionManager
import com.vitorpamplona.amethyst.desktop.ui.note.DesktopPollCard
import com.vitorpamplona.amethyst.desktop.ui.note.NoteCard
import com.vitorpamplona.amethyst.desktop.ui.note.SpamCheckedNoteRender
import com.vitorpamplona.amethyst.desktop.ui.note.WoTBadge
import com.vitorpamplona.amethyst.desktop.ui.rememberDisplayData
import com.vitorpamplona.quartz.nip23LongContent.LongTextNoteEvent
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
@Composable
fun SearchResultsList(
@@ -74,6 +78,10 @@ fun SearchResultsList(
onNavigateToProfile: (String) -> Unit,
onNavigateToThread: (String) -> Unit,
localCache: DesktopLocalCache? = null,
relayManager: DesktopRelayConnectionManager? = null,
account: AccountState.LoggedIn? = null,
myPubKeyHex: String? = null,
onHashtagClick: ((String) -> Unit)? = null,
modifier: Modifier = Modifier,
listState: LazyListState = rememberLazyListState(),
) {
@@ -89,7 +97,8 @@ fun SearchResultsList(
// Group notes by kind
val textNotes = notes.filter { it.kind == 1 }
val articles = notes.filter { it.kind == LongTextNoteEvent.KIND }
val otherNotes = notes.filter { it.kind != 1 && it.kind != LongTextNoteEvent.KIND }
val polls = notes.filter { it.kind == PollEvent.KIND }
val otherNotes = notes.filter { it.kind != 1 && it.kind != LongTextNoteEvent.KIND && it.kind != PollEvent.KIND }
// Per-section collapsed state (absent = expanded)
val collapsedSections = remember { mutableStateMapOf<String, Boolean>() }
@@ -252,6 +261,56 @@ fun SearchResultsList(
}
}
// Polls section (interactive cards — read tallies + vote)
if (polls.isNotEmpty()) {
item(key = "divider-polls") { HorizontalDivider(Modifier.padding(vertical = 4.dp)) }
val collapsed = collapsedSections["polls"] == true
stickyHeader(key = "header-polls") {
SortableHeader(
title = "Polls",
count = polls.size,
icon = MaterialSymbols.Poll,
options = SearchSortOrder.EVENT_OPTIONS,
selected = eventSortOrder,
onSelect = { state.updateEventSortOrder(it) },
collapsed = collapsed,
onToggleCollapse = { collapsedSections["polls"] = !collapsed },
)
}
if (!collapsed) {
items(polls.take(5), key = { "poll-${it.id}" }) { event ->
PollSearchItem(
event = event as PollEvent,
localCache = localCache,
relayManager = relayManager,
account = account,
myPubKeyHex = myPubKeyHex,
onNavigateToThread = onNavigateToThread,
onNavigateToProfile = onNavigateToProfile,
onHashtagClick = onHashtagClick,
)
}
if (polls.size > 5) {
item(key = "polls-expand") {
ExpandableSection(
remaining = polls.drop(5),
) { event ->
PollSearchItem(
event = event as PollEvent,
localCache = localCache,
relayManager = relayManager,
account = account,
myPubKeyHex = myPubKeyHex,
onNavigateToThread = onNavigateToThread,
onNavigateToProfile = onNavigateToProfile,
onHashtagClick = onHashtagClick,
)
}
}
}
}
}
// Other section
if (otherNotes.isNotEmpty()) {
item(key = "divider-other") { HorizontalDivider(Modifier.padding(vertical = 4.dp)) }
@@ -320,6 +379,49 @@ private fun wotBadgeFor(userHex: String): (@Composable androidx.compose.foundati
}
}
@Composable
private fun PollSearchItem(
event: PollEvent,
localCache: DesktopLocalCache?,
relayManager: DesktopRelayConnectionManager?,
account: AccountState.LoggedIn?,
myPubKeyHex: String?,
onNavigateToThread: (String) -> Unit,
onNavigateToProfile: (String) -> Unit,
onHashtagClick: ((String) -> Unit)?,
) {
SpamCheckedNoteRender(
displayedEvent = event,
noteIdHex = event.id,
localCache = localCache,
) {
if (localCache != null && relayManager != null) {
// Interactive: read tallies + vote. Note is resolved from the cache; option
// rendering comes from the event, so an empty (unconsumed) note still renders.
DesktopPollCard(
note = localCache.getOrCreateNote(event.id),
event = event,
relayManager = relayManager,
localCache = localCache,
account = account,
myPubKeyHex = myPubKeyHex,
onNavigateToThread = onNavigateToThread,
onNavigateToProfile = onNavigateToProfile,
onHashtagClick = onHashtagClick,
)
} else {
// Read-only fallback when the live cache/relay manager isn't available.
NoteCard(
note = event.rememberDisplayData(localCache),
localCache = localCache,
onClick = { onNavigateToThread(event.id) },
onAuthorClick = onNavigateToProfile,
onMentionClick = onNavigateToProfile,
)
}
}
}
@Composable
private fun SortableHeader(
title: String,
desktopApp/src/jvmTest/kotlin/com/vitorpamplona/amethyst/desktop/cache/DesktopLocalCachePollTest.kt
Vendored
+113
View File
@@ -0,0 +1,113 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.amethyst.desktop.cache
import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair
import com.vitorpamplona.quartz.nip01Core.relay.normalizer.NormalizedRelayUrl
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
import com.vitorpamplona.quartz.nip88Polls.poll.PollEvent
import com.vitorpamplona.quartz.nip88Polls.response.PollResponseEvent
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* NIP-88 poll consumption: a kind-1068 poll becomes a renderable Note, and a kind-1018
* response is linked into that poll Note's `pollState()` tally. Second identical response
* (a relay echo) must not double-count.
*/
class DesktopLocalCachePollTest {
private val relayUrl = NormalizedRelayUrl("wss://relay.test/")
private fun signedPoll(
signer: NostrSignerSync,
createdAt: Long,
): PollEvent =
signer.sign(
createdAt = createdAt,
kind = PollEvent.KIND,
tags =
arrayOf(
arrayOf("option", "0", "Yes"),
arrayOf("option", "1", "No"),
arrayOf("polltype", "singlechoice"),
),
content = "Pick one",
)
private fun signedResponse(
signer: NostrSignerSync,
pollId: String,
option: String,
createdAt: Long,
): PollResponseEvent =
signer.sign(
createdAt = createdAt,
kind = PollResponseEvent.KIND,
tags =
arrayOf(
arrayOf("e", pollId),
arrayOf("response", option),
),
content = "",
)
@Test
fun `a poll response is linked into the poll's tally`() {
val cache = DesktopLocalCache()
val author = NostrSignerSync(KeyPair())
val voter = NostrSignerSync(KeyPair())
val poll = signedPoll(author, createdAt = 1_700_000_000)
assertTrue(cache.consume(poll, relayUrl, wasVerified = true), "poll should be consumed")
val response = signedResponse(voter, poll.id, option = "0", createdAt = 1_700_000_100)
assertTrue(cache.consume(response, relayUrl, wasVerified = true), "response should be consumed")
val pollNote = cache.getNoteIfExists(poll.id)
assertTrue(pollNote != null, "poll note must exist")
val tally = pollNote.pollState().responses.value
assertEquals(1, tally.totalVotes())
assertEquals("0", tally.winning())
}
@Test
fun `a duplicate response is not counted twice`() {
val cache = DesktopLocalCache()
val author = NostrSignerSync(KeyPair())
val voter = NostrSignerSync(KeyPair())
val poll = signedPoll(author, createdAt = 1_700_000_000)
cache.consume(poll, relayUrl, wasVerified = true)
val response = signedResponse(voter, poll.id, option = "1", createdAt = 1_700_000_100)
assertTrue(cache.consume(response, relayUrl, wasVerified = true))
// Same signed event echoed back by another relay — id-dedup must reject it.
assertTrue(!cache.consume(response, relayUrl, wasVerified = true))
val tally =
cache
.getNoteIfExists(poll.id)!!
.pollState()
.responses.value
assertEquals(1, tally.totalVotes())
}
}
desktopApp/src/jvmTest/kotlin/com/vitorpamplona/amethyst/desktop/filters/FilterBuildersTest.kt
+7 -7
View File
@@ -37,7 +37,7 @@ class FilterBuildersTest {
fun testTextNotesGlobal() {
val filter = FilterBuilders.textNotesGlobal(limit = 50)
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(50, filter.limit)
assertNull(filter.authors)
assertNull(filter.tags)
@@ -51,7 +51,7 @@ class FilterBuildersTest {
val until = 1640995200L // 2022-01-01
val filter = FilterBuilders.textNotesGlobal(limit = 100, since = since, until = until)
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(100, filter.limit)
assertEquals(since, filter.since)
assertEquals(until, filter.until)
@@ -62,7 +62,7 @@ class FilterBuildersTest {
val authors = listOf(testPubKey, testPubKey2)
val filter = FilterBuilders.textNotesFromAuthors(authors, limit = 25)
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(authors, filter.authors)
assertEquals(25, filter.limit)
assertNull(filter.tags)
@@ -74,7 +74,7 @@ class FilterBuildersTest {
val since = 1609459200L
val filter = FilterBuilders.textNotesFromAuthors(authors, limit = 10, since = since)
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(authors, filter.authors)
assertEquals(10, filter.limit)
assertEquals(since, filter.since)
@@ -432,7 +432,7 @@ class FilterBuildersTest {
val filter = FilterBuilders.textNotesGlobal(limit = 50)
assertTrue(!filter.isEmpty())
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(50, filter.limit)
}
@@ -442,7 +442,7 @@ class FilterBuildersTest {
val filter = FilterBuilders.textNotesFromAuthors(followedUsers, limit = 50)
assertTrue(!filter.isEmpty())
assertEquals(listOf(1, 6, 16), filter.kinds)
assertEquals(listOf(1, 6, 16, 1068), filter.kinds)
assertEquals(followedUsers, filter.authors)
assertEquals(50, filter.limit)
}
@@ -458,7 +458,7 @@ class FilterBuildersTest {
assertTrue(!contactListFilter.isEmpty())
assertEquals(listOf(0), metadataFilter.kinds)
assertEquals(listOf(1, 6, 16), postsFilter.kinds)
assertEquals(listOf(1, 6, 16, 1068), postsFilter.kinds)
assertEquals(listOf(3), contactListFilter.kinds)
}
geode/src/main/kotlin/com/vitorpamplona/geode/RelayIndexingStrategy.kt
+8
View File
@@ -57,6 +57,14 @@ fun relayIndexingStrategy(
// index unconditionally; without it the filter walks the whole
// time index.
indexEventsByPubkeyAlone = true,
// The tag ∩ author ∩ kind shape (DM rooms, reports-by-follows,
// follows-scoped community feeds — 65 client assembler call sites)
// otherwise reads every row for the tag/kind before filtering the
// author. TagAuthorIndexBenchmark @ 1M events: 14.2 ms -> 0.66 ms
// (~21x, growing with corpus size) with insert cost inside run noise
// (49.0 vs 47.4 µs/event). Existing DBs build the index on next open
// via ensureOptionalIndexes.
indexTagsWithKindAndPubkey = true,
indexFullTextSearch = fullTextSearch,
// Tokenize off the commit path; NostrServer drives the catch-up
// worker and search queries drain it first, so NIP-50 stays
quartz/RELAY.md
+2
View File
@@ -83,6 +83,8 @@ val store = EventStore(
By default, all single-letter tags with values are indexed. Override `shouldIndex(kind, tag)` for custom behavior. More indexes = faster queries but larger database.
Flag flips are safe on existing databases: any flag-gated index the strategy wants but the on-disk schema lacks is created on the next open (idempotent `CREATE INDEX IF NOT EXISTS`, one-time build cost) — no schema version bump involved. Disabling a flag never drops an existing index.
`indexFullTextSearch` defaults to `true` and controls the NIP-50 full-text index (`event_fts`). Set it to `false` when search is served elsewhere (e.g. a Vespa backend, or a `SearchEventSource` as shown below): inserts skip the FTS tokenization cost, no `event_fts` table/trigger is created, and any filter carrying a non-empty `search` term returns no matches.
## Non-Storage Relays (search, redirector, computed)
quartz/build.gradle.kts
+4 -2
View File
@@ -17,7 +17,7 @@ kotlin {
}
jvm {
compilerOptions {
jvmTarget.set(JvmTarget.JVM_21)
jvmTarget.set(JvmTarget.JVM_17)
}
}
@@ -33,7 +33,7 @@ kotlin {
.toInt()
compilerOptions {
jvmTarget.set(JvmTarget.JVM_21)
jvmTarget.set(JvmTarget.JVM_17)
}
optimization {
@@ -94,6 +94,8 @@ kotlin {
// Forward the negentropy-benchmark corpus size to the test JVM.
System.getProperty("negBenchN")?.let { systemProperty("negBenchN", it) }
System.getProperty("followBenchScale")?.let { systemProperty("followBenchScale", it) }
System.getProperty("tagBenchScale")?.let { systemProperty("tagBenchScale", it) }
System.getProperty("fsBenchScale")?.let { systemProperty("fsBenchScale", it) }
// Opt-in JFR profiling of a benchmark run (-PnegProfile=/tmp/neg.jfr).
(project.findProperty("negProfile") as? String)?.let {
jvmArgs("-XX:+FlightRecorder", "-XX:StartFlightRecording=filename=$it,settings=profile,dumponexit=true")
quartz/plans/2026-07-21-sqlite-query-scaling.md
+168
View File
@@ -0,0 +1,168 @@
# SQLite query scaling: contentless FTS + rowid search, tag-path merge, pooled statements
**Status: shipped.** Follow-up to a scale-curve report (`SQLite vs Vespa`,
throughput vs corpus size) showing the app-side SQLite store degrading with
corpus size on several query shapes — NIP-50 search ~18× (25k→400k), batch
ingest 16×, author-timeline 13×, follow-feed 2.4× — while point reads stayed
flat. Two of those (author-timeline, follow-feed) were already addressed
(the pubkey-alone index and `MergeQueryExecutor`) and mostly reflect running
the report under the *client* `DefaultIndexingStrategy` rather than
`relayIndexingStrategy()`. This change adds the **large-IN tag watcher** to the
merge executor, fixes the **FTS delete path** (which degraded with corpus
size) and shrinks the FTS index, fixes **NIP-50 search ordering** (it was
sorting by `created_at`, not relevance), and fixes a **statement-cache** miss
the merge paths hit.
It does **not** fix NIP-50 *search* latency: bm25 relevance scoring (like the
old `created_at` sort) must visit every match, so search cost still grows with
the match set (§1). Corpus-independent search is an external-engine job.
Everything here is read/size work; the write path and on-disk index set are
unchanged except the FTS table, which gets *smaller* and deletes faster.
## 1. FTS — contentless index (fast deletes + smaller), NOT a search-scaling fix
The old index was `fts5(event_header_row_id, content)`: it stored a second
copy of the tokenized text, its rowid was auto-assigned (unrelated to the
event), and — the real problem — the `fts_foreign_key` delete trigger deleted
by the `event_header_row_id` *column*, which FTS5 cannot seek, so it **scanned
the whole index per delete**.
Changes (`FullTextSearchModule`, `QueryBuilder`, DB version 4→5):
- **rowid = `event_headers.row_id`.** Set explicitly on every insert; the
delete trigger, reindex, and catch-up paths key off it. Deletes become an
O(log n) primary-key seek instead of an O(n) column scan. This is the win:
every event removal fires the trigger — replaceable rotation, kind-5,
expiration, right-to-vanish — so on the old schema deletion throughput
degraded with corpus size.
- **Contentless** (`fts5(content, content='', contentless_delete=1)`). The
indexed text is *derived* (`SearchableEvent.indexableContent()`, not a raw
column), so FTS5 **external-content** — which re-reads the source column from
the base table — cannot express it; **contentless** is the correct primitive.
It drops the stored content copy (index shrinks) and `contentless_delete=1`
keeps the delete trigger working.
- **Segment compaction.** `reindexAll` finishes with `'optimize'`, and the
periodic `SQLiteEventStore.optimize()` (geode's maintenance tick) folds in a
bounded `'merge'`, so incremental / deferred-catch-up inserts don't leave the
index as many small segments.
Delete cost — `FtsSearchScalingBenchmark.deleteByColumnVsByRowid`, 500 deletes,
in-memory:
| rows | by column (old) | by rowid (new) |
|---|---:|---:|
| 2k | 91.8 ms | 6.6 ms |
| 8k | 362.0 ms | 4.7 ms |
By-column grows ~linearly with the table (O(n)/delete); by-rowid is flat —
~78× at 8k rows and widening.
**Search ordering fixed to relevance (NIP-50), which the store was getting
wrong.** NIP-50 says results are returned "in descending order by quality of
search result ... not by the usual `.created_at`", with the limit applied after
the score — but the store sorted search by `created_at DESC` (pre-existing).
*Every* search filter now orders by FTS5 bm25 (`ORDER BY event_fts.rank`,
`created_at DESC` as a tie-break): the tag-free shape (`makeSimpleSearch`) and
`search + tag` (the `prepareRowIDSubQueries`/`makeQueryIn` path, which carries
the rank column through the row-id subquery via `projectRank` and applies the
LIMIT by rank). Only the negentropy snapshot keeps `created_at` — a sync set,
not a ranked result. Verified against stronger-but-older matches outranking
weaker-but-newer ones, tag-scoped, with the limit cutting by score
(`SearchRelevanceOrderTest`, `Fts5CapabilityProbe`).
An earlier draft of *this* change instead added a `searchOrderByRowId` flag
(order by the FTS rowid for O(limit) search); that is *ingestion* order — wrong
events under a limit once ingestion diverges from time order, and not relevance
either — so it was removed.
This is a **correctness** fix, not a scaling one: bm25 (like the created_at
sort) must score every match, so search latency still grows with the match set
(the report's 18× curve) and `optimize` doesn't change the asymptotics
(measured within noise at 100k/200k). Corpus-independent search is genuinely an
external-engine job (the Vespa side of the report), not this index.
Migration (v4→v5): the old rowids can't be remapped, so `event_fts` is dropped
and rebuilt — synchronous stores rebuild in the upgrade transaction (client
corpora are small), deferred stores reset the catch-up watermark to 0 and let
the background worker repopulate (no long migration transaction).
`ContentlessFtsMigrationTest` fabricates a real v4 DB (old schema + garbage
rows + `user_version=4`) and asserts the reopen rebuilds search, wipes the
stale rows, maps rowid→row_id, and keeps the delete trigger working.
## 2. Tag watcher — extend `MergeQueryExecutor` to the tag path
`kinds=[7] AND #e=[hundreds of note ids] LIMIT n` (reactions/replies) had the
same shape the follow-feed fix already solved for authors: the per-value
streams come sorted off `(tag_hash[, kind], created_at)`, but their union does
not, so SQLite collected every matching row and TEMP-B-TREE-sorted to the limit
— O(matching history), growing with the corpus (`TagAuthorIndexBenchmark`:
12.8 ms cold at 200k → 14.2 ms at 1M).
`MergeQueryExecutor` now opens one lazy newest-first cursor per `(value[,
kind])` off `query_by_tags_hash_kind` (or `query_by_tags_hash` when there is no
kind, gated by `indexTagsByCreatedAtAlone`) and heap-merges to the limit —
**O(limit + streams)**. Unlike the author path, one event can carry several
queried tag values, so the tag merge **dedups by event id** through a `seen`
set (the author path, one pubkey per event, skips it). Eligibility is narrow
(one non-`d` tag key, ≥2 values, a limit, no ids/authors/d-tag/search, no
AND-tags); everything else falls through to the single-SQL plan. Counts and
deletes are unchanged (still single-SQL). `TagMergeCorrectnessTest` pins it
against a Kotlin reference including cross-stream dedup, since/until windows,
the raw path, and tie handling (tag cursors order off `event_tags`, which has
no id column, so same-second ties are a valid newest-N but not id-exact).
## 3. Pooled statement cache — so the merges actually cache
`StatementCachingConnection` kept **one** handle per SQL string. The k-way
merge opens *many* identical-SQL cursors at once (one per author/tag stream),
so every stream past the first missed the cache and prepared uncached, and a
repeated follow-feed / reactions REQ re-prepared all of them each poll. The
cache now keeps a small **pool** per SQL (bounded by a global cap, default
raised 256→512), so concurrent same-SQL checkouts reuse cached handles and
repeated polls reuse their per-stream cursors. `StatementCachingConnectionTest`
covers sequential reuse, concurrent distinctness/independence, freed-handle
reuse, and cap overflow → uncached fallback.
## Correctness / scope
- DB version bump 4→5 with a real-upgrade test.
- New capability test (`Fts5CapabilityProbe`) pins the FTS5 features the
contentless index needs (contentless_delete, absent-rowid delete no-op,
rowid ordering, merge/optimize) to the bundled SQLite (3.50.1), so a driver
downgrade fails loudly.
- All existing `store.sqlite` tests pass unchanged except `QueryAssemblerTest`,
whose asserted EXPLAIN output updated for the new join column
(`event_fts.rowid`) and the contentless table's virtual-index marker
(`0:M2`→`0:M1`).
- Search ordering: bm25 relevance for every search REQ shape — tag-free,
`search + tag`, and multi-filter all-search (e.g. the client's
search-across-kinds, unioned then deduped by event keeping the best score).
Mixed search/non-search multi-filter REQs stay `created_at` (a non-search
branch has no defined relevance).
## Audit follow-ups (post-review hardening)
A two-reviewer adversarial audit of the branch found no correctness/data-loss/
crash bugs; it produced these robustness fixes and one gap-closure:
- **Multi-filter search ordering** (gap): a REQ of several filters that all
carry a search term was `created_at`-ordered (the union path). Now
relevance-ordered via `unionSubqueriesIfNeeded(projectRank)` — each branch
projects `rank`, `UNION ALL` + `GROUP BY row_id MIN(rank)` dedups across
branches keeping the best score. Only when every branch is a search branch
(and FTS is on); count/delete unions stay single-column.
- **Merge stream-prep leak** (F1): `prepareAuthorStreams`/`prepareTagStreams`
now build cursors through `buildStreams`, which closes any already-prepared
statements if a later prepare throws — otherwise a mid-loop failure stranded
checked-out, un-reset handles (read locks) in the pooled connection.
- **Stream-count overflow** (F3): `authors×kinds` / `values×kinds` computed as
`Long` so a pathological product can't wrap `Int` back into the eligible
band and route a huge fan-out into the merge.
- **Finalizer footgun** (F2): the pooled statement's `finalize()` was renamed
`finalizeStatement()` — a no-arg `finalize()` is the JVM's `Object.finalize`,
so the GC could double-close the native handle after explicit close.
Migration cost noted (low/operational, not correctness): the synchronous
v4→v5 rebuild runs in the upgrade transaction; atomic and kill-safe (rolls
back to v4), but a very large client store pays a one-time first-open stall.
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/concord/cord02Community/ConcordCommunityState.kt
+2 -2
View File
@@ -33,7 +33,7 @@ import com.vitorpamplona.quartz.concord.cord04Roles.RoleEntity
import com.vitorpamplona.quartz.concord.cord04Roles.asFloor
/** A channel id paired with its current folded definition. */
class ConcordChannel(
data class ConcordChannel(
val channelIdHex: String,
val definition: ChannelEntity,
)
@@ -49,7 +49,7 @@ class ConcordChannel(
* "Every member keeps the entire Control Plane in sync — it is small and must
* stay complete." Recompute this whenever the known editions change.
*/
class ConcordCommunityState(
data class ConcordCommunityState(
val ownerPubKey: String,
val metadata: MetadataEntity?,
val channels: Map<String, ConcordChannel>,
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/concord/cord04Roles/AuthorityResolver.kt
+1 -1
View File
@@ -44,7 +44,7 @@ import com.vitorpamplona.quartz.nip01Core.core.toHexKey
* assigned Role and holds [ConcordPermissions.MANAGE_ROLES]. Cycles that never
* touch the owner can never bootstrap themselves.
*/
class AuthorityResolver private constructor(
data class AuthorityResolver private constructor(
private val ownerLower: String,
private val roles: Map<String, RoleEntity>,
private val memberRoles: Map<String, Set<String>>,
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/concord/cord04Roles/ControlEntities.kt
+5 -5
View File
@@ -64,7 +64,7 @@ object ConcordJson {
* drops the role, and with it every authority (grant) that depends on it.
*/
@Serializable
class RoleScope(
data class RoleScope(
val kind: String = "server",
@SerialName("channel_id") val channelId: String? = null,
)
@@ -75,7 +75,7 @@ class RoleScope(
* ranks higher; no role may claim position 0 (reserved for the owner).
*/
@Serializable
class RoleEntity(
data class RoleEntity(
val name: String = "",
val position: Long = 0,
/** u64 permission bitfield as a decimal string. */
@@ -94,7 +94,7 @@ class RoleEntity(
* terminates at the owner (see [AuthorityResolver]).
*/
@Serializable
class GrantEntity(
data class GrantEntity(
val member: String = "",
@SerialName("role_ids") val roleIds: List<String> = emptyList(),
)
@@ -105,7 +105,7 @@ class GrantEntity(
* A [deleted] channel is terminal — its id is never reused.
*/
@Serializable
class ChannelEntity(
data class ChannelEntity(
val name: String = "",
val private: Boolean = false,
val voice: Boolean = false,
@@ -122,7 +122,7 @@ class ChannelEntity(
* community name too.
*/
@Serializable
class MetadataEntity(
data class MetadataEntity(
val name: String = "",
val icon: ImagePointer? = null,
val banner: ImagePointer? = null,
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/commands/toClient/EoseMessage.kt
+16
View File
@@ -25,6 +25,22 @@ class EoseMessage(
) : Message {
override fun label() = LABEL
/**
* Wire form is `["EOSE","<subId>"]` — sent once per REQ, so it is on
* the per-subscription floor. Splice it directly when [subId] needs no
* escaping (the common case: client-chosen sub ids are short ASCII),
* skipping the generic serializer's node tree. Byte-identical output;
* any exotic subId falls back.
*/
override fun toJson(): String {
if (!isEscapeFreeAscii(subId)) return super.toJson()
return buildString(subId.length + 12) {
append("[\"EOSE\",\"")
append(subId)
append("\"]")
}
}
companion object {
const val LABEL = "EOSE"
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/commands/toClient/OkMessage.kt
+19
View File
@@ -29,6 +29,25 @@ class OkMessage(
) : Message {
override fun label() = LABEL
/**
* Wire form is `["OK","<eventId>",<true|false>,"<message>"]` — sent
* once per published EVENT. [eventId] is validated hex (always
* escape-free); splice directly when [message] also needs no escaping,
* which covers the empty-string success ack and the plain-ASCII
* rejection reasons. Byte-identical output; a reason with quotes or
* non-ASCII falls back to the generic serializer.
*/
override fun toJson(): String {
if (!isEscapeFreeAscii(message)) return super.toJson()
return buildString(eventId.length + message.length + 20) {
append("[\"OK\",\"")
append(eventId)
append(if (success) "\",true,\"" else "\",false,\"")
append(message)
append("\"]")
}
}
companion object {
const val LABEL = "OK"
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/commands/toClient/WireJson.kt
+36
View File
@@ -0,0 +1,36 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.commands.toClient
/**
* True when every char of [s] is printable ASCII (0x20–0x7e) and not a JSON
* metacharacter (`"` / `\`) — i.e. exactly the bytes a JSON string encoder
* would emit verbatim between the quotes. Frame builders use this to gate a
* direct-`buildString` fast path against the generic serializer: when it holds
* the spliced output is byte-identical, and any exotic value (control chars,
* quotes, non-ASCII) falls back to the escaping serializer.
*/
internal fun isEscapeFreeAscii(s: String): Boolean {
for (c in s) {
if (c < ' ' || c > '~' || c == '"' || c == '\\') return false
}
return true
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/filters/FilterIndex.kt
+117 -35
View File
@@ -22,6 +22,11 @@ package com.vitorpamplona.quartz.nip01Core.relay.filters
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import kotlinx.collections.immutable.PersistentMap
import kotlinx.collections.immutable.PersistentSet
import kotlinx.collections.immutable.persistentHashMapOf
import kotlinx.collections.immutable.persistentHashSetOf
import kotlinx.collections.immutable.toPersistentHashSet
import kotlin.concurrent.atomics.AtomicReference
import kotlin.concurrent.atomics.ExperimentalAtomicApi
@@ -62,9 +67,10 @@ import kotlin.concurrent.atomics.ExperimentalAtomicApi
* copy-on-write CAS loops, mirroring the
* `nip86RelayManagement.server.BanStore` pattern. Reads in
* [candidatesFor] and [forEach] are wait-free single-load atomic.
* Writes (subscription register / unregister) copy the inner maps
* — fine for this workload because writes are subscription-rate
* (rare) while reads are event-rate (frequent).
* Writes (subscription register / unregister) build the next snapshot
* from persistent (HAMT) maps — O(keys × log S) with structural
* sharing rather than a full O(S) copy of both maps, since on a relay
* a write happens on every REQ open and close.
*
* ## What the index does NOT cover
*
@@ -109,14 +115,26 @@ class FilterIndex<S : Any> {
private object Unindexed : BucketKey
/**
* Single immutable snapshot. [buckets] maps a key to the set of
* subscribers registered under it; [assignments] is the reverse
* map used by [unregister] to find a subscriber's keys without
* scanning every bucket.
* Single immutable snapshot. Subscribers are held in one map per
* indexable dimension so [candidatesFor] — called once per accepted
* ingest event, the hot read — can probe each dimension with the
* event's own field (`event.id`, `event.pubKey`, `tag[0]`/`tag[1]`,
* `event.kind`) and allocate no key-wrapper objects. [assignments] is
* the reverse map ([S] → the [BucketKey]s it occupies) used by
* [unregister]; the wrappers live only here, built on the rare
* register path.
*
* Persistent (HAMT) maps/sets: a register/unregister produces the
* next snapshot in O(keys × log S) with structural sharing, instead
* of copying full maps — registration happens on every REQ open/close.
*/
private data class State<S>(
val buckets: Map<BucketKey, Set<S>> = emptyMap(),
val assignments: Map<S, Set<BucketKey>> = emptyMap(),
val ids: PersistentMap<HexKey, PersistentSet<S>> = persistentHashMapOf(),
val authors: PersistentMap<HexKey, PersistentSet<S>> = persistentHashMapOf(),
val tags: PersistentMap<String, PersistentMap<String, PersistentSet<S>>> = persistentHashMapOf(),
val kinds: PersistentMap<Int, PersistentSet<S>> = persistentHashMapOf(),
val unindexed: PersistentSet<S> = persistentHashSetOf(),
val assignments: PersistentMap<S, PersistentSet<BucketKey>> = persistentHashMapOf(),
)
private val state: AtomicReference<State<S>> = AtomicReference(State())
@@ -181,18 +199,23 @@ class FilterIndex<S : Any> {
while (true) {
val current = state.load()
val keys = current.assignments[subscriber] ?: return
val newBuckets = current.buckets.toMutableMap()
var ids = current.ids
var authors = current.authors
var tags = current.tags
var kinds = current.kinds
var unindexed = current.unindexed
for (key in keys) {
val cur = newBuckets[key] ?: continue
val next = cur - subscriber
if (next.isEmpty()) {
newBuckets.remove(key)
} else {
newBuckets[key] = next
when (key) {
is IdKey -> ids = ids.removeSub(key.id, subscriber)
is AuthorKey -> authors = authors.removeSub(key.author, subscriber)
is KindKey -> kinds = kinds.removeSub(key.kind, subscriber)
is TagKey -> tags = tags.removeTagSub(key.letter, key.value, subscriber)
Unindexed -> unindexed = unindexed.remove(subscriber)
}
}
val newAssignments = current.assignments - subscriber
if (state.compareAndSet(current, State(newBuckets, newAssignments))) return
val next =
State(ids, authors, tags, kinds, unindexed, current.assignments.remove(subscriber))
if (state.compareAndSet(current, next)) return
}
}
@@ -202,19 +225,22 @@ class FilterIndex<S : Any> {
* candidate to handle negative constraints.
*
* Iteration order is insertion-stable per call but otherwise
* unspecified.
* unspecified. Allocates only the result set — dimensions are
* probed with the event's own fields, no key wrappers.
*/
fun candidatesFor(event: Event): Set<S> {
val s = state.load()
if (s.buckets.isEmpty()) return emptySet()
if (s.assignments.isEmpty()) return emptySet()
val result = LinkedHashSet<S>()
s.buckets[Unindexed]?.let { result.addAll(it) }
s.buckets[IdKey(event.id)]?.let { result.addAll(it) }
s.buckets[AuthorKey(event.pubKey)]?.let { result.addAll(it) }
s.buckets[KindKey(event.kind)]?.let { result.addAll(it) }
for (tag in event.tags) {
if (tag.size >= 2 && tag[0].length == 1) {
s.buckets[TagKey(tag[0], tag[1])]?.let { result.addAll(it) }
if (s.unindexed.isNotEmpty()) result.addAll(s.unindexed)
s.ids[event.id]?.let { result.addAll(it) }
s.authors[event.pubKey]?.let { result.addAll(it) }
s.kinds[event.kind]?.let { result.addAll(it) }
if (s.tags.isNotEmpty()) {
for (tag in event.tags) {
if (tag.size >= 2 && tag[0].length == 1) {
s.tags[tag[0]]?.get(tag[1])?.let { result.addAll(it) }
}
}
}
return result
@@ -235,19 +261,75 @@ class FilterIndex<S : Any> {
keys: List<BucketKey>,
) {
if (keys.isEmpty()) return
val keySet = keys.toSet()
val keySet = keys.toPersistentHashSet()
while (true) {
val current = state.load()
val newBuckets = current.buckets.toMutableMap()
var ids = current.ids
var authors = current.authors
var tags = current.tags
var kinds = current.kinds
var unindexed = current.unindexed
for (key in keySet) {
val cur = newBuckets[key] ?: emptySet()
if (subscriber in cur) continue
newBuckets[key] = cur + subscriber
when (key) {
is IdKey -> ids = ids.addSub(key.id, subscriber)
is AuthorKey -> authors = authors.addSub(key.author, subscriber)
is KindKey -> kinds = kinds.addSub(key.kind, subscriber)
is TagKey -> tags = tags.addTagSub(key.letter, key.value, subscriber)
Unindexed -> unindexed = unindexed.add(subscriber)
}
}
val existing = current.assignments[subscriber]
val merged = if (existing == null) keySet else existing + keySet
val newAssignments = current.assignments + (subscriber to merged)
if (state.compareAndSet(current, State(newBuckets, newAssignments))) return
val merged = existing?.addAll(keySet) ?: keySet
val next = State(ids, authors, tags, kinds, unindexed, current.assignments.put(subscriber, merged))
if (state.compareAndSet(current, next)) return
}
}
// Per-dimension add/remove of one subscriber, returning the same map
// instance when nothing changed so the CAS builds minimal new nodes.
private fun <K> PersistentMap<K, PersistentSet<S>>.addSub(
key: K,
sub: S,
): PersistentMap<K, PersistentSet<S>> {
val cur = this[key] ?: persistentHashSetOf()
val next = cur.add(sub)
return if (next === cur) this else put(key, next)
}
private fun <K> PersistentMap<K, PersistentSet<S>>.removeSub(
key: K,
sub: S,
): PersistentMap<K, PersistentSet<S>> {
val cur = this[key] ?: return this
val next = cur.remove(sub)
return when {
next === cur -> this
next.isEmpty() -> remove(key)
else -> put(key, next)
}
}
private fun PersistentMap<String, PersistentMap<String, PersistentSet<S>>>.addTagSub(
letter: String,
value: String,
sub: S,
): PersistentMap<String, PersistentMap<String, PersistentSet<S>>> {
val inner = this[letter] ?: persistentHashMapOf()
val newInner = inner.addSub(value, sub)
return if (newInner === inner) this else put(letter, newInner)
}
private fun PersistentMap<String, PersistentMap<String, PersistentSet<S>>>.removeTagSub(
letter: String,
value: String,
sub: S,
): PersistentMap<String, PersistentMap<String, PersistentSet<S>>> {
val inner = this[letter] ?: return this
val newInner = inner.removeSub(value, sub)
return when {
newInner === inner -> this
newInner.isEmpty() -> remove(letter)
else -> put(letter, newInner)
}
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/RelaySession.kt
+24 -2
View File
@@ -49,6 +49,7 @@ import com.vitorpamplona.quartz.utils.Log
import com.vitorpamplona.quartz.utils.cache.LargeCache
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.CoroutineStart
import kotlinx.coroutines.Job
import kotlinx.coroutines.channels.ClosedSendChannelException
import kotlinx.coroutines.launch
@@ -291,8 +292,16 @@ class RelaySession(
// Policy may rewrite filters to match the user's access level.
val filters = (result as PolicyResult.Accepted).cmd.filters
// UNDISPATCHED: the stored replay runs inline on this coroutine —
// the reader-pool acquire doesn't suspend when a connection is
// free, so EVENT frames and EOSE go out without a scheduler hop
// (SmallReqFloorBenchmark: the hop was most of the dispatch
// slice on small REQs). The coroutine first parks at the live
// tail (awaitCancellation), which is when launch returns and the
// job lands in [subscriptions]; commands on this connection are
// processed sequentially, so nothing can target the sub earlier.
val job =
scope.launch {
scope.launch(start = CoroutineStart.UNDISPATCHED) {
try {
if (policy.filtersOutgoingEvents) {
// Screened path: every event is materialized so the
@@ -331,7 +340,20 @@ class RelaySession(
},
)
},
onEachLive = { event -> send(EventMessage(cmd.subId, event)) },
// Live events arrive with their wire body already
// serialized (once per event, shared across every
// matching subscription): splice it into the same
// per-sub frame prefix as the stored replay, no
// per-event EventMessage or re-serialize.
onEachLive = { _, body ->
sendRaw(
buildString(framePrefix.length + body.length + 1) {
append(framePrefix)
append(body)
append(']')
},
)
},
onEose = { send(EoseMessage(cmd.subId)) },
)
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/backend/LiveEventStore.kt
+73 -63
View File
@@ -74,13 +74,57 @@ class LiveEventStore(
* One live REQ subscription. Carries the filters (for the
* post-index `match` re-check needed for negative constraints
* like `since` / `until` / `tagsAll`) and the delivery callback
* the index dispatches into. Identity-keyed inside [FilterIndex].
* the index dispatches into. [deliver] receives the event and its
* pre-serialized wire body (memoized once per fanout across all
* matching subscribers). Identity-keyed inside [FilterIndex].
*/
private class LiveSubscription(
val filters: List<Filter>,
val deliver: (Event) -> Unit,
val deliver: (Event, String) -> Unit,
)
/**
* Replay-dedupe set for one REQ. During the historical replay the
* store's ids are [record]ed here so the concurrent live path can
* drop an event the replay also emitted; after EOSE the set is
* [release]d and the live path forwards everything.
*
* Written from the replay coroutine and read from the [IngestQueue]
* drain coroutine (via `fanout`), so every access takes a tiny spin
* lock — [locked] is `inline`, so the per-row `record` / `isDuplicate`
* calls allocate no closure. The backing `HashSet` is created empty
* up front (so the register-before-replay race guarantee holds) but
* the JVM defers its table allocation to the first `add`, so a
* zero-row replay costs only the empty set object, not a sized table.
* It MUST stay a mutable set under a lock, never a copy-on-add
* immutable set — `set + id` per row made large replays O(n²).
*/
private class SeenIds {
private val lock = AtomicBoolean(false)
private var ids: HashSet<String>? = HashSet()
private inline fun <R> locked(block: () -> R): R {
while (lock.exchange(true)) {
while (lock.load()) { }
}
try {
return block()
} finally {
lock.store(false)
}
}
fun record(id: String) {
locked { ids?.add(id) }
}
fun isDuplicate(id: String): Boolean = locked { ids?.contains(id) ?: false }
fun release() {
locked { ids = null }
}
}
/**
* Fire-and-forget enqueue: hand [event] to the [IngestQueue] and
* fire [onComplete] once the writer's batch has a per-row
@@ -149,9 +193,17 @@ class LiveEventStore(
* batch writer.
*/
private fun fanout(event: Event) {
for (sub in index.candidatesFor(event)) {
val candidates = index.candidatesFor(event)
if (candidates.isEmpty()) return
// Serialize the wire body at most once for this event, no matter
// how many subscriptions match it — the old path re-serialized the
// whole event per matching subscriber, so a note landing in N live
// feeds paid N identical Jackson passes. Lazy so a fanout that
// matches nothing (index over-approximates) serializes nothing.
var body: String? = null
for (sub in candidates) {
if (sub.filters.any { it.match(event) }) {
sub.deliver(event)
sub.deliver(event, body ?: event.toJson().also { body = it })
}
}
}
@@ -178,61 +230,33 @@ class LiveEventStore(
onEose: () -> Unit,
) {
drainFtsIfSearching(filters)
// During the historical replay, record ids the store has
// emitted so the live path can dedupe. The index registers
// *before* the replay starts (otherwise an event accepted
// mid-replay would slip past the live path entirely — same
// race the previous SharedFlow-based implementation closed
// with `onSubscription`).
//
// The set is read from the [IngestQueue] drain coroutine (in
// `deliver`, called synchronously from `fanout`) and written
// from this coroutine (the historical-replay closure below),
// so access is guarded by a tiny spin lock (contains/add,
// never I/O). It MUST be a mutable set under a lock, not an
// immutable Set under an AtomicReference with copy-on-add:
// `set + id` copies the whole set per streamed event, which
// made large replays accidentally O(n²) — a 100k-event REQ
// crawled at ~700 events/s and the rate degraded as the
// response grew (see the plan doc's giant-REQ finding).
//
// Once cleared to null after EOSE, `deliver` short-circuits
// and every live event is forwarded.
val seenLock = AtomicBoolean(false)
var seenIds: HashSet<String>? = HashSet(1024)
fun <R> seenLocked(block: () -> R): R {
while (seenLock.exchange(true)) {
while (seenLock.load()) { }
}
try {
return block()
} finally {
seenLock.store(false)
}
}
// The index registers *before* the replay starts (otherwise an
// event accepted mid-replay would slip past the live path entirely
// — same race the previous SharedFlow-based implementation closed
// with `onSubscription`), and [SeenIds] bridges the two coroutines:
// the replay records ids here, the live `deliver` drops duplicates,
// and after EOSE the set is released so every live event forwards.
val seen = SeenIds()
val sub =
LiveSubscription(
filters = filters,
deliver = { event ->
val duplicate = seenLocked { seenIds?.contains(event.id) ?: false }
if (duplicate) return@LiveSubscription
onEach(event)
deliver = { event, _ ->
if (!seen.isDuplicate(event.id)) onEach(event)
},
)
index.register(filters, sub)
try {
store.query<Event>(filters.strippingSearchExtensions()) { event ->
seenLocked { seenIds?.add(event.id) }
seen.record(event.id)
onEach(event)
}
onEose()
// Drop the dedupe set so the live path stops paying for
// it. From this point the index drives delivery and
// duplicates are no longer possible.
seenLocked { seenIds = null }
seen.release()
// Suspend until the caller's coroutine is cancelled
// (e.g. NIP-01 CLOSE or connection drop). The `finally`
// unregisters from the index.
@@ -255,42 +279,28 @@ class LiveEventStore(
ctx: RequestContext,
filters: List<Filter>,
onEachStored: (RawEvent) -> Unit,
onEachLive: (Event) -> Unit,
onEachLive: (Event, String) -> Unit,
onEose: () -> Unit,
) {
drainFtsIfSearching(filters)
val seenLock = AtomicBoolean(false)
var seenIds: HashSet<String>? = HashSet(1024)
fun <R> seenLocked(block: () -> R): R {
while (seenLock.exchange(true)) {
while (seenLock.load()) { }
}
try {
return block()
} finally {
seenLock.store(false)
}
}
val seen = SeenIds()
val sub =
LiveSubscription(
filters = filters,
deliver = { event ->
val duplicate = seenLocked { seenIds?.contains(event.id) ?: false }
if (duplicate) return@LiveSubscription
onEachLive(event)
deliver = { event, body ->
if (!seen.isDuplicate(event.id)) onEachLive(event, body)
},
)
index.register(filters, sub)
try {
store.rawQuery(filters.strippingSearchExtensions()) { raw ->
seenLocked { seenIds?.add(raw.id) }
seen.record(raw.id)
onEachStored(raw)
}
onEose()
seenLocked { seenIds = null }
seen.release()
awaitCancellation()
} finally {
index.unregister(sub)
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/relay/server/backend/SessionBackend.kt
+2 -2
View File
@@ -78,9 +78,9 @@ interface SessionBackend {
ctx: RequestContext,
filters: List<Filter>,
onEachStored: (RawEvent) -> Unit,
onEachLive: (Event) -> Unit,
onEachLive: (Event, String) -> Unit,
onEose: () -> Unit,
): Unit = query(ctx, filters, onEachLive, onEose)
): Unit = query(ctx, filters, { onEachLive(it, it.toJson()) }, onEose)
/** Answers a NIP-45 COUNT with an exact cardinality for the caller in [ctx]. */
suspend fun count(
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/EventIndexesModule.kt
+32 -7
View File
@@ -147,13 +147,38 @@ class EventIndexesModule(
*/
fun migrateV2AddPubkeyIndex(db: SQLiteConnection) {
if (!indexStrategy.indexEventsByPubkeyAlone) return
val orderBy =
if (indexStrategy.useAndIndexIdOnOrderBy) {
"created_at DESC, id ASC"
} else {
"created_at DESC"
}
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_pubkey_created ON event_headers (pubkey, $orderBy)")
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_pubkey_created ON event_headers (pubkey, ${orderByColumns()})")
}
private fun orderByColumns() =
if (indexStrategy.useAndIndexIdOnOrderBy) {
"created_at DESC, id ASC"
} else {
"created_at DESC"
}
/**
* Materializes any flag-gated index the current [indexStrategy] wants
* but the on-disk schema predates. Flags are runtime configuration, not
* schema — a deployment can flip one without a `user_version` bump — so
* this runs idempotently on every open. The first open after enabling a
* flag pays a one-time index build over the existing rows; subsequent
* opens are no-ops. A disabled flag never drops an existing index (that
* stays an operator decision).
*/
fun ensureOptionalIndexes(db: SQLiteConnection) {
if (indexStrategy.indexEventsByCreatedAtAlone) {
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_created_at_id ON event_headers (${orderByColumns()})")
}
if (indexStrategy.indexEventsByPubkeyAlone) {
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_pubkey_created ON event_headers (pubkey, ${orderByColumns()})")
}
if (indexStrategy.indexTagsByCreatedAtAlone) {
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_tags_hash ON event_tags (tag_hash, created_at DESC)")
}
if (indexStrategy.indexTagsWithKindAndPubkey) {
db.execSQL("CREATE INDEX IF NOT EXISTS query_by_tags_hash_kind_pubkey ON event_tags (tag_hash, kind, pubkey_hash, created_at DESC)")
}
}
val sqlInsertHeader =
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/FullTextSearchModule.kt
+146 -19
View File
@@ -31,6 +31,37 @@ import com.vitorpamplona.quartz.utils.EventFactory
/**
* NIP-50 full-text search index over event content.
*
* The index is a **contentless** FTS5 table (`content=''`,
* `contentless_delete=1`) whose `rowid` is the event's
* `event_headers.row_id`. Two consequences:
*
* - **Deletes.** The `fts_foreign_key` trigger deletes by `rowid` (an FTS5
* primary-key seek, O(log n)). The old `fts5(event_header_row_id,
* content)` schema deleted by a *regular* column, which FTS5 cannot seek —
* it scans the whole index per delete (O(n)), so deletion throughput
* degraded with corpus size. Every event removal fires this trigger
* (replaceable rotation, kind-5, expiration, right-to-vanish), so the
* seek matters. Measured `Fts5CapabilityProbe`/`FtsSearchScalingBenchmark`:
* ~78× at 8k rows and widening with the table.
* - **Size.** A contentless table stores only the inverted index, not a
* second copy of the tokenized text. The indexed text is *derived*
* ([SearchableEvent.indexableContent], not any raw column), so FTS5
* external-content — which reads the source column from the base table —
* cannot express it; contentless is the correct primitive.
*
* Search results are ordered by **relevance**, per NIP-50 ("descending order
* by quality of search result ... not by the usual `.created_at`", limit
* applied after the score) — via FTS5 bm25 (`ORDER BY event_fts.rank`), with
* `created_at DESC` only as a tie-break. This holds for *every* search filter:
* the tag-free shape ([QueryBuilder.makeSimpleSearch]), `search + tag` (whose
* row-id subquery carries the rank through via `projectRank`), and a
* multi-filter all-search REQ (unioned, deduped by event keeping the best
* score). Only the negentropy snapshot — and a multi-filter REQ mixing search
* and non-search branches — keep `created_at` (a sync set / a branch with no
* defined relevance). bm25 must score every match, so search latency still grows with the
* match set regardless of ordering; corpus-independent search needs an
* external engine, not this index.
*
* When [enabled] is `false` the module becomes an inert no-op: no
* `event_fts` virtual table and no `fts_foreign_key` delete trigger are
* created, inserts skip the per-event tokenization cost, and the reindex
@@ -52,29 +83,53 @@ class FullTextSearchModule(
) : IModule {
val tableName = "event_fts"
val triggerName = "fts_foreign_key"
val eventHeaderRowIdName = "event_header_row_id"
/**
* The FTS column that links back to `event_headers`. It is the implicit
* `rowid` of the (contentless) FTS table, which we set equal to
* `event_headers.row_id` on every insert — so the join is
* `event_headers.row_id = event_fts.rowid`, with no stored column.
*/
val rowIdColumn = "rowid"
val contentName = "content"
val stateTableName = "fts_catchup_state"
/**
* Whether the on-disk `event_fts` is an FTS5 table (vs the fts4/fts3
* fallback). Only FTS5 supports the contentless schema and the
* `merge`/`optimize` maintenance commands; the fallback stores content
* and skips maintenance. Read only on the (single-threaded) writer.
* Cached lazily from `sqlite_master` so a reopened DB — where [create]
* never runs — still resolves it.
*/
private var isFts5: Boolean? = null
override fun create(db: SQLiteConnection) {
if (!enabled) return
val ftsVersion = versionFinder(db)
db.execSQL(
"""
CREATE VIRTUAL TABLE $tableName
USING fts$ftsVersion($eventHeaderRowIdName, $contentName)
""",
)
isFts5 = ftsVersion >= 5
// FTS5: contentless index (no stored content copy) with delete
// support. fts4/fts3 (bundled driver never selects them) fall back to
// a plain content-storing table — rowid-explicit insert and
// delete-by-rowid work there too, only the size win is FTS5-only.
val columns =
if (ftsVersion >= 5) {
"$contentName, content='', contentless_delete=1"
} else {
contentName
}
db.execSQL("CREATE VIRTUAL TABLE $tableName USING fts$ftsVersion($columns)")
// Foreign key cleanup for full text search
// Foreign key cleanup for full text search. Deletes by the FTS rowid
// (= event_headers.row_id); a header with no FTS row (non-searchable
// kind) deletes nothing, which is a harmless no-op.
db.execSQL(
"""
CREATE TRIGGER $triggerName
AFTER DELETE ON event_headers
FOR EACH ROW
BEGIN
DELETE FROM $tableName
WHERE old.row_id = $tableName.$eventHeaderRowIdName;
DELETE FROM $tableName WHERE $tableName.rowid = old.row_id;
END;
""",
)
@@ -82,6 +137,16 @@ class FullTextSearchModule(
createStateTable(db)
}
private fun resolveIsFts5(db: SQLiteConnection): Boolean {
isFts5?.let { return it }
val sql =
db.prepare("SELECT sql FROM sqlite_master WHERE type = 'table' AND name = ?").use { stmt ->
stmt.bindText(1, tableName)
if (stmt.step()) stmt.getText(0) else ""
}
return sql.contains("fts5", ignoreCase = true).also { isFts5 = it }
}
/**
* Watermark for the deferred path: everything with
* `row_id <= last_row_id` is guaranteed indexed. Idempotent — also
@@ -124,13 +189,13 @@ class FullTextSearchModule(
val insertFTS =
"""
INSERT OR ROLLBACK INTO $tableName ($eventHeaderRowIdName, $contentName)
INSERT OR ROLLBACK INTO $tableName (rowid, $contentName)
VALUES (?, ?)
""".trimIndent()
val deleteFTSByRowId =
"""
DELETE FROM $tableName WHERE $eventHeaderRowIdName = ?
DELETE FROM $tableName WHERE rowid = ?
""".trimIndent()
fun insert(
@@ -200,7 +265,19 @@ class FullTextSearchModule(
dropTrigger(db)
drop(db)
create(db)
populateAll(db)
// The rebuild just wrote every row as its own tiny segment; compact
// them into one so the first search after a reindex isn't a scan over
// hundreds of segments.
optimize(db)
}
/**
* Scan every stored searchable event and insert its derived content into
* the (already created, empty) FTS index, keyed by `event_headers.row_id`.
* The caller owns the transaction and the create/drop lifecycle.
*/
private fun populateAll(db: SQLiteConnection) {
val kinds = searchableKindsPresent(db)
if (kinds.isEmpty()) return
@@ -230,6 +307,58 @@ class FullTextSearchModule(
}
}
/**
* v4 → v5 migration: the pre-v5 index was `fts5(event_header_row_id,
* content)` with an auto-assigned rowid unrelated to `event_headers`, and
* it stored a second copy of the content. v5 is the contentless,
* rowid = row_id schema. The old rowids can't be remapped in place, so the
* table is dropped and repopulated. Runs inside the migration transaction.
*
* - **synchronous** stores rebuild now (client corpora are small).
* - **deferred** stores reset the catch-up watermark to 0 so the relay's
* background worker repopulates without a long migration transaction.
*/
fun migrateV4ToContentless(db: SQLiteConnection) {
if (!enabled) return
dropTrigger(db)
drop(db)
create(db)
if (deferIndexing) {
db.prepare("UPDATE $stateTableName SET last_row_id = 0 WHERE id = 1").use { it.step() }
} else {
populateAll(db)
optimize(db)
}
}
/**
* Full FTS5 segment compaction — merges the b-tree segments left by
* incremental inserts into one, so a `MATCH` touches a single segment
* instead of dozens. Expensive (rewrites the whole index); call it once
* after a rebuild, not per batch. No-op on the fts4/fts3 fallback.
*/
fun optimize(db: SQLiteConnection) {
if (!enabled || !resolveIsFts5(db)) return
db.prepare("INSERT INTO $tableName($tableName) VALUES ('optimize')").use { it.step() }
}
/**
* Bounded incremental segment merge — does at most [pages] pages of merge
* work, so it stays cheap enough to run on a periodic maintenance tick
* while the index keeps growing from deferred catch-up. No-op on the
* fts4/fts3 fallback.
*/
fun mergeSegments(
db: SQLiteConnection,
pages: Int = 16,
) {
if (!enabled || !resolveIsFts5(db)) return
db.prepare("INSERT INTO $tableName($tableName, rank) VALUES ('merge', ?)").use { stmt ->
stmt.bindLong(1, pages.toLong())
stmt.step()
}
}
/**
* Process one batch of a resumable rebuild: re-derive the FTS rows
* for up to [batchSize] events whose `row_id > ` [afterRowId] and
@@ -329,13 +458,11 @@ class FullTextSearchModule(
// Unlike [reindexBatch] there is NO per-row delete here: rows past
// the watermark were never indexed (deferred mode skips insert()),
// and the watermark advances atomically with the FTS rows it
// covers, so a crash replay is impossible. The delete would also
// be ruinous — `event_header_row_id` is a plain FTS5 column, so
// deleting by it scans the whole FTS table per row, which turned
// the first catch-up implementation O(n²). Consequence: switching
// a database back and forth between deferred and synchronous
// strategies requires a [reindexAll] in between (same rule as a
// searchable-kinds change).
// covers, so a crash replay is impossible. (Delete-by-rowid is now
// O(log n) on the contentless index, so this is purely about not
// doing redundant work.) Consequence: switching a database back and
// forth between deferred and synchronous strategies requires a
// [reindexAll] in between (same rule as a searchable-kinds change).
val limit = batchSize.coerceAtLeast(1)
val kinds = searchableKindsPresent(db)
var last = watermark
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/IndexingStrategy.kt
+15 -3
View File
@@ -74,9 +74,21 @@ interface IndexingStrategy {
* Activate this if you see too many Tag-centric Filters without
* kind AND pubkey at the same time.
*
* This is a rarely used index (reports by your follows or
* NIP-04 DMs for instance) that becomes quite large without
* major gains.
* This shape (reports by your follows, NIP-04 DM rooms, follows-scoped
* community feeds) is not rare on the client side: the 2026-07 filter
* assembler survey counted 65 call sites building
* `kinds + authors + tags`. Without this index the plan seeks
* `(tag_hash, kind)` and reads every row for that tag/kind before
* filtering the author.
*
* Measured by `TagAuthorIndexBenchmark` (jvmTest prodbench): the
* DM-room query drops 9.4 ms → 0.6 ms (~15×) at 200k events and
* 14.2 ms → 0.66 ms (~21×) at 1M — the gap grows with corpus size —
* while batch-insert cost stays inside run noise (49.0 vs 47.4
* µs/event at 1M). geode enables it; the client default stays off
* because a client store's per-tag row counts are bounded by one
* user's data. Flipping it on an existing DB is safe: the index is
* built on next open by `EventIndexesModule.ensureOptionalIndexes`.
*
* Keep in mind that activating too many indexes increases the size of the
* DB so much that the indexes themselves won't fit in memory, requiring
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/MergeQueryExecutor.kt
+246 -57
View File
@@ -24,52 +24,94 @@ import androidx.sqlite.SQLiteConnection
import androidx.sqlite.SQLiteStatement
/**
* k-way merge executor for the **home-feed** query shape:
* `authors=[…] (+ kinds=[…]) [+ since/until] limit=N` ordered newest-first.
* k-way merge executor for the two "wide fan-out, newest-N" query shapes
* whose single-SQL plan reads O(matching history) rather than O(limit):
*
* SQLite serves this by seeking every `(kind, pubkey)` combo and feeding
* *all* matching rows through a LIMIT-bounded sorter — so it reads O(the
* followed set's whole matching history). For prolific follows on a cold
* on-disk DB that's the `follow-feed` regression (relayBench: 97 ms vs
* strfry 17 ms). See `quartz/plans/2026-07-04-follow-feed-read-tradeoff.md`.
* 1. **home-feed** — `authors=[…] (+ kinds=[…]) [+ since/until] limit=N`.
* SQLite seeks every `(kind, pubkey)` combo and feeds *all* matching rows
* through a LIMIT-bounded sorter — O(the followed set's whole matching
* history). For prolific follows on a cold on-disk DB that was the
* `follow-feed` regression (relayBench: 97 ms vs strfry 17 ms). See
* `quartz/plans/2026-07-04-follow-feed-read-tradeoff.md`.
* 2. **tag watcher** — `#<x>=[hundreds of values] (+ kinds=[…])
* [+ since/until] limit=N`, the reactions/replies archetype
* (`kinds=[7] AND #e=[note ids]`). The per-value streams come sorted off
* `(tag_hash[, kind], created_at)`, but their union does not, so SQLite
* collects every matching row and TEMP-B-TREE sorts to the limit — the
* tag-index analogue of the follow-feed shape. Measured by
* `TagAuthorIndexBenchmark` (jvmTest prodbench): `#e IN 300, limit 500`
* cost 12.8 ms cold at 200k events and 14.2 ms at 1M, growing with
* matching history.
*
* Each `(kind, pubkey)` is already a newest-first stream off the
* `query_by_kind_pubkey_created (kind, pubkey, created_at DESC)` index
* (or `query_by_pubkey_created` for authors-only). This opens one lazy
* cursor per stream and merges their heads, stopping at the limit — so it
* reads only **O(limit + streams)** rows regardless of how much history the
* authors have, and it reuses the existing indexes (no write/size cost).
* Each stream is already a newest-first cursor off an existing index:
* - authors: `query_by_kind_pubkey_created (kind, pubkey, created_at DESC)`
* (or `query_by_pubkey_created` for authors-only);
* - tags: `query_by_tags_hash_kind (tag_hash, kind, created_at DESC)`
* (or `query_by_tags_hash` for the no-kind case, gated by
* [IndexingStrategy.indexTagsByCreatedAtAlone]).
*
* The merge opens one lazy cursor per stream, merges their heads newest-first,
* and stops at the limit — reading **O(limit + streams)** rows regardless of
* how much history the authors/tags have, and reusing the existing indexes
* (no write/size cost). With the pooled statement cache
* ([StatementCachingConnection]) the per-stream cursors are prepared once and
* reused across repeated polls of the same REQ.
*
* Merge order is `created_at DESC`, tie-broken by `id ASC`. NIP-01 leaves
* same-`created_at` ties unspecified, so the returned set is a valid
* newest-N either way. The `id ASC` tie-break is exact — byte-for-byte the
* same events the single-SQL path returns — only when the store indexes id
* ([IndexingStrategy.useAndIndexIdOnOrderBy]): then each per-stream cursor
* streams in `(created_at DESC, id ASC)` straight off the index, so a
* stream's same-second head really is its id-minimum. Without that index
* the per-stream cursor yields same-second rows in rowid order, so the
* result is still a valid newest-N but may differ from the single-SQL path
* exactly at a same-second boundary.
* same-`created_at` ties unspecified, so the returned set is a valid newest-N
* either way. The `id ASC` tie-break is exact — byte-for-byte the same events
* the single-SQL path returns — only when the store indexes id
* ([IndexingStrategy.useAndIndexIdOnOrderBy]) **and** the stream cursor can
* order by id off the index. The author streams can (id is on
* `event_headers`); the tag streams cannot (the cursor orders off
* `event_tags`, which has no id column), so a tag stream yields same-second
* rows in rowid order — still a valid newest-N, but same-second ties may
* differ from an id-ordered reference.
*
* **Cross-stream duplicates.** An author appears in exactly one author stream
* (one pubkey per event), so the home-feed merge never double-counts. A single
* event can carry several of the queried tag values (or a repeated tag), so it
* can surface in several tag streams — the single-SQL path dedups with
* `SELECT DISTINCT`. The tag merge therefore dedups by event id through a
* `seen` set; the author merge skips that set entirely.
*/
internal object MergeQueryExecutor {
/** Author-stream projection: a single `event_headers` scan, unqualified. */
const val COLS = "id, pubkey, created_at, kind, tags, content, sig"
/** Tag-stream projection: `event_tags` joins `event_headers`, so qualify. */
private const val EH_COLS =
"event_headers.id, event_headers.pubkey, event_headers.created_at, event_headers.kind, event_headers.tags, event_headers.content, event_headers.sig"
/**
* Above this many streams, fall back to the single-SQL plan: the
* per-stream cursor setup stops paying off, and huge author lists are
* collecting a lot no matter what. `kinds.size × authors.size`.
* per-stream cursor setup stops paying off, and huge fan-outs are
* collecting a lot no matter what. `kinds.size × (authors|values).size`.
*/
const val MAX_STREAMS = 2048
/**
* Stream count if [filter] is merge-eligible, else `-1`. Eligible = a
* simple (no tag/search/id/d-tag) query with authors + a limit, whose
* per-stream index exists. `kinds` optional: with it, one stream per
* `(kind, author)`; without, one per author (needs the pubkey index).
* Stream count if [filter] is merge-eligible under *either* shape, else
* `-1`. Routing check for [QueryBuilder]; [run] re-derives which shape.
*/
fun streamCount(
filter: QueryBuilder.FilterWithDTags,
indexStrategy: IndexingStrategy,
): Int {
val authorStreams = authorStreamCount(filter, indexStrategy)
if (authorStreams > 0) return authorStreams
return tagStreamCount(filter, indexStrategy)
}
/**
* Author-shape stream count, or `-1`. Eligible = a simple (no tag/search/
* id/d-tag) query with authors + a limit, whose per-stream index exists.
* `kinds` optional: with it, one stream per `(kind, author)`; without, one
* per author (needs the pubkey index).
*/
fun authorStreamCount(
filter: QueryBuilder.FilterWithDTags,
indexStrategy: IndexingStrategy,
): Int {
if (!filter.isSimpleQuery()) return -1
if (filter.ids != null) return -1
@@ -83,27 +125,69 @@ internal object MergeQueryExecutor {
// distinct set.
val distinctAuthors = authors.distinct().size
val kinds = filter.kinds
// Long product: a pathological authors×kinds could overflow Int and
// wrap back into the eligible band, routing a huge fan-out here.
val streams =
if (kinds != null && kinds.isNotEmpty()) {
distinctAuthors * kinds.distinct().size
distinctAuthors.toLong() * kinds.distinct().size
} else {
// authors-only needs the (pubkey, created_at) index to stream.
if (!indexStrategy.indexEventsByPubkeyAlone) return -1
distinctAuthors
distinctAuthors.toLong()
}
// A single stream is already the optimal single index seek — let the
// normal path handle it; only merge when there's something to merge.
return if (streams in 2..MAX_STREAMS) streams else -1
return if (streams in 2..MAX_STREAMS.toLong()) streams.toInt() else -1
}
/** Prepares one bound, newest-first cursor per stream. */
private fun prepareStreams(
/**
* Tag-shape stream count, or `-1`. Eligible = a single non-`d` tag key
* with `IN` (any-of) semantics and ≥2 distinct values, plus a limit, no
* ids/authors/d-tag/search, and no `AND`-tags (`tagsAll`) — the large-IN
* watcher shape. `kinds` optional: with it, one stream per
* `(value, kind)` off `query_by_tags_hash_kind`; without, one per value
* off `query_by_tags_hash` (needs [IndexingStrategy.indexTagsByCreatedAtAlone]).
*
* Authors are excluded on purpose: `tag ∩ author ∩ kind` is a covered
* single seek under [IndexingStrategy.indexTagsWithKindAndPubkey], not a
* fan-out, and mixing an author predicate into per-tag streams would not
* reduce the read.
*/
fun tagStreamCount(
filter: QueryBuilder.FilterWithDTags,
indexStrategy: IndexingStrategy,
): Int {
if (filter.ids != null) return -1
if (filter.authors != null) return -1
if (filter.dTags != null) return -1
if (filter.search != null && filter.search.isNotEmpty()) return -1
if (filter.limit == null || filter.limit <= 0) return -1
// AND-tags can't be expressed as a union of per-value streams.
if (filter.nonDTagsAll != null && filter.nonDTagsAll.isNotEmpty()) return -1
val inTags = filter.nonDTagsIn ?: return -1
// A second tag key would AND across keys — not a single union.
if (inTags.size != 1) return -1
val values = inTags.values.first().distinct()
if (values.size < 2) return -1
val kinds = filter.kinds?.distinct()?.takeIf { it.isNotEmpty() }
val streams =
if (kinds != null) {
values.size.toLong() * kinds.size
} else {
if (!indexStrategy.indexTagsByCreatedAtAlone) return -1
values.size.toLong()
}
return if (streams in 2..MAX_STREAMS.toLong()) streams.toInt() else -1
}
/** Prepares one bound, newest-first cursor per author stream. */
private fun prepareAuthorStreams(
db: SQLiteConnection,
filter: QueryBuilder.FilterWithDTags,
indexStrategy: IndexingStrategy,
): List<SQLiteStatement> {
// Dedup so a repeated pubkey/kind can't open two identical cursors and
// double-emit (see streamCount).
// double-emit (see authorStreamCount).
val authors = filter.authors!!.distinct()
val kinds = filter.kinds?.distinct()?.takeIf { it.isNotEmpty() }
val since = filter.since
@@ -117,8 +201,7 @@ internal object MergeQueryExecutor {
val orderBy =
if (indexStrategy.useAndIndexIdOnOrderBy) "created_at DESC, id ASC" else "created_at DESC"
val stmts = ArrayList<SQLiteStatement>((kinds?.size ?: 1) * authors.size)
if (kinds != null) {
return if (kinds != null) {
val sql =
buildString {
append("SELECT ").append(COLS)
@@ -128,16 +211,14 @@ internal object MergeQueryExecutor {
if (since != null) append(" AND created_at >= ?")
append(" ORDER BY ").append(orderBy)
}
for (kind in kinds) {
for (author in authors) {
val stmt = db.prepare(sql)
var p = 1
stmt.bindLong(p++, kind.toLong())
stmt.bindText(p++, author)
if (until != null) stmt.bindLong(p++, until)
if (since != null) stmt.bindLong(p++, since)
stmts.add(stmt)
}
buildStreams(kinds.size * authors.size) { i ->
val stmt = db.prepare(sql)
var p = 1
stmt.bindLong(p++, kinds[i / authors.size].toLong())
stmt.bindText(p++, authors[i % authors.size])
if (until != null) stmt.bindLong(p++, until)
if (since != null) stmt.bindLong(p++, since)
stmt
}
} else {
val sql =
@@ -149,30 +230,135 @@ internal object MergeQueryExecutor {
if (since != null) append(" AND created_at >= ?")
append(" ORDER BY ").append(orderBy)
}
for (author in authors) {
buildStreams(authors.size) { i ->
val stmt = db.prepare(sql)
var p = 1
stmt.bindText(p++, author)
stmt.bindText(p++, authors[i])
if (until != null) stmt.bindLong(p++, until)
if (since != null) stmt.bindLong(p++, since)
stmts.add(stmt)
stmt
}
}
}
/** Prepares one bound, newest-first cursor per tag-value stream. */
private fun prepareTagStreams(
db: SQLiteConnection,
filter: QueryBuilder.FilterWithDTags,
hasher: TagNameValueHasher,
): List<SQLiteStatement> {
val entry = filter.nonDTagsIn!!.entries.first()
val tagName = entry.key
val values = entry.value.distinct()
val kinds = filter.kinds?.distinct()?.takeIf { it.isNotEmpty() }
val since = filter.since
val until = filter.until
// The tag cursors stream off event_tags (which has no id column), so
// the tie order can only be created_at DESC — see the class doc.
return if (kinds != null) {
val sql =
buildString {
append("SELECT ").append(EH_COLS)
append(" FROM event_tags INDEXED BY query_by_tags_hash_kind")
append(" JOIN event_headers ON event_headers.row_id = event_tags.event_header_row_id")
append(" WHERE event_tags.tag_hash = ? AND event_tags.kind = ?")
if (until != null) append(" AND event_tags.created_at <= ?")
if (since != null) append(" AND event_tags.created_at >= ?")
append(" ORDER BY event_tags.created_at DESC")
}
buildStreams(values.size * kinds.size) { i ->
val stmt = db.prepare(sql)
var p = 1
stmt.bindLong(p++, hasher.hash(tagName, values[i / kinds.size]))
stmt.bindLong(p++, kinds[i % kinds.size].toLong())
if (until != null) stmt.bindLong(p++, until)
if (since != null) stmt.bindLong(p++, since)
stmt
}
} else {
val sql =
buildString {
append("SELECT ").append(EH_COLS)
append(" FROM event_tags INDEXED BY query_by_tags_hash")
append(" JOIN event_headers ON event_headers.row_id = event_tags.event_header_row_id")
append(" WHERE event_tags.tag_hash = ?")
if (until != null) append(" AND event_tags.created_at <= ?")
if (since != null) append(" AND event_tags.created_at >= ?")
append(" ORDER BY event_tags.created_at DESC")
}
buildStreams(values.size) { i ->
val stmt = db.prepare(sql)
var p = 1
stmt.bindLong(p++, hasher.hash(tagName, values[i]))
if (until != null) stmt.bindLong(p++, until)
if (since != null) stmt.bindLong(p++, since)
stmt
}
}
return stmts
}
/**
* Runs the merge, calling [onRow] with each winning cursor positioned on
* the row to emit, newest-first, up to `limit`. [onRow] must read the
* current row (it stays valid until the next step).
* Runs the merge for whichever shape [filter] matches, calling [onRow]
* with each winning cursor positioned on the row to emit, newest-first,
* up to `limit`. [onRow] must read the current row (it stays valid until
* the next step). [hasher] is only consulted for the tag shape.
*/
fun run(
db: SQLiteConnection,
filter: QueryBuilder.FilterWithDTags,
indexStrategy: IndexingStrategy,
hasher: (SQLiteConnection) -> TagNameValueHasher,
onRow: (SQLiteStatement) -> Unit,
) {
if (authorStreamCount(filter, indexStrategy) > 0) {
// One pubkey per event ⇒ author streams never overlap: no dedup.
mergeStreams(prepareAuthorStreams(db, filter, indexStrategy), filter.limit!!, dedup = false, onRow)
} else {
// A single event can match several tag values ⇒ dedup by id.
mergeStreams(prepareTagStreams(db, filter, hasher(db)), filter.limit!!, dedup = true, onRow)
}
}
/**
* Prepares [count] cursors via [prepareOne], closing any already-prepared
* statements if a later prepare throws — otherwise a mid-loop failure would
* strand checked-out, un-reset handles in the pooled connection (dead
* slots holding read locks). On success the caller ([mergeStreams]) owns
* closing them.
*/
private inline fun buildStreams(
count: Int,
prepareOne: (Int) -> SQLiteStatement,
): List<SQLiteStatement> {
val stmts = ArrayList<SQLiteStatement>(count)
try {
for (i in 0 until count) stmts.add(prepareOne(i))
} catch (e: Throwable) {
for (s in stmts) {
try {
s.close()
} catch (_: Throwable) {
}
}
throw e
}
return stmts
}
/**
* Heap-free k-way merge over the prepared [stmts]: repeatedly emits the
* newest live head (`created_at DESC`, tie `id ASC`) until [limit] rows
* are emitted or every stream is drained. When [dedup] is set an event id
* already emitted is skipped (its cursor still advances), so a row that
* surfaces in several streams is emitted once.
*/
private fun mergeStreams(
stmts: List<SQLiteStatement>,
limit: Int,
dedup: Boolean,
onRow: (SQLiteStatement) -> Unit,
) {
val stmts = prepareStreams(db, filter, indexStrategy)
try {
val k = stmts.size
val headCreatedAt = LongArray(k)
@@ -188,8 +374,8 @@ internal object MergeQueryExecutor {
}
}
val seen = if (dedup) HashSet<String>() else null
var emitted = 0
val limit = filter.limit!!
while (emitted < limit) {
// Pick the newest live head: created_at DESC, then id ASC.
var best = -1
@@ -204,8 +390,11 @@ internal object MergeQueryExecutor {
}
if (best == -1) break
onRow(stmts[best]) // cursor is still on the head row
emitted++
// Emit unless this id was already emitted by another stream.
if (seen == null || seen.add(headId[best]!!)) {
onRow(stmts[best]) // cursor is still on the head row
emitted++
}
// Advance the winner to its next row.
if (stmts[best].step()) {
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryBuilder.kt
+85 -26
View File
@@ -49,7 +49,7 @@ class QueryBuilder(
val merge = filter.toFilterWithDTags()
if (MergeQueryExecutor.streamCount(merge, indexStrategy) > 0) {
val out = ArrayList<T>(merge.limit!!)
MergeQueryExecutor.run(db, merge, indexStrategy) { out.add(it.toEvent()) }
MergeQueryExecutor.run(db, merge, indexStrategy, hasher) { out.add(it.toEvent()) }
return out
}
return db.runQuery(toSql(filter, hasher(db)))
@@ -62,7 +62,7 @@ class QueryBuilder(
) {
val merge = filter.toFilterWithDTags()
if (MergeQueryExecutor.streamCount(merge, indexStrategy) > 0) {
MergeQueryExecutor.run(db, merge, indexStrategy) { onEach(it.toEvent()) }
MergeQueryExecutor.run(db, merge, indexStrategy, hasher) { onEach(it.toEvent()) }
return
}
db.runQuery(toSql(filter, hasher(db)), onEach)
@@ -102,7 +102,7 @@ class QueryBuilder(
val merge = filter.toFilterWithDTags()
if (MergeQueryExecutor.streamCount(merge, indexStrategy) > 0) {
val out = ArrayList<RawEvent>(merge.limit!!)
MergeQueryExecutor.run(db, merge, indexStrategy) { out.add(it.toRawEvent()) }
MergeQueryExecutor.run(db, merge, indexStrategy, hasher) { out.add(it.toRawEvent()) }
return out
}
return db.runRawQuery(toSql(filter, hasher(db)))
@@ -115,7 +115,7 @@ class QueryBuilder(
) {
val merge = filter.toFilterWithDTags()
if (MergeQueryExecutor.streamCount(merge, indexStrategy) > 0) {
MergeQueryExecutor.run(db, merge, indexStrategy) { onEach(it.toRawEvent()) }
MergeQueryExecutor.run(db, merge, indexStrategy, hasher) { onEach(it.toRawEvent()) }
return
}
db.runRawQuery(toSql(filter, hasher(db)), onEach)
@@ -202,13 +202,17 @@ class QueryBuilder(
)
}
val rowIdSubqueries = prepareRowIDSubQueries(filter, hasher)
// A search term that survived the simple-search branch above always
// combines with tags here (search + #t etc.). NIP-50 still orders by
// relevance, so carry the FTS rank through the subquery and out.
val rankSearch = newFilter.search != null && newFilter.search.isNotEmpty()
val rowIdSubqueries = prepareRowIDSubQueries(filter, hasher, projectRank = rankSearch)
return if (rowIdSubqueries == null) {
QuerySpec(makeEverythingQuery())
} else {
QuerySpec(
makeQueryIn(rowIdSubqueries.sql),
makeQueryIn(rowIdSubqueries.sql, orderByRank = rankSearch),
rowIdSubqueries.args,
)
}
@@ -220,7 +224,13 @@ class QueryBuilder(
): QuerySpec {
if (filters.size == 1) return toSql(filters.first(), hasher)
val rowIdSubqueries = unionSubqueriesIfNeeded(filters, hasher)
// A multi-filter search REQ (e.g. the client's search-across-kinds,
// all filters sharing one term) must still be NIP-50 relevance-ordered.
// Only when EVERY branch is a search branch (and FTS is on, so each has
// a rank column) — a non-search branch has no defined relevance, so a
// mixed REQ falls back to created_at.
val rankSearch = fts.enabled && filters.all { !it.search.isNullOrEmpty() }
val rowIdSubqueries = unionSubqueriesIfNeeded(filters, hasher, projectRank = rankSearch)
return if (rowIdSubqueries == null) {
QuerySpec(
@@ -229,7 +239,7 @@ class QueryBuilder(
)
} else {
QuerySpec(
makeQueryIn(rowIdSubqueries.sql),
makeQueryIn(rowIdSubqueries.sql, orderByRank = rankSearch),
rowIdSubqueries.args,
)
}
@@ -427,7 +437,7 @@ class QueryBuilder(
val sql =
buildString {
append("SELECT event_headers.id, event_headers.created_at FROM event_headers")
append("\nINNER JOIN ${fts.tableName} ON event_headers.row_id = ${fts.tableName}.${fts.eventHeaderRowIdName}")
append("\nINNER JOIN ${fts.tableName} ON event_headers.row_id = ${fts.tableName}.rowid")
if (clause.conditions.isNotEmpty()) {
append("\nWHERE ${clause.conditions}")
}
@@ -470,14 +480,20 @@ class QueryBuilder(
private fun makeEverythingQuery() = "SELECT id, pubkey, created_at, kind, tags, content, sig FROM event_headers ORDER BY created_at DESC${if (indexStrategy.useAndIndexIdOnOrderBy) ", id ASC" else ""}"
private fun makeQueryIn(rowIdQuery: String) =
"""
// [orderByRank] presents the joined result in NIP-50 relevance order: the
// subquery (built with `projectRank`) exposes the FTS bm25 score as a
// `rank` column, and `created_at DESC` is only a tie-break. Off, it keeps
// the default newest-first ordering for every non-search shape.
private fun makeQueryIn(
rowIdQuery: String,
orderByRank: Boolean = false,
) = """
SELECT id, pubkey, created_at, kind, tags, content, sig FROM event_headers
INNER JOIN (
$rowIdQuery
) AS filtered
ON event_headers.row_id = filtered.row_id
ORDER BY created_at DESC${if (indexStrategy.useAndIndexIdOnOrderBy) ", id ASC" else ""}
ORDER BY ${if (orderByRank) "filtered.rank, " else ""}created_at DESC${if (indexStrategy.useAndIndexIdOnOrderBy) ", id ASC" else ""}
""".trimIndent()
private fun <T : Event> SQLiteConnection.runQuery(query: QuerySpec): List<T> =
@@ -713,16 +729,34 @@ class QueryBuilder(
fun unionSubqueriesIfNeeded(
filters: List<Filter>,
hasher: TagNameValueHasher,
// See [prepareRowIDSubQueries]. When set, every branch is a search
// branch that also projects a `rank` column; the union keeps one row
// per event with its BEST (min) bm25 score so the caller can present
// the whole multi-filter search REQ NIP-50-ranked. Callers must only
// pass true when all filters carry a search term (else a branch has no
// rank column). Off for count/delete, which stay single-column.
projectRank: Boolean = false,
): QuerySpec? {
val inner =
filters.mapNotNull { filter ->
prepareRowIDSubQueries(filter, hasher)
prepareRowIDSubQueries(filter, hasher, projectRank)
}
if (inner.isEmpty()) return null
return if (inner.size == 1) {
inner.first()
if (inner.size == 1) return inner.first()
return if (projectRank) {
// UNION ALL keeps every (row_id, rank) so an event matching two
// branches under different terms isn't dropped before MIN; GROUP BY
// then dedups by event keeping the best score.
QuerySpec(
sql =
"SELECT row_id, MIN(rank) as rank FROM (\n " +
inner.joinToString("\n UNION ALL\n ") { "SELECT row_id, rank FROM (${it.sql})" } +
"\n ) GROUP BY row_id",
args = inner.flatMap { it.args },
)
} else {
QuerySpec(
sql = inner.joinToString("\n UNION\n ") { "SELECT row_id FROM (${it.sql})" },
@@ -748,6 +782,13 @@ class QueryBuilder(
fun prepareRowIDSubQueries(
filter: Filter,
hasher: TagNameValueHasher,
// When set on a search filter, the subquery also projects the FTS
// bm25 score as a `rank` column and orders its own LIMIT by relevance,
// so the caller ([makeQueryIn] with `orderByRank`) can present results
// NIP-50-ranked. Off (the default) for count/delete/union/negentropy,
// which never expose a second column (a two-column subquery breaks
// `row_id IN (…)`) and don't rank.
projectRank: Boolean = false,
): QuerySpec? {
if (filter.isEmpty()) return null
@@ -761,6 +802,10 @@ class QueryBuilder(
val mustJoinSearch = filter.search != null && fts.enabled
// Only emit the rank column when there is actually an FTS join to take
// it from; a `projectRank` request on a tag-only filter is ignored.
val emitRank = projectRank && mustJoinSearch
val nonDTagsIn = filter.tags?.filter { it.key != "d" } ?: emptyMap()
val nonDTagsAll = filter.tagsAll?.filter { it.key != "d" } ?: emptyMap()
@@ -789,7 +834,11 @@ class QueryBuilder(
buildString {
// always do tags if there are any
if (reverseLookup) {
append("SELECT DISTINCT(event_tags.event_header_row_id) as row_id FROM event_tags")
append("SELECT DISTINCT(event_tags.event_header_row_id) as row_id")
// rank is functionally determined by the row_id (one FTS
// row per event), so it doesn't change what DISTINCT folds.
if (emitRank) append(", ${fts.tableName}.rank as rank")
append(" FROM event_tags")
// it's quite rare to have 2 tags in the filter, but possible
nonDTagsIn.keys.forEachIndexed { index, tagName ->
@@ -815,13 +864,15 @@ class QueryBuilder(
}
if (mustJoinSearch) {
append(" INNER JOIN ${fts.tableName} ON ${fts.tableName}.${fts.eventHeaderRowIdName} = event_tags.event_header_row_id")
append(" INNER JOIN ${fts.tableName} ON ${fts.tableName}.rowid = event_tags.event_header_row_id")
}
} else if (mustJoinSearch) {
append("SELECT ${fts.tableName}.${fts.eventHeaderRowIdName} as row_id FROM ${fts.tableName}")
append("SELECT ${fts.tableName}.rowid as row_id")
if (emitRank) append(", ${fts.tableName}.rank as rank")
append(" FROM ${fts.tableName}")
if (hasHeaders) {
append(" INNER JOIN event_headers ON event_headers.row_id = ${fts.tableName}.${fts.eventHeaderRowIdName}")
append(" INNER JOIN event_headers ON event_headers.row_id = ${fts.tableName}.rowid")
}
} else {
// no tags and no search.
@@ -934,15 +985,17 @@ class QueryBuilder(
append(" WHERE ${clause.conditions}")
}
if (filter.limit != null) {
if (reverseLookup) {
if (emitRank) {
// NIP-50: the LIMIT keeps the most RELEVANT rows, not
// the newest, so the inner cut is by rank too.
append(" ORDER BY rank")
} else if (reverseLookup) {
append(" ORDER BY event_tags.created_at DESC")
append(" LIMIT ")
append(filter.limit)
} else {
append(" ORDER BY event_headers.created_at DESC")
append(" LIMIT ")
append(filter.limit)
}
append(" LIMIT ")
append(filter.limit)
}
}
@@ -1001,11 +1054,17 @@ class QueryBuilder(
val sql =
buildString {
append("SELECT event_headers.id, event_headers.pubkey, event_headers.created_at, event_headers.kind, event_headers.tags, event_headers.content, event_headers.sig FROM event_headers")
append("\nINNER JOIN ${fts.tableName} ON event_headers.row_id = ${fts.tableName}.${fts.eventHeaderRowIdName}")
append("\nINNER JOIN ${fts.tableName} ON event_headers.row_id = ${fts.tableName}.rowid")
if (clause.conditions.isNotEmpty()) {
append("\nWHERE ${clause.conditions}")
}
append("\nORDER BY event_headers.created_at DESC")
// NIP-50: search results are ordered by relevance ("quality of
// search result"), not created_at, and the limit is applied
// after the score. FTS5 exposes bm25 as the `rank` column (more
// negative = more relevant), so ORDER BY rank ascending is
// best-match-first. created_at DESC is only a tie-break so
// equally-relevant matches come newest-first deterministically.
append("\nORDER BY ${fts.tableName}.rank, event_headers.created_at DESC")
if (indexStrategy.useAndIndexIdOnOrderBy) {
append(", event_headers.id ASC")
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SQLiteEventStore.kt
+18 -1
View File
@@ -60,7 +60,7 @@ class SQLiteEventStore(
val extraPragmas: List<String> = emptyList(),
) {
companion object {
const val DATABASE_VERSION = 4
const val DATABASE_VERSION = 5
}
val seedModule = SeedModule()
@@ -160,6 +160,11 @@ class SQLiteEventStore(
setUserVersion(this, DATABASE_VERSION)
}
}
// Flag-gated indexes are runtime config, not schema: a
// deployment that flips an IndexingStrategy flag on an
// existing DB gets the index built here (idempotent,
// one-time cost), with no user_version bump involved.
eventIndexModule.ensureOptionalIndexes(db)
},
)
}
@@ -208,6 +213,12 @@ class SQLiteEventStore(
// watermark seeds at the current MAX(row_id).
fullTextSearchModule.createStateTable(db)
}
4 -> {
// Upgrade from version 4 to 5: the FTS index became a
// contentless table keyed by event_headers.row_id. The old
// rowids can't be remapped, so drop and repopulate.
fullTextSearchModule.migrateV4ToContentless(db)
}
}
}
}
@@ -269,6 +280,12 @@ class SQLiteEventStore(
pool.useWriter { db ->
db.execSQL("PRAGMA analysis_limit = 400;")
db.execSQL("PRAGMA optimize;")
// Fold a bounded FTS segment merge into the same periodic
// maintenance tick: incremental (and deferred catch-up) inserts
// leave the NIP-50 index as many small segments, and a MATCH
// queries every one. Bounded so the tick stays cheap; a no-op when
// there is nothing to merge or FTS is off.
fullTextSearchModule.mergeSegments(db)
}
/**
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/StatementCachingConnection.kt
+55 -25
View File
@@ -34,46 +34,72 @@ import androidx.sqlite.SQLiteStatement
* real reset + clearBindings happens on the next checkout). Statements are
* only truly finalized when the connection itself closes.
*
* Each SQL string caches a small **pool** of handles rather than a single
* one, so overlapping checkouts of the *same* SQL all reuse cached handles.
* That is exactly the k-way-merge query shape ([MergeQueryExecutor]): it
* opens one identical-SQL cursor per author/tag stream — dozens to hundreds
* live at once — which a single-handle cache could not serve (every stream
* past the first fell back to an uncached prepare). The pool lets a repeated
* follow-feed / reactions-watcher REQ reuse its per-stream cursors instead
* of re-preparing them each poll.
*
* Constraints, by design of the call sites:
* - **Not thread-safe** — same contract as the underlying connection,
* which the pool already serializes (single writer under a mutex).
* - **No overlapping use of the same SQL** — checking out one SQL string
* twice without closing the first use would alias one native handle.
* Insert/query paths never nest the same statement; a checkout while
* the previous one is still open falls back to an uncached statement.
* which the pool already serializes (single writer under a mutex; each
* reader held by one coroutine at a time).
*/
class StatementCachingConnection(
private val delegate: SQLiteConnection,
/**
* Ceiling on retained statements. Query SQL embeds one `?` per filter
* element, so shape variety is client-controlled — without a cap a
* long-lived relay connection would accumulate native handles without
* bound. Once full, unseen SQL just prepares uncached. 256 comfortably
* covers the write path's fixed set plus the recurring filter shapes.
* Ceiling on retained statements across all SQL strings. Query SQL
* embeds one `?` per filter element, so shape variety is
* client-controlled — without a cap a long-lived relay connection would
* accumulate native handles without bound. Once full, unseen SQL (or an
* extra concurrent copy of a cached SQL) just prepares uncached. 512
* covers the write path's fixed set, the recurring single-shot filter
* shapes, and a few hundred concurrent per-stream merge cursors.
*/
private val maxCachedStatements: Int = 256,
private val maxCachedStatements: Int = 512,
) : SQLiteConnection by delegate {
private val cache = HashMap<String, CachedStatement>()
// One reusable pool per SQL string. Several entries of the same pool may
// be checked out simultaneously (the merge path); a `prepare` reuses the
// first free entry, grows the pool while under the global cap, and only
// then falls back to an uncached statement.
private val cache = HashMap<String, ArrayList<CachedStatement>>()
private var cachedCount = 0
override fun prepare(sql: String): SQLiteStatement {
val cached =
cache[sql] ?: run {
if (cache.size >= maxCachedStatements) return delegate.prepare(sql)
CachedStatement(delegate.prepare(sql)).also { cache[sql] = it }
val pool = cache[sql]
if (pool != null) {
for (i in pool.indices) {
val stmt = pool[i]
if (!stmt.checkedOut) {
stmt.checkedOut = true
stmt.clearBindings()
return stmt
}
}
// Every pooled handle for this SQL is in use — grow if the global
// budget allows, else serve an uncached statement.
if (cachedCount >= maxCachedStatements) return delegate.prepare(sql)
return CachedStatement(delegate.prepare(sql)).also {
it.checkedOut = true
pool.add(it)
cachedCount++
}
if (cached.checkedOut) {
// Same SQL prepared while the previous handle is still in use —
// stay correct with a plain uncached statement.
return delegate.prepare(sql)
}
cached.checkedOut = true
cached.clearBindings()
return cached
if (cachedCount >= maxCachedStatements) return delegate.prepare(sql)
return CachedStatement(delegate.prepare(sql)).also {
it.checkedOut = true
cache[sql] = arrayListOf(it)
cachedCount++
}
}
override fun close() {
cache.values.forEach { runCatching { it.finalize() } }
cache.values.forEach { pool -> pool.forEach { runCatching { it.finalizeStatement() } } }
cache.clear()
cachedCount = 0
delegate.close()
}
@@ -94,6 +120,10 @@ class StatementCachingConnection(
checkedOut = false
}
fun finalize() = delegate.close()
// Not named `finalize`: a no-arg `finalize()` is treated by the JVM as
// Object.finalize(), so the GC would call it and double-close the
// native handle after our explicit close(). This is only ever invoked
// explicitly from the connection's close().
fun finalizeStatement() = delegate.close()
}
}
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip50Search/SearchQuery.kt
+12
View File
@@ -202,8 +202,20 @@ fun Filter.strippingSearchExtensions(): Filter {
/**
* Applies [strippingSearchExtensions] to every filter, returning this
* same list when no filter carried extension tokens.
*
* This runs on every REQ/COUNT/snapshot, and the overwhelming majority
* carry no `search` term at all, so the no-search case must not allocate:
* bail before building any list when nothing could be stripped.
*/
fun List<Filter>.strippingSearchExtensions(): List<Filter> {
var hasSearch = false
for (i in indices) {
if (!this[i].search.isNullOrEmpty()) {
hasSearch = true
break
}
}
if (!hasSearch) return this
var changed = false
val out =
map {
quartz/src/commonMain/kotlin/com/vitorpamplona/quartz/nip89AppHandlers/clientTag/NostrSignerWithClientTag.kt
+15
View File
@@ -117,3 +117,18 @@ class NostrSignerWithClientTag(
return tags + arrayOf(clientTag)
}
}
/**
* The same signer with the NIP-89 client-tag decorator peeled off, or the receiver unchanged when it
* carries no such decorator.
*
* The client tag says "this app composed this event", so it belongs only on templates this app
* authored. When we sign on someone else's behalf — a napplet, an nSite, a web app over NIP-07, a
* client using us as a NIP-46 bunker — the template is theirs, and appending a tag rewrites the very
* bytes they are about to have hashed into an id. NIP-07 callers routinely re-check the returned
* event against the template they submitted (block/buzz compares `JSON.stringify(tags)` outright)
* and reject the result as an invalid signature when it does not match.
*
* Any decoration under this one (metering, NIP-13 mining) is preserved, as is [NostrSigner.pubKey].
*/
fun NostrSigner.withoutClientTag(): NostrSigner = if (this is NostrSignerWithClientTag) inner else this
quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/QueryAssemblerTest.kt
+17 -17
View File
@@ -238,9 +238,9 @@ class QueryAssemblerTest : BaseDBTest() {
INNER JOIN (
SELECT row_id FROM (SELECT event_headers.row_id as row_id FROM event_headers ORDER BY event_headers.created_at DESC LIMIT 10)
UNION
SELECT row_id FROM (SELECT event_fts.event_header_row_id as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.event_header_row_id WHERE (event_headers.kind IN ("1", "1111")) AND (event_headers.pubkey = "7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d") AND (event_fts MATCH "keywords") ORDER BY event_headers.created_at DESC LIMIT 100)
SELECT row_id FROM (SELECT event_fts.rowid as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.rowid WHERE (event_headers.kind IN ("1", "1111")) AND (event_headers.pubkey = "7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d") AND (event_fts MATCH "keywords") ORDER BY event_headers.created_at DESC LIMIT 100)
UNION
SELECT row_id FROM (SELECT event_fts.event_header_row_id as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.event_header_row_id WHERE (event_headers.kind = "20") AND (event_fts MATCH "cats") ORDER BY event_headers.created_at DESC LIMIT 30)
SELECT row_id FROM (SELECT event_fts.rowid as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.rowid WHERE (event_headers.kind = "20") AND (event_fts MATCH "cats") ORDER BY event_headers.created_at DESC LIMIT 30)
) AS filtered
ON event_headers.row_id = filtered.row_id
ORDER BY $orderBy
@@ -252,13 +252,13 @@ class QueryAssemblerTest : BaseDBTest() {
│ │ └── SCAN (subquery-1)
│ ├── UNION USING TEMP B-TREE
│ │ ├── CO-ROUTINE (subquery-3)
│ │ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
│ │ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
│ │ │ ├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
│ │ │ └── USE TEMP B-TREE FOR ORDER BY
│ │ └── SCAN (subquery-3)
│ └── UNION USING TEMP B-TREE
│ ├── CO-ROUTINE (subquery-5)
│ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
│ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
│ │ ├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
│ │ └── USE TEMP B-TREE FOR ORDER BY
│ └── SCAN (subquery-5)
@@ -275,9 +275,9 @@ class QueryAssemblerTest : BaseDBTest() {
INNER JOIN (
SELECT row_id FROM (SELECT event_headers.row_id as row_id FROM event_headers ORDER BY event_headers.created_at DESC LIMIT 10)
UNION
SELECT row_id FROM (SELECT event_fts.event_header_row_id as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.event_header_row_id WHERE (event_headers.kind IN ("1", "1111")) AND (event_headers.pubkey = "7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d") AND (event_fts MATCH "keywords") ORDER BY event_headers.created_at DESC LIMIT 100)
SELECT row_id FROM (SELECT event_fts.rowid as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.rowid WHERE (event_headers.kind IN ("1", "1111")) AND (event_headers.pubkey = "7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d") AND (event_fts MATCH "keywords") ORDER BY event_headers.created_at DESC LIMIT 100)
UNION
SELECT row_id FROM (SELECT event_fts.event_header_row_id as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.event_header_row_id WHERE (event_headers.kind = "20") AND (event_fts MATCH "cats") ORDER BY event_headers.created_at DESC LIMIT 30)
SELECT row_id FROM (SELECT event_fts.rowid as row_id FROM event_fts INNER JOIN event_headers ON event_headers.row_id = event_fts.rowid WHERE (event_headers.kind = "20") AND (event_fts MATCH "cats") ORDER BY event_headers.created_at DESC LIMIT 30)
) AS filtered
ON event_headers.row_id = filtered.row_id
ORDER BY $orderBy
@@ -290,13 +290,13 @@ class QueryAssemblerTest : BaseDBTest() {
│ │ └── SCAN (subquery-1)
│ ├── UNION USING TEMP B-TREE
│ │ ├── CO-ROUTINE (subquery-3)
│ │ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
│ │ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
│ │ │ ├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
│ │ │ └── USE TEMP B-TREE FOR ORDER BY
│ │ └── SCAN (subquery-3)
│ └── UNION USING TEMP B-TREE
│ ├── CO-ROUTINE (subquery-5)
│ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
│ │ ├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
│ │ ├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
│ │ └── USE TEMP B-TREE FOR ORDER BY
│ └── SCAN (subquery-5)
@@ -712,10 +712,10 @@ class QueryAssemblerTest : BaseDBTest() {
assertEquals(
"""
SELECT event_headers.id, event_headers.pubkey, event_headers.created_at, event_headers.kind, event_headers.tags, event_headers.content, event_headers.sig FROM event_headers
INNER JOIN event_fts ON event_headers.row_id = event_fts.event_header_row_id
INNER JOIN event_fts ON event_headers.row_id = event_fts.rowid
WHERE (event_fts MATCH "keywords") AND (event_headers.pubkey IN ("7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d", "f3ac434d61bc0f491a814782ccfdf9c439dae1f0bde9097ad4a245f4c495cd14", "12ae0fd81c85e1e7d9ed096397dc3129849425fe6f8afce7213ebf38ddfc6ca9"))
ORDER BY event_headers.created_at DESC, event_headers.id ASC
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
ORDER BY event_fts.rank, event_headers.created_at DESC, event_headers.id ASC
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
└── USE TEMP B-TREE FOR ORDER BY
""".trimIndent(),
@@ -725,10 +725,10 @@ class QueryAssemblerTest : BaseDBTest() {
assertEquals(
"""
SELECT event_headers.id, event_headers.pubkey, event_headers.created_at, event_headers.kind, event_headers.tags, event_headers.content, event_headers.sig FROM event_headers
INNER JOIN event_fts ON event_headers.row_id = event_fts.event_header_row_id
INNER JOIN event_fts ON event_headers.row_id = event_fts.rowid
WHERE (event_fts MATCH "keywords") AND (event_headers.pubkey IN ("7c5eb72a4584fdaaeaa145b25c92ea9917704224951219dbd43acef9e91fb88d", "f3ac434d61bc0f491a814782ccfdf9c439dae1f0bde9097ad4a245f4c495cd14", "12ae0fd81c85e1e7d9ed096397dc3129849425fe6f8afce7213ebf38ddfc6ca9"))
ORDER BY event_headers.created_at DESC
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
ORDER BY event_fts.rank, event_headers.created_at DESC
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
└── USE TEMP B-TREE FOR ORDER BY
""".trimIndent(),
@@ -741,14 +741,14 @@ class QueryAssemblerTest : BaseDBTest() {
fun testKindAndSearch() =
forEachDB { db ->
val filter = Filter(kinds = listOf(1, 1111, 10000), search = "keywords")
val orderBy = if (db.indexStrategy.useAndIndexIdOnOrderBy) "event_headers.created_at DESC, event_headers.id ASC" else "event_headers.created_at DESC"
val orderBy = if (db.indexStrategy.useAndIndexIdOnOrderBy) "event_fts.rank, event_headers.created_at DESC, event_headers.id ASC" else "event_fts.rank, event_headers.created_at DESC"
assertEquals(
"""
SELECT event_headers.id, event_headers.pubkey, event_headers.created_at, event_headers.kind, event_headers.tags, event_headers.content, event_headers.sig FROM event_headers
INNER JOIN event_fts ON event_headers.row_id = event_fts.event_header_row_id
INNER JOIN event_fts ON event_headers.row_id = event_fts.rowid
WHERE (event_fts MATCH "keywords") AND (event_headers.kind IN ("1", "1111", "10000"))
ORDER BY $orderBy
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M2
├── SCAN event_fts VIRTUAL TABLE INDEX 0:M1
├── SEARCH event_headers USING INTEGER PRIMARY KEY (rowid=?)
└── USE TEMP B-TREE FOR ORDER BY
""".trimIndent(),
quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/SearchRelevanceOrderTest.kt
+202
View File
@@ -0,0 +1,202 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
import kotlin.test.assertEquals
/**
* NIP-50: search results are ordered by "quality of search result" (relevance),
* **not** by `created_at`, and the limit is applied after the score. The store
* uses FTS5 bm25 (`ORDER BY event_fts.rank`), so a stronger match outranks a
* newer one.
*/
class SearchRelevanceOrderTest {
private val signer = NostrSignerSync()
private fun note(
content: String,
createdAt: Long,
) = signer.sign(TextNoteEvent.build(content, createdAt = createdAt))
@Test
fun strongerMatchOutranksNewer() =
runBlocking {
val store = EventStore(dbName = null)
try {
// Older, but the term appears 3× in a short doc → most relevant.
val strong = note("bitcoin bitcoin bitcoin", createdAt = 1_000)
// Newer, term once buried in a long doc → least relevant.
val weak = note("bitcoin is one topic among many other unrelated words here padding", createdAt = 9_000)
// Newer still, but does not match at all.
val nonMatch = note("completely different subject entirely", createdAt = 9_999)
store.insert(weak)
store.insert(strong)
store.insert(nonMatch)
val results = store.query<Event>(Filter(search = "bitcoin", limit = 10)).map { it.id }
// Relevance order (strong before weak) — the opposite of
// created_at DESC (which would put weak first) — and the
// non-matching note is absent.
assertEquals(listOf(strong.id, weak.id), results)
} finally {
store.close()
}
}
@Test
fun limitAppliesAfterRelevanceScore() =
runBlocking {
val store = EventStore(dbName = null)
try {
// Three docs of decreasing relevance but increasing created_at,
// so created_at order and relevance order are exact opposites.
val best = note("apple apple apple apple", createdAt = 1)
val mid = note("apple apple filler words", createdAt = 2)
val worst = note("apple among lots of other unrelated filler words here", createdAt = 3)
store.insert(best)
store.insert(mid)
store.insert(worst)
// limit=2 after scoring keeps the two MOST RELEVANT, not the
// two newest.
val top2 = store.query<Event>(Filter(search = "apple", limit = 2)).map { it.id }
assertEquals(listOf(best.id, mid.id), top2)
} finally {
store.close()
}
}
/** A `#t` tag makes this a `search + tag` filter — the combined path. */
private var idSeq = 0
private fun hexId(n: Int): String {
val s = n.toString(16)
return "0".repeat(64 - s.length) + s
}
private fun tagged(
content: String,
createdAt: Long,
topic: String,
): Event =
EventFactory.create(
hexId(++idSeq),
"00".repeat(32),
createdAt,
1,
arrayOf(arrayOf("t", topic)),
content,
"0".repeat(128),
)
private fun evk(
content: String,
createdAt: Long,
kind: Int,
): Event = EventFactory.create(hexId(++idSeq), "00".repeat(32), createdAt, kind, arrayOf(), content, "0".repeat(128))
@Test
fun multiFilterSearchIsRelevanceOrderedAcrossBranches() =
runBlocking {
val store = EventStore(dbName = null)
try {
// Relevance A > B > C, created_at A < B < C (reverse), and the
// branches split by kind: A,C are kind 1 (TextNote), B is kind
// 1111 (Comment) — both searchable kinds.
val a = evk("apple apple apple apple", 1, 1)
val b = evk("apple apple apple", 2, 1111)
val c = evk("apple filler filler filler filler filler", 3, 1)
store.batchInsert(listOf(a, b, c))
// The client's search-across-kinds shape: one term, two filters.
val filters =
listOf(
Filter(search = "apple", kinds = listOf(1), limit = 100),
Filter(search = "apple", kinds = listOf(1111), limit = 100),
)
val ids = store.query<Event>(filters).map { it.id }
assertEquals(listOf(a.id, b.id, c.id), ids, "multi-filter search must be relevance-ordered across branches")
} finally {
store.close()
}
}
@Test
fun multiFilterSearchDedupsEventsMatchingSeveralBranches() =
runBlocking {
val store = EventStore(dbName = null)
try {
val x = evk("banana banana banana", 1, 1)
val y = evk("banana one", 2, 1)
store.batchInsert(listOf(x, y))
// Overlapping branches: both kind-1 events match BOTH filters.
// GROUP BY row_id must fold each to a single ranked row.
val filters =
listOf(
Filter(search = "banana", kinds = listOf(1, 6), limit = 100),
Filter(search = "banana", kinds = listOf(1, 2), limit = 100),
)
val ids = store.query<Event>(filters).map { it.id }
assertEquals(ids.size, ids.toSet().size, "no event may appear twice across branches")
assertEquals(listOf(x.id, y.id), ids, "deduped, relevance-ordered")
assertEquals(2, store.count(filters), "count parity across the union")
} finally {
store.close()
}
}
@Test
fun searchWithATagIsAlsoRelevanceOrdered() =
runBlocking {
val store = EventStore(dbName = null)
try {
// All tagged #t=nostr; relevance decreases as created_at rises,
// so created_at order would be the exact reverse of relevance.
val strong = tagged("nostr nostr nostr", createdAt = 1, topic = "nostr")
val weak = tagged("nostr among many other unrelated filler words here padding", createdAt = 2, topic = "nostr")
// Matches the term but wrong tag → excluded by the tag filter.
val wrongTag = tagged("nostr nostr nostr nostr", createdAt = 3, topic = "other")
// Right tag but doesn't match the term → excluded by search.
val noMatch = tagged("bitcoin only", createdAt = 4, topic = "nostr")
store.batchInsert(listOf(strong, weak, wrongTag, noMatch))
val filter = Filter(search = "nostr", tags = mapOf("t" to listOf("nostr")), limit = 10)
val ids = store.query<Event>(filter).map { it.id }
assertEquals(listOf(strong.id, weak.id), ids, "search + tag must be relevance-ordered, tag-scoped")
// limit after score keeps the most relevant one.
val top1 = store.query<Event>(filter.copy(limit = 1)).map { it.id }
assertEquals(listOf(strong.id), top1)
} finally {
store.close()
}
}
}
quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/StatementCachingConnectionTest.kt
+113
View File
@@ -0,0 +1,113 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import androidx.sqlite.SQLiteStatement
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
import kotlin.test.AfterTest
import kotlin.test.BeforeTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertNotSame
import kotlin.test.assertSame
class StatementCachingConnectionTest {
private lateinit var conn: StatementCachingConnection
private val sql = "SELECT ? AS v"
@BeforeTest
fun setup() {
conn = StatementCachingConnection(BundledSQLiteDriver().open(":memory:"))
}
@AfterTest
fun tearDown() {
conn.close()
}
private fun readOne(stmt: SQLiteStatement): Long {
stmt.step()
return stmt.getLong(0)
}
@Test
fun sequentialSameSqlReusesTheSameHandle() {
var first: SQLiteStatement? = null
conn.prepare(sql).use { stmt ->
stmt.bindLong(1, 7)
assertEquals(7, readOne(stmt))
first = stmt
}
// Closed (returned to pool) — the next prepare of the same SQL must
// hand back the very same cached handle, not a fresh prepare.
conn.prepare(sql).use { stmt ->
assertSame(first, stmt)
stmt.bindLong(1, 9)
assertEquals(9, readOne(stmt))
}
}
@Test
fun concurrentSameSqlHandlesAreDistinctAndIndependent() {
// The k-way-merge shape: many identical-SQL cursors live at once.
val a = conn.prepare(sql)
val b = conn.prepare(sql)
val c = conn.prepare(sql)
assertNotSame(a, b)
assertNotSame(b, c)
assertNotSame(a, c)
a.bindLong(1, 1)
b.bindLong(1, 2)
c.bindLong(1, 3)
// Each cursor keeps its own bindings/position even while the others
// are open — no aliasing of one native handle.
assertEquals(1, readOne(a))
assertEquals(2, readOne(b))
assertEquals(3, readOne(c))
a.close()
b.close()
c.close()
// After release, a fresh concurrent burst reuses the pooled handles.
val reused = conn.prepare(sql)
assertSame(a, reused, "pool should hand back a freed handle before preparing anew")
reused.close()
}
@Test
fun overflowingTheGlobalCapFallsBackToUncached() {
val small = StatementCachingConnection(BundledSQLiteDriver().open(":memory:"), maxCachedStatements = 2)
try {
val live = (0 until 5).map { small.prepare(sql) }
// All five must be usable even though only two can be cached; the
// extra three are plain uncached statements.
live.forEachIndexed { i, stmt ->
stmt.bindLong(1, i.toLong())
assertEquals(i.toLong(), readOne(stmt))
}
live.forEach { it.close() }
} finally {
small.close()
}
}
}
quartz/src/commonTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/TagMergeCorrectnessTest.kt
+295
View File
@@ -0,0 +1,295 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* Correctness guard for the tag-stream path of [MergeQueryExecutor]: the
* `#<x>=[values] (+ kinds) [+ since/until] limit=N` watcher shape
* (reactions/replies). The merge must return the same newest-N a
* `SELECT DISTINCT … ORDER BY created_at DESC LIMIT N` would, deduping events
* that carry several of the queried tag values.
*
* Where `created_at` is distinct the order is fully determined and asserted
* against a Kotlin reference. Where it ties, the tag cursors can only order by
* `created_at` (event_tags has no id column), so the result is a valid
* newest-N but not id-exact — those cases assert set + size instead.
*/
class TagMergeCorrectnessTest {
private val hex = "0123456789abcdef"
private fun mix(seed: Long): Long {
var z = seed + -0x61c8864680b583ebL
z = (z xor (z ushr 30)) * -0x40a7b892e31b1a47L
z = (z xor (z ushr 27)) * -0x6b2fb644ecceee15L
return z xor (z ushr 31)
}
private fun hex64(
salt: Long,
index: Int,
): String {
val out = CharArray(64)
for (w in 0 until 4) {
val v = mix(salt * 1_000_003 + index.toLong() * 4 + w)
for (b in 0 until 8) {
val byte = ((v ushr (b * 8)) and 0xFF).toInt()
out[(w * 8 + b) * 2] = hex[byte ushr 4]
out[(w * 8 + b) * 2 + 1] = hex[byte and 0xF]
}
}
return out.concatToString()
}
private val sig = "0".repeat(128)
private var idSeq = 0
private fun ev(
createdAt: Long,
kind: Int,
eTags: List<String>,
): Event =
EventFactory.create(
hex64(7, idSeq++),
hex64(1, idSeq),
createdAt,
kind,
eTags.map { arrayOf("e", it) }.toTypedArray(),
"",
sig,
)
private val newestFirst =
Comparator<Event> { a, b ->
if (a.createdAt != b.createdAt) b.createdAt.compareTo(a.createdAt) else a.id.compareTo(b.id)
}
private fun reference(
all: List<Event>,
values: Set<String>,
kinds: Set<Int>?,
since: Long?,
until: Long?,
limit: Int,
): List<String> =
all
.asSequence()
.filter { e -> e.tags.any { it.size >= 2 && it[0] == "e" && it[1] in values } }
.filter { kinds == null || it.kind in kinds }
.filter { since == null || it.createdAt >= since }
.filter { until == null || it.createdAt <= until }
.sortedWith(newestFirst)
.map { it.id }
.distinct()
.take(limit)
.toList()
private fun mergeEligible(
store: EventStore,
filter: Filter,
): Boolean =
MergeQueryExecutor.streamCount(
with(store.store.queryBuilder) { filter.toFilterWithDTags() },
store.store.queryBuilder.indexStrategy,
) > 0
private fun newStore() =
EventStore(
dbName = null,
indexStrategy =
DefaultIndexingStrategy(
indexTagsByCreatedAtAlone = true,
useAndIndexIdOnOrderBy = true,
indexFullTextSearch = false,
),
)
@Test
fun distinctCreatedAt_withKinds_matchesReference() =
runBlocking {
val store = newStore()
val notes = (0 until 6).map { hex64(2, it) }
val all = ArrayList<Event>()
var t = 1_700_000_000L
for (round in 0 until 40) {
// reactions (kind 7) and replies (kind 1) to a rotating note
all.add(ev(t++, 7, listOf(notes[round % notes.size])))
all.add(ev(t++, 1, listOf(notes[(round + 1) % notes.size])))
}
// Noise: kind-7 to notes NOT in the query set, and other tags.
for (i in 0 until 100) all.add(ev(t++, 7, listOf(hex64(9, i))))
store.batchInsert(all)
val queried = notes.take(3)
val filter = Filter(kinds = listOf(7), tags = mapOf("e" to queried), limit = 25)
assertTrue(mergeEligible(store, filter), "tag watcher must be merge-eligible")
val merged = store.query<Event>(filter).map { it.id }
assertEquals(reference(all, queried.toSet(), setOf(7), null, null, 25), merged)
store.close()
}
@Test
fun crossStreamDuplicate_emittedOnce() =
runBlocking {
val store = newStore()
val a = hex64(3, 0)
val b = hex64(3, 1)
val all = ArrayList<Event>()
var t = 1_700_000_000L
// Events tagging BOTH a and b — they appear in both value streams
// and must be emitted exactly once.
repeat(5) { all.add(ev(t++, 1, listOf(a, b))) }
// Events tagging only one of them.
repeat(5) { all.add(ev(t++, 1, listOf(a))) }
repeat(5) { all.add(ev(t++, 1, listOf(b))) }
store.batchInsert(all)
val filter = Filter(kinds = listOf(1), tags = mapOf("e" to listOf(a, b)), limit = 500)
assertTrue(mergeEligible(store, filter))
val merged = store.query<Event>(filter).map { it.id }
assertEquals(merged.size, merged.toSet().size, "no event may be emitted twice")
assertEquals(15, merged.size, "5 both + 5 a-only + 5 b-only = 15 distinct")
assertEquals(reference(all, setOf(a, b), setOf(1), null, null, 500), merged)
store.close()
}
@Test
fun noKinds_usesTagCreatedAtIndex() =
runBlocking {
val store = newStore()
val notes = (0 until 5).map { hex64(4, it) }
val all = ArrayList<Event>()
var t = 1_700_000_000L
for (round in 0 until 30) all.add(ev(t++, (round % 3) + 1, listOf(notes[round % notes.size])))
for (i in 0 until 60) all.add(ev(t++, 1, listOf(hex64(8, i))))
store.batchInsert(all)
val queried = notes.take(3)
val filter = Filter(tags = mapOf("e" to queried), limit = 20)
assertTrue(mergeEligible(store, filter), "no-kind tag watcher must be merge-eligible with the tag index")
val merged = store.query<Event>(filter).map { it.id }
assertEquals(reference(all, queried.toSet(), null, null, null, 20), merged)
store.close()
}
@Test
fun withSinceAndUntil_boundsTheWindow() =
runBlocking {
val store = newStore()
val notes = (0 until 4).map { hex64(5, it) }
val all = ArrayList<Event>()
val base = 1_700_000_000L
for (i in 0 until 300) all.add(ev(base + i.toLong(), 7, listOf(notes[i % notes.size])))
store.batchInsert(all)
val since = base + 50
val until = base + 250
val filter = Filter(kinds = listOf(7), tags = mapOf("e" to notes), since = since, until = until, limit = 500)
assertTrue(mergeEligible(store, filter))
val merged = store.query<Event>(filter).map { it.id }
val ref = reference(all, notes.toSet(), setOf(7), since, until, 500)
assertEquals(ref, merged)
assertTrue(merged.isNotEmpty() && ref.size < 300)
store.close()
}
@Test
fun rawPathMatchesDecoded() =
runBlocking {
val store = newStore()
val notes = (0 until 6).map { hex64(6, it) }
val all = ArrayList<Event>()
var t = 1_700_000_000L
for (round in 0 until 25) all.add(ev(t++, 7, listOf(notes[round % notes.size])))
store.batchInsert(all)
val filter = Filter(kinds = listOf(7), tags = mapOf("e" to notes.take(3)), limit = 15)
val decoded = store.query<Event>(filter).map { it.id }
val raw = store.store.rawQuery(filter).map { it.id }
assertEquals(decoded, raw, "the zero-decode raw path must match the decoded query")
store.close()
}
@Test
fun tiedCreatedAt_matchesReferenceAsSet() =
runBlocking {
val store = newStore()
val notes = (0 until 4).map { hex64(10, it) }
val all = ArrayList<Event>()
// Many events share a created_at — the tag cursor can't id-order
// within a second, so assert a valid newest-N by set + size.
var t = 1_700_000_000L
for (block in 0 until 15) {
val ts = t
for (n in notes) all.add(ev(ts, 7, listOf(n)))
t += 1
}
store.batchInsert(all)
val filter = Filter(kinds = listOf(7), tags = mapOf("e" to notes), limit = 22)
assertTrue(mergeEligible(store, filter))
val merged = store.query<Event>(filter).map { it.id }
assertEquals(22, merged.size)
assertEquals(merged.size, merged.toSet().size)
// The whole result must sit within the newest slice the reference
// would return once ties are resolved either way: everything in
// `merged` must be at or above the created_at cutoff.
val ids = merged.toSet()
val chosen = all.filter { it.id in ids }
val cutoff = chosen.minOf { it.createdAt }
val eligibleAboveCutoff = all.filter { it.createdAt > cutoff }.map { it.id }.toSet()
assertTrue(eligibleAboveCutoff.all { it in ids }, "every event newer than the cutoff must be included")
store.close()
}
@Test
fun ineligibleShapesFallThrough() =
runBlocking {
val store = newStore()
store.batchInsert(listOf(ev(1_700_000_000L, 7, listOf(hex64(2, 0)))))
// Single value → single seek, not a merge.
assertFalse(mergeEligible(store, Filter(kinds = listOf(7), tags = mapOf("e" to listOf(hex64(2, 0))), limit = 10)))
// No limit → not merge-eligible.
assertFalse(mergeEligible(store, Filter(kinds = listOf(7), tags = mapOf("e" to listOf(hex64(2, 0), hex64(2, 1))))))
// Authors present → covered-index seek shape, not a tag merge.
assertFalse(
mergeEligible(
store,
Filter(kinds = listOf(7), authors = listOf(hex64(1, 1)), tags = mapOf("e" to listOf(hex64(2, 0), hex64(2, 1))), limit = 10),
),
)
store.close()
}
}
quartz/src/jvmMain/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsQueryPlanner.kt
+116 -26
View File
@@ -25,6 +25,7 @@ import com.vitorpamplona.quartz.nip01Core.core.isAddressable
import com.vitorpamplona.quartz.nip01Core.core.isReplaceable
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.store.sqlite.TagNameValueHasher
import java.nio.file.DirectoryStream
import java.nio.file.Files
import java.nio.file.Path
import kotlin.io.path.exists
@@ -36,16 +37,23 @@ import kotlin.io.path.exists
*
* Step-2 coverage:
* - `ids` → direct canonical opens
* - `tagsAll`/`tags` → tag index union (first key)
* - `kinds` → kind index union
* - `authors` → author index union
* - `tagsAll`/`tags`/`kinds`/`authors` → cheapest index tree drives
* (capped entry-count comparison), the rest post-filter
* - otherwise → full scan via every `idx/kind/<k>/` subtree
*
* The planner is intentionally dumb about selectivity — "first available
* driver wins". A cost-based picker (smallest listing) can slot in
* later without changing callers. All FilterMatcher semantics (tag
* AND/OR, since/until, id, author, kind cross-checks) are enforced in
* the orchestrator, so picking a loose driver is correctness-safe.
* Driver choice is cost-based: every legal driver (each `tagsAll` value
* alone — AND semantics make any single value a complete driver — each
* `tags` key's value union, the kind set, the author set) opens a lazy
* directory iterator, all are drained in lockstep, and the first to
* exhaust — the smallest listing — drives. A giant tree (`idx/kind/1/`
* with a million entries) is therefore never read past ~the smallest
* candidate's size. Before the pick, the fixed tags → kinds → authors
* order sent `authors + kinds + limit` — the most common CLI shape —
* through the kind tree: 149 ms fixed-order vs 4.0 ms cost-based at 30k
* events per `FsDriverSelectionBenchmark` (floor: author-only at
* 1.4 ms). All FilterMatcher semantics (tag AND/OR,
* since/until, id, author, kind cross-checks) are enforced in the
* orchestrator, so any driver pick is correctness-safe.
*/
internal class FsQueryPlanner(
private val layout: FsLayout,
@@ -74,19 +82,100 @@ internal class FsQueryPlanner(
return ftsDriver(search)
}
firstTagKey(filter)?.let { (name, values) ->
return mergeDesc(values.map { v -> walkDir(layout.tagValueDir(name, v, hasher.hash(name, v))) })
val candidates = driverCandidates(filter)
if (candidates.isEmpty()) return allKindsDriver()
return mergeDesc(cheapestDriver(candidates).map { walkDir(it) })
}
/**
* Every set of index directories that, walked and post-filtered, yields
* a superset of the filter's matches:
* - each `tagsAll` value alone (AND semantics — every match carries it),
* - each `tags` key's full value union (OR within a key, AND across),
* - the kind set, and the author set.
* Listed in the old fixed-priority order so [cheapestDriver] keeps that
* order on cost ties.
*/
private fun driverCandidates(filter: Filter): List<List<Path>> {
val out = ArrayList<List<Path>>()
filter.tagsAll?.forEach { (name, values) ->
values.forEach { v -> out.add(listOf(layout.tagValueDir(name, v, hasher.hash(name, v)))) }
}
filter.tags?.forEach { (name, values) ->
if (values.isNotEmpty()) {
out.add(values.map { v -> layout.tagValueDir(name, v, hasher.hash(name, v)) })
}
}
filter.kinds?.takeIf { it.isNotEmpty() }?.let { kinds ->
out.add(kinds.map { layout.kindDir(it) })
}
filter.authors?.takeIf { it.isNotEmpty() }?.let { authors ->
out.add(authors.map { layout.authorDir(it) })
}
return out
}
/**
* Smallest candidate by lockstep listing drain: one lazy directory
* iterator per candidate, all advanced [COST_BATCH] entries per round —
* the first to exhaust its listing is the smallest, so a giant tree is
* never read past ~the smallest candidate's size (a candidate that
* exhausts on round one costs the others one batch each). A candidate
* whose dirs are all missing exhausts immediately: driving from an empty
* mandatory predicate correctly yields an empty result. If every
* candidate survives [COST_CAP] entries, all are huge and relative
* driver choice stops mattering — the first (old fixed-priority order)
* wins.
*/
private fun cheapestDriver(candidates: List<List<Path>>): List<Path> {
if (candidates.size == 1) return candidates[0]
val cursors = candidates.map { EntryCursor(it) }
try {
var advanced = 0L
while (advanced < COST_CAP) {
for (i in cursors.indices) {
if (!cursors[i].skip(COST_BATCH)) return candidates[i]
}
advanced += COST_BATCH
}
return candidates[0]
} finally {
cursors.forEach { it.close() }
}
}
/** Lazy entry iterator over a candidate's directories, in order. */
private class EntryCursor(
dirs: List<Path>,
) : AutoCloseable {
private val remaining = ArrayDeque(dirs)
private var stream: DirectoryStream<Path>? = null
private var iter: Iterator<Path> = emptyList<Path>().iterator()
/** Advances up to [n] entries; false when the listing ends first. */
fun skip(n: Int): Boolean {
var left = n
while (left > 0) {
if (iter.hasNext()) {
iter.next()
left--
continue
}
close()
val dir = remaining.removeFirstOrNull() ?: return false
if (!Files.isDirectory(dir)) continue
stream = Files.newDirectoryStream(dir)
iter = stream!!.iterator()
}
return true
}
filter.kinds?.let { kinds ->
return mergeDesc(kinds.map { walkDir(layout.kindDir(it)) })
override fun close() {
stream?.close()
stream = null
iter = emptyList<Path>().iterator()
}
filter.authors?.let { authors ->
return mergeDesc(authors.map { walkDir(layout.authorDir(it)) })
}
return allKindsDriver()
}
/**
@@ -301,14 +390,15 @@ internal class FsQueryPlanner(
var top: Candidate,
)
// ---- helpers ------------------------------------------------------
private companion object {
/** Entries each candidate's cursor advances per lockstep round. */
const val COST_BATCH = 64
/** First tag filter with at least one value, preferring `tagsAll`. */
private fun firstTagKey(filter: Filter): Pair<String, List<String>>? {
filter.tagsAll?.firstNonEmpty()?.let { return it }
filter.tags?.firstNonEmpty()?.let { return it }
return null
/**
* Stop draining once every candidate has survived this many
* entries: past it they are all huge, relative choice stops
* mattering, and the first candidate in priority order wins.
*/
const val COST_CAP = 65_536L
}
private fun Map<String, List<String>>.firstNonEmpty(): Pair<String, List<String>>? = entries.firstOrNull { it.value.isNotEmpty() }?.let { it.key to it.value }
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FollowFeedReadBenchmark.kt
+1 -1
View File
@@ -202,7 +202,7 @@ class FollowFeedReadBenchmark {
runBlocking {
store.store.pool.useReader { c ->
var n = 0
MergeQueryExecutor.run(c, filter, store.store.queryBuilder.indexStrategy) { n++ }
MergeQueryExecutor.run(c, filter, store.store.queryBuilder.indexStrategy, store.store.seedModule::hasher) { n++ }
n
}
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/FtsSearchScalingBenchmark.kt
+185
View File
@@ -0,0 +1,185 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
import androidx.sqlite.SQLiteConnection
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.store.sqlite.DefaultIndexingStrategy
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
/**
* Two measurements around the contentless FTS index, motivating the v4→v5
* schema change and being honest about what it does and does not fix.
*
* 1. **Delete scaling — the win.** The old `fts5(event_header_row_id,
* content)` schema had the `fts_foreign_key` trigger delete by a regular
* FTS column, which FTS5 cannot seek (it scans, O(n) per delete). The
* contentless schema keys deletes off the rowid (= event_headers.row_id),
* an O(log n) primary-key seek. Every event removal fires this trigger
* (replaceable rotation, kind-5, expiration, right-to-vanish).
* 2. **Search scaling — the limit.** `MATCH … ORDER BY rank LIMIT n` (NIP-50
* relevance ordering, bm25) must score *every* matching document, so
* search cost grows with the match set regardless of ordering (created_at
* has the same shape). Segment `optimize` compacts the index but does not
* change that; corpus-independent search needs an external engine. Shown
* fragmented vs optimized to size the (secondary) compaction effect.
*
* Size search with `-DftsBenchScale=N` (default 1). Not an assertion test.
*/
class FtsSearchScalingBenchmark {
companion object {
val SCALE = System.getProperty("ftsBenchScale")?.toInt() ?: 1
val SIZES = listOf(50_000, 100_000, 200_000).map { it * SCALE }
const val NEEDLE = "zzneedle"
}
private val hex = "0123456789abcdef"
private fun mix(seed: Long): Long {
var z = seed + -0x61c8864680b583ebL
z = (z xor (z ushr 30)) * -0x40a7b892e31b1a47L
z = (z xor (z ushr 27)) * -0x6b2fb644ecceee15L
return z xor (z ushr 31)
}
private fun hex64(index: Int): String {
val out = CharArray(64)
for (w in 0 until 4) {
val v = mix(index.toLong() * 4 + w + 99)
for (b in 0 until 8) {
val byte = ((v ushr (b * 8)) and 0xFF).toInt()
out[(w * 8 + b) * 2] = hex[byte ushr 4]
out[(w * 8 + b) * 2 + 1] = hex[byte and 0xF]
}
}
return String(out)
}
private val vocab = (0 until 400).map { "word$it" }
private val sig = "0".repeat(128)
private fun seed(n: Int): List<Event> {
val base = 1_700_000_000L
val events = ArrayList<Event>(n)
for (i in 0 until n) {
val r = mix(i.toLong())
val content =
buildString {
for (w in 0 until 8) append(vocab[((r ushr (w * 3)) and 0x1FF).toInt() % vocab.size]).append(' ')
// ~1% carry the searched term.
if (i % 100 == 0) append(NEEDLE)
}
events.add(EventFactory.create(hex64(i), hex64(i % 5000), base + i.toLong(), 1, emptyArray(), content, sig))
}
return events
}
private inline fun timeMs(block: () -> Unit): Double {
val start = System.nanoTime()
block()
return (System.nanoTime() - start) / 1e6
}
private fun SQLiteConnection.exec(sql: String) = prepare(sql).use { it.step() }
@Test
fun deleteByColumnVsByRowid() {
// Old-schema (delete by FTS column) vs contentless (delete by rowid),
// 500 deletes at two table sizes. By-column should grow with the
// table; by-rowid should stay flat.
println("─ FtsSearchScalingBenchmark.delete (500 deletes) ─")
println(" %-9s %14s %14s".format("rows", "byColumn", "byRowid"))
for (n in listOf(2_000 * SCALE, 8_000 * SCALE)) {
val db = BundledSQLiteDriver().open(":memory:")
try {
db.exec("CREATE VIRTUAL TABLE col USING fts5(event_header_row_id, content)")
db.exec("CREATE VIRTUAL TABLE row USING fts5(content, content='', contentless_delete=1)")
for (i in 1..n) {
db.prepare("INSERT INTO col(event_header_row_id, content) VALUES (?, 'alpha beta gamma')").use {
it.bindLong(1, i.toLong())
it.step()
}
db.prepare("INSERT INTO row(rowid, content) VALUES (?, 'alpha beta gamma')").use {
it.bindLong(1, i.toLong())
it.step()
}
}
val byColumn =
timeMs {
for (i in 1..500) {
db.prepare("DELETE FROM col WHERE event_header_row_id = ?").use {
it.bindLong(1, i.toLong())
it.step()
}
}
}
val byRowid =
timeMs {
for (i in 1..500) {
db.prepare("DELETE FROM row WHERE rowid = ?").use {
it.bindLong(1, i.toLong())
it.step()
}
}
}
println(" %-9s %11.2f ms %11.2f ms".format("${n / 1000}k", byColumn, byRowid))
} finally {
db.close()
}
}
}
@Test
fun searchScaling() =
runBlocking {
println("─ FtsSearchScalingBenchmark.search (created_at DESC, limit=50) ─")
println(" %-9s %16s %16s".format("corpus", "fragmented", "optimized"))
for (size in SIZES) {
val store = EventStore(dbName = null, indexStrategy = DefaultIndexingStrategy())
try {
seed(size).chunked(10_000).forEach { store.batchInsert(it) }
val f = Filter(search = NEEDLE, limit = 50)
val frag = time(store, f)
store.store.reindexFullTextSearch() // rebuild + optimize()
val opt = time(store, f)
println(" %-9s %13.2f ms %13.2f ms".format("${size / 1000}k", frag, opt))
} finally {
store.close()
}
}
}
private suspend fun time(
store: EventStore,
filter: Filter,
): Double {
repeat(3) { store.query<Event>(filter) }
val runs = 20
val start = System.nanoTime()
repeat(runs) { store.query<Event>(filter) }
return (System.nanoTime() - start) / 1e6 / runs
}
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/SmallReqFloorBenchmark.kt
+89 -2
View File
@@ -32,11 +32,14 @@ import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.CoroutineStart
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.withTimeout
import java.util.concurrent.atomic.AtomicInteger
import kotlin.test.Test
import kotlin.test.assertEquals
@@ -65,6 +68,8 @@ class SmallReqFloorBenchmark {
const val AUTHORS = 2_500 // ~20 events per author, matching author-archive
const val ROUNDS = 400
const val WARMUP = 100
const val IDLE_SUBS = 1_000
const val FANOUT_SUBS = 200
}
private fun hexId(seed: Int): String = seed.toString(16).padStart(64, '0')
@@ -122,16 +127,19 @@ class SmallReqFloorBenchmark {
}
// --- B: backend queryRaw to EOSE (live machinery included) ---
// UNDISPATCHED mirrors the production path (RelaySession.handleReq
// starts the query coroutine undispatched), so B−A is the live
// machinery itself, not a benchmark-only scheduler hop.
suspend fun timeBackend(round: Int): Long {
val eose = CompletableDeferred<Long>()
val t0 = System.nanoTime()
val job =
scope.launch {
scope.launch(start = CoroutineStart.UNDISPATCHED) {
live.queryRaw(
ctx = ctx,
filters = listOf(filterFor(round)),
onEachStored = {},
onEachLive = {},
onEachLive = { _, _ -> },
onEose = { eose.complete(System.nanoTime() - t0) },
)
}
@@ -143,6 +151,33 @@ class SmallReqFloorBenchmark {
val b = LongArray(ROUNDS)
repeat(ROUNDS) { b[it] = timeBackend(it) }
// --- B@1k: same, with 1000 idle live subscriptions parked ---
// Register/unregister cost scales with the live population
// (FilterIndex mutates a shared snapshot per REQ open/close),
// which the single-sub stage can't see. Each idle sub filters
// on an author absent from the corpus: 0-row replay, then parks
// at the live tail and stays registered.
val idleJobs =
(0 until IDLE_SUBS).map { i ->
val ready = CompletableDeferred<Unit>()
val job =
scope.launch(start = CoroutineStart.UNDISPATCHED) {
live.queryRaw(
ctx = ctx,
filters = listOf(Filter(authors = listOf(hexId(1_000_000 + i)), kinds = listOf(1), limit = 1)),
onEachStored = {},
onEachLive = { _, _ -> },
onEose = { ready.complete(Unit) },
)
}
ready.await()
job
}
repeat(WARMUP) { timeBackend(it) }
val b1k = LongArray(ROUNDS)
repeat(ROUNDS) { b1k[it] = timeBackend(it) }
idleJobs.forEach { it.cancel() }
// --- C: full session dispatch, REQ json in → EOSE frame out ---
suspend fun timeSession(round: Int): Long {
val eose = CompletableDeferred<Long>()
@@ -160,15 +195,67 @@ class SmallReqFloorBenchmark {
val c = LongArray(ROUNDS)
repeat(ROUNDS) { c[it] = timeSession(it) }
// --- fanout: one live event → FANOUT_SUBS live subscriptions ---
// All subs register on `live` directly (via queryRaw, same backend
// we submit into) and filter an author with no stored events (0-row
// replay, then park live). Submitting one matching event fans out to
// every sub; the body is serialized once and spliced per sub, so
// this measures the shared-serialization path (#2). skipVerify so
// the synthetic sig is accepted. Fewer rounds than A–C: each round
// is FANOUT_SUBS deliveries and a real group-commit insert.
val fanAuthor = hexId(9_000_001)
val delivered = AtomicInteger(0)
var fanDone = CompletableDeferred<Long>()
var fanStart = 0L
val fanJobs =
(0 until FANOUT_SUBS).map {
val ready = CompletableDeferred<Unit>()
val job =
scope.launch(start = CoroutineStart.UNDISPATCHED) {
live.queryRaw(
ctx = ctx,
filters = listOf(Filter(authors = listOf(fanAuthor), kinds = listOf(1))),
onEachStored = {},
onEachLive = { _, _ ->
if (delivered.incrementAndGet() == FANOUT_SUBS) {
fanDone.complete(System.nanoTime() - fanStart)
}
},
onEose = { ready.complete(Unit) },
)
}
ready.await()
job
}
val fanRounds = 60
val fanWarmup = 15
val fan = LongArray(fanRounds)
var fanSeq = 0
repeat(fanWarmup + fanRounds) { r ->
delivered.set(0)
fanDone = CompletableDeferred()
val ev = EventFactory.create<Event>(hexId(9_500_000 + fanSeq), fanAuthor, 1_700_000_000L + fanSeq, 1, emptyArray(), "fanout $fanSeq", sig)
fanSeq++
fanStart = System.nanoTime()
live.submit(ev, skipVerify = true) {}
val nanos = withTimeout(30_000) { fanDone.await() }
if (r >= fanWarmup) fan[r - fanWarmup] = nanos
}
fanJobs.forEach { it.cancel() }
assertEquals(true, rowsA > 0, "author filters must return rows")
val mA = median(a)
val mB = median(b)
val mB1k = median(b1k)
val mC = median(c)
println("SmallReqFloorBenchmark @ ${EVENTS / 1000}k events, ~${rowsA / ROUNDS} rows/req, medians of $ROUNDS")
println(" A raw store query: ${"%6.3f".format(mA)} ms")
println(" B backend queryRaw→EOSE: ${"%6.3f".format(mB)} ms (live machinery +${"%6.3f".format(mB - mA)})")
println(" B@${IDLE_SUBS} idle subs: ${"%6.3f".format(mB1k)} ms (population cost +${"%6.3f".format(mB1k - mB)})")
println(" C session REQ→EOSE: ${"%6.3f".format(mC)} ms (dispatch+frames +${"%6.3f".format(mC - mB)})")
val mFan = median(fan)
println(" fanout 1→$FANOUT_SUBS live subs: ${"%6.3f".format(mFan)} ms (${"%.2f".format(mFan * 1000 / FANOUT_SUBS)} µs/sub; body serialized once)")
server.close()
scope.cancel()
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/relay/prodbench/TagAuthorIndexBenchmark.kt
+184
View File
@@ -0,0 +1,184 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.relay.prodbench
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.store.sqlite.DefaultIndexingStrategy
import com.vitorpamplona.quartz.nip01Core.store.sqlite.EventStore
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.runBlocking
import kotlin.test.Test
/**
* Measures the two tag-path query shapes the client filter-assembler survey
* (2026-07) found hot but that no existing benchmark covers:
*
* 1. **tag ∩ author (DM-room shape)** — `kinds=[4] AND authors=[peer] AND
* #p=[me] LIMIT n`. 65 assembler call sites build this shape (every
* NIP-04 chat room, reports-by-follows, follows-scoped community feeds).
* [com.vitorpamplona.quartz.nip01Core.store.sqlite.IndexingStrategy.indexTagsWithKindAndPubkey]
* gates a covering `(tag_hash, kind, pubkey_hash, created_at)` index for
* it, but the flag is off everywhere (including geode). Without it the
* plan seeks `(tag_hash, kind)` and reads EVERY DM the user has ever
* received before filtering to the one peer. This compares query latency
* with the flag off vs on, and the batch-insert cost the extra index adds.
*
* 2. **large-IN tag watcher (reactions shape)** — `kinds=[7] AND
* #e=[hundreds of note ids] LIMIT n`. The per-value streams come sorted
* off `(tag_hash, kind, created_at)`, but their union does not, so SQLite
* collects every matching row and TEMP-B-TREE sorts to the limit — the
* tag-index analogue of the follow-feed regression
* [MergeQueryExecutor] fixed for author streams. Reported with and
* without a `since` bound to show what EOSE-warm steady state hides.
*
* Size the seed with `-DtagBenchScale=N` (default 1 ≈ ~200k events).
*/
class TagAuthorIndexBenchmark {
companion object {
val SCALE = System.getProperty("tagBenchScale")?.toInt() ?: 1
}
private val hex = "0123456789abcdef"
private fun mix(seed: Long): Long {
var z = seed + -0x61c8864680b583ebL
z = (z xor (z ushr 30)) * -0x40a7b892e31b1a47L
z = (z xor (z ushr 27)) * -0x6b2fb644ecceee15L
return z xor (z ushr 31)
}
private fun hex64(
salt: Long,
index: Int,
): String {
val out = CharArray(64)
for (w in 0 until 4) {
val v = mix(salt * 1_000_003 + index.toLong() * 4 + w)
for (b in 0 until 8) {
val byte = ((v ushr (b * 8)) and 0xFF).toInt()
out[(w * 8 + b) * 2] = hex[byte ushr 4]
out[(w * 8 + b) * 2 + 1] = hex[byte and 0xF]
}
}
return String(out)
}
private val sig = "0".repeat(128)
private var idSeq = 0
private fun ev(
pubkey: String,
createdAt: Long,
kind: Int,
tags: Array<Array<String>>,
): Event = EventFactory.create(hex64(7, idSeq++), pubkey, createdAt, kind, tags, "", sig)
private fun seedEvents(): List<Event> {
idSeq = 0
val base = 1_700_000_000L
val span = 3_000_000L // ~35 days
val me = hex64(9, 0)
val events = ArrayList<Event>(220_000 * SCALE)
// DM inbox: 200 peers, 300 DMs each → 60k kind-4 rows sharing the
// same (p:me) tag hash. The room query wants one peer's 300.
val peers = (0 until 200).map { hex64(2, it) }
for ((i, peer) in peers.withIndex()) {
repeat(300 * SCALE) {
val ts = base + (mix(i * 131L + it) and 0x7fffffff) % span
events.add(ev(peer, ts, 4, arrayOf(arrayOf("p", me))))
}
}
// Notification noise: 2000 authors mention me in kind-1 notes, so
// (p:me) spans multiple kinds like a real inbox does.
repeat(40_000 * SCALE) {
val author = hex64(3, it % 2_000)
val ts = base + (mix(it * 17L) and 0x7fffffff) % span
events.add(ev(author, ts, 1, arrayOf(arrayOf("p", me))))
}
// Reactions: 100k kind-7 events spread over 5000 target notes, for
// the large-IN watcher shape.
val noteIds = (0 until 5_000).map { hex64(5, it) }
repeat(100_000 * SCALE) {
val author = hex64(4, it % 3_000)
val ts = base + (mix(it * 29L) and 0x7fffffff) % span
events.add(ev(author, ts, 7, arrayOf(arrayOf("e", noteIds[it % noteIds.size]))))
}
return events
}
@Test
fun compareTagAuthorIndex() =
runBlocking {
val events = seedEvents()
val me = hex64(9, 0)
val peers = (0 until 200).map { hex64(2, it) }
val noteIds = (0 until 5_000).map { hex64(5, it) }
println("─ TagAuthorIndexBenchmark: ${events.size} events (scale=$SCALE) ─")
val strategies =
listOf(
"flag-off" to DefaultIndexingStrategy(indexFullTextSearch = false),
"flag-on " to DefaultIndexingStrategy(indexFullTextSearch = false, indexTagsWithKindAndPubkey = true),
)
for ((label, strategy) in strategies) {
val store = EventStore(dbName = null, indexStrategy = strategy)
val t0 = System.nanoTime()
events.chunked(10_000).forEach { store.batchInsert(it) }
val insertMs = (System.nanoTime() - t0) / 1e6
println(" ═ $label ═ insert: %.0f ms (%.1f µs/event)".format(insertMs, insertMs * 1000 / events.size))
// 1. DM room: one peer's DMs out of the whole (p:me) inbox.
val room = Filter(kinds = listOf(4), authors = listOf(peers[42]), tags = mapOf("p" to listOf(me)), limit = 100)
time(store, "dm-room (#p ∩ author ∩ kind, limit 100)", room)
// 2. Reactions watcher: 300 note ids, cold (no since).
val watcher = Filter(kinds = listOf(7), tags = mapOf("e" to noteIds.take(300)), limit = 500)
time(store, "reactions (#e IN 300, limit 500, cold)", watcher)
// 3. Same watcher, EOSE-warm (since bounds the window).
val warm = watcher.copy(since = 1_700_000_000L + 2_900_000L)
time(store, "reactions (#e IN 300, limit 500, since)", warm)
store.close()
}
}
private suspend fun time(
store: EventStore,
label: String,
filter: Filter,
) {
repeat(3) { store.query<Event>(filter) }
val runs = 10
var rows = 0
val start = System.nanoTime()
repeat(runs) { rows = store.query<Event>(filter).size }
val ms = (System.nanoTime() - start) / 1e6 / runs
println(" %-42s %8.2f ms (%d rows)".format(label, ms, rows))
}
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/fs/FsDriverSelectionBenchmark.kt
+159
View File
@@ -0,0 +1,159 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.fs
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.utils.EventFactory
import kotlinx.coroutines.runBlocking
import java.nio.file.Files
import java.nio.file.Path
import kotlin.io.path.exists
import kotlin.test.Test
/**
* Guards [FsQueryPlanner]'s cost-based driver pick on the
* `authors + kinds + limit` shape — the most common CLI query (27
* assembler call sites; every `amy feed`-style author timeline over
* non-replaceable kinds).
*
* Under the pre-pick fixed order (tags → kinds → authors),
* `Filter(authors=[pk], kinds=[1], limit=n)` drove from `idx/kind/1/`
* (the biggest tree in any real store) and post-filtered the author:
* 149 ms at 30k events. The lockstep pick drives from the author tree
* and runs at ~4 ms. The benchmark times:
*
* - **planner (cost-based pick)**: the filter as the planner runs it —
* should sit near the floor, far below a kind-tree walk.
* - **author-driver emulation**: the author tree walked via an
* authors-only query with the kind check applied by the caller — the
* reference the pick is expected to match or beat.
* - **author-only floor**: `authors + limit` with no kind, the cheapest
* possible walk of the same tree.
*
* Size the seed with `-DfsBenchScale=N` (default 1 ≈ ~30k events; each
* event is a file + ~3 hardlinks, so seeding dominates wall time).
*/
class FsDriverSelectionBenchmark {
companion object {
val SCALE = System.getProperty("fsBenchScale")?.toInt() ?: 1
}
private val hex = "0123456789abcdef"
private fun mix(seed: Long): Long {
var z = seed + -0x61c8864680b583ebL
z = (z xor (z ushr 30)) * -0x40a7b892e31b1a47L
z = (z xor (z ushr 27)) * -0x6b2fb644ecceee15L
return z xor (z ushr 31)
}
private fun hex64(
salt: Long,
index: Int,
): String {
val out = CharArray(64)
for (w in 0 until 4) {
val v = mix(salt * 1_000_003 + index.toLong() * 4 + w)
for (b in 0 until 8) {
val byte = ((v ushr (b * 8)) and 0xFF).toInt()
out[(w * 8 + b) * 2] = hex[byte ushr 4]
out[(w * 8 + b) * 2 + 1] = hex[byte and 0xF]
}
}
return String(out)
}
private val sig = "0".repeat(128)
private var idSeq = 0
private fun ev(
pubkey: String,
createdAt: Long,
kind: Int,
): Event = EventFactory.create(hex64(7, idSeq++), pubkey, createdAt, kind, emptyArray(), "", sig)
@Test
fun compareDrivers() =
runBlocking {
val root: Path = Files.createTempDirectory("fs-driver-bench-")
val store = FsEventStore(root)
try {
val base = 1_700_000_000L
val span = 3_000_000L
val target = hex64(9, 0)
// Background: 300 authors × 100 kind-1 notes.
val bg = ArrayList<Event>(30_000 * SCALE + 300)
repeat(30_000 * SCALE) {
val author = hex64(1, it % 300)
bg.add(ev(author, base + (mix(it * 31L) and 0x7fffffff) % span, 1))
}
// Target author: 200 kind-1 notes + 50 kind-7 reactions.
repeat(200) { bg.add(ev(target, base + (mix(it * 131L) and 0x7fffffff) % span, 1)) }
repeat(50) { bg.add(ev(target, base + (mix(it * 61L) and 0x7fffffff) % span, 7)) }
val t0 = System.nanoTime()
store.transaction { bg.forEach { insert(it) } }
val insertMs = (System.nanoTime() - t0) / 1e6
println("─ FsDriverSelectionBenchmark: ${bg.size} events (scale=$SCALE), seed %.0f ms ─".format(insertMs))
// The planner's own pick — expected to choose the author
// tree over the ~30k-entry kind-1 tree.
val kindDriven = Filter(authors = listOf(target), kinds = listOf(1), limit = 50)
time(store, "planner (cost-based pick)") { store.query<Event>(kindDriven).size }
// Reference: author tree walked explicitly, kind checked
// by the caller — the pick should match or beat this.
time(store, "author-driver emulation") {
store
.query<Event>(Filter(authors = listOf(target), limit = 250))
.asSequence()
.filter { it.kind == 1 }
.take(50)
.count()
}
// Floor: author-only shape, the cheapest walk of the tree.
val authorOnly = Filter(authors = listOf(target), limit = 50)
time(store, "author-only floor") { store.query<Event>(authorOnly).size }
} finally {
store.close()
if (root.exists()) {
Files.walk(root).use { it.sorted(Comparator.reverseOrder()).forEach { p -> Files.deleteIfExists(p) } }
}
}
}
private inline fun time(
store: FsEventStore,
label: String,
run: () -> Int,
) {
repeat(3) { run() }
val runs = 10
var rows = 0
val start = System.nanoTime()
repeat(runs) { rows = run() }
val ms = (System.nanoTime() - start) / 1e6 / runs
println(" %-32s %8.2f ms (%d rows)".format(label, ms, rows))
}
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/ContentlessFtsMigrationTest.kt
+132
View File
@@ -0,0 +1,132 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import androidx.sqlite.SQLiteConnection
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync
import com.vitorpamplona.quartz.nip10Notes.TextNoteEvent
import com.vitorpamplona.quartz.utils.Secp256k1Instance
import kotlinx.coroutines.runBlocking
import java.nio.file.Files
import java.nio.file.Path
import kotlin.io.path.deleteIfExists
import kotlin.test.AfterTest
import kotlin.test.BeforeTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertTrue
/**
* Verifies the real v4 → v5 FTS upgrade path: a database written with the old
* `fts5(event_header_row_id, content)` index (auto-assigned rowid, content
* stored) must, on next open, drop that table and rebuild a contentless index
* keyed by `event_headers.row_id`, with search intact.
*
* The v4 state is fabricated by opening a fresh v5 store, then rewriting its
* `event_fts` to the old schema (with deliberately wrong content, to prove the
* rebuild wipes it) and stamping `user_version = 4`.
*/
class ContentlessFtsMigrationTest {
private val signer = NostrSignerSync()
private lateinit var dbFile: Path
private fun path() = dbFile.toAbsolutePath().toString()
@BeforeTest
fun setup() {
Secp256k1Instance
dbFile = Files.createTempFile("contentless-fts-migration-", ".db")
Files.deleteIfExists(dbFile)
}
@AfterTest
fun tearDown() {
listOf("", "-wal", "-shm", "-journal").forEach { Path.of(dbFile.toString() + it).deleteIfExists() }
}
@Test
fun upgradesOldFtsSchemaAndRebuildsSearch() =
runBlocking {
val alpha = signer.sign(TextNoteEvent.build("uniqalpha searchable body", createdAt = 1_700_000_000L))
val beta = signer.sign(TextNoteEvent.build("uniqbeta searchable body", createdAt = 1_700_000_100L))
// 1. Fresh v5 store, seed events, close.
EventStore(dbName = path(), relay = null).also {
it.insert(alpha)
it.insert(beta)
it.close()
}
// 2. Rewrite event_fts to the pre-v5 schema with WRONG content and
// downgrade user_version to 4 — the state a v4 database is in.
BundledSQLiteDriver().open(path()).use { db ->
db.exec("DROP TRIGGER IF EXISTS fts_foreign_key")
db.exec("DROP TABLE IF EXISTS event_fts")
db.exec("CREATE VIRTUAL TABLE event_fts USING fts5(event_header_row_id, content)")
db.exec(
"""
CREATE TRIGGER fts_foreign_key AFTER DELETE ON event_headers FOR EACH ROW
BEGIN DELETE FROM event_fts WHERE old.row_id = event_fts.event_header_row_id; END
""".trimIndent(),
)
// Stale/garbage rows: a real v4 index would hold correct data,
// but seeding garbage proves the migration rebuilds from
// event_headers rather than trusting the old table.
db.exec("INSERT INTO event_fts(event_header_row_id, content) VALUES (1, 'uniqstale garbage')")
db.exec("PRAGMA user_version = 4")
}
// 3. Reopen with current code → onUpgrade(4→5) → migrateV4ToContentless.
val store = EventStore(dbName = path(), relay = null)
try {
// Rebuilt from event_headers: real content is searchable...
assertEquals(alpha.id, store.query<Event>(Filter(search = "uniqalpha")).single().id)
assertEquals(beta.id, store.query<Event>(Filter(search = "uniqbeta")).single().id)
// ...and the old garbage is gone.
assertTrue(store.query<Event>(Filter(search = "uniqstale")).isEmpty())
// The new rowid IS event_headers.row_id: the join returns the
// right event for each FTS rowid.
store.store.pool.useReader { db ->
db
.prepare(
"SELECT h.id FROM event_fts f JOIN event_headers h ON h.row_id = f.rowid ORDER BY f.rowid",
).use { stmt ->
val ids = ArrayList<String>()
while (stmt.step()) ids.add(stmt.getText(0))
assertEquals(listOf(alpha.id, beta.id), ids, "FTS rowid must map to event_headers.row_id")
}
}
// The rebuilt delete trigger still cleans up FTS on delete.
store.store.delete(beta.id)
assertTrue(store.query<Event>(Filter(search = "uniqbeta")).isEmpty())
assertEquals(alpha.id, store.query<Event>(Filter(search = "uniqalpha")).single().id)
} finally {
store.close()
}
}
private fun SQLiteConnection.exec(sql: String) = prepare(sql).use { it.step() }
}
quartz/src/jvmTest/kotlin/com/vitorpamplona/quartz/nip01Core/store/sqlite/Fts5CapabilityProbe.kt
+123
View File
@@ -0,0 +1,123 @@
/*
* Copyright (c) 2025 Vitor Pamplona
*
* Permission is hereby granted, free of charge, to any person obtaining a copy of
* this software and associated documentation files (the "Software"), to deal in
* the Software without restriction, including without limitation the rights to use,
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
* Software, and to permit persons to whom the Software is furnished to do so,
* subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in all
* copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
package com.vitorpamplona.quartz.nip01Core.store.sqlite
import androidx.sqlite.driver.bundled.BundledSQLiteDriver
import kotlin.test.Test
import kotlin.test.assertEquals
/**
* Pins the FTS5 features the [FullTextSearchModule] contentless index depends
* on to the bundled SQLite. All three shipped between SQLite 3.43 and 3.44
* (2023); if the bundled driver is ever downgraded below that, this fails
* loudly instead of the store silently breaking search on delete.
*
* - `content=''` **contentless** table with `contentless_delete=1`: lets the
* index drop the duplicated content column yet still delete rows by rowid
* (the `fts_foreign_key` trigger needs it).
* - explicit `rowid` on insert (= `event_headers.row_id`): the join key and
* the O(log n) delete key.
* - bm25 `rank` on a contentless table, through a join: NIP-50 relevance order.
* - `'merge'` / `'optimize'` maintenance commands: segment compaction.
*/
class Fts5CapabilityProbe {
@Test
fun contentlessDeleteAndRowidOrderingAreSupported() {
val db = BundledSQLiteDriver().open(":memory:")
try {
db.execSQL("CREATE VIRTUAL TABLE cl USING fts5(content, content='', contentless_delete=1)")
db.execSQL("INSERT INTO cl(rowid, content) VALUES (100, 'hello world')")
db.execSQL("INSERT INTO cl(rowid, content) VALUES (50, 'hello there')")
db.execSQL("INSERT INTO cl(rowid, content) VALUES (200, 'hello again')")
db.execSQL("DELETE FROM cl WHERE rowid = 50")
// Deleting an absent rowid must be a harmless no-op: the store's
// fts_foreign_key trigger fires on EVERY event_headers delete, but
// only searchable events ever got an FTS row.
db.execSQL("DELETE FROM cl WHERE rowid = 999999")
val order = ArrayList<Long>()
db.prepare("SELECT rowid FROM cl WHERE cl MATCH 'hello' ORDER BY rowid DESC LIMIT 5").use {
while (it.step()) order.add(it.getLong(0))
}
// Deleted 50 is gone; the rest come back newest-rowid first.
assertEquals(listOf(200L, 100L), order)
} finally {
db.close()
}
}
@Test
fun bm25RankWorksOnContentlessTableInAJoin() {
// NIP-50 orders by relevance, not created_at. Verify FTS5 bm25 `rank`
// works on a contentless table and is reachable through the same
// join-back-to-base-table shape the store's search query uses.
val db = BundledSQLiteDriver().open(":memory:")
try {
db.execSQL("CREATE TABLE headers (row_id INTEGER PRIMARY KEY, created_at INTEGER, tag TEXT)")
db.execSQL("CREATE VIRTUAL TABLE fts USING fts5(content, content='', contentless_delete=1)")
// row 10: term appears 3× in a short doc (most relevant).
// row 20: term once in a long doc (least relevant) but NEWER.
db.execSQL("INSERT INTO headers VALUES (10, 100, 'A')")
db.execSQL("INSERT INTO fts(rowid, content) VALUES (10, 'needle needle needle')")
db.execSQL("INSERT INTO headers VALUES (20, 999, 'B')")
db.execSQL("INSERT INTO fts(rowid, content) VALUES (20, 'needle alpha beta gamma delta epsilon zeta eta')")
// created_at DESC would return B (999) first; relevance returns A.
val byRank = ArrayList<String>()
db
.prepare(
"""
SELECT headers.tag FROM headers
INNER JOIN fts ON headers.row_id = fts.rowid
WHERE fts MATCH 'needle'
ORDER BY fts.rank
LIMIT 10
""".trimIndent(),
).use { while (it.step()) byRank.add(it.getText(0)) }
assertEquals(listOf("A", "B"), byRank, "bm25 rank must put the more relevant (shorter, higher-tf) doc first")
} finally {
db.close()
}
}
@Test
fun segmentMergeAndOptimizeAreSupported() {
val db = BundledSQLiteDriver().open(":memory:")
try {
db.execSQL("CREATE VIRTUAL TABLE m USING fts5(content)")
db.execSQL("INSERT INTO m(rowid, content) VALUES (1, 'a b c')")
db.execSQL("INSERT INTO m(rowid, content) VALUES (2, 'd e f')")
// Bounded incremental merge, then a full optimize — both must parse
// and run without error on the bundled build.
db.execSQL("INSERT INTO m(m, rank) VALUES ('merge', -16)")
db.execSQL("INSERT INTO m(m) VALUES ('optimize')")
val hits = ArrayList<Long>()
db.prepare("SELECT rowid FROM m WHERE m MATCH 'e' ORDER BY rowid").use {
while (it.step()) hits.add(it.getLong(0))
}
assertEquals(listOf(2L), hits)
} finally {
db.close()
}
}
}
relayBench/src/main/kotlin/com/vitorpamplona/relaybench/Scenarios.kt
+47 -3
View File
@@ -70,11 +70,11 @@ object Scenarios {
}
val topAuthors = notesByAuthor.entries.sortedWith(compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key }).map { it.key }
val hottestThread =
val hotNotes =
eTagRefs.entries
.sortedWith(compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key })
.firstOrNull()
?.key
.map { it.key }
val hottestThread = hotNotes.firstOrNull()
val mostMentioned =
pTagRefs.entries
.sortedWith(compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key })
@@ -86,6 +86,23 @@ object Scenarios {
.firstOrNull()
?.key
// The author that most often tags the most-mentioned pubkey — a
// conversation pair for the tag ∩ author (DM-room) query shape.
val conversationPeer =
mostMentioned?.let { me ->
val byAuthor = HashMap<String, Int>()
for (e in events) {
if (e.kind != 1 || e.pubKey == me) continue
if (e.tags.any { it.size >= 2 && it[0] == "p" && it[1] == me }) {
byAuthor.merge(e.pubKey, 1, Int::plus)
}
}
byAuthor.entries
.sortedWith(compareByDescending<Map.Entry<String, Int>> { it.value }.thenBy { it.key })
.firstOrNull()
?.key
}
// Evenly spread sample of note ids — a "fetch these 100 events" batch.
val idSample =
if (noteIds.size <= 100) {
@@ -159,6 +176,33 @@ object Scenarios {
),
)
}
if (mostMentioned != null && conversationPeer != null) {
// The tag ∩ author ∩ kind shape (65 client assembler call
// sites: NIP-04 DM rooms, reports-by-follows, follows-scoped
// community feeds). Modeled on kind 1 because public corpora
// carry no DMs; the index path exercised is identical.
add(
Scenario(
"conversation",
"notes by one author tagging the most-mentioned pubkey (DM-room shape)",
Filter(kinds = listOf(1), authors = listOf(conversationPeer), tags = mapOf("p" to listOf(mostMentioned)), limit = 500),
),
)
}
if (hotNotes.size > 1) {
// Large-IN tag watcher: per-value streams come sorted off the
// tag index but their union does not, exposing whether the
// store collects+sorts or merges. 150 values stays inside
// strfry's default 200-element filter cap.
val watched = hotNotes.take(150)
add(
Scenario(
"reactions-watch",
"reactions on the ${watched.size} hottest notes (visible-feed reaction watcher)",
Filter(kinds = listOf(7), tags = mapOf("e" to watched), limit = 500),
),
)
}
topHashtag?.let {
add(
Scenario(