mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-10-05 19:28:25 +00:00
test(search): pin the local search engine against a live relay's answers
Adds a recorded corpus of 64 real events from search-staging.brainstorm.world — a vespa-relay, the same software this token language was ported from — and 18 tests over it: 8 on the engine in :commons, 10 on `LocalCache.filter` in :amethyst. Recorded, not fetched. `tools/search-parity/fetch_fixtures.py` drives `amy fetch` against the relay by hand; the tests read the committed fixture, so `./gradlew test` stays offline and the pre-push hook does not depend on somebody else's uptime. The fixture stores each case's filter FIELDS rather than a prebuilt filter, so the test rebuilds the Filter in view of the reader and a wrong rebuild cannot quietly make the assertions vacuous. What the relay can and cannot referee turned out to be the whole design, and it was measured rather than assumed: - **NIP-01 it can.** Given kinds, #t, since and until there is exactly one right answer, and across all 64 events our matcher agrees with the relay about every one it chose to return — 0 violations. That is now a hard assertion, with a converse test so it cannot pass by matching everything. - **NIP-50 it cannot.** This relay retrieves topically: asked for `bitcoin` it returns a block-height summary that never says "bitcoin". Eight of 64 events carry no literal occurrence of the term that fetched them. Asserting our substring matcher reproduces that would encode someone else's semantic expansion as a requirement on a lexical one — a test that fails on correct code. So text results are deliberately not compared, and the divergence is pinned as a range instead: zero would mean the relay turned lexical and the comparison should be rewritten, a quarter would mean we regressed. The fixture uses the `include:spam` lens, which waives the web-of-trust gate. Also measured: it makes the corpus reproducible, where `observer:<pubkey>` ties every answer to one account's moving trust graph — but it does not make retrieval lexical, and in fact widens the divergence from 4 events to 8 by letting more topical matches through. The LocalCache tests cover what the relay knows nothing about and where the bugs actually were: the regular/addressable split, the viewer-policy predicate composing with rather than replacing the filter, and the result cap keeping the newest — the ordering whose absence let `take(limit)` run before the sort. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017yKjw2WqwZpSzsqcYZMnkV
This commit is contained in:
Claude
+171
@@ -0,0 +1,171 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.model
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.jsonArray
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import org.junit.Assert.assertEquals
|
||||
import org.junit.Assert.assertTrue
|
||||
import org.junit.BeforeClass
|
||||
import org.junit.Test
|
||||
import java.io.File
|
||||
|
||||
/**
|
||||
* `LocalCache.filter` over a real corpus — 60-odd notes recorded from a live vespa-relay by
|
||||
* `tools/search-parity/fetch_fixtures.py`, the same fixture the engine-level parity test in
|
||||
* `:commons` reads.
|
||||
*
|
||||
* The engine test proves our NIP-01 matcher agrees with the relay about individual events. This
|
||||
* one covers what sits above the matcher and the relay knows nothing about: the split between
|
||||
* regular and addressable notes, the viewer-policy predicate, the result cap, and the ordering the
|
||||
* cap depends on — where a bug had `take(limit)` running before the sort and silently dropping the
|
||||
* newest matches.
|
||||
*
|
||||
* `LocalCache` is a process-wide object and JUnit's method order is hash-based, so the corpus is
|
||||
* loaded once and every assertion here is read-only.
|
||||
*/
|
||||
class LocalCacheSearchParityTest {
|
||||
companion object {
|
||||
private lateinit var corpus: List<Event>
|
||||
|
||||
@BeforeClass
|
||||
@JvmStatic
|
||||
fun loadCorpus() {
|
||||
val file =
|
||||
listOf(File("../tools/search-parity/fixture.json"), File("tools/search-parity/fixture.json"))
|
||||
.firstOrNull { it.isFile }
|
||||
?: error("tools/search-parity/fixture.json is missing; run tools/search-parity/fetch_fixtures.py")
|
||||
|
||||
val root = Json { ignoreUnknownKeys = true }.parseToJsonElement(file.readText()).jsonObject
|
||||
corpus =
|
||||
root["cases"]!!
|
||||
.jsonArray
|
||||
.flatMap { it.jsonObject["events"]!!.jsonArray }
|
||||
.map { Event.fromJson(it.toString()) }
|
||||
.distinctBy { it.id }
|
||||
|
||||
// LocalCache.consume refuses the main thread; a plain JVM test has no Looper, so the
|
||||
// check passes and the events land synchronously.
|
||||
corpus.forEach { LocalCache.justConsumeMyOwnEvent(it) }
|
||||
}
|
||||
}
|
||||
|
||||
private fun idsFor(filter: Filter) = LocalCache.filter(filter).mapNotNull { it.event?.id }
|
||||
|
||||
@Test
|
||||
fun theCorpusLoadedAndIsWorthAssertingOver() {
|
||||
assertTrue("fixture produced no events", corpus.size >= 40)
|
||||
// Both halves of the filter path — regular notes and addressables — must be represented,
|
||||
// or the split below is only tested on one branch.
|
||||
assertTrue("no regular notes in the corpus", corpus.any { it.kind == 1 })
|
||||
assertTrue("no addressable notes in the corpus", corpus.any { it.kind == 30023 })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aKindFilterReturnsExactlyTheEventsOfThatKind() {
|
||||
val expected = corpus.filter { it.kind == 1 }.map { it.id }.toSet()
|
||||
val found = idsFor(Filter(kinds = listOf(1))).toSet()
|
||||
assertTrue("kind:1 missed ${(expected - found).size} of ${expected.size}", expected.all { it in found })
|
||||
assertTrue("kind:1 returned events of another kind", found.all { id -> corpus.first { it.id == id }.kind == 1 })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun anAddressableKindComesBackThroughTheAddressablePath() {
|
||||
val expected = corpus.filter { it.kind == 30023 }.map { it.id }.toSet()
|
||||
if (expected.isEmpty()) return
|
||||
val found = idsFor(Filter(kinds = listOf(30023))).toSet()
|
||||
assertTrue("long-form articles did not come back: ${expected - found}", expected.all { it in found })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun anAuthorFilterNarrowsToThatAuthor() {
|
||||
val author = corpus.groupBy { it.pubKey }.maxByOrNull { it.value.size }!!.key
|
||||
val found = LocalCache.filter(Filter(authors = listOf(author)))
|
||||
assertTrue("no results for the corpus' most prolific author", found.isNotEmpty())
|
||||
assertTrue("an author filter returned somebody else", found.all { it.event?.pubKey == author })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aTagFilterNarrowsToCarriersOfThatTag() {
|
||||
val tagged = corpus.filter { e -> e.tags.any { it.size > 1 && it[0] == "t" } }
|
||||
if (tagged.isEmpty()) return
|
||||
val value = tagged.first().tags.first { it.size > 1 && it[0] == "t" }[1]
|
||||
val found = LocalCache.filter(Filter(tags = mapOf("t" to listOf(value))))
|
||||
assertTrue(
|
||||
"a #t filter returned an event without the tag",
|
||||
found.all { note ->
|
||||
note.event?.tags?.any { it.size > 1 && it[0] == "t" && it[1] == value } == true
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aWindowIsInclusiveAtBothEnds() {
|
||||
val sorted = corpus.map { it.createdAt }.sorted()
|
||||
val since = sorted[sorted.size / 4]
|
||||
val until = sorted[sorted.size * 3 / 4]
|
||||
val found = LocalCache.filter(Filter(since = since, until = until))
|
||||
assertTrue("a window returned something outside it", found.all { (it.event?.createdAt ?: 0) in since..until })
|
||||
// Inclusive: the events sitting exactly on each bound are in.
|
||||
val onBounds = corpus.filter { it.createdAt == since || it.createdAt == until }.map { it.id }
|
||||
val foundIds = found.mapNotNull { it.event?.id }.toSet()
|
||||
assertTrue("a bound event was excluded", onBounds.all { it in foundIds })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theResultCapKeepsTheNewestNotWhicheverTheWalkReachedFirst() {
|
||||
// The bug this pins: `take(limit)` ran before the sort, so the cap dropped whatever the
|
||||
// hash walk happened to reach last — the newest as often as not.
|
||||
val all = LocalCache.filter(Filter(kinds = listOf(1)))
|
||||
if (all.size < 3) return
|
||||
val cap = all.size / 2
|
||||
val capped = LocalCache.filter(Filter(kinds = listOf(1), limit = cap))
|
||||
|
||||
assertEquals("the cap was not honoured", cap, capped.size)
|
||||
val newest = all.take(cap).mapNotNull { it.event?.id }
|
||||
assertEquals("the cap did not keep the newest results", newest, capped.mapNotNull { it.event?.id })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun thePredicateComposesWithTheFilterRatherThanReplacingIt() {
|
||||
val author = corpus.groupBy { it.pubKey }.maxByOrNull { it.value.size }!!.key
|
||||
val byKind = LocalCache.filter(Filter(kinds = listOf(1)))
|
||||
val byBoth = LocalCache.filter(Filter(kinds = listOf(1))) { it.event?.pubKey == author }
|
||||
|
||||
assertTrue("the predicate widened the result", byBoth.size <= byKind.size)
|
||||
assertTrue("the predicate was not applied", byBoth.all { it.event?.pubKey == author })
|
||||
assertTrue("the filter was dropped when a predicate was given", byBoth.all { it.event?.kind == 1 })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aPredicateThatRefusesEverythingReturnsNothing() {
|
||||
assertTrue(LocalCache.filter(Filter(kinds = listOf(1))) { false }.isEmpty())
|
||||
}
|
||||
|
||||
@Test
|
||||
fun resultsComeBackNewestFirst() {
|
||||
val found = LocalCache.filter(Filter(kinds = listOf(1))).mapNotNull { it.event?.createdAt }
|
||||
assertEquals("results are not sorted newest-first", found.sortedDescending(), found)
|
||||
}
|
||||
}
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.search
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.core.Event
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.Filter
|
||||
import kotlinx.serialization.json.Json
|
||||
import kotlinx.serialization.json.JsonObject
|
||||
import kotlinx.serialization.json.jsonArray
|
||||
import kotlinx.serialization.json.jsonObject
|
||||
import kotlinx.serialization.json.jsonPrimitive
|
||||
import java.io.File
|
||||
|
||||
/**
|
||||
* Real answers from a live vespa-relay (`search-staging.brainstorm.world`), recorded once by
|
||||
* `tools/search-parity/fetch_fixtures.py` and read from disk here.
|
||||
*
|
||||
* Recorded rather than fetched so `./gradlew test` stays offline and deterministic — a suite that
|
||||
* reaches the network fails on a train, and this one runs in the pre-push hook.
|
||||
*/
|
||||
data class ParityCase(
|
||||
val name: String,
|
||||
val terms: String,
|
||||
val search: String,
|
||||
val filter: Filter,
|
||||
val events: List<Event>,
|
||||
)
|
||||
|
||||
object SearchParityFixture {
|
||||
private val json = Json { ignoreUnknownKeys = true }
|
||||
|
||||
val cases: List<ParityCase> by lazy { load() }
|
||||
|
||||
/**
|
||||
* Read from the repository rather than a module's resources: the same corpus backs the
|
||||
* engine-level tests here and the `LocalCache` tests in `:amethyst`, and copying 60-odd real
|
||||
* events into two modules would guarantee they drift apart.
|
||||
*/
|
||||
fun load(): List<ParityCase> {
|
||||
val file = resolveFixture()
|
||||
val root = json.parseToJsonElement(file.readText()).jsonObject
|
||||
return root["cases"]!!.jsonArray.map { it.jsonObject.toCase() }
|
||||
}
|
||||
|
||||
/** Gradle runs a module's tests from its own directory, so try there and then the repo root. */
|
||||
private fun resolveFixture(): File =
|
||||
listOf(File("../tools/search-parity/fixture.json"), File("tools/search-parity/fixture.json"))
|
||||
.firstOrNull { it.isFile }
|
||||
?: error("tools/search-parity/fixture.json is missing; regenerate with tools/search-parity/fetch_fixtures.py")
|
||||
|
||||
private fun JsonObject.toCase(): ParityCase {
|
||||
val flags = this["flags"]!!.jsonArray.map { it.jsonPrimitive.content }
|
||||
return ParityCase(
|
||||
name = this["name"]!!.jsonPrimitive.content,
|
||||
terms = this["terms"]!!.jsonPrimitive.content,
|
||||
search = this["search"]!!.jsonPrimitive.content,
|
||||
filter = flags.toFilter(),
|
||||
// Through the real parser, so the corpus exercises the same code the app runs.
|
||||
events = this["events"]!!.jsonArray.map { Event.fromJson(it.toString()) },
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* The amy flags a case was fetched with, back as the NIP-01 filter they stood for. The fixture
|
||||
* stores the fields rather than a pre-built filter so the reconstruction is visible here and a
|
||||
* silently-wrong rebuild cannot make the parity assertions vacuous.
|
||||
*/
|
||||
private fun List<String>.toFilter(): Filter {
|
||||
var kinds: List<Int>? = null
|
||||
var tags: MutableMap<String, List<String>>? = null
|
||||
var since: Long? = null
|
||||
var until: Long? = null
|
||||
var i = 0
|
||||
while (i < size) {
|
||||
val value = getOrNull(i + 1)
|
||||
when (this[i]) {
|
||||
"--kind" -> kinds = value?.split(",")?.map(String::toInt)
|
||||
"--since" -> since = value?.toLong()
|
||||
"--until" -> until = value?.toLong()
|
||||
"--tag" -> {
|
||||
val (name, v) = value!!.split("=", limit = 2)
|
||||
tags = (tags ?: mutableMapOf()).apply { put(name, listOf(v)) }
|
||||
}
|
||||
}
|
||||
i += 2
|
||||
}
|
||||
return Filter(kinds = kinds, tags = tags, since = since, until = until)
|
||||
}
|
||||
}
|
||||
+189
@@ -0,0 +1,189 @@
|
||||
/*
|
||||
* Copyright (c) 2025 Vitor Pamplona
|
||||
*
|
||||
* Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
* this software and associated documentation files (the "Software"), to deal in
|
||||
* the Software without restriction, including without limitation the rights to use,
|
||||
* copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
|
||||
* Software, and to permit persons to whom the Software is furnished to do so,
|
||||
* subject to the following conditions:
|
||||
*
|
||||
* The above copyright notice and this permission notice shall be included in all
|
||||
* copies or substantial portions of the Software.
|
||||
*
|
||||
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
* FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
||||
* COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
|
||||
* AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||
* WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
*/
|
||||
package com.vitorpamplona.amethyst.commons.search
|
||||
|
||||
import com.vitorpamplona.quartz.nip01Core.relay.filters.FilterMatcher
|
||||
import com.vitorpamplona.quartz.nip50Search.EventSearchMatcher
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* The local search engine, checked against a real relay's answers over a real corpus.
|
||||
*
|
||||
* ## What this can and cannot prove
|
||||
*
|
||||
* The oracle is `search-staging.brainstorm.world`, a vespa-relay — the same software Amethyst's
|
||||
* token language was ported from. It is a **valid oracle for NIP-01 filtering** and **not one for
|
||||
* NIP-50 text matching**, and the difference is the whole design of this file.
|
||||
*
|
||||
* NIP-01 is a closed specification: given `kinds`, `#t`, `since` and `until`, either an event
|
||||
* satisfies the filter or it does not, and there is exactly one right answer. Every event the
|
||||
* relay chose to return for a filter must therefore satisfy that filter under our matcher too.
|
||||
* Across the recorded corpus that holds for 64 of 64 events, and [everyEventTheRelayReturnedSatisfiesTheFilterItAnsweredFor]
|
||||
* pins it.
|
||||
*
|
||||
* NIP-50 is not closed. This relay retrieves topically, not lexically: asked for `bitcoin` it
|
||||
* returns a block-height summary that never says "bitcoin", and asked for `nostr` it returns "made
|
||||
* my display name refer to my npub's last characters". Eight of the 64 recorded events — 12% —
|
||||
* contain no literal occurrence of the term that fetched them.
|
||||
*
|
||||
* The fixture is recorded with the `include:spam` lens, which waives the web-of-trust gate. That
|
||||
* was measured, not assumed: waiving it makes the corpus reproducible (the alternative,
|
||||
* `observer:<pubkey>`, pins every answer to one account's trust graph) but it does not make
|
||||
* retrieval lexical — the divergence actually widens, from 4 events to 8, because dropping the
|
||||
* gate lets more topical matches through. Asserting that
|
||||
* our substring matcher reproduces those results would encode someone else's semantic expansion as
|
||||
* a requirement on a lexical matcher — a test that fails on correct code. So this file deliberately
|
||||
* makes **no** claim that our text results equal the relay's, and
|
||||
* [theRelayRetrievesSemanticallyWhichIsWhyTextResultsAreNotCompared] documents the divergence as a
|
||||
* fact about the corpus rather than a failure.
|
||||
*
|
||||
* What the corpus is still worth beyond that: 64 events of real, adversarial content — newlines,
|
||||
* currency symbols, URLs, multi-byte text, empty tag arrays — to run the rest of the engine over.
|
||||
*/
|
||||
class SearchRelayParityTest {
|
||||
private val cases = SearchParityFixture.cases
|
||||
|
||||
@Test
|
||||
fun theFixtureIsPresentAndNotSilentlyEmpty() {
|
||||
// A relay that was down would otherwise turn every assertion below into a no-op.
|
||||
assertTrue(cases.isNotEmpty(), "no cases in the fixture")
|
||||
assertTrue(cases.all { it.events.isNotEmpty() }, "a case recorded zero events: ${cases.filter { it.events.isEmpty() }.map { it.name }}")
|
||||
assertTrue(cases.sumOf { it.events.size } >= 40, "the corpus is too small to be worth asserting over")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun everyEventTheRelayReturnedSatisfiesTheFilterItAnsweredFor() {
|
||||
// The core parity claim: our NIP-01 matcher agrees with a production relay about its own
|
||||
// answers, over data neither of us invented.
|
||||
cases.forEach { case ->
|
||||
case.events.forEach { event ->
|
||||
assertTrue(
|
||||
FilterMatcher.match(
|
||||
event = event,
|
||||
kinds = case.filter.kinds,
|
||||
tags = case.filter.tags,
|
||||
since = case.filter.since,
|
||||
until = case.filter.until,
|
||||
),
|
||||
"case ${case.name}: the relay returned ${event.id.take(8)} (kind ${event.kind}, " +
|
||||
"created_at ${event.createdAt}) but our matcher rejects it for " +
|
||||
"kinds=${case.filter.kinds} tags=${case.filter.tags} " +
|
||||
"since=${case.filter.since} until=${case.filter.until}",
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun aFilterThatExcludesAnEventRejectsIt() {
|
||||
// The converse, so the assertion above cannot pass by matching everything.
|
||||
val corpus = cases.flatMap { it.events }.distinctBy { it.id }
|
||||
val impossible = corpus.count { FilterMatcher.match(it, kinds = listOf(31337)) }
|
||||
assertEquals(0, impossible, "a kind filter nothing in the corpus carries still matched")
|
||||
|
||||
val ancient = corpus.count { FilterMatcher.match(it, until = 1L) }
|
||||
assertEquals(0, ancient, "an until of 1970 still matched real events")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theWindowedCasesReallyExerciseTheirBounds() {
|
||||
// Guards against a window so wide it would pass whatever the matcher did.
|
||||
val since = cases.first { it.name == "window_since" }
|
||||
assertTrue(since.events.all { it.createdAt >= since.filter.since!! })
|
||||
val until = cases.first { it.name == "window_until" }
|
||||
assertTrue(until.events.all { it.createdAt <= until.filter.until!! })
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theHashtagCaseCarriesTheTagOnEveryEvent() {
|
||||
val case = cases.first { it.name == "tag_hashtag" }
|
||||
assertTrue(case.filter.tags?.containsKey("t") == true, "the fixture lost its tag filter")
|
||||
case.events.forEach { event ->
|
||||
assertTrue(
|
||||
event.tags.any { it.size > 1 && it[0] == "t" && it[1] == "bitcoin" },
|
||||
"${event.id.take(8)} came back for #t=bitcoin without carrying it",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun ourOwnQueryLanguageBuildsFiltersThatAcceptTheRelaysAnswers() {
|
||||
// End to end over real data: text through QueryParser and SearchFilterBuilder must produce
|
||||
// filters that still accept what the relay returned for the same words and kinds.
|
||||
val case = cases.first { it.name == "tag_hashtag" }
|
||||
val query = QueryParser.parse("#bitcoin")
|
||||
val filters = SearchFilterBuilder.build(query, kinds = listOf(1), limit = 100)
|
||||
assertTrue(filters.isNotEmpty())
|
||||
|
||||
val tagArm = filters.first { it.tags?.containsKey("t") == true }
|
||||
case.events.forEach { event ->
|
||||
assertTrue(
|
||||
FilterMatcher.match(event, kinds = tagArm.kinds, tags = tagArm.tags),
|
||||
"our #t arm rejects ${event.id.take(8)}, which the relay returned for the same tag",
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theSearchMatcherSurvivesRealContent() {
|
||||
// Real notes carry newlines, `$`, URLs and multi-byte text. The matcher must answer
|
||||
// without throwing and must agree with a plain substring test on the content it holds.
|
||||
val corpus = cases.flatMap { it.events }.distinctBy { it.id }
|
||||
listOf("bitcoin", "BITCOIN", "sat/vByte", "$", "\"open source\"", "").forEach { term ->
|
||||
val matcher = EventSearchMatcher(term)
|
||||
corpus.forEach { event ->
|
||||
val matched = matcher.match(event)
|
||||
if (term.isBlank()) {
|
||||
assertTrue(matched, "an empty search must constrain nothing")
|
||||
} else if (!term.startsWith("\"") && event.content.contains(term, true)) {
|
||||
assertTrue(matched, "term $term is literally in ${event.id.take(8)} but did not match")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
fun theRelayRetrievesSemanticallyWhichIsWhyTextResultsAreNotCompared() {
|
||||
// Not a failure — a recorded fact about the oracle, kept executable so that if the relay
|
||||
// ever becomes purely lexical the divergence drops to zero and this test tells us.
|
||||
val lexicalMisses =
|
||||
cases.sumOf { case ->
|
||||
val terms =
|
||||
case.terms
|
||||
.replace("\"", " ")
|
||||
.split(" ")
|
||||
.filter { it.isNotBlank() }
|
||||
case.events.count { event ->
|
||||
val matcher = EventSearchMatcher(terms.joinToString(" "))
|
||||
!matcher.match(event)
|
||||
}
|
||||
}
|
||||
val total = cases.sumOf { it.events.size }
|
||||
assertTrue(
|
||||
lexicalMisses in 1..(total / 4),
|
||||
"expected a small semantic divergence from the relay, got $lexicalMisses of $total — " +
|
||||
"0 would mean the relay turned lexical (drop this test and compare results directly); " +
|
||||
"a quarter or more means our matcher regressed",
|
||||
)
|
||||
}
|
||||
}
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Record real answers from a live vespa-relay so the local search engine can be tested
|
||||
against them offline.
|
||||
|
||||
Run by hand when the fixture needs refreshing; the test that consumes its output never
|
||||
touches the network, so `./gradlew test` stays hermetic and offline.
|
||||
|
||||
./gradlew :cli:installDist
|
||||
tools/search-parity/fetch_fixtures.py > commons/src/jvmTest/resources/search-parity-fixture.json
|
||||
|
||||
Each case pins the filter FIELDS (kinds/tags/window) as flags rather than folding them into
|
||||
the search string, so the test can rebuild the identical NIP-01 Filter and assert our matcher
|
||||
agrees with the relay about every event it chose to return.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
RELAY = os.environ.get("RELAY", "wss://search-staging.brainstorm.world")
|
||||
AMY = os.environ.get("AMY", os.path.join(os.getcwd(), "cli/build/install/amy/bin/amy"))
|
||||
# The relay gates and ranks results through the searcher's web of trust, and answers an
|
||||
# anonymous query with nothing at all. `include:spam` waives that gate, which is what makes
|
||||
# this fixture reproducible: the alternative, `observer:<pubkey>`, ties every recorded answer
|
||||
# to one account's trust graph and re-records differently as that graph moves.
|
||||
#
|
||||
# It does NOT make the relay lexical. Measured on this corpus, results still include events
|
||||
# carrying no literal occurrence of the term — asked for `nostr` it returns "made my display
|
||||
# name refer to my npub's last characters". Retrieval is semantic; see the parity test for why
|
||||
# that means text results are not compared.
|
||||
LENS = os.environ.get("LENS", "include:spam")
|
||||
LIMIT = os.environ.get("LIMIT", "8")
|
||||
TIMEOUT = os.environ.get("TIMEOUT", "30")
|
||||
|
||||
# name, search terms, filter flags
|
||||
CASES = [
|
||||
("text_single", "bitcoin", ["--kind", "1"]),
|
||||
("text_two_words", "bitcoin lightning", ["--kind", "1"]),
|
||||
("text_longform", "nostr", ["--kind", "30023"]),
|
||||
("tag_hashtag", "bitcoin", ["--kind", "1", "--tag", "t=bitcoin"]),
|
||||
("window_since", "nostr", ["--kind", "1", "--since", "1700000000"]),
|
||||
("window_until", "nostr", ["--kind", "1", "--until", "1800000000"]),
|
||||
("kinds_union", "nostr", ["--kind", "1,30023"]),
|
||||
("phrase_quoted", '"open source"', ["--kind", "1"]),
|
||||
]
|
||||
|
||||
|
||||
def fetch(terms, flags):
|
||||
search = f"{terms} {LENS}"
|
||||
cmd = [AMY, "fetch", *flags, "--search", search, "--limit", LIMIT,
|
||||
"--relay", RELAY, "--timeout", TIMEOUT, "--json"]
|
||||
try:
|
||||
out = subprocess.run(cmd, capture_output=True, text=True, timeout=int(TIMEOUT) + 30)
|
||||
return search, json.loads(out.stdout).get("events", [])
|
||||
except Exception as e: # a relay that is down must not silently produce an empty fixture
|
||||
print(f"FAILED {' '.join(shlex.quote(c) for c in cmd)}: {e}", file=sys.stderr)
|
||||
raise
|
||||
|
||||
|
||||
def main():
|
||||
if not os.access(AMY, os.X_OK):
|
||||
sys.exit("amy not built: ./gradlew :cli:installDist")
|
||||
cases = []
|
||||
for name, terms, flags in CASES:
|
||||
search, events = fetch(terms, flags)
|
||||
print(f"{name:16s} {len(events):3d} events", file=sys.stderr)
|
||||
cases.append({"name": name, "terms": terms, "flags": flags,
|
||||
"search": search, "events": events})
|
||||
json.dump({"relay": RELAY, "lens": LENS, "cases": cases},
|
||||
sys.stdout, indent=1, sort_keys=True)
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user