mirror of
https://github.com/vitorpamplona/amethyst.git
synced 2026-10-05 19:28:25 +00:00
Claude
3b20aede9a
Adds a recorded corpus of 64 real events from search-staging.brainstorm.world — a vespa-relay, the same software this token language was ported from — and 18 tests over it: 8 on the engine in :commons, 10 on `LocalCache.filter` in :amethyst. Recorded, not fetched. `tools/search-parity/fetch_fixtures.py` drives `amy fetch` against the relay by hand; the tests read the committed fixture, so `./gradlew test` stays offline and the pre-push hook does not depend on somebody else's uptime. The fixture stores each case's filter FIELDS rather than a prebuilt filter, so the test rebuilds the Filter in view of the reader and a wrong rebuild cannot quietly make the assertions vacuous. What the relay can and cannot referee turned out to be the whole design, and it was measured rather than assumed: - **NIP-01 it can.** Given kinds, #t, since and until there is exactly one right answer, and across all 64 events our matcher agrees with the relay about every one it chose to return — 0 violations. That is now a hard assertion, with a converse test so it cannot pass by matching everything. - **NIP-50 it cannot.** This relay retrieves topically: asked for `bitcoin` it returns a block-height summary that never says "bitcoin". Eight of 64 events carry no literal occurrence of the term that fetched them. Asserting our substring matcher reproduces that would encode someone else's semantic expansion as a requirement on a lexical one — a test that fails on correct code. So text results are deliberately not compared, and the divergence is pinned as a range instead: zero would mean the relay turned lexical and the comparison should be rewritten, a quarter would mean we regressed. The fixture uses the `include:spam` lens, which waives the web-of-trust gate. Also measured: it makes the corpus reproducible, where `observer:<pubkey>` ties every answer to one account's moving trust graph — but it does not make retrieval lexical, and in fact widens the divergence from 4 events to 8 by letting more topical matches through. The LocalCache tests cover what the relay knows nothing about and where the bugs actually were: the regular/addressable split, the viewer-policy predicate composing with rather than replacing the filter, and the result cap keeping the newest — the ordering whose absence let `take(limit)` run before the sort. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017yKjw2WqwZpSzsqcYZMnkV
77 lines
3.3 KiB
Python
Executable File
77 lines
3.3 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Record real answers from a live vespa-relay so the local search engine can be tested
|
|
against them offline.
|
|
|
|
Run by hand when the fixture needs refreshing; the test that consumes its output never
|
|
touches the network, so `./gradlew test` stays hermetic and offline.
|
|
|
|
./gradlew :cli:installDist
|
|
tools/search-parity/fetch_fixtures.py > commons/src/jvmTest/resources/search-parity-fixture.json
|
|
|
|
Each case pins the filter FIELDS (kinds/tags/window) as flags rather than folding them into
|
|
the search string, so the test can rebuild the identical NIP-01 Filter and assert our matcher
|
|
agrees with the relay about every event it chose to return.
|
|
"""
|
|
import json
|
|
import os
|
|
import shlex
|
|
import subprocess
|
|
import sys
|
|
|
|
RELAY = os.environ.get("RELAY", "wss://search-staging.brainstorm.world")
|
|
AMY = os.environ.get("AMY", os.path.join(os.getcwd(), "cli/build/install/amy/bin/amy"))
|
|
# The relay gates and ranks results through the searcher's web of trust, and answers an
|
|
# anonymous query with nothing at all. `include:spam` waives that gate, which is what makes
|
|
# this fixture reproducible: the alternative, `observer:<pubkey>`, ties every recorded answer
|
|
# to one account's trust graph and re-records differently as that graph moves.
|
|
#
|
|
# It does NOT make the relay lexical. Measured on this corpus, results still include events
|
|
# carrying no literal occurrence of the term — asked for `nostr` it returns "made my display
|
|
# name refer to my npub's last characters". Retrieval is semantic; see the parity test for why
|
|
# that means text results are not compared.
|
|
LENS = os.environ.get("LENS", "include:spam")
|
|
LIMIT = os.environ.get("LIMIT", "8")
|
|
TIMEOUT = os.environ.get("TIMEOUT", "30")
|
|
|
|
# name, search terms, filter flags
|
|
CASES = [
|
|
("text_single", "bitcoin", ["--kind", "1"]),
|
|
("text_two_words", "bitcoin lightning", ["--kind", "1"]),
|
|
("text_longform", "nostr", ["--kind", "30023"]),
|
|
("tag_hashtag", "bitcoin", ["--kind", "1", "--tag", "t=bitcoin"]),
|
|
("window_since", "nostr", ["--kind", "1", "--since", "1700000000"]),
|
|
("window_until", "nostr", ["--kind", "1", "--until", "1800000000"]),
|
|
("kinds_union", "nostr", ["--kind", "1,30023"]),
|
|
("phrase_quoted", '"open source"', ["--kind", "1"]),
|
|
]
|
|
|
|
|
|
def fetch(terms, flags):
|
|
search = f"{terms} {LENS}"
|
|
cmd = [AMY, "fetch", *flags, "--search", search, "--limit", LIMIT,
|
|
"--relay", RELAY, "--timeout", TIMEOUT, "--json"]
|
|
try:
|
|
out = subprocess.run(cmd, capture_output=True, text=True, timeout=int(TIMEOUT) + 30)
|
|
return search, json.loads(out.stdout).get("events", [])
|
|
except Exception as e: # a relay that is down must not silently produce an empty fixture
|
|
print(f"FAILED {' '.join(shlex.quote(c) for c in cmd)}: {e}", file=sys.stderr)
|
|
raise
|
|
|
|
|
|
def main():
|
|
if not os.access(AMY, os.X_OK):
|
|
sys.exit("amy not built: ./gradlew :cli:installDist")
|
|
cases = []
|
|
for name, terms, flags in CASES:
|
|
search, events = fetch(terms, flags)
|
|
print(f"{name:16s} {len(events):3d} events", file=sys.stderr)
|
|
cases.append({"name": name, "terms": terms, "flags": flags,
|
|
"search": search, "events": events})
|
|
json.dump({"relay": RELAY, "lens": LENS, "cases": cases},
|
|
sys.stdout, indent=1, sort_keys=True)
|
|
print()
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|