Adds a thin `amy geochat` verb over the quartz + commons geohash-chat code so the
interop path is exercisable headlessly (and against a real Bitchat client):
`send` builds/signs/mines/publishes a kind-20000 message with the per-geohash
ephemeral identity; `listen` holds a live subscription (kinds 20000/20001 are
ephemeral, so relays broadcast but don't store them) and reports messages plus
distinct present pubkeys; `keys` shows the derived per-geohash pubkey.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172JoMccseEKenyWan6txWV
The rename-echo assertion was flaky: firing two GCE commits (set-image + rename)
3s apart races whitenoise's per-epoch processing, so the second commit sometimes
lands as "unprocessable" and the name never propagates — a false negative.
The interop itself is confirmed working: in a harness run whitenoise's own log shows
`background_sync_group_image_cache_if_needed` parsing amy's image extension and
attempting a Blossom fetch (404 only because the harness sets the image without
uploading a blob), and test_09 (whitenoise messaging in the same group) passes right
after the image commit — both impossible if the image extension had been rejected.
Rewrite test_17 to the robust form: A sets the image, then sends ONE application
message at the post-image epoch; whitenoise can only decrypt it if it applied the
image-bearing commit, so wait_for_message B receiving it is direct proof the extension
parsed. Also skip cleanly when A is no longer a member of the group.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JL3GXW1fmHa3xWfQjLLqfp
whitenoise-rs became a workspace (core `whitenoise` crate at root, wn/wnd moved to
crates/whitenoise-cli) and grew native --discovery-relays / --default-account-relays
flags, which broke the headless harness's patch/build assumptions.
- Repath the mock-keyring patch to crates/whitenoise-cli/src/bin/wnd.rs and match the
new main() (env-gated Whitenoise::initialize_mock_keyring_store(); the fn is pub
under the integration-tests feature).
- Drop the discovery-env / defaults-env patches: start_daemon now passes the native
--discovery-relays / --default-account-relays flags instead.
- Build wn/wnd via `cargo build -p whitenoise-cli --features whitenoise/integration-tests`
(the binaries left the root crate; integration-tests brings in the mock keyring).
- The skip-unprocessable-retry patch still applies (root crate, 19-line offset).
Enables the end-to-end test_17_group_image_commit (amy sets a group icon -> whitenoise
must still process the commit) added in the previous commit to actually run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JL3GXW1fmHa3xWfQjLLqfp
Add `amy marmot group set-image <gid> <file> [--server URL]` and
`amy marmot group clear-image <gid>`: encrypt the avatar with the MIP-01 v2 scheme
(interoperable with mdk/whitenoise), optionally push the ciphertext to Blossom signed
by the keypair derived from image_upload_key, and commit the image fields into the
group's NostrGroupData extension. Thin assembly over quartz
(MarmotGroupImageEncryption) + commons (updateGroupMetadata) per the CLI rules.
Add headless interop test_17_group_image_commit: A (amy) sets a group image on a group
whitenoise is a member of, then renames. The rename is a later MLS epoch, so wn can
only observe the new name if it first applied the image-bearing GCE commit — proving
the image extension stays parseable on mdk (which rejects trailing bytes). This is the
end-to-end guard for the group-icon interop fix.
Note: the marmot interop harness currently can't build current whitenoise-rs — its
patches target the pre-restructure `src/bin/wnd.rs` layout, but whitenoise-rs has moved
to a `crates/whitenoise-cli/` workspace. Refreshing those patches is separate harness
maintenance; the interop correctness is otherwise covered by the pinned-mdk source
analysis and the MarmotGroupImageTest wire regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JL3GXW1fmHa3xWfQjLLqfp
Two roster gaps against the reference client (Armada):
1. Moderators showed as "Admin" and role definitions were often empty. The
displayed roles came from ConcordCommunityState.roles, which was built from the
RAW structural fold heads — so a rogue higher-version edition on a role's
coordinate (e.g. marking the Admin role deleted) corrupted or emptied the roster,
and even when present the UI collapsed every role-holder to a single "Admin"
badge. Now state.roles comes from the authority-gated resolver
(AuthorityResolver.roles(), exposed alongside rolesFor()), and ConcordMembersScreen
renders each member's actual most-privileged role name (Admin / Moderator / custom).
2. Member count was a fraction of the real one (e.g. 10 vs ~44). CORD-02 §5: "an
author seen publishing is observably present, auto-included even if their Join
never arrived." The roster only counted Guestbook joiners + the privileged roster,
omitting the bulk of members who never post a Join. ConcordCommunitySession now
tracks observedAuthors from every decrypted channel message and folds them into
allMembers() and the roster.
Also: amy's `concord roles/grant/ban/...` now register the control-plane stream key
before draining (like `channels`/`read`/`send` already do), so the mod verbs aren't
served an empty fold on NIP-42-gated relays — used to ground-truth the resolved roles.
Verified via amy against live Soapbox: `concord roles` now returns Admin (pos 1) and
Moderator (pos 2) instead of []. quartz + commons concord suites green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`wot` claimed the whole web-of-trust concept for what is actually a cheap
single-hop metric — the count of your follows who also follow X. The real
computed web of trust is `graperank`. Renamed the command (and WotCommand ->
FofCommand) to `fof`, reframing its KDoc/usage away from "trust," and kept
`wot` as a deprecation alias that warns and points at both `fof` and
`graperank`.
Also documents the command for the first time: `amy wot` had no --help block
and no README row. Added both (help block + three README rows for
get/list/sync), so the follows-of-follows score and its relationship to
graperank are now discoverable. JSON output shape is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
Audited every graperank verb/flag in the code against the three doc
surfaces and fixed the drift:
- Main.kt `--help`: rewrote the GrapeRank block to match the terse house
style (one line per command, key flags on continuation lines) — it had
grown to ~68 lines of prose. Now ~33 lines, reordered into pipeline
order (crawl -> score -> publish, then rank/status/refresh, then the
provider/operator discovery group), with the per-verb timeout-semantics
prose dropped (it lives in the KDoc). Verbs and primary flags verified
against the dispatch and each function's arg reads.
- README: added the missing `graperank crawl` and `graperank score` rows
(stage 1 and 2 of the pipeline — crawl had no table row at all, score
was only mentioned inline) and labelled the three pipeline stages.
No stale references remain (no `graperank sync`, no removed publish/
bench flags, `operator providers`/`update` only as documented aliases).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
`graperank sync` is gone — `crawl` is the only spelling. Because the dispatch
treats any non-verb first token as an OBSERVER, this exposed a latent leniency
in the shared resolver: decodePublicKeyAsHexOrNull's fallback runs Hex.decode
without a length check, so a short bech32/hex-ish word like `sync` decodes to a
bogus few-byte "pubkey" instead of failing — turning `graperank sync` into a
silent network crawl over garbage. Context.requireUserHex now requires the
resolved value to be exactly 64 hex chars (a pubkey is always 32 bytes), so a
mistyped OBSERVER/USER on any amy command errors cleanly (exit 2) instead of
silently scoring/fetching against a garbage key. `graperank sync` now returns
`bad_args` rather than crawling.
Also drops the never-built `graperank compare` idea from the roadmap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
- New `graperank status`: read-only local inventory with no network, no
signing, and no side effects — WoT record counts in the store (the "do I
need to crawl again?" answer), reachability-cache size + newest-record
age, operator/service-key state, and persisted card + retraction counts
per observer. Reads the reachability cache through a throwaway signer so
a fresh machine's status never lazily creates the operator master.
- Flag consistency: --relay-concurrency and --concurrency are now accepted
interchangeably on `graperank refresh`, `graperank publish`, and
`relay probe` (each keeps its documented spelling as canonical), and the
help text states what --timeout means per verb (per-REQ drain 10s for
crawl/score, idle watchdog 30s for refresh/publish, per wave 15s for
probe, drain 8s for rank/register/providers).
- `graperank sync` now prints a deprecation warning before running crawl —
the name points at the wrong concept since the negentropy record refresh
became `graperank refresh` — and is removed from the docs; removal comes
in a later release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
Concord relays gate a plane's kind-1059 wraps behind NIP-42 and serve them
only to a connection authenticated AS the plane's derived stream key — the
member is neither the wrap's author (the stream key) nor its recipient (a
throwaway ephemeral key), so an account AUTH is refused and every plane REQ
came back empty (no channels, no messages), even though the community folded
its name.
Mirror the app's per-plane AUTH (60475c10c0) in the CLI:
- Context.registerConcordStreamKeys(relays, secrets) records the derived
control/channel stream secrets, scoped to the community's relays.
- The RelayAuthenticator provider now signs one kind-22242 per registered
stream key alongside the account AUTH — locally, from the raw derived key
(NostrSignerSync), never the account, so no user identity is exposed.
- The concord channels/read/send verbs register their control + channel
stream keys before draining/publishing.
- drain() gains pendingOnAuthRequired: an auth-required CLOSED keeps the relay
pending instead of terminal, so the post-auth subscription re-fire delivers
the events rather than the one-shot drain returning empty. The concord verbs
opt in; all other drains are unchanged.
Verified end-to-end against the live Soapbox community (relay.ditto.pub /
relay.dreamith.to): `amy concord channels` now folds the name + 9 channels
consistently and `amy concord read` returns messages. `channels` also emits
the folded icon/banner/description pointers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- `graperank update` -> `graperank refresh`: the verb says what it does
(refresh the WoT record kinds from each author's outbox) and stops
colliding with the sync/crawl naming tangle. `update` stays as an alias.
- New `graperank unregister PROVIDER [--service KIND:TAG] [--relay URL]`:
the missing inverse of `register` — removes matching entries (public +
private) from the account's kind:10040 and re-publishes it; without
narrowing flags every entry for that provider key is dropped.
- `graperank operator providers` -> `operator keys`: it lists the
observer -> service-key map, and the old name collided with
`graperank providers` (the kind:10040 read). `providers` stays as an
alias; the JSON key `providers` becomes `keys` in both the listing and
`operator status` (breaking, matches the rename).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
The census probes the whole known relay universe and feeds the shared NIP-66
reachability cache (kind:30166) that every reachability-aware command reads —
it was never graperank-specific, so it moves from GrapeRankCommand to
RelayCommands as `amy relay probe`. `amy graperank probe` stays as an alias,
and flags, JSON output, and behaviour are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
Every `amy graperank` / `graperank score` run now reconciles its result into
the local event store as NIP-85 kind:30382 cards signed by the per-observer
service key (cutoff --min-rank, default 2): changed ranks are re-signed,
unchanged ones skipped (no event-id churn), dropped targets retracted with a
kind:5 the store applies on insert. The store is the source of truth, so the
score of a run is durable and reusable instead of ephemeral stdout.
New verbs complete the crawl -> score -> publish pipeline:
- `graperank publish [OBSERVER] [--relay URL[,URL...]]` — transport only:
one NIP-77 up-only reconcile per operator relay over the provider key's
kind:30382 + kind:5, converging the relay to the local set (deletions
propagate, lost cards restored, full-set publish fallback for relays
without negentropy); also refreshes the observer's kind:10040 pointer.
- `graperank rank USER [--provider P] [--refresh]` — the consumer side:
newest card per provider from the local store, with a relay drain on miss.
GrapeRankPublisher (quartz) is reworked accordingly: reconcileAndPublish is
replaced by reconcileLocal (sign+insert+retract against the store) and
syncToRelays (NegentropyStoreSync up-only + blastPublish fallback), with a
commonTest covering upsert/skip/retract and re-carding a retracted target.
BREAKING (--json / flags): `--publish`, `--publish-limit`, `--publish-relay`
and `--bench-sign` are removed from the score command. Its JSON drops
published/publish_rejected/deleted/delete_rejected/skipped_unchanged/
publish_truncated/published_kind/published_to/publish_error/observer_10040/
bench_signed/bench_sign_ms and gains provider_pubkey/min_rank/cards_total/
cards_signed/cards_unchanged/cards_retracted/cards_ms. Publishing moved to
the new `graperank publish` verb.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013WSzVX9RoUxyV3nT3dfc56
The relay-auth fix added an `interactive` flag to
RelayAuthenticator.signWithAllLoggedInUsers; update the desktop and cli
implementers (which don't prompt) to the 3-arg lambda.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts the Kover coverage aggregation (introduced in dca345212b) from every
module, the root aggregation block, the Sonar coverage import, the version
catalog, and BUILDING.md. With the plugin present the 16 GB CI runner OOM-killed
during the Android test+build job even with Kover opt-in disabled; removing it
entirely clears the OOM. No other CI changes — the diff versus main is exactly
the inverse of dca345212b.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Apply Kover (Apache-2.0, build-time only) to the KMP/JVM modules and aggregate at the root
- include amethyst playDebug in the aggregated Kover coverage
The miner enumerates the nonce space deterministically (the random base is
overwritten before the first hash), so naively racing N copies of the search
duplicates the exact same candidate sequence N times. PoWMiner.mine() now
races workers over disjoint slices instead: each worker's nonce carries a
distinct fixed prefix while only the bytes after it are enumerated, making the
aggregate hash rate scale with cores (~3.8x on a 4-core box).
The hot loop also switches from sha256() to sha256Into() with a reused
32-byte buffer, so hashing no longer allocates per attempt.
amy wiring:
- `pow mine` and `post --pow` mine on all cores by default; `pow mine
--threads N` overrides.
- `pow bench` measures the all-cores rate (what mining now uses, also the
basis for expected_seconds) alongside a new hashes_per_second_single_core.
- PoWEstimator benchmarks with sha256Into to match the miner, and gains a
workers overload that prices in cross-core contention.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UhaF5scnAvhP9wr9R7bTWM
Durability & correctness:
- Keep the job entry + disk checkpoint alive until the publish continuation
completes (was: dropped at mining-complete); failed publishes keep their
checkpoint for restart retry and surface through a failures SharedFlow
- Move draft deletion into the publish continuations in every composer so a
cancelled or process-killed mining job can't destroy the only copy of a post
- Persist gift-wrap mining: split NIP17Factory into createSeals (signer
interaction, runs inline) + wrapSeal (pure CPU, runs on the queue), new
REPLAY_WRAPS records restore pending DM wraps after process death
- Clamp synced NIP-78 difficulty (PoWPolicy.MAX_DIFFICULTY), require a sane
target in PoWMiner, bound PoWRankEvaluator against short ids
- Reaction double-tap while mining now toggles (dedupeKey + cancelByKey)
instead of publishing duplicates
- Restore checkpoints for every loaded account, not just the active one
- logOff purges the account's checkpoints and cancels its queued jobs
- Private notes: composer chip now gates on the gift-wrap kind and the
per-post override reaches sendPrivateNote
Coverage:
- Public/live chat (kinds 42 + 1311) and voice replies (1244 + kind-1 audio
replies) now route through the mining gate
Perf:
- FGS start() dedupes with a running flag; PendingIntents built once
- Banner 1 Hz clock only ticks while a job shows elapsed time
- PoWEstimator benchmark is single-flight behind a Mutex
UX / l10n:
- Post-mining failures toast with retry information
- Count strings converted to <plurals>; elapsed time via DateUtils; settings
estimate uses localized units; shared powKindLabelRes replaces three
duplicated kind→label maps; dead pow_chip_* strings removed
- CLI: pow mine lowercases the pubkey before mining (uppercase hex mined an
id that never matches the signed event) and validates via quartz Hex
New queue tests: checkpoint lifetime, failure reporting, cancelByKey toggle,
per-owner cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
- `amy notes post TEXT --pow BITS [--pow-timeout SECS]` mines the note
pre-signature via quartz's PoWMiner (blocking — the CLI process is the
job), exits 124 on timeout with nothing published, and adds additive
--json keys: pow, pow_target, pow_millis.
- `amy pow check EVENT-JSON|-` reports actual_bits, committed_target,
has_commitment and effective_pow (capped at the commitment per
NIP-13's anti-lucky-spam rule) plus id+sig validity.
- `amy pow mine --target N [--pubkey HEX] [--timeout SECS] TEMPLATE|-`
mines an unsigned template for any pubkey — NIP-13's delegated PoW:
ids don't commit to signatures, so a headless box can mine for a
phone and hand the template back for signing.
- `amy pow bench` prints the machine's hash rate and expected seconds
at 16/20/24/28 bits (commons PoWEstimator).
- cli/tests/pow/pow-headless.sh: relay-free harness covering bench,
mine (commitment shape + 124 timeout), and the delegated round trip
(mine → sign via `amy event` → pow check ≥ target). 6/6 passing.
No logic added to cli/ — thin assembly over quartz nip13Pow + commons
PoWEstimator per the CLI architecture rules.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ADb3dez9jPk6QqyQ1rTx4V
Adds the moderation verbs over the tested ConcordModeration write path:
- roles <community> list live roles + current banlist
- role <community> <name> <pos> PERM... define a role (perms by name), prints role_id
- grant <community> <user> <roleId> grant a role to a member
- ban / unban <community> <user> banlist add/remove
Each drains the Control Plane, chains the next edition onto the current head,
and publishes it; authority is enforced on fold by every client. Users resolve
via npub/nprofile/hex/nip05 (ctx.requireUserHex).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
CachingDns duplicated 20% of what the app's SurgeDns already did better
(stale-while-revalidate, single-flight, jittered 24-48h positive TTL,
persistence, poison filtering). Consolidate on SurgeDns and fix what the
verification pass found along the way:
- move SurgeDns + SurgeDnsStore (+ their 41 tests) to quartz jvmAndroid so
the CLI can share them; delete CachingDns
- negative TTL 10s -> 10min (class default): a dead domain burns 10-30s of
getaddrinfo per re-dial and both the relay pool and a crawl re-dial dead
hosts continuously; the relay pool's own backoff already reaches 5min
- stop the call-failure listeners from erasing negative entries they were
just written from: callFailed caused by UnknownHostException must not
invalidate (it deleted every negative entry milliseconds after creation,
silently defeating the negative TTL entirely). The invalidation remains
for its real purpose - stale positives with rotated IPs
- new SurgeDns.staleAll(): soft-expire everything WITHOUT discarding -
positives serve stale and revalidate in the background on next use,
negatives re-try on first touch. Wired to network-identity changes in
AppModules (same trigger as Tor's onNetworkChange), replacing nothing:
the full-clear invalidate() was never actually called on network change
- amy: SurgeDns wired into the CLI OkHttp client, snapshot persisted at
~/.amy/shared/dns-cache.bin across runs (best-effort load/save)
Verified: all SurgeDns/SurgeDnsStore tests pass in the new location incl.
two new staleAll tests; :amethyst compiles (main + unit tests); cli suite
green. Caveat recorded in the plan doc: behind an HTTP CONNECT proxy OkHttp
never consults the client resolver (hostname goes to the proxy), so this
layer pays off on direct-connect deployments only - which also means the
branch's A/B numbers never depended on it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
NIP-29 metadata/roster events (39000-39003) "are addressable events signed by
the relay keypair directly ... as stated by the NIP-11 `self` pubkey", and
"relays shouldn't accept these events if they're signed by anyone else". So the
authoritative test for a genuine group is `39000.author == relay.self` — which
also rejects a stray user-published 39000 even on a real NIP-29 relay, something
the earlier supported_nips heuristic could not.
Add `isRelaySignedRelayGroup(channel)`: strict `author == self` when the relay
publishes `self`, falling back to `supported_nips ∋ 29` when it omits `self`, and
false when it has neither. Apply it at the surfaces that show unsolicited groups:
- Discovery feed: replace the relay-level supported_nips filter with the
per-channel self-key check in matches(); the screen now warms each candidate
relay's NIP-11 and re-invalidates the feed as each doc resolves.
- On-relay group list: filter to relay-signed groups, warming that relay's
NIP-11 so genuine groups fill in and fakes stay hidden.
- CLI `relaygroup browse`/`info`: fetch the relay's NIP-11 (new
Context.relayInfo) and drop 39xxx not signed by `self`; browse reports the
dropped count.
Explicit user actions (a received invite link, opening an naddr) are left
untouched — hiding those would be user-hostile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
Phase-B throughput plateaus at ~100-150 users/s at every crawl scale while
CPU/FDs/network sit idle, pointing at the per-relay 16-permit gates on the
hot backbone/fallback relays every batch touches. The 16-vs-100 benchmark
that chose the default predates the multithreaded-crawl fix, so expose the
starting cap for A/B runs; the NOTICE/CLOSED demotion ladder still protects
relays that push back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Cold hop-3 A/B (fresh store per leg, same observer): 48 workers beat 24
twice at identical contact-list counts (579s->559s and 474s->396s). The
old result that made 64 look 2x slower predated the multithreaded
dispatcher fix - more coroutines on one starved event-loop thread. Plan
doc gains the full A/B table (kept: background-park degating, sweep
dedup, tail overlap, dc48; rejected: 5s fast window).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
A saved crawl log couldn't attribute wall time to the finishing tail
(aggregator recovery vs report deletions vs parked drains) without
external timestamps. Prefix every crawl progress line with [t+SSSs].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Any graperank crawl/probe now creates ~/.amy/operator/ (the reachability
cache signs with the operator monitor key), and listAccounts counted it as
an account - so a single-account user's very first crawl left every later
command failing with "multiple accounts (operator, <name>)". Add it to
RESERVED_NAMES alongside shared/current. Hit for real while network-testing
the connect-storm changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Most of a from-scratch crawl's wall clock was connection setup, re-paid
serially: the relay pool tears down a relay's socket ~300ms after its last
drain unsubscribes, OkHttp allowed only 256 concurrent WS handshakes, and
every dial re-ran an uncached blocking getaddrinfo (10-30s per dead domain,
once per per-user path URL of the same host). Open the connections once, in
parallel, and keep them:
- GrapeRankCrawler: the warm-pool trick (never-matching REQ that only holds
the socket) now covers the whole candidate universe instead of the top 20 -
seeded at crawl start from the reachability cache's live set (one parallel
connection storm, Config.knownLiveRelays) and refreshed each round with
newly learned outbox relays, capped by Config.preconnectCap (FD-budget
aware, --preconnect-cap / --no-preconnect).
- CachingDns (quartz jvmAndroid): 10-min positive + negative DNS cache with
in-flight per-host dedup; dead domains fail in microseconds instead of
re-burning resolver timeouts, path URLs of one host resolve once.
- cli Context: dispatcher and pre-connect caps derived from the process's
open-files limit (UnixOperatingSystemMXBean), warning when ulimit is low.
- amy graperank probe: relay census - mass-connects every relay the store
knows (kind:10002 universe deduped per authority + cached verdicts) in
waves, records live/dead with real measured rtt-open into the NIP-66
reachability cache (RelayProber + RelayReachabilityStore.recordProbed),
so the next crawl skips dead relays and waits once for the slow-but-alive.
Single-server limits (FDs, ephemeral ports, DNS, threads, conntrack) and the
design are documented in quartz/plans/2026-07-10-graperank-connect-storm.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zEYRGKF943RgLaHTViJaB
Full Concord stack through the CLI, thin over commons ConcordActions + Context:
- ConcordStore (~/.amy/<account>/concord.json, 0600): joined communities +
their secrets for re-derivation across runs
- concord create — mint community + publish genesis, save locally
- concord list — list joined communities
- concord channels — drain + fold the Control Plane, list channels
- concord send — post an encrypted kind-9 message to a channel
- concord read — drain + decrypt a channel's messages (oldest-first)
- concord invite — mint + publish a shareable invite link (bundle 33301)
- concord join URL — fetch + decrypt the bundle with the fragment token, save
Verified end-to-end against a local `amy serve` (geode) relay: Alice creates a
community and mints an invite; Bob joins from the URL alone, both post to
#general, and both read the identical decrypted message list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzJ2Cwo8tg4oZq43oRa3ig
Measured: relays deliver their events in ~0.6s then sit ~4.6s (86% of drain wall)
before sending EOSE — mostly relay-side (our pipeline adds only ~200ms). So instead
of waiting the full 10s fast window then parking, close a drain that has delivered
>=1 event and then gone silent for eoseIdleMs, treating it as complete ("eose-idle").
awaitTerminalOrQuiescent: the idle timer arms only AFTER the first event, so a relay
merely slow to answer still gets the full timeoutMs and is never cut prematurely; a
still-streaming relay keeps resetting the window. eose-idle paginates if the page was
capped and clears timeout strikes (it delivered), but joins notAnswered (no clean
EOSE, so its missing authors are retried elsewhere). Off by default (eoseIdleMs=0),
CLI --eose-idle-ms, so it can be A/B'd against the plain fast window.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Two changes:
1. The reachability flush re-wrote the SEEDED known-dead relays with a fresh
created_at every run, refreshing their TTL without a re-probe — so a relay
marked dead once (and thereafter skipped, never re-dialed) would stay
blacklisted forever as long as crawls kept running, defeating the TTL's
re-probe. Stats.deadRelays now reports only relays actually dialed this run
(deadRelays - knownDeadRelays); seeded records keep their original timestamp
and age out on schedule so the next run re-probes them.
2. Rename GrapeRankDataCrawler -> GrapeRankCrawler (file + all references).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Wire RelayReachabilityStore into the crawler and the WoT updater so liveness is
shared across procedures and runs instead of each rediscovering dead relays.
- OperatorKeys.monitorKey(): a dedicated machine monitor identity derived from the
operator master (domain "relay-monitor:"), independent of any account — the
30166 records are published under this, not the observer key.
- Context.reachability: a RelayReachabilityStore over the shared store, signed by
the monitor key.
- Crawler: Config.knownDeadRelays seeds deadRelays before the run; Stats now
returns the final dead/live sets. GrapeRankCommand seeds from snapshot().dead
and flushes the crawl's verdicts back via reachability.record().
- Updater: Config.knownDead skips proven-dead relays from the reconcile plan — a
dead relay cannot serve its authors, so reconciling it only burns a timeout.
Live author-advertised relays are always synced.
All behind --no-reachability-cache. TTL'd (24h), so a recovered relay is retried
once its record ages out — a "skip for now", never a permanent ignore, keeping
the outbox rule that every live advertised relay is tried.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
The network-only WoT data traversal is a crawl, not a sync — and main now
ships negentropy sync (`amy sync`, `graperank update`), so the old verb name
was ambiguous. Rename the subcommand and its handler to `crawl`, keeping
`sync` as a back-compat alias so existing scripts keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Moves the `amy graperank update` logic out of the CLI and into quartz as
GrapeRankUpdater, alongside GrapeRankDataCrawler in experimental/graperank,
so Android and any other quartz consumer can run the same refresh.
Given an INostrClient + IEventStore it reads every kind:10002 in the store,
inverts them into a write-relay -> authors map (the outbox model), then runs
one NIP-77 negentropy reconcile per write relay scoped to its authors:
bidirectional content sync into/from the store, deletion settle over the
residual (applyDown downloads the relay's kind:5 when an uploaded record was
rejected because the author retracted it), and a full paged-download fallback
when a relay can't reconcile. Bounds and directions are a Config; per-relay
and aggregate outcomes are returned as a Result.
The CLI `graperank update` is now a thin wrapper: it parses flags, builds the
Config, and renders GrapeRankUpdater.Result as text/JSON — no sync logic left
in cli/ (all reconcile/window/back-pressure/deletion logic lives in quartz's
relay-client accessories, which GrapeRankUpdater composes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Adds a store-driven refresh of the record kinds a GrapeRank score is a
function of (0 profiles / 3 follows / 10002 outbox lists / 1984 reports).
It reads every kind:10002 already in the local store, inverts them into a
write-relay -> authors map (the outbox model), then runs one NIP-77
negentropy reconcile per write relay scoped to exactly the authors who
publish there. Bidirectional by default; each group then settles deletions
over the reconcile residual via quartz's negentropySettleDeletions, whose
applyDown direction downloads the relay's covering kind:5 when an uploaded
record was rejected because the author retracted it.
When negentropy can't reconcile a relay (no NIP-77, an over-cap minimal
window, a mid-sync disconnect), the group falls back to a full paged
download (Context.drainAllPages) of the same authors+kinds so those
records are still refreshed.
Thin assembly only: reconcile, windowing, back-pressure, and deletion
settle all live in the quartz relay-client accessories, mirroring
SyncCommand; this only routes ids to Context.drain / drainAllPages /
publish and inverts the relay list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TdEvjsZ81XuUtdJsVzmHxt
Correctness:
- Discovery sort no longer reads createdAt live in the comparator (TimSort
"contract violated" crash risk); orders by member count then the shared
sortedByDefaultFeedOrder snapshot.
- One shared resolver (relayGroupDiscoveryChannelFor) is used by the feed
match, sort AND the row, so a 39000 seen on >1 relay always binds one
channel — no more "sorted by relay B, rendered with relay A's empty roster".
- Roster (39001/39002) arrivals now re-inject the group's 39000 into the feed
and re-invalidate the datasource, so a group where a follow is an admin/
member surfaces instead of staying hidden / frozen at 0 members.
- Group replies route to the group's host (resolved from the channel), never
falling through to signAndComputeBroadcast — fixes the outbox leak when the
parent note had no relay provenance.
- Messages list (inline mode) updates group rows incrementally: the additive
path now handles group-scoped messages, not just public/ephemeral/DM.
- Optimistic null-relay group sends attach only to an unambiguous single
channel, so a message to the "_" group on relay A no longer bleeds into "_"
on relay B.
Consistency:
- AllFollows discovery also REQs #t/#g (not just authors), matching the local
AnyOf constraint; muted-authors maps to ByPeople; community stays AllGroups
— fetch and display now agree.
- Global drops the 1-week `since` floor so long-lived group metadata is fetched.
Performance:
- #d metadata backfill scans the group cache once per filter assembly (grouped
by relay), not once per relay.
- Discovery rows warm content only; the directory subscription already streams
metadata/rosters for the relay.
- memberCount is memoized (members ∪ admins recomputed on roster change, not per
read); thread re-sort extracted.
- Discovery list gets rememberFeedContentPadding, contentType and animateItem.
CLI:
- `relaygroup edit` preserves current name/about/tags when only a flag changes.
Cleanup: dropped dead relayKeys() branches and the redundant AnyOf single-lens
collapse in the discovery constraint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B5MLY4hq5LXJ2D5WeLRyXj
The probe does blocking DNS + TCP connect, and dead-domain DNS lookups hang well
past the connect timeout. On the shared Dispatchers.IO those hanging lookups
starved the crawl's own IO: an A/B at hop-3 showed probe-on 981s vs probe-off
517s, the entire +464s landing on the finishing drain (rounds were identical).
Coverage was unchanged (91.84% vs 91.74%), so the probe classification is correct
— it was purely IO contention.
Give the probe its own fixed daemon pool (128 threads) so its blocking work can
never touch the crawl's IO, and align the culler's concurrency to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSW59hJtP4Yn8fnRUxc7F5
Claude
Vitor PamplonaandGitHub
davotoulaandClaude Opus 4.8