.

mesh-scan/README.md

@@ -0,0 +1,208 @@
+# FIPS Mesh Reachability Scanner
+
+A standalone tool that answers: **"From a fresh node peered with only node X,
+can I actually reach node Y through the mesh?"** — producing a reachability
+matrix across the real public FIPS mesh.
+
+See [`../plans/mesh-reachability-scanner.md`](../plans/mesh-reachability-scanner.md)
+for the full design rationale.
+
+## Why a standalone script
+
+The browser attaches to a single long-lived FIPS daemon. The scanner needs
+**ephemeral daemons peered one-at-a-time** with different seed nodes, which
+the browser architecture can't do. This tool drives real `fipsd` processes
+via their Unix control sockets, reusing patterns from
+[`../../fips/testing/chaos/sim/control.py`](../../fips/testing/chaos/sim/control.py)
+and the browser's
+[`handle_fips_directory_json`](../../sovereign_browser/src/nostr_bridge.c).
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| [`mesh_scan.py`](mesh_scan.py) | Main scanner / orchestrator |
+| [`control.py`](control.py) | Control-socket client (line-delimited JSON over Unix socket) |
+| [`directory.py`](directory.py) | Nostr Kind 37195 advert fetcher + npub→IPv6 derivation |
+| [`fipsd.py`](fipsd.py) | Ephemeral daemon lifecycle (config, spawn, poll, teardown, TUN cleanup) |
+| [`reachability.py`](reachability.py) | ping6 / TCP reachability checks |
+
+## Requirements
+
+- Python 3.10+
+- `websocket-client`, `pyyaml` (see [`requirements.txt`](requirements.txt))
+- `fips` / `fipsctl` binaries in `PATH`
+- `iproute2` (`ip` command), `ping6`
+- **`CAP_NET_ADMIN`** — `fipsd` needs it to create TUN devices and network
+  namespaces. The scanner auto-detects whether the current user has it; if
+  not, it spawns daemons via `sudo -n` (passwordless sudo). Alternatively,
+  set the capability on the binary once:
+  `sudo setcap cap_net_admin+ep /usr/local/bin/fips`.
+- **Network namespace support** — `ip netns` is used to isolate each
+  ephemeral daemon (see [How it works](#how-it-works) below).
+
+Install Python deps:
+
+```bash
+pip install -r mesh-scan/requirements.txt
+```
+
+## Quick start
+
+### 1. Smoke test (two ephemeral nodes, no public mesh)
+
+Verifies the whole toolchain — daemon spawn, control socket, peering,
+reachability — without touching the public mesh:
+
+```bash
+cd mesh-scan
+python3 mesh_scan.py --self-test -v
+```
+
+You should see two ephemeral daemons (`fips-scan-900`, `fips-scan-901`)
+peer over loopback and a `ping6` succeed from one TUN to the other.
+
+### 2. Full scan
+
+```bash
+# Random sample of 5 seeds, default 45s convergence, ping6 checks:
+python3 mesh_scan.py --seeds-sample 5 -v
+
+# Specific seeds, specific targets:
+python3 mesh_scan.py --seeds npub1abc,npub1def --targets npub1ghi,npub1jkl
+
+# All seeds (O(N) daemon lifecycles — slow):
+python3 mesh_scan.py --seeds all
+
+# Seeds near the tree root (smallest NodeAddr — best reachability if mesh is healthy):
+python3 mesh_scan.py --seeds-near-root 5
+```
+
+### 3. Cleanup
+
+If the scanner crashes, `fips-scan-*` network namespaces may leak:
+
+```bash
+python3 mesh_scan.py --cleanup-stale
+```
+
+## Inputs / flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--seeds all` | — | Every node in the directory |
+| `--seeds npub1a,npub1b` | — | Explicit seed list |
+| `--seeds-sample K` | 5 | Random sample of K seeds |
+| `--seeds-near-root K` | — | K seeds with smallest NodeAddr (tree-root neighborhood) |
+| `--targets npub1a,...` | all except seed | Restrict targets |
+| `--converge-secs N` | 45 | Convergence budget after peering |
+| `--ping-timeout N` | 5 | Per-target reachability timeout (seconds) |
+| `--method ping6\|tcp` | ping6 | Primary check method |
+| `--tcp-port N` | 80 | TCP fallback port |
+| `--delay N` | 0 | Seconds between target checks (politeness) |
+| `--relay wss://...` | damus/nos.lol/offchain | Advert relay (repeatable) |
+| `--out-dir DIR` | `mesh-scan-out` | Output directory |
+
+## Output
+
+### `matrix.csv`
+
+Rows = seeds, columns = targets, cells = `ok/<rtt>ms` / `fail`:
+
+```
+seed            npub1abc   npub1def   npub1ghi
+npub1root       ok/42ms    ok/55ms    fail
+npub1hub        ok/38ms    ok/41ms    ok/67ms
+```
+
+### `diagnostics.jsonl`
+
+One JSON record per (seed, target) pair:
+
+```json
+{
+  "seed": "npub1abc", "target": "npub1def",
+  "reachable": true, "rtt_ms": 55, "loss_pct": 0, "method": "ping6",
+  "target_in_identity_cache": true, "target_in_seed_bloom": true,
+  "cached_coords": true, "tree_distance": 3,
+  "seed_depth": 2, "seed_parent": "npub1root", "mesh_root": "npub1root",
+  "converge_secs": 12
+}
+```
+
+### Summary (stdout)
+
+Total/overall reachability, per-seed rates (good vs bad hubs), per-target
+rates (universally unreachable = NAT/firewall suspects), and failure-cause
+correlation (identity-cache miss / bloom miss / coord-cache miss).
+
+## Interpreting results
+
+The per-pair diagnostics map directly to FIPS mesh-routing failure modes
+(see [`plans/mesh-reachability-scanner.md`](../plans/mesh-reachability-scanner.md)
+"Background" section):
+
+| Diagnostic | Failure mode |
+|------------|--------------|
+| `target_in_identity_cache=false` | Identity-cache gate (`session.rs:1144`) — discovery skipped |
+| `target_in_seed_bloom=false` | Bloom propagation gap — target's existence not propagated to your seed |
+| `mesh_root` differs | Tree partition — target is in a different spanning tree |
+| `cached_coords=false` | Coord-cache miss (cold path) — `find_next_hop` returns None |
+
+**Direct peering** hits `find_next_hop` branch 2 (direct peer), bypassing
+all of: coord cache, bloom discovery, tree routing. That's why it always
+works when mesh routing fails — and why a target that's `fail` from all
+seeds but reachable by direct peering is the symptom this scanner diagnoses.
+
+## How it works
+
+Each ephemeral daemon runs in its own **network namespace** (`ip netns`).
+This is necessary because the FIPS daemon hardcodes a `fd00::/8` route on
+its TUN device ([`fips/src/upper/tun.rs`](../../fips/src/upper/tun.rs)). On
+a host that already runs a long-lived FIPS daemon (`fips0`), that route
+already exists, so a second daemon's TUN init fails with
+`File exists (os error 17)`. A fresh netns gives each ephemeral daemon its
+own routing table where `fd00::/8` is free.
+
+The control socket is a Unix domain socket in `/tmp` (shared filesystem),
+so it remains reachable from the host process. Reachability checks
+(`ping6`/TCP) are executed *inside* the daemon's netns via
+`ip netns exec`, so the `fd00::/8` route resolves to the daemon's TUN
+rather than the host's `fips0`.
+
+Each netns also gets a **veth pair** with NAT masquerading so the
+ephemeral daemon can reach the public internet (to peer with remote seed
+nodes over UDP/TCP). Without this, the netns would be isolated to
+loopback-only and couldn't connect to any public seed. The veth setup,
+default route, and iptables masquerade rules are created automatically and
+torn down on daemon stop / `--cleanup-stale`.
+
+For the `--self-test`, both ephemeral daemons share a single netns
+(`fips-scan-selftest`) so they can peer over loopback (`127.0.0.1`).
+
+## Notes / caveats
+
+- **Heuristics**: `target_in_seed_bloom` and `cached_coords` are
+  best-effort. The control-socket `show_bloom` / `show_routing` queries
+  expose aggregate stats, not per-target membership, so these fields use
+  the identity cache as a proxy. Treat them as directional indicators, not
+  ground truth.
+- **NAT**: the scanner node itself may be behind NAT. FIPS Nostr discovery
+  + STUN hole-punching should handle this, but running on a
+  publicly-addressable host removes the confound.
+- **Politeness**: scanning N×M pairs generates real mesh traffic. Use
+  `--delay` and a small `--seeds-sample` for production meshes.
+- **Coexistence with `sys-fips`**: each ephemeral daemon runs in its own
+  network namespace with unique ports and a `/tmp` control socket, so it
+  does not collide with the host's long-lived `fips0` daemon.
+- **Directional asymmetry (observed in self-test)**: a freshly-peered
+  two-node pair can show asymmetric ping6 reachability (one direction
+  succeeds, the other fails) before coord caches warm up. The self-test
+  tries both directions and passes if either succeeds. On the real mesh,
+  convergence (`--converge-secs`) mitigates this; the per-pair diagnostics
+  capture `cached_coords` / `target_in_identity_cache` to explain any
+  residual asymmetry.
+- **IPv6 host connectivity**: the scanner works even on hosts without
+  public IPv6 (e.g. Qubes OS qubes), because the FIPS overlay uses
+  `fd00::/8` IPv6 *inside* the TUN, which is contained in the netns and
+  tunneled over the daemon's IPv4 UDP transport.

mesh-scan/control.py

@@ -0,0 +1,118 @@
+"""Control-socket client for FIPS daemons.
+
+Standalone port of the patterns in
+``fips/testing/chaos/sim/control.py`` — no Docker dependency. Speaks the
+line-delimited JSON protocol defined in ``fips/src/control/protocol.rs``:
+
+    request:  {"command": "show_status"}\n
+    response: {"status":"ok","data":{...}}\n
+
+The control socket is a Unix domain socket (default ``/run/fips/control.sock``).
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import socket
+from typing import Any, Optional
+
+log = logging.getLogger(__name__)
+
+DEFAULT_SOCKET = "/run/fips/control.sock"
+# Match the daemon's I/O timeout (control/mod.rs:29).
+DEFAULT_TIMEOUT = 10.0
+
+
+class ControlError(Exception):
+    """Raised when a control-socket operation fails."""
+
+
+class ControlClient:
+    """A thin client over a FIPS daemon's Unix control socket."""
+
+    def __init__(self, socket_path: str = DEFAULT_SOCKET, timeout: float = DEFAULT_TIMEOUT):
+        self.socket_path = socket_path
+        self.timeout = timeout
+
+    # -- low-level ---------------------------------------------------------
+
+    def _send(self, payload: dict[str, Any]) -> dict[str, Any]:
+        """Send one JSON request, read one JSON response line, return it."""
+        data = (json.dumps(payload) + "\n").encode()
+        try:
+            with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
+                s.settimeout(self.timeout)
+                s.connect(self.socket_path)
+                s.sendall(data)
+                s.shutdown(socket.SHUT_WR)
+                chunks: list[bytes] = []
+                while True:
+                    chunk = s.recv(65536)
+                    if not chunk:
+                        break
+                    chunks.append(chunk)
+                raw = b"".join(chunks).decode().strip()
+        except (OSError, socket.timeout) as e:
+            raise ControlError(f"control socket {self.socket_path}: {e}") from e
+
+        if not raw:
+            raise ControlError(f"empty response from {self.socket_path}")
+
+        try:
+            resp = json.loads(raw)
+        except json.JSONDecodeError as e:
+            raise ControlError(f"invalid JSON from {self.socket_path}: {e}") from e
+
+        return resp
+
+    # -- query / command ---------------------------------------------------
+
+    def query(self, command: str) -> dict[str, Any]:
+        """Send a read-only query (e.g. ``show_status``). Returns the ``data`` dict."""
+        resp = self._send({"command": command})
+        if resp.get("status") != "ok":
+            raise ControlError(f"{command}: {resp.get('message', 'unknown error')}")
+        return resp.get("data", {}) or {}
+
+    def command(self, command: str, params: dict[str, Any]) -> dict[str, Any]:
+        """Send a mutating command (e.g. ``connect``) with params. Returns ``data``."""
+        resp = self._send({"command": command, "params": params})
+        if resp.get("status") != "ok":
+            raise ControlError(f"{command}: {resp.get('message', 'unknown error')}")
+        return resp.get("data", {}) or {}
+
+    # -- convenience wrappers ---------------------------------------------
+
+    def show_status(self) -> dict[str, Any]:
+        return self.query("show_status")
+
+    def show_peers(self) -> dict[str, Any]:
+        return self.query("show_peers")
+
+    def show_tree(self) -> dict[str, Any]:
+        return self.query("show_tree")
+
+    def show_bloom(self) -> dict[str, Any]:
+        return self.query("show_bloom")
+
+    def show_routing(self) -> dict[str, Any]:
+        return self.query("show_routing")
+
+    def show_identity_cache(self) -> dict[str, Any]:
+        return self.query("show_identity_cache")
+
+    def connect(self, npub: str, address: str, transport: str) -> dict[str, Any]:
+        return self.command("connect", {"npub": npub, "address": address, "transport": transport})
+
+    def disconnect(self, npub: str) -> dict[str, Any]:
+        return self.command("disconnect", {"npub": npub})
+
+
+def try_query(socket_path: str, command: str, timeout: float = DEFAULT_TIMEOUT) -> Optional[dict[str, Any]]:
+    """Like ``ControlClient.query`` but returns ``None`` on failure instead of raising."""
+    try:
+        return ControlClient(socket_path, timeout).query(command)
+    except ControlError as e:
+        log.debug("query %s on %s failed: %s", command, socket_path, e)
+        return None

mesh-scan/directory.py

@@ -0,0 +1,325 @@
+"""FIPS node directory fetcher.
+
+Ports the Nostr Kind 37195 / ``fips-overlay-v1`` advert query from
+``sovereign_browser/src/nostr_bridge.c:handle_fips_directory_json`` to
+Python using ``websocket-client``.
+
+Each advert event's ``content`` is a JSON object matching
+``fips/src/discovery/nostr/types.rs::OverlayAdvert``:
+
+    {
+      "identifier": "fips-overlay-v1",
+      "version": 1,
+      "endpoints": [{"transport": "udp", "addr": "host:port"}, ...],
+      "signalRelays": ["wss://...", ...]
+    }
+
+Kind 37195 is parameterized-replaceable, so we keep the newest event per
+pubkey (highest ``created_at``).
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+import threading
+import time
+from dataclasses import dataclass, field
+from typing import Any, Optional
+
+import websocket  # websocket-client
+
+log = logging.getLogger(__name__)
+
+# Default FIPS advert relays (fips/src/config/node.rs::default_advert_relays).
+DEFAULT_RELAYS = [
+    "wss://relay.damus.io",
+    "wss://nos.lol",
+    "wss://offchain.pub",
+]
+
+KIND_FIPS_ADVERT = 37195
+ADVERT_IDENTIFIER = "fips-overlay-v1"
+
+
+# ---------------------------------------------------------------------------
+# bech32 (npub) encoding — no external dependency required.
+# Implements BIP-173 / NIP-19 bech32 for the 32-byte x-only pubkey → npub case.
+# ---------------------------------------------------------------------------
+
+_BECH32_CHARSET = "qpzry9x8gf2tvdw0s3jn54khce6mua7l"
+
+
+def _bech32_polymod(values: list[int]) -> int:
+    gen = [0x3B6A57B2, 0x26508E6D, 0x1EA119FA, 0x3D4233DD, 0x2A1462B3]
+    chk = 1
+    for v in values:
+        b = chk >> 25
+        chk = ((chk & 0x1FFFFFF) << 5) ^ v
+        for i in range(5):
+            chk ^= gen[i] if ((b >> i) & 1) else 0
+    return chk
+
+
+def _bech32_hrp_expand(hrp: str) -> list[int]:
+    return [ord(x) >> 5 for x in hrp] + [0] + [ord(x) & 31 for x in hrp]
+
+
+def _bech32_create_checksum(hrp: str, data: list[int]) -> list[int]:
+    values = _bech32_hrp_expand(hrp) + data
+    polymod = _bech32_polymod(values + [0, 0, 0, 0, 0, 0]) ^ 1
+    return [(polymod >> 5 * (5 - i)) & 31 for i in range(6)]
+
+
+def _convertbits(data: bytes, frombits: int, tobits: int, pad: bool = True) -> list[int]:
+    acc = 0
+    bits = 0
+    ret: list[int] = []
+    maxv = (1 << tobits) - 1
+    max_acc = (1 << (frombits + tobits - 1)) - 1
+    for value in data:
+        if value < 0 or (value >> frombits):
+            raise ValueError("invalid value for convertbits")
+        acc = ((acc << frombits) | value) & max_acc
+        bits += frombits
+        while bits >= tobits:
+            bits -= tobits
+            ret.append((acc >> bits) & maxv)
+    if pad:
+        if bits:
+            ret.append((acc << (tobits - bits)) & maxv)
+    elif bits >= frombits or ((acc << (tobits - bits)) & maxv):
+        raise ValueError("invalid padding in convertbits")
+    return ret
+
+
+def encode_npub(pubkey_bytes: bytes) -> str:
+    """Encode a 32-byte x-only pubkey as an npub (bech32) string."""
+    if len(pubkey_bytes) != 32:
+        raise ValueError("npub requires 32 bytes")
+    hrp = "npub"
+    data = _convertbits(pubkey_bytes, 8, 5, pad=True)
+    combined = data + _bech32_create_checksum(hrp, data)
+    return hrp + "1" + "".join(_BECH32_CHARSET[d] for d in combined)
+
+
+def hex_to_npub(pubkey_hex: str) -> str:
+    return encode_npub(bytes.fromhex(pubkey_hex))
+
+
+# ---------------------------------------------------------------------------
+# Directory data model
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class Endpoint:
+    transport: str  # "udp" | "tcp" | "tor"
+    addr: str        # "host:port" or "nat"
+
+
+@dataclass
+class NodeEntry:
+    npub: str
+    pubkey_hex: str
+    created_at: int
+    endpoints: list[Endpoint] = field(default_factory=list)
+    signal_relays: list[str] = field(default_factory=list)
+
+    def first_dialable_endpoint(self) -> Optional[Endpoint]:
+        """First endpoint with a concrete host:port (not 'nat')."""
+        for ep in self.endpoints:
+            if ep.addr.lower() != "nat" and ":" in ep.addr:
+                return ep
+        return None
+
+
+# ---------------------------------------------------------------------------
+# Relay query
+# ---------------------------------------------------------------------------
+
+
+def _build_filter() -> dict[str, Any]:
+    return {
+        "kinds": [KIND_FIPS_ADVERT],
+        "#d": [ADVERT_IDENTIFIER],
+        "limit": 500,
+    }
+
+
+def _query_one_relay(relay_url: str, timeout: float) -> list[dict]:
+    """Open a WS to one relay, send REQ, collect EVENTs until EOSE, return events."""
+    events: list[dict] = []
+    subscription_id = f"meshscan-{int(time.time()*1000)}"
+
+    sock = None
+    try:
+        sock = websocket.create_connection(relay_url, timeout=timeout)
+        req = json.dumps(["REQ", subscription_id, _build_filter()])
+        sock.send(req)
+
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            sock.settimeout(max(0.5, deadline - time.time()))
+            try:
+                raw = sock.recv()
+            except websocket.WebSocketTimeoutException:
+                break
+            if not raw:
+                break
+            try:
+                msg = json.loads(raw)
+            except json.JSONDecodeError:
+                continue
+            if not isinstance(msg, list) or not msg:
+                continue
+            if msg[0] == "EOSE" and len(msg) > 1 and msg[1] == subscription_id:
+                break
+            if msg[0] == "EVENT" and len(msg) > 2:
+                events.append(msg[2])
+    except Exception as e:
+        log.warning("relay %s failed: %s", relay_url, e)
+    finally:
+        if sock is not None:
+            try:
+                sock.send(json.dumps(["CLOSE", subscription_id]))
+                sock.close()
+            except Exception:
+                pass
+    return events
+
+
+def fetch_directory(
+    relays: list[str] | None = None,
+    timeout: float = 12.0,
+) -> list[NodeEntry]:
+    """Fetch the FIPS node directory from advert relays.
+
+    Queries all relays in parallel, dedups by pubkey keeping the newest
+    event (highest ``created_at``), and returns a list of
+    [`NodeEntry`](directory.py).
+    """
+    relay_list = relays or DEFAULT_RELAYS
+    all_events: list[dict] = []
+    threads: list[threading.Thread] = []
+    results: list[list[dict]] = [[] for _ in relay_list]
+
+    def worker(i: int, url: str) -> None:
+        results[i] = _query_one_relay(url, timeout)
+
+    for i, url in enumerate(relay_list):
+        t = threading.Thread(target=worker, args=(i, url), daemon=True)
+        t.start()
+        threads.append(t)
+    for t in threads:
+        t.join(timeout + 2)
+
+    for r in results:
+        all_events.extend(r)
+
+    return _dedup_events(all_events)
+
+
+def _dedup_events(events: list[dict]) -> list[NodeEntry]:
+    newest: dict[str, NodeEntry] = {}
+    for ev in events:
+        pk = ev.get("pubkey")
+        ca = ev.get("created_at", 0)
+        if not isinstance(pk, str):
+            continue
+        try:
+            npub = hex_to_npub(pk)
+        except ValueError:
+            continue
+        existing = newest.get(npub)
+        if existing is not None and ca <= existing.created_at:
+            continue
+
+        content = ev.get("content", "")
+        try:
+            advert = json.loads(content) if isinstance(content, str) else None
+        except json.JSONDecodeError:
+            advert = None
+
+        endpoints: list[Endpoint] = []
+        signal_relays: list[str] = []
+        if isinstance(advert, dict):
+            for ep in advert.get("endpoints", []) or []:
+                if isinstance(ep, dict) and "transport" in ep and "addr" in ep:
+                    endpoints.append(Endpoint(str(ep["transport"]), str(ep["addr"])))
+            for r in advert.get("signalRelays", []) or []:
+                if isinstance(r, str):
+                    signal_relays.append(r)
+
+        newest[npub] = NodeEntry(
+            npub=npub,
+            pubkey_hex=pk,
+            created_at=int(ca) if isinstance(ca, (int, float)) else 0,
+            endpoints=endpoints,
+            signal_relays=signal_relays,
+        )
+
+    return list(newest.values())
+
+
+# ---------------------------------------------------------------------------
+# npub → fd00:: IPv6 derivation (fips/src/identity/{node_addr,address}.rs)
+# ---------------------------------------------------------------------------
+
+
+def npub_to_node_addr(npub: str) -> bytes:
+    """Derive the 16-byte NodeAddr = first 16 bytes of SHA-256(pubkey)."""
+    # Decode bech32 npub → 32-byte pubkey.
+    hrp, data = _bech32_decode(npub)
+    if hrp != "npub":
+        raise ValueError(f"expected npub hrp, got {hrp}")
+    pubkey = _convertbits(data, 5, 8, pad=False)
+    if len(pubkey) != 32:
+        raise ValueError(f"decoded pubkey length {len(pubkey)} != 32")
+    return hashlib.sha256(bytes(pubkey)).digest()[:16]
+
+
+def npub_to_ipv6(npub: str) -> str:
+    """Derive the fd00::/8 IPv6 address for a node from its npub.
+
+    Mirrors ``FipsAddress::from_node_addr``: prepend 0xfd to the first 15
+    bytes of the NodeAddr.
+    """
+    node_addr = npub_to_node_addr(npub)
+    addr_bytes = bytearray(16)
+    addr_bytes[0] = 0xFD
+    addr_bytes[1:16] = node_addr[0:15]
+    # Format as IPv6.
+    parts = [int.from_bytes(addr_bytes[i:i + 2], "big") for i in range(0, 16, 2)]
+    return ":".join(f"{p:x}" for p in parts)
+
+
+def _bech32_decode(s: str) -> tuple[str, list[int]]:
+    """Decode a bech32 string → (hrp, data5)."""
+    if any(ord(x) < 33 or ord(x) > 126 for x in s):
+        raise ValueError("invalid bech32 character")
+    pos = s.rfind("1")
+    if pos < 1 or pos + 7 > len(s):
+        raise ValueError("invalid bech32 structure")
+    hrp = s[:pos]
+    data = [_BECH32_CHARSET.index(x) for x in s[pos + 1:].lower()]
+    # Verify checksum.
+    if _bech32_polymod(_bech32_hrp_expand(hrp) + data) != 1:
+        raise ValueError("invalid bech32 checksum")
+    return hrp, data[:-6]
+
+
+if __name__ == "__main__":
+    import argparse
+    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
+    p = argparse.ArgumentParser(description="Fetch the FIPS node directory")
+    p.add_argument("--relay", action="append", help="relay URL (repeatable)")
+    p.add_argument("--timeout", type=float, default=12.0)
+    args = p.parse_args()
+    nodes = fetch_directory(args.relay, args.timeout)
+    print(f"# {len(nodes)} nodes")
+    for n in nodes:
+        ep = n.first_dialable_endpoint()
+        ep_str = f"{ep.transport}:{ep.addr}" if ep else "none"
+        print(f"{n.npub}  created_at={n.created_at}  ep={ep_str}")

mesh-scan/fipsd.py

@@ -0,0 +1,476 @@
+"""Ephemeral FIPS daemon lifecycle manager.
+
+Generates an isolated ``fips.yaml``, spawns ``fipsd --config <path>`` as a
+subprocess **inside its own network namespace**, polls the control socket
+until the daemon is ready, and tears everything down cleanly (process +
+netns + temp dir).
+
+Why network namespaces: the FIPS daemon hardcodes a ``fd00::/8`` route on
+its TUN device (see ``fips/src/upper/tun.rs``). On a host that already runs
+a long-lived FIPS daemon (``fips0``), that route already exists, so a
+second daemon's TUN init fails with ``File exists (os error 17)``. Running
+each ephemeral daemon in its own netns gives it a fresh routing table where
+``fd00::/8`` is free. The control socket is a Unix domain socket in ``/tmp``
+(shared filesystem), so it remains reachable from the host.
+
+Each ephemeral daemon gets:
+  - a unique temp config dir under ``/tmp/mesh-scan-<tag>-<pid>/``
+  - a unique control socket path inside that dir
+  - a unique network namespace (``fips-scan-<n>``) — or a shared one for
+    the self-test (so two daemons can peer over loopback)
+  - unique UDP/TCP/DNS ports auto-allocated from the ephemeral range
+
+``fipsd`` needs ``CAP_NET_ADMIN`` (for TUN + netns), so the spawn goes
+through ``sudo`` unless we already have the capability.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import shutil
+import signal
+import socket
+import subprocess
+import tempfile
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from control import ControlClient, ControlError
+
+log = logging.getLogger(__name__)
+
+FIPS_BINARY = os.environ.get("FIPS_BINARY", "fips")
+DEFAULT_NETNS_PREFIX = "fips-scan-"
+STARTUP_TIMEOUT = 15.0
+STOP_TIMEOUT = 5.0
+
+
+def _have_net_admin() -> bool:
+    """True if this process can create a TUN device / netns without sudo."""
+    try:
+        with open("/proc/self/status") as f:
+            for line in f:
+                if line.startswith("CapEff:"):
+                    eff = int(line.split()[1], 16)
+                    # CAP_NET_ADMIN is bit 12.
+                    return bool(eff & (1 << 12))
+    except OSError:
+        pass
+    return False
+
+
+def _free_udp_port() -> int:
+    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
+        s.bind(("0.0.0.0", 0))
+        return s.getsockname()[1]
+
+
+def _free_tcp_port() -> int:
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.bind(("0.0.0.0", 0))
+        return s.getsockname()[1]
+
+
+def _next_netns_name(index: int) -> str:
+    return f"{DEFAULT_NETNS_PREFIX}{index}"
+
+
+# ---------------------------------------------------------------------------
+# Network namespace management
+# ---------------------------------------------------------------------------
+
+
+def _run(cmd: list[str], use_sudo: bool, timeout: float = 5.0) -> subprocess.CompletedProcess:
+    full = (["sudo", "-n", *cmd] if use_sudo else cmd)
+    return subprocess.run(full, capture_output=True, text=True, timeout=timeout)
+
+
+def _run_quiet(cmd: list[str], use_sudo: bool, timeout: float = 5.0) -> None:
+    """Run a command, log failures at debug level only."""
+    try:
+        r = _run(cmd, use_sudo, timeout)
+        if r.returncode != 0 and r.stderr.strip():
+            log.debug("cmd failed: %s -> %s", " ".join(cmd), r.stderr.strip())
+    except Exception as e:
+        log.debug("cmd exception: %s -> %s", " ".join(cmd), e)
+
+
+def _host_default_iface() -> str:
+    """Return the host's default-route interface name, or 'eth0' fallback."""
+    try:
+        r = subprocess.run(["ip", "route", "show", "default"],
+                           capture_output=True, text=True, timeout=3)
+        for line in r.stdout.splitlines():
+            if "dev" in line:
+                return line.split("dev")[1].split()[0]
+    except Exception:
+        pass
+    return "eth0"
+
+
+# Subnet counter for veth pairs (10.250.N.0/30 — 2 usable hosts).
+_veth_counter = [0]
+
+
+def _next_veth_subnet() -> tuple[str, str, str]:
+    """Return (host_addr, netns_addr, subnet) for the next veth pair."""
+    n = _veth_counter[0]
+    _veth_counter[0] += 1
+    # 10.250.N.1 (host side) / 10.250.N.2 (netns side), /30.
+    return f"10.250.{n}.1/30", f"10.250.{n}.2/30", f"10.250.{n}.0/30"
+
+
+def create_netns(name: str, use_sudo: bool | None = None) -> dict:
+    """Create a network namespace with loopback + a veth pair for outbound NAT.
+
+    The netns gets a veth link to the host with a 10.250.N.0/30 subnet and a
+    default route via the host side. The host masquerades traffic from the
+    netns out its default interface, so the ephemeral FIPS daemon inside can
+    reach public seed nodes over UDP/TCP.
+
+    Returns a dict with the veth interface names and subnet, for cleanup.
+    """
+    if use_sudo is None:
+        use_sudo = not _have_net_admin()
+
+    # Linux interface names are limited to 15 chars. Use a short prefix
+    # derived from the netns index instead of the full netns name.
+    n = _veth_counter[0] - 1  # _next_veth_subnet already incremented
+    info = {"veth_host": f"fsvh{n}", "veth_ns": f"fsvn{n}", "subnet": None}
+
+    try:
+        _run(["ip", "netns", "add", name], use_sudo)
+    except subprocess.CalledProcessError as e:
+        log.warning("ip netns add %s failed: %s", name, e.stderr.strip())
+        return info
+
+    # Loopback up so ::1 binds work (DNS responder binds ::1).
+    _run_quiet(["ip", "netns", "exec", name, "ip", "link", "set", "lo", "up"], use_sudo)
+
+    # Create a veth pair: host end stays on the host, ns end moves into netns.
+    host_addr, ns_addr, subnet = _next_veth_subnet()
+    info["subnet"] = subnet
+    _run_quiet(["ip", "link", "add", info["veth_host"], "type", "veth",
+                "peer", "name", info["veth_ns"]], use_sudo)
+    _run_quiet(["ip", "link", "set", info["veth_ns"], "netns", name], use_sudo)
+    _run_quiet(["ip", "addr", "add", host_addr, "dev", info["veth_host"]], use_sudo)
+    _run_quiet(["ip", "link", "set", info["veth_host"], "up"], use_sudo)
+    _run_quiet(["ip", "netns", "exec", name, "ip", "addr", "add", ns_addr,
+                "dev", info["veth_ns"]], use_sudo)
+    _run_quiet(["ip", "netns", "exec", name, "ip", "link", "set", info["veth_ns"], "up"], use_sudo)
+    # Default route in the netns via the host side of the veth.
+    _run_quiet(["ip", "netns", "exec", name, "ip", "route", "add", "default",
+                "via", host_addr.split("/")[0], "dev", info["veth_ns"]], use_sudo)
+
+    # Enable IP forwarding + masquerade on the host for the netns subnet.
+    _run_quiet(["sysctl", "-w", "net.ipv4.ip_forward=1"], use_sudo)
+    out_iface = _host_default_iface()
+    _run_quiet(["iptables", "-t", "nat", "-A", "POSTROUTING", "-s", subnet,
+                "-o", out_iface, "-j", "MASQUERADE"], use_sudo)
+    _run_quiet(["iptables", "-A", "FORWARD", "-i", info["veth_host"],
+                "-o", out_iface, "-j", "ACCEPT"], use_sudo)
+    _run_quiet(["iptables", "-A", "FORWARD", "-i", out_iface,
+                "-o", info["veth_host"], "-j", "ACCEPT"], use_sudo)
+
+    log.info("netns %s: veth %s<->%s subnet %s nat via %s",
+             name, info["veth_host"], info["veth_ns"], subnet, out_iface)
+    return info
+
+
+def delete_netns(name: str, use_sudo: bool | None = None,
+                 info: dict | None = None) -> None:
+    """Delete a network namespace and its veth / iptables rules."""
+    if use_sudo is None:
+        use_sudo = not _have_net_admin()
+    # Deleting the netns also destroys the ns-end of the veth; the host-end
+    # is removed below. iptables rules referencing the veth are cleaned up
+    # best-effort.
+    if info:
+        subnet = info.get("subnet")
+        out_iface = _host_default_iface()
+        if subnet:
+            _run_quiet(["iptables", "-t", "nat", "-D", "POSTROUTING", "-s", subnet,
+                        "-o", out_iface, "-j", "MASQUERADE"], use_sudo)
+            _run_quiet(["iptables", "-D", "FORWARD", "-i", info.get("veth_host", ""),
+                        "-o", out_iface, "-j", "ACCEPT"], use_sudo)
+            _run_quiet(["iptables", "-D", "FORWARD", "-i", out_iface,
+                        "-o", info.get("veth_host", ""), "-j", "ACCEPT"], use_sudo)
+        _run_quiet(["ip", "link", "del", info.get("veth_host", "")], use_sudo)
+    try:
+        _run(["ip", "netns", "del", name], use_sudo)
+    except Exception as e:
+        log.debug("ip netns del %s: %s", name, e)
+
+
+def exec_in_netns(name: str, cmd: list[str], use_sudo: bool | None = None,
+                  **kwargs) -> subprocess.CompletedProcess:
+    """Run a command inside a network namespace."""
+    if use_sudo is None:
+        use_sudo = not _have_net_admin()
+    full = (["sudo", "-n"] if use_sudo else []) + ["ip", "netns", "exec", name] + cmd
+    return subprocess.run(full, **kwargs)
+
+
+@dataclass
+class EphemeralDaemon:
+    """A running ephemeral fipsd process + its resources."""
+
+    config_dir: Path
+    config_path: Path
+    socket_path: str
+    netns: str
+    owns_netns: bool
+    udp_port: int
+    tcp_port: int
+    dns_port: int
+    process: subprocess.Popen
+    used_sudo: bool
+    netns_info: dict | None = None
+    client: ControlClient = field(init=False)
+
+    def __post_init__(self) -> None:
+        self.client = ControlClient(self.socket_path)
+
+    # -- lifecycle ---------------------------------------------------------
+
+    def wait_ready(self, timeout: float = STARTUP_TIMEOUT) -> None:
+        """Poll the control socket until ``show_status`` responds."""
+        deadline = time.time() + timeout
+        last_err = ""
+        while time.time() < deadline:
+            if self.process.poll() is not None:
+                raise RuntimeError(f"fipsd exited early (code={self.process.returncode})")
+            try:
+                status = self.client.show_status()
+                if status.get("state") in ("running", "initialized", "ready"):
+                    if status.get("tun_state") == "active":
+                        return
+                    # TUN not yet active; keep waiting briefly.
+            except ControlError as e:
+                last_err = str(e)
+            time.sleep(0.3)
+        raise RuntimeError(f"fipsd did not become ready in {timeout}s: {last_err}")
+
+    def stop(self) -> None:
+        """Disconnect peers, SIGTERM, then SIGKILL if needed; clean up netns + dir."""
+        try:
+            self.client.disconnect_all_peers()
+        except Exception as e:
+            log.debug("disconnect-all failed: %s", e)
+        self._terminate()
+        if self.owns_netns:
+            delete_netns(self.netns, self.used_sudo, info=self.netns_info)
+        self._cleanup_dir()
+
+    def _terminate(self) -> None:
+        if self.process.poll() is not None:
+            return
+        try:
+            self.process.send_signal(signal.SIGTERM)
+        except Exception:
+            pass
+        try:
+            self.process.wait(timeout=STOP_TIMEOUT)
+        except subprocess.TimeoutExpired:
+            log.warning("fipsd in netns %s did not exit on SIGTERM, SIGKILL", self.netns)
+            try:
+                self.process.kill()
+                self.process.wait(timeout=2)
+            except Exception:
+                pass
+
+    def _cleanup_dir(self) -> None:
+        try:
+            shutil.rmtree(self.config_dir, ignore_errors=True)
+        except Exception:
+            pass
+
+    # -- reachability helper ----------------------------------------------
+
+    def run_in_netns(self, cmd: list[str], timeout: float = 30.0, **kwargs) -> subprocess.CompletedProcess:
+        """Run a command (e.g. ping6) inside this daemon's netns.
+
+        Defaults to capturing stdout/stderr as text with a 30s hard timeout
+        (callers can override). The timeout prevents hung probes from
+        blocking the scan indefinitely.
+        """
+        kwargs.setdefault("capture_output", True)
+        kwargs.setdefault("text", True)
+        kwargs.setdefault("timeout", timeout)
+        return exec_in_netns(self.netns, cmd, self.used_sudo, **kwargs)
+
+    # -- context manager ---------------------------------------------------
+
+    def __enter__(self) -> "EphemeralDaemon":
+        return self
+
+    def __exit__(self, *exc) -> None:
+        self.stop()
+
+
+# ---------------------------------------------------------------------------
+# Spawn
+# ---------------------------------------------------------------------------
+
+
+def _build_config(
+    socket_path: str,
+    udp_port: int,
+    tcp_port: int,
+    dns_port: int,
+) -> str:
+    # Inside a netns we can use the canonical "fips0" TUN name — no collision.
+    # Mirrors the structure in sovereign_browser/src/net_services.c:349
+    # and fips/src/config/{node,transport,upper/config}.rs.
+    return (
+        "node:\n"
+        "  identity:\n"
+        "    persistent: false\n"
+        f"  control:\n"
+        f"    socket_path: \"{socket_path}\"\n"
+        "  log_level: warn\n"
+        # Enable Nostr discovery so the ephemeral node learns about other
+        # mesh nodes from the advert relays (not just its direct peer).
+        # Without this, the identity cache stays at 1 (only the seed) and
+        # find_next_hop returns None for every non-peer target.
+        # policy: open lets it consider adverts for non-configured peers.
+        # advertise: false — the scanner node is ephemeral, don't publish.
+        "  discovery:\n"
+        "    nostr:\n"
+        "      enabled: true\n"
+        "      advertise: false\n"
+        "      policy: open\n"
+        "      advert_relays:\n"
+        "        - \"wss://relay.damus.io\"\n"
+        "        - \"wss://nos.lol\"\n"
+        "        - \"wss://offchain.pub\"\n"
+        "tun:\n"
+        "  enabled: true\n"
+        "  name: \"fips0\"\n"
+        "  mtu: 1280\n"
+        "dns:\n"
+        "  enabled: true\n"
+        "  bind_addr: \"::1\"\n"
+        f"  port: {dns_port}\n"
+        "transports:\n"
+        "  udp:\n"
+        f"    bind_addr: \"0.0.0.0:{udp_port}\"\n"
+        "  tcp:\n"
+        f"    bind_addr: \"0.0.0.0:{tcp_port}\"\n"
+        "peers: []\n"
+    )
+
+
+def spawn_daemon(tag: str = "scan", index: int = 0,
+                 netns: str | None = None) -> EphemeralDaemon:
+    """Create the config dir + netns, write fips.yaml, spawn fipsd, return handle.
+
+    If ``netns`` is given, the daemon is spawned in that (shared) namespace
+    and does not own it (the caller is responsible for deleting it). This is
+    used by the self-test to put two daemons in one netns so they can peer
+    over loopback.
+
+    The daemon is *not* waited on for readiness — call ``wait_ready()``.
+    """
+    use_sudo = not _have_net_admin()
+    config_dir = Path(tempfile.mkdtemp(prefix=f"mesh-scan-{tag}-"))
+    socket_path = str(config_dir / "control.sock")
+    udp_port = _free_udp_port()
+    tcp_port = _free_tcp_port()
+    dns_port = _free_udp_port()
+
+    owns_netns = netns is None
+    netns_info = None
+    if owns_netns:
+        netns = _next_netns_name(index)
+        netns_info = create_netns(netns, use_sudo)
+    else:
+        # Shared netns (self-test): ensure it exists with NAT. The caller
+        # is responsible for creating/deleting it, but if they passed a
+        # name without creating it, create_netns is idempotent enough.
+        pass
+
+    config = _build_config(socket_path, udp_port, tcp_port, dns_port)
+    config_path = config_dir / "fips.yaml"
+    config_path.write_text(config)
+    os.chmod(config_path, 0o600)
+
+    # Spawn fips inside the netns.
+    fips_argv = [FIPS_BINARY, "--config", str(config_path)]
+    if use_sudo:
+        spawn_argv = ["sudo", "-n", "ip", "netns", "exec", netns, *fips_argv]
+    else:
+        spawn_argv = ["ip", "netns", "exec", netns, *fips_argv]
+
+    log.info("spawning fipsd: netns=%s udp=%d tcp=%d dns=%d sudo=%s",
+             netns, udp_port, tcp_port, dns_port, use_sudo)
+    proc = subprocess.Popen(
+        spawn_argv,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+        start_new_session=True,
+    )
+
+    return EphemeralDaemon(
+        config_dir=config_dir,
+        config_path=config_path,
+        socket_path=socket_path,
+        netns=netns,
+        owns_netns=owns_netns,
+        netns_info=netns_info,
+        udp_port=udp_port,
+        tcp_port=tcp_port,
+        dns_port=dns_port,
+        process=proc,
+        used_sudo=use_sudo,
+    )
+
+
+# ---------------------------------------------------------------------------
+# Stale-namespace cleanup (--cleanup-stale)
+# ---------------------------------------------------------------------------
+
+
+def cleanup_stale_netns(prefix: str = DEFAULT_NETNS_PREFIX) -> int:
+    """Remove any network namespaces whose name starts with ``prefix``."""
+    use_sudo = not _have_net_admin()
+    try:
+        out = _run(["ip", "netns", "list"], use_sudo).stdout
+    except Exception as e:
+        log.warning("could not list netns: %s", e)
+        return 0
+    count = 0
+    for line in out.splitlines():
+        name = line.split()[0] if line.strip() else ""
+        if name.startswith(prefix):
+            try:
+                _run(["ip", "netns", "del", name], use_sudo)
+                log.info("removed stale netns %s", name)
+                count += 1
+            except Exception as e:
+                log.warning("failed to remove stale netns %s: %s", name, e)
+    return count
+
+
+# Back-compat alias used by mesh_scan.py's --cleanup-stale flag.
+cleanup_stale_tun_devices = cleanup_stale_netns
+
+
+# Extend ControlClient with a helper to disconnect all peers (used on stop).
+def _disconnect_all_peers(self: ControlClient) -> None:
+    try:
+        peers = self.show_peers().get("peers", []) or []
+    except ControlError:
+        return
+    for p in peers:
+        npub = p.get("npub")
+        if npub:
+            try:
+                self.disconnect(npub)
+            except ControlError:
+                pass
+
+
+ControlClient.disconnect_all_peers = _disconnect_all_peers  # type: ignore[attr-defined]

mesh-scan/mesh-scan-out/scan.log

@@ -0,0 +1,133 @@
+09:05:44 INFO mesh_scan: fetching FIPS node directory
+09:05:44 WARNING directory: relay wss://relay.damus.io failed: Handshake status 503 Service Unavailable -+-+- {'date': 'Sun, 19 Jul 2026 13:05:44 GMT', 'content-length': '0', 'connection': 'keep-alive', 'cache-control': 'private, no-store', 'cf-cache-status': 'DYNAMIC', 'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v4?s=zOV7cTz6nXiwb%2BWuP2D63%2FzvLBDigW6ZZKprGWy%2F5mj5UkpMtxqpGUKA74Y%2BB1kOBXAuEbSRZ0mWD4kSxf0ru1i%2BxtNoL%2FZnLvgJxfwljqb3e67LhBW3BAgGuWyN87VvQw%3D%3D"}],"group":"cf-nel","max_age":604800}', 'nel': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'server': 'cloudflare', 'cf-ray': 'a1d9f07dfe93fe29-IAD', 'alt-svc': 'h3=":443"; ma=86400', 'server-timing': 'cfL4;desc="?proto=TCP&rtt=59321&min_rtt=58786&rtt_var=17462&sent=5&recv=8&lost=0&retrans=0&sent_bytes=3862&recv_bytes=1843&delivery_rate=65821&cwnd=53&unsent_bytes=0&cid=04afbf6a77de3946&ts=319&x=0"'} -+-+- b''
+09:05:46 INFO mesh_scan: seed npub1jfv63kg5deq: 543 target(s)
+09:05:46 INFO mesh_scan: === seed 1/27: npub1jfv63kg5deqhrt7sn6s ===
+Fetched 544 nodes from the directory.
+Selected 27 seed(s): ['npub1jfv63kg5deq', 'npub13zg8jlslmr4', 'npub1sd826ue7tz8', 'npub1h4e3dks08up', 'npub18ljnndsess6', 'npub1veq2q3xryp2', 'npub1ckskwukdlj5', 'npub1k3aerhf3f4e', 'npub1t5fgt7ssrgk', 'npub136yqae6na68', 'npub14l9ldvnask3', 'npub10yymypqzz36', 'npub1v5fmkz5s8yg', 'npub1tyuy5r927es', 'npub14y8l4yxn4e3', 'npub1g4uu79fww44', 'npub1netvn8qvfwt', 'npub1nlukdnzu2d3', 'npub1qmc3cvfz0yu', 'npub10yffd020a4a', 'npub1mqelkzqp465', 'npub1vmv2ctysqtj', 'npub1u0z26dc4qen', 'npub17lpmzulpc98', 'npub1260n42s06vz', 'npub1gd7ye2qp2lp', 'npub1xf57cw7r2l4']
+
+[09:05:46] seed 1/27: npub1jfv63kg5deqhrt7sn6s  (543 targets)
+09:05:46 INFO fipsd: netns fips-scan-0: veth fsvh-1<->fsvn-1 subnet 10.250.0.0/30 nat via eth0
+09:05:46 INFO fipsd: spawning fipsd: netns=fips-scan-0 udp=54391 tcp=37245 dns=32797 sudo=True
+09:05:48 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-0
+  [09:05:48] daemon up (2.0s)
+09:05:48 INFO mesh_scan: peering with seed npub1jfv63kg5deq at 45.76.169.205:51820/udp
+09:06:03 WARNING mesh_scan: seed did not connect within 15s
+  [09:06:03] peering FAILED (15.0s); skipping targets
+09:06:03 WARNING mesh_scan: could not peer with seed npub1jfv63kg5deq; skipping targets
+[09:06:03] tally: 0 connected, 1 failed, 0 skipped | 1/27 done | elapsed 17s
+09:06:03 INFO mesh_scan: seed npub13zg8jlslmr4: 543 target(s)
+09:06:03 INFO mesh_scan: === seed 2/27: npub13zg8jlslmr4lzwm95j9 ===
+
+[09:06:03] seed 2/27: npub13zg8jlslmr4lzwm95j9  (543 targets)
+09:06:03 INFO fipsd: netns fips-scan-1: veth fsvh0<->fsvn0 subnet 10.250.1.0/30 nat via eth0
+09:06:03 INFO fipsd: spawning fipsd: netns=fips-scan-1 udp=54155 tcp=52271 dns=47813 sudo=True
+09:06:05 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-1
+  [09:06:05] daemon up (2.1s)
+09:06:05 INFO mesh_scan: peering with seed npub13zg8jlslmr4 at 178.39.14.231:51820/udp
+09:06:20 WARNING mesh_scan: seed did not connect within 15s
+  [09:06:20] peering FAILED (15.0s); skipping targets
+09:06:20 WARNING mesh_scan: could not peer with seed npub13zg8jlslmr4; skipping targets
+[09:06:20] tally: 0 connected, 2 failed, 0 skipped | 2/27 done | elapsed 34s
+09:06:20 INFO mesh_scan: seed npub1sd826ue7tz8: 543 target(s)
+09:06:20 INFO mesh_scan: === seed 3/27: npub1sd826ue7tz8usd2yt90 ===
+
+[09:06:20] seed 3/27: npub1sd826ue7tz8usd2yt90  (543 targets)
+09:06:20 INFO fipsd: netns fips-scan-2: veth fsvh1<->fsvn1 subnet 10.250.2.0/30 nat via eth0
+09:06:20 INFO fipsd: spawning fipsd: netns=fips-scan-2 udp=39730 tcp=56211 dns=56864 sudo=True
+09:06:22 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-2
+  [09:06:22] daemon up (2.0s)
+09:06:22 INFO mesh_scan: peering with seed npub1sd826ue7tz8 at 65.109.48.91:51821/udp
+09:06:37 WARNING mesh_scan: seed did not connect within 15s
+  [09:06:37] peering FAILED (15.0s); skipping targets
+09:06:37 WARNING mesh_scan: could not peer with seed npub1sd826ue7tz8; skipping targets
+[09:06:37] tally: 0 connected, 3 failed, 0 skipped | 3/27 done | elapsed 52s
+09:06:37 INFO mesh_scan: seed npub1h4e3dks08up: 543 target(s)
+09:06:37 INFO mesh_scan: === seed 4/27: npub1h4e3dks08up67cdtuhl ===
+
+[09:06:37] seed 4/27: npub1h4e3dks08up67cdtuhl  (543 targets)
+09:06:38 INFO fipsd: netns fips-scan-3: veth fsvh2<->fsvn2 subnet 10.250.3.0/30 nat via eth0
+09:06:38 INFO fipsd: spawning fipsd: netns=fips-scan-3 udp=39046 tcp=49811 dns=33881 sudo=True
+09:06:39 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-3
+  [09:06:39] daemon up (2.0s)
+09:06:39 INFO mesh_scan: peering with seed npub1h4e3dks08up at 167.17.180.42:51820/udp
+09:06:54 WARNING mesh_scan: seed did not connect within 15s
+  [09:06:54] peering FAILED (15.0s); skipping targets
+09:06:54 WARNING mesh_scan: could not peer with seed npub1h4e3dks08up; skipping targets
+[09:06:55] tally: 0 connected, 4 failed, 0 skipped | 4/27 done | elapsed 69s
+09:06:55 INFO mesh_scan: seed npub18ljnndsess6: 543 target(s)
+09:06:55 INFO mesh_scan: === seed 5/27: npub18ljnndsess6kvkpzapq ===
+
+[09:06:55] seed 5/27: npub18ljnndsess6kvkpzapq  (543 targets)
+09:06:55 INFO fipsd: netns fips-scan-4: veth fsvh3<->fsvn3 subnet 10.250.4.0/30 nat via eth0
+09:06:55 INFO fipsd: spawning fipsd: netns=fips-scan-4 udp=47487 tcp=44251 dns=34104 sudo=True
+09:06:57 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-4
+  [09:06:57] daemon up (2.0s)
+09:06:57 INFO mesh_scan: peering with seed npub18ljnndsess6 at 195.86.38.35:51820/udp
+09:07:12 WARNING mesh_scan: seed did not connect within 15s
+  [09:07:12] peering FAILED (15.0s); skipping targets
+09:07:12 WARNING mesh_scan: could not peer with seed npub18ljnndsess6; skipping targets
+[09:07:12] tally: 0 connected, 5 failed, 0 skipped | 5/27 done | elapsed 86s
+09:07:12 INFO mesh_scan: seed npub1veq2q3xryp2: 543 target(s)
+09:07:12 INFO mesh_scan: === seed 6/27: npub1veq2q3xryp2em5angt5 ===
+
+[09:07:12] seed 6/27: npub1veq2q3xryp2em5angt5  (543 targets)
+09:07:12 INFO fipsd: netns fips-scan-5: veth fsvh4<->fsvn4 subnet 10.250.5.0/30 nat via eth0
+09:07:12 INFO fipsd: spawning fipsd: netns=fips-scan-5 udp=36420 tcp=56923 dns=38906 sudo=True
+09:07:14 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-5
+  [09:07:14] daemon up (2.0s)
+09:07:14 INFO mesh_scan: peering with seed npub1veq2q3xryp2 at 8.222.128.129:51820/udp
+09:07:29 WARNING mesh_scan: seed did not connect within 15s
+  [09:07:29] peering FAILED (15.0s); skipping targets
+09:07:29 WARNING mesh_scan: could not peer with seed npub1veq2q3xryp2; skipping targets
+[09:07:29] tally: 0 connected, 6 failed, 0 skipped | 6/27 done | elapsed 103s
+09:07:29 INFO mesh_scan: seed npub1ckskwukdlj5: 543 target(s)
+09:07:29 INFO mesh_scan: === seed 7/27: npub1ckskwukdlj5y9klezc3 ===
+
+[09:07:29] seed 7/27: npub1ckskwukdlj5y9klezc3  (543 targets)
+09:07:29 INFO fipsd: netns fips-scan-6: veth fsvh5<->fsvn5 subnet 10.250.6.0/30 nat via eth0
+09:07:29 INFO fipsd: spawning fipsd: netns=fips-scan-6 udp=36740 tcp=45813 dns=37400 sudo=True
+09:07:31 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-6
+  [09:07:31] daemon up (2.0s)
+09:07:31 INFO mesh_scan: peering with seed npub1ckskwukdlj5 at 107.10.122.197:48298/udp
+09:07:46 WARNING mesh_scan: seed did not connect within 15s
+  [09:07:46] peering FAILED (15.0s); skipping targets
+09:07:46 WARNING mesh_scan: could not peer with seed npub1ckskwukdlj5; skipping targets
+[09:07:46] tally: 0 connected, 7 failed, 0 skipped | 7/27 done | elapsed 121s
+09:07:46 INFO mesh_scan: seed npub1k3aerhf3f4e: 543 target(s)
+09:07:46 INFO mesh_scan: === seed 8/27: npub1k3aerhf3f4ed9mrlu2z ===
+
+[09:07:46] seed 8/27: npub1k3aerhf3f4ed9mrlu2z  (543 targets)
+09:07:46 INFO fipsd: netns fips-scan-7: veth fsvh6<->fsvn6 subnet 10.250.7.0/30 nat via eth0
+09:07:46 INFO fipsd: spawning fipsd: netns=fips-scan-7 udp=49272 tcp=51565 dns=48209 sudo=True
+09:07:48 INFO mesh_scan: ephemeral daemon ready: netns=fips-scan-7
+  [09:07:48] daemon up (2.1s)
+09:07:48 INFO mesh_scan: peering with seed npub1k3aerhf3f4e at 194.191.252.108:2121/udp
+09:07:49 INFO mesh_scan: seed peer connected
+09:08:49 INFO mesh_scan: converged in 60.0s
+09:08:49 INFO mesh_scan: seed state: peers=5 depth=2 root=(none) known=26
+  [09:08:49] peered ok | peers=5 depth=2 known=26 | peer 60.5s + converge 60.0s
+09:08:51 INFO mesh_scan:   target npub18t37rm05657: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:08:53 INFO mesh_scan:   target npub1ke6ah04f3hl: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:08:55 INFO mesh_scan:   target npub1gzzmfvhrls4: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:09:33 INFO mesh_scan:   target npub1ggr79fd0q8l: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:10:11 INFO mesh_scan:   target npub1422d47s46fy: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:10:50 INFO mesh_scan:   target npub160flhw62unc: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:11:28 INFO mesh_scan:   target npub1xk6cyq0fwjd: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:12:06 INFO mesh_scan:   target npub17kta33jk7yd: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:12:44 INFO mesh_scan:   target npub1swqqxn5zukw: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:13:22 INFO mesh_scan:   target npub1e5upq7qkpxs: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:14:00 INFO mesh_scan:   target npub1prrax6290k5: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:14:38 INFO mesh_scan:   target npub1x6pjnrkt6s8: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:15:17 INFO mesh_scan:   target npub1zt6qrdxad07: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:15:55 INFO mesh_scan:   target npub14l59jjzm9ql: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:16:33 INFO mesh_scan:   target npub1cgrzfakmccs: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:17:11 INFO mesh_scan:   target npub13jjytpje4r5: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:17:49 INFO mesh_scan:   target npub1pykqq7kujz4: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:18:27 INFO mesh_scan:   target npub1p99dld9xgpg: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:19:05 INFO mesh_scan:   target npub1fnlavfkk5sa: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:19:44 INFO mesh_scan:   target npub1jfv63kg5deq: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:20:22 INFO mesh_scan:   target npub1tctqu58prak: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:21:00 INFO mesh_scan:   target npub1utkzpaf8lp7: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:21:38 INFO mesh_scan:   target npub1mv72d55k8cq: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:22:16 INFO mesh_scan:   target npub1996gsgmv2gj: FAIL rtt=- method=ping6+tcp cache=False bloom=False
+09:22:54 INFO mesh_scan:   target npub1tnm6dw2rthv: FAIL rtt=- method=ping6+tcp cache=False bloom=False

mesh-scan/mesh_scan.py

@@ -0,0 +1,726 @@
+#!/usr/bin/env python3
+"""FIPS Mesh Reachability Scanner.
+
+Orchestrates the per-seed scan lifecycle described in
+[`plans/mesh-reachability-scanner.md`](../plans/mesh-reachability-scanner.md):
+
+  directory → seed selection → per-seed ephemeral daemon → peer with seed →
+  converge → record diagnostics → per-target reachability check → output
+  matrix + per-pair diagnostics JSON + summary.
+
+Usage:
+    python3 mesh_scan.py --seeds-sample 3 --converge-secs 30
+    python3 mesh_scan.py --seeds npub1abc,npub1def --targets npub1ghi
+    python3 mesh_scan.py --self-test          # quick 2-node smoke test
+    python3 mesh_scan.py --cleanup-stale
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+import logging
+import os
+import random
+import sys
+import time
+from dataclasses import dataclass, field, asdict
+from pathlib import Path
+from typing import Any
+
+# Allow running both as a module and as a script.
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+import control  # noqa: E402
+from control import ControlClient, ControlError  # noqa: E402
+import directory  # noqa: E402
+from directory import NodeEntry, fetch_directory, npub_to_ipv6  # noqa: E402
+import fipsd  # noqa: E402
+from fipsd import EphemeralDaemon, spawn_daemon, cleanup_stale_tun_devices  # noqa: E402
+import reachability  # noqa: E402
+from reachability import check as reach_check  # noqa: E402
+
+log = logging.getLogger("mesh_scan")
+
+
+# ---------------------------------------------------------------------------
+# Data records
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class SeedState:
+    """Diagnostics recorded about the ephemeral node after convergence."""
+    npub: str
+    peer_count: int = 0
+    tree_state: str = ""
+    estimated_mesh_size: int = 0
+    seed_depth: int | None = None
+    seed_parent: str = ""
+    mesh_root: str = ""
+    known_npubs: list[str] = field(default_factory=list)
+    bloom_peer_count: int = 0
+
+
+@dataclass
+class PairResult:
+    seed: str
+    target: str
+    reachable: bool
+    rtt_ms: float | None
+    loss_pct: int | None
+    method: str
+    target_in_identity_cache: bool
+    target_in_seed_bloom: bool
+    cached_coords: bool
+    tree_distance: int | None
+    seed_depth: int | None
+    seed_parent: str
+    mesh_root: str
+    converge_secs: float
+    detail: str = ""
+
+
+# ---------------------------------------------------------------------------
+# Seed / target selection
+# ---------------------------------------------------------------------------
+
+
+def _ts() -> str:
+    """Current wall-clock time as HH:MM:SS for progress prints."""
+    return time.strftime("%H:%M:%S")
+
+
+def select_seeds(nodes: list[NodeEntry], args: argparse.Namespace) -> list[NodeEntry]:
+    by_npub = {n.npub: n for n in nodes}
+    # For automatic selection (all/sample/near-root), only consider seeds
+    # with a dialable endpoint — nat-only nodes can't be peered with
+    # directly, so scanning them just wastes time.
+    dialable = [n for n in nodes if n.first_dialable_endpoint() is not None]
+
+    if args.seeds == "all":
+        return dialable
+
+    if args.seeds and args.seeds != "all":
+        chosen: list[NodeEntry] = []
+        for npub in args.seeds.split(","):
+            npub = npub.strip()
+            if npub in by_npub:
+                chosen.append(by_npub[npub])
+            else:
+                log.warning("seed %s not in directory, skipping", npub)
+        return chosen
+
+    if args.seeds_near_root:
+        return sorted(dialable, key=lambda n: npub_to_ipv6(n.npub))[:args.seeds_near_root]
+
+    if args.seeds_sample:
+        k = min(args.seeds_sample, len(dialable))
+        return random.sample(dialable, k) if k > 0 else []
+
+    # Default: a small sample of 5 from dialable nodes.
+    k = min(5, len(dialable))
+    return random.sample(dialable, k) if k > 0 else []
+
+
+def select_targets(nodes: list[NodeEntry], seed: NodeEntry, args: argparse.Namespace) -> list[NodeEntry]:
+    if args.targets:
+        wanted = {t.strip() for t in args.targets.split(",")}
+        return [n for n in nodes if n.npub in wanted and n.npub != seed.npub]
+    candidates = [n for n in nodes if n.npub != seed.npub]
+    if args.targets_sample:
+        k = min(args.targets_sample, len(candidates))
+        return random.sample(candidates, k) if k > 0 else []
+    return candidates
+
+
+# ---------------------------------------------------------------------------
+# Per-seed lifecycle
+# ---------------------------------------------------------------------------
+
+
+def _peer_connected(client: ControlClient, seed_npub: str) -> bool:
+    try:
+        peers = client.show_peers().get("peers", []) or []
+    except ControlError:
+        return False
+    for p in peers:
+        if p.get("npub") == seed_npub and p.get("connectivity") == "connected":
+            return True
+    return False
+
+
+def _peer_with_seed(daemon: EphemeralDaemon, seed: NodeEntry, timeout: float = 15.0) -> bool:
+    ep = seed.first_dialable_endpoint()
+    if ep is None:
+        log.warning("seed %s has no dialable endpoint (only nat?)", seed.npub[:16])
+        return False
+    log.info("peering with seed %s at %s/%s", seed.npub[:16], ep.addr, ep.transport)
+    try:
+        daemon.client.connect(seed.npub, ep.addr, ep.transport)
+    except ControlError as e:
+        log.warning("connect command failed: %s", e)
+        return False
+
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        if _peer_connected(daemon.client, seed.npub):
+            log.info("seed peer connected")
+            return True
+        time.sleep(0.5)
+    log.warning("seed did not connect within %.0fs", timeout)
+    return False
+
+
+def _converged(client: ControlClient) -> tuple[bool, dict[str, Any]]:
+    """Check convergence criteria. Returns (converged, status_dict)."""
+    try:
+        status = client.show_status()
+    except ControlError:
+        return False, {}
+    peer_count = status.get("peer_count", 0)
+    if peer_count < 1:
+        return False, status
+    # tree_state isn't always present; treat non-empty tree with a root as stable.
+    try:
+        tree = client.show_tree()
+    except ControlError:
+        tree = {}
+    root = tree.get("root")
+    peer_tree_count = tree.get("peer_tree_count", 0)
+    if not root or peer_tree_count < 1:
+        return False, status
+    return True, {"status": status, "tree": tree}
+
+
+def _wait_convergence(daemon: EphemeralDaemon, budget: float) -> tuple[float, dict[str, Any]]:
+    """Poll until converged or budget exhausted. Returns (elapsed_secs, snapshot).
+
+    Convergence requires:
+      - tree is stable (has a root + peer tree entries)
+      - identity cache has stopped growing for 3 consecutive 5s polls
+      - identity cache has at least 2 entries (not just the direct peer)
+    The minimum-count gate prevents premature exit before Nostr discovery
+    has had time to populate the cache from the advert relays.
+    """
+    start = time.time()
+    deadline = start + budget
+    last_snapshot: dict[str, Any] = {}
+    stable_count = 0
+    last_known_count = -1
+    poll_interval = 5.0
+    min_cache = 2
+    while time.time() < deadline:
+        converged, snap = _converged(daemon.client)
+        if converged:
+            try:
+                ic = daemon.client.show_identity_cache().get("count", 0)
+            except ControlError:
+                ic = 0
+            if ic == last_known_count and ic >= min_cache:
+                stable_count += 1
+            else:
+                stable_count = 0
+            last_known_count = ic
+            if stable_count >= 3:
+                last_snapshot = snap
+                return time.time() - start, last_snapshot
+        last_snapshot = snap
+        time.sleep(poll_interval)
+    return time.time() - start, last_snapshot
+
+
+def _record_seed_state(daemon: EphemeralDaemon, seed_npub: str) -> SeedState:
+    try:
+        status = daemon.client.show_status()
+    except ControlError:
+        status = {}
+    try:
+        tree = daemon.client.show_tree()
+    except ControlError:
+        tree = {}
+    try:
+        ic = daemon.client.show_identity_cache()
+    except ControlError:
+        ic = {}
+    try:
+        bloom = daemon.client.show_bloom()
+    except ControlError:
+        bloom = {}
+
+    known = [e.get("npub") for e in (ic.get("entries", []) or []) if e.get("npub")]
+    return SeedState(
+        npub=seed_npub,
+        peer_count=status.get("peer_count", 0),
+        tree_state=str(status.get("state", "")),
+        estimated_mesh_size=status.get("estimated_mesh_size", 0),
+        seed_depth=tree.get("depth"),
+        seed_parent=tree.get("parent_display_name", "") or "",
+        mesh_root=tree.get("root_npub", "") or "",
+        known_npubs=known,
+        bloom_peer_count=len(bloom.get("peer_filters", []) or []),
+    )
+
+
+def _target_in_bloom(daemon: EphemeralDaemon, target_npub: str) -> bool:
+    """Best-effort: is the target's node_addr in any peer's bloom filter?
+
+    The bloom query doesn't expose per-element membership directly, so this
+    is a heuristic: we treat 'target known to at least one peer filter' as
+    a proxy. Falls back to False if unavailable.
+    """
+    try:
+        bloom = daemon.client.show_bloom()
+    except ControlError:
+        return False
+    # show_bloom gives per-peer filter stats, not membership. We approximate
+    # by checking the identity cache (a node we know about is more likely to
+    # be in propagated filters). This is documented as a heuristic.
+    try:
+        ic = daemon.client.show_identity_cache()
+    except ControlError:
+        return False
+    for e in ic.get("entries", []) or []:
+        if e.get("npub") == target_npub:
+            return True
+    return False
+
+
+def _target_has_cached_coords(daemon: EphemeralDaemon, target_npub: str) -> bool:
+    """Inferred from show_routing: is the target in the coord cache?
+
+    show_routing gives a count, not per-target entries, so we approximate:
+    if the target is in the identity cache AND there are coord_cache_entries,
+    we assume coords may be cached. (Documented heuristic.)
+    """
+    try:
+        routing = daemon.client.show_routing()
+    except ControlError:
+        return False
+    return routing.get("coord_cache_entries", 0) > 0 and _target_in_bloom(daemon, target_npub)
+
+
+def _tree_distance_to_target(daemon: EphemeralDaemon, target_npub: str) -> int | None:
+    """Tree distance to a target, if it appears in show_tree peers."""
+    try:
+        tree = daemon.client.show_tree()
+    except ControlError:
+        return None
+    for p in tree.get("peers", []) or []:
+        if p.get("display_name") == target_npub or p.get("npub") == target_npub:
+            d = p.get("distance_to_us")
+            if isinstance(d, (int, float)):
+                return int(d)
+    return None
+
+
+def scan_seed(
+    seed: NodeEntry,
+    targets: list[NodeEntry],
+    index: int,
+    args: argparse.Namespace,
+    total_seeds: int = 0,
+) -> tuple[list[PairResult], SeedState | None]:
+    """Run the full per-seed lifecycle. Returns (pair_results, seed_state)."""
+    log.info("=== seed %d/%d: %s ===", index + 1, total_seeds, seed.npub[:24])
+    t0 = time.time()
+    print(f"\n[{_ts()}] seed {index + 1}/{total_seeds}: {seed.npub[:24]}  "
+          f"({len(targets)} targets)", flush=True)
+    pair_results: list[PairResult] = []
+    # Skip seeds with no dialable endpoint (only nat) — we can't peer with
+    # them directly, so there's no point spawning a daemon.
+    if seed.first_dialable_endpoint() is None:
+        print(f"  [{_ts()}] skipped: no dialable endpoint (nat only) "
+              f"({time.time()-t0:.1f}s)", flush=True)
+        log.warning("seed %s has no dialable endpoint (only nat?); skipping", seed.npub[:16])
+        return pair_results, None
+    daemon: EphemeralDaemon | None = None
+    try:
+        daemon = spawn_daemon(tag=seed.npub[:12], index=index)
+        daemon.wait_ready(args.startup_timeout)
+        log.info("ephemeral daemon ready: netns=%s", daemon.netns)
+        print(f"  [{_ts()}] daemon up ({time.time()-t0:.1f}s)", flush=True)
+
+        t_peer = time.time()
+        if not _peer_with_seed(daemon, seed, timeout=args.peer_timeout):
+            print(f"  [{_ts()}] peering FAILED ({time.time()-t_peer:.1f}s); "
+                  f"skipping targets", flush=True)
+            log.warning("could not peer with seed %s; skipping targets", seed.npub[:16])
+            return pair_results, None
+
+        t_conv = time.time()
+        converge_secs, snap = _wait_convergence(daemon, args.converge_secs)
+        log.info("converged in %.1fs", converge_secs)
+
+        seed_state = _record_seed_state(daemon, seed.npub)
+        log.info("seed state: peers=%d depth=%s root=%s known=%d",
+                 seed_state.peer_count, seed_state.seed_depth,
+                 seed_state.mesh_root[:16] or "(none)", len(seed_state.known_npubs))
+        print(f"  [{_ts()}] peered ok | peers={seed_state.peer_count} "
+              f"depth={seed_state.seed_depth} known={len(seed_state.known_npubs)} | "
+              f"peer {time.time()-t_peer:.1f}s + converge {time.time()-t_conv:.1f}s",
+              flush=True)
+
+        # Reachability checks run inside the daemon's netns so the fd00::/8
+        # route resolves to the daemon's TUN, not the host's fips0.
+        runner = daemon.run_in_netns
+        for target in targets:
+            target_ipv6 = npub_to_ipv6(target.npub)
+            res = reach_check(target_ipv6, timeout=args.ping_timeout,
+                              tcp_port=args.tcp_port, prefer=args.method,
+                              runner=runner)
+            in_cache = target.npub in seed_state.known_npubs
+            in_bloom = _target_in_bloom(daemon, target.npub)
+            cached_coords = _target_has_cached_coords(daemon, target.npub)
+            tree_dist = _tree_distance_to_target(daemon, target.npub)
+
+            pr = PairResult(
+                seed=seed.npub,
+                target=target.npub,
+                reachable=res.reachable,
+                rtt_ms=res.rtt_ms,
+                loss_pct=res.loss_pct,
+                method=res.method,
+                target_in_identity_cache=in_cache,
+                target_in_seed_bloom=in_bloom,
+                cached_coords=cached_coords,
+                tree_distance=tree_dist,
+                seed_depth=seed_state.seed_depth,
+                seed_parent=seed_state.seed_parent,
+                mesh_root=seed_state.mesh_root,
+                converge_secs=converge_secs,
+                detail=res.detail,
+            )
+            pair_results.append(pr)
+            status = "ok" if pr.reachable else "FAIL"
+            log.info("  target %s: %s rtt=%s method=%s cache=%s bloom=%s",
+                     target.npub[:16], status,
+                     f"{res.rtt_ms:.0f}ms" if res.rtt_ms else "-",
+                     res.method, in_cache, in_bloom)
+            if args.delay > 0:
+                time.sleep(args.delay)
+
+        # Per-seed summary line (always visible on stdout).
+        ok_count = sum(1 for r in pair_results if r.reachable)
+        total = len(pair_results)
+        rate = 100 * ok_count / total if total else 0
+        print(f"  [{_ts()}] result: {ok_count}/{total} reachable ({rate:.0f}%) "
+              f"| total {time.time()-t0:.1f}s", flush=True)
+        return pair_results, seed_state
+
+    except Exception as e:
+        log.exception("seed scan failed: %s", e)
+        return pair_results, None
+    finally:
+        if daemon is not None:
+            try:
+                daemon.stop()
+            except Exception as e:
+                log.warning("daemon cleanup failed: %s", e)
+
+
+# ---------------------------------------------------------------------------
+# Output
+# ---------------------------------------------------------------------------
+
+
+def write_matrix_csv(results: list[PairResult], path: Path) -> None:
+    seeds = list(dict.fromkeys(r.seed for r in results))
+    targets = list(dict.fromkeys(r.target for r in results))
+    with path.open("w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["seed"] + [t[:16] for t in targets])
+        for s in seeds:
+            row = [s[:16]]
+            for t in targets:
+                pr = next((r for r in results if r.seed == s and r.target == t), None)
+                if pr is None:
+                    row.append("")
+                elif pr.reachable:
+                    row.append(f"ok/{pr.rtt_ms:.0f}ms" if pr.rtt_ms else "ok")
+                else:
+                    row.append("fail")
+            w.writerow(row)
+
+
+def write_diagnostics_jsonl(results: list[PairResult], path: Path) -> None:
+    with path.open("w") as f:
+        for pr in results:
+            f.write(json.dumps(asdict(pr)) + "\n")
+
+
+def print_summary(results: list[PairResult], seed_states: dict[str, SeedState]) -> None:
+    total = len(results)
+    if total == 0:
+        print("\nNo pairs scanned.")
+        return
+    ok = sum(1 for r in results if r.reachable)
+    print("\n" + "=" * 60)
+    print("MESH REACHABILITY SUMMARY")
+    print("=" * 60)
+    print(f"Seeds scanned: {len(seed_states)}")
+    print(f"Total pairs:   {total}")
+    print(f"Reachable:     {ok}/{total} ({100*ok/total:.1f}%)")
+
+    # Per-seed rates.
+    print("\nPer-seed reachability:")
+    for s, state in seed_states.items():
+        seed_results = [r for r in results if r.seed == s]
+        s_ok = sum(1 for r in seed_results if r.reachable)
+        n = len(seed_results)
+        rate = 100 * s_ok / n if n else 0
+        print(f"  {s[:24]}  {s_ok}/{n} ({rate:.0f}%)  peers={state.peer_count} depth={state.seed_depth}")
+
+    # Per-target rates.
+    targets = list(dict.fromkeys(r.target for r in results))
+    print("\nPer-target reachability (universally unreachable = NAT/firewall suspect):")
+    for t in targets:
+        t_results = [r for r in results if r.target == t]
+        t_ok = sum(1 for r in t_results if r.reachable)
+        n = len(t_results)
+        rate = 100 * t_ok / n if n else 0
+        flag = "  <-- unreachable from all seeds" if t_ok == 0 and n > 0 else ""
+        print(f"  {t[:24]}  {t_ok}/{n} ({rate:.0f}%){flag}")
+
+    # Failure-cause correlation.
+    fails = [r for r in results if not r.reachable]
+    if fails:
+        cache_miss = sum(1 for r in fails if not r.target_in_identity_cache)
+        bloom_miss = sum(1 for r in fails if not r.target_in_seed_bloom)
+        coord_miss = sum(1 for r in fails if not r.cached_coords)
+        print(f"\nFailure correlation ({len(fails)} failures):")
+        print(f"  identity-cache miss: {cache_miss}")
+        print(f"  bloom miss:          {bloom_miss}")
+        print(f"  coord-cache miss:    {coord_miss}")
+    print("=" * 60)
+
+
+# ---------------------------------------------------------------------------
+# Self-test: two ephemeral nodes peered directly
+# ---------------------------------------------------------------------------
+
+
+def self_test(args: argparse.Namespace) -> int:
+    """Smoke test: spawn two ephemeral daemons, peer them, verify they communicate.
+
+    This validates the whole toolchain (daemon spawn, control socket, peering,
+    reachability) without touching the public mesh. Both daemons share a
+    single network namespace so they can peer over loopback (127.0.0.1).
+    """
+    log.info("SELF-TEST: spawning two ephemeral daemons (shared netns) and peering them")
+    shared_netns = "fips-scan-selftest"
+    a = b = None
+    try:
+        # Create the shared namespace up front; both daemons join it.
+        shared_info = fipsd.create_netns(shared_netns)
+        a = spawn_daemon(tag="selftest-a", index=900, netns=shared_netns)
+        a.wait_ready(args.startup_timeout)
+        b = spawn_daemon(tag="selftest-b", index=901, netns=shared_netns)
+        b.wait_ready(args.startup_timeout)
+
+        sa = a.client.show_status()
+        sb = b.client.show_status()
+        npub_a = sa.get("npub")
+        npub_b = sb.get("npub")
+        ipv6_a = sa.get("ipv6_addr")
+        ipv6_b = sb.get("ipv6_addr")
+        log.info("node A: npub=%s ipv6=%s netns=%s", npub_a, ipv6_a, a.netns)
+        log.info("node B: npub=%s ipv6=%s netns=%s", npub_b, ipv6_b, b.netns)
+
+        if not (npub_a and npub_b and ipv6_a and ipv6_b):
+            log.error("could not read both nodes' status")
+            return 1
+
+        # Peer B → A using A's UDP port (loopback inside the shared netns).
+        a_ep = f"127.0.0.1:{a.udp_port}"
+        log.info("connecting B -> A at %s/udp", a_ep)
+        try:
+            b.client.connect(npub_a, a_ep, "udp")
+        except ControlError as e:
+            log.error("connect command failed: %s", e)
+            return 1
+
+        # Wait for the peer to come up on both sides.
+        deadline = time.time() + 20
+        connected = False
+        while time.time() < deadline:
+            if _peer_connected(b.client, npub_a) and _peer_connected(a.client, npub_b):
+                connected = True
+                break
+            time.sleep(0.5)
+        if not connected:
+            log.error("nodes did not peer within 20s")
+            return 1
+        log.info("nodes peered (connected on both sides)")
+
+        # Give the tree a moment to settle.
+        time.sleep(5)
+
+        # Reachability: try both directions. The ephemeral nodes are direct
+        # peers, so find_next_hop branch 2 (direct peer) should deliver. We
+        # try B->A and A->B because freshly-peered two-node trees can show
+        # directional asymmetry before coord caches warm up; either
+        # direction succeeding proves the overlay carries traffic.
+        results = []
+        for label, src, dst_ipv6 in [("B->A", b, ipv6_a), ("A->B", a, ipv6_b)]:
+            log.info("ping6 %s (%s) from %s netns", label, dst_ipv6, src.netns)
+            res = reachability.ping6(dst_ipv6, timeout=5, count=3, runner=src.run_in_netns)
+            log.info("%s ping6: reachable=%s rtt=%s loss=%s",
+                     label, res.reachable, res.rtt_ms, res.loss_pct)
+            results.append((label, res))
+            if res.reachable:
+                rtt_str = f"{res.rtt_ms:.3f}ms" if res.rtt_ms is not None else "n/a"
+                print(f"\nSELF-TEST PASS: {label} ping6 succeeds "
+                      f"({dst_ipv6}) rtt={rtt_str}")
+                return 0
+
+        # Fallback: TCP probe in each direction (ephemeral nodes don't serve
+        # HTTP by default, so this is unlikely to succeed — included for
+        # completeness).
+        for label, src, dst_ipv6 in [("B->A", b, ipv6_a), ("A->B", a, ipv6_b)]:
+            tcp_res = reachability.tcp_probe(dst_ipv6, port=80, timeout=5,
+                                             runner=src.run_in_netns)
+            if tcp_res.reachable:
+                print(f"\nSELF-TEST PASS (tcp): {label} reaches {dst_ipv6} "
+                      f"rtt={tcp_res.rtt_ms:.1f}ms")
+                return 0
+
+        print("\nSELF-TEST FAIL: neither direction reached the other via ping6.")
+        for label, res in results:
+            print(f"  {label}: {res.detail[:160]}")
+        print("Nodes peered successfully but ICMPv6 echo did not complete in either")
+        print("direction. This can happen when coord caches are cold; the peering")
+        print("itself succeeded, which validates spawn/control/peer toolchain.")
+        return 2
+    finally:
+        for d in (b, a):
+            if d is not None:
+                try:
+                    d.stop()
+                except Exception:
+                    pass
+        # Delete the shared netns (daemons didn't own it).
+        fipsd.delete_netns(shared_netns, info=shared_info)
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(description="FIPS Mesh Reachability Scanner")
+    p.add_argument("--seeds", help="'all' or comma-separated npubs")
+    p.add_argument("--seeds-sample", type=int, help="random sample of K seeds")
+    p.add_argument("--seeds-near-root", type=int, help="K seeds with smallest NodeAddr")
+    p.add_argument("--targets", help="comma-separated target npubs (default: all except seed)")
+    p.add_argument("--targets-sample", type=int, help="random sample of K targets per seed")
+    p.add_argument("--converge-secs", type=float, default=45.0)
+    p.add_argument("--ping-timeout", type=int, default=5)
+    p.add_argument("--tcp-port", type=int, default=80, help="TCP fallback port")
+    p.add_argument("--method", choices=["ping6", "tcp"], default="ping6")
+    p.add_argument("--delay", type=float, default=0.0, help="seconds between target checks")
+    p.add_argument("--startup-timeout", type=float, default=15.0)
+    p.add_argument("--peer-timeout", type=float, default=15.0)
+    p.add_argument("--relay", action="append", help="advert relay URL (repeatable)")
+    p.add_argument("--relay-timeout", type=float, default=12.0)
+    p.add_argument("--out-dir", default="mesh-scan-out", help="output directory")
+    p.add_argument("--seed", type=int, default=None, help="RNG seed for --seeds-sample")
+    p.add_argument("--self-test", action="store_true", help="run a 2-node smoke test and exit")
+    p.add_argument("--cleanup-stale", action="store_true", help="remove fips-scan-* TUN devices and exit")
+    p.add_argument("-v", "--verbose", action="count", default=0)
+    args = p.parse_args(argv)
+
+    level = logging.DEBUG if args.verbose >= 2 else (logging.INFO if args.verbose >= 1 else logging.WARNING)
+    logging.basicConfig(level=level, format="%(asctime)s %(levelname)s %(name)s: %(message)s",
+                        datefmt="%H:%M:%S")
+
+    if args.cleanup_stale:
+        n = fipsd.cleanup_stale_netns()
+        print(f"removed {n} stale namespace(s)")
+        return 0
+
+    # Auto-cleanup any stale namespaces from previous runs before starting.
+    n = fipsd.cleanup_stale_netns()
+    if n:
+        print(f"Auto-cleanup: removed {n} stale namespace(s) from previous run(s).")
+
+    if args.self_test:
+        try:
+            return self_test(args)
+        finally:
+            fipsd.cleanup_stale_netns()
+
+    if args.seed is not None:
+        random.seed(args.seed)
+
+    log.info("fetching FIPS node directory")
+    nodes = fetch_directory(args.relay, args.relay_timeout)
+    print(f"Fetched {len(nodes)} nodes from the directory.")
+    if not nodes:
+        print("No nodes found. Exiting.")
+        return 1
+
+    seeds = select_seeds(nodes, args)
+    if not seeds:
+        print("No seeds selected. Exiting.")
+        return 1
+    print(f"Selected {len(seeds)} seed(s): {[s.npub[:16] for s in seeds]}")
+
+    out_dir = Path(args.out_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    all_results: list[PairResult] = []
+    seed_states: dict[str, SeedState] = {}
+    tally = {"connected": 0, "failed": 0, "skipped": 0}
+    scan_start = time.time()
+    matrix_path = out_dir / "matrix.csv"
+    diag_path = out_dir / "diagnostics.jsonl"
+
+    # Write diagnostics incrementally (append after each seed) so a Ctrl-C
+    # doesn't lose all data. The matrix is rewritten (overwrite) each time
+    # since it's a full table.
+    diag_path.unlink(missing_ok=True)
+
+    def write_incremental() -> None:
+        write_matrix_csv(all_results, matrix_path)
+        write_diagnostics_jsonl(all_results, diag_path)
+
+    interrupted = False
+    try:
+        for i, seed in enumerate(seeds):
+            targets = select_targets(nodes, seed, args)
+            log.info("seed %s: %d target(s)", seed.npub[:16], len(targets))
+            results, state = scan_seed(seed, targets, i, args, total_seeds=len(seeds))
+            all_results.extend(results)
+            if state is not None:
+                seed_states[seed.npub] = state
+                tally["connected"] += 1
+            elif not results:
+                tally["failed" if seed.first_dialable_endpoint() else "skipped"] += 1
+            else:
+                tally["connected"] += 1
+            # Write incremental output after each seed.
+            write_incremental()
+            elapsed = time.time() - scan_start
+            print(f"[{_ts()}] tally: {tally['connected']} connected, "
+                  f"{tally['failed']} failed, {tally['skipped']} skipped | "
+                  f"{i+1}/{len(seeds)} done | elapsed {elapsed:.0f}s", flush=True)
+    except KeyboardInterrupt:
+        interrupted = True
+        print("\n[INTERRUPTED] Writing collected data and exiting...", flush=True)
+
+    write_incremental()
+    print_summary(all_results, seed_states)
+    if interrupted:
+        print(f"\n[INTERRUPTED] Partial results: {len(all_results)} pairs from "
+              f"{len(seed_states)} seeds.")
+    print(f"\nOutputs written to {out_dir}/ (matrix.csv, diagnostics.jsonl)")
+    # Auto-cleanup any leaked namespaces on exit.
+    fipsd.cleanup_stale_netns()
+    return 0 if not interrupted else 130
+
+
+if __name__ == "__main__":
+    sys.exit(main())

mesh-scan/reachability.py

@@ -0,0 +1,188 @@
+"""Reachability checks for FIPS mesh targets.
+
+Builds on the pattern in
+[`test_fips.sh`](../fips_playground/test_fips.sh) (curl over ``.fips``) and
+[`test_qube_pair.sh`](../fips_playground/test_qube_pair.sh) (ping6 / TCP
+probe). Two methods:
+
+  - **Primary**: ``ping6 -c 3 -W <timeout> <target.ipv6>`` through the
+    ephemeral daemon's TUN. Records success/fail/RTT/loss.
+  - **Fallback**: open a TCP connection to ``[<target.ipv6>]:<port>`` with a
+    short timeout (used when ICMPv6 is unavailable or filtered).
+
+The target IPv6 is computed directly from the npub via
+[`directory.npub_to_ipv6`](directory.py) so we don't depend on the daemon's
+DNS resolver being wired into the host's ``/etc/resolv.conf``.
+
+Because each ephemeral daemon runs in its own network namespace (see
+[`fipsd.py`](fipsd.py)), the checks are executed *inside* that namespace via
+``ip netns exec`` so the ``fd00::/8`` route resolves to the daemon's TUN
+rather than the host's ``fips0``.
+"""
+
+from __future__ import annotations
+
+import logging
+import re
+import shlex
+import socket
+import subprocess
+import time
+from dataclasses import dataclass
+from typing import Callable, Optional
+
+log = logging.getLogger(__name__)
+
+# A runner executes a command list and returns a CompletedProcess.
+# Defaults to a plain subprocess.run; fipsd.EphemeralDaemon.run_in_netns
+# wraps it in `ip netns exec <netns>`.
+Runner = Callable[[list[str]], subprocess.CompletedProcess]
+
+
+def _default_runner(cmd: list[str]) -> subprocess.CompletedProcess:
+    return subprocess.run(cmd, capture_output=True, text=True,
+                          timeout=30)
+
+
+@dataclass
+class ReachabilityResult:
+    reachable: bool
+    method: str            # "ping6" | "tcp" | "none"
+    rtt_ms: float | None = None
+    loss_pct: int | None = None
+    detail: str = ""
+
+
+def ping6(target_ipv6: str, timeout: int = 5, count: int = 3,
+          runner: Optional[Runner] = None) -> ReachabilityResult:
+    """Ping the target's fd00:: IPv6 through the daemon's TUN.
+
+    ``runner`` executes the command inside the daemon's network namespace
+    (so the ``fd00::/8`` route hits the right TUN). If omitted, runs on the
+    host (only correct when the host has no competing ``fd00::/8`` route).
+    """
+    cmd = ["ping6", "-c", str(count), "-W", str(timeout), target_ipv6]
+    run = runner or _default_runner
+    log.debug("ping6: %s", " ".join(shlex.quote(c) for c in cmd))
+    try:
+        proc = run(cmd)
+    except subprocess.TimeoutExpired:
+        return ReachabilityResult(False, "ping6", detail="ping6 command timed out")
+    except FileNotFoundError:
+        return ReachabilityResult(False, "ping6", detail="ping6 not installed")
+
+    out = (proc.stdout or "") + (proc.stderr or "")
+    if proc.returncode == 0:
+        rtt = _parse_rtt(out)
+        loss = _parse_loss(out)
+        return ReachabilityResult(True, "ping6", rtt_ms=rtt, loss_pct=loss, detail=out.strip())
+
+    loss = _parse_loss(out)
+    return ReachabilityResult(False, "ping6", rtt_ms=None, loss_pct=loss, detail=out.strip())
+
+
+def tcp_probe(target_ipv6: str, port: int = 80, timeout: float = 5.0,
+              runner: Optional[Runner] = None) -> ReachabilityResult:
+    """Open a TCP connection to ``[target_ipv6]:port``.
+
+    When ``runner`` is given (netns), the probe is done with a small Python
+    one-liner executed inside the namespace so the source address and
+    routing come from the daemon's TUN. When omitted, a direct socket
+    connect is attempted on the host.
+    """
+    if runner is not None:
+        script = (
+            "import socket,sys,time;"
+            "s=socket.socket(socket.AF_INET6,socket.SOCK_STREAM);"
+            f"s.settimeout({timeout});"
+            "t=time.time();"
+            f"rc=s.connect_ex(('{target_ipv6}',{port},0,0));"
+            "dt=(time.time()-t)*1000.0;"
+            "print(rc,dt);s.close()"
+        )
+        cmd = ["python3", "-c", script]
+        log.debug("tcp(netns): [%s]:%d", target_ipv6, port)
+        try:
+            proc = runner(cmd)
+        except Exception as e:
+            return ReachabilityResult(False, "tcp", detail=f"netns tcp runner failed: {e}")
+        out = (proc.stdout or "").strip()
+        try:
+            rc_str, dt_str = out.split()
+            rc = int(rc_str)
+            rtt = float(dt_str)
+        except Exception:
+            return ReachabilityResult(False, "tcp", detail=f"tcp parse failed: {out!r}")
+        if rc == 0:
+            return ReachabilityResult(True, "tcp", rtt_ms=rtt, loss_pct=0,
+                                      detail=f"connected to [{target_ipv6}]:{port}")
+        return ReachabilityResult(False, "tcp", detail=f"tcp connect_ex rc={rc}")
+
+    # Host-side direct connect.
+    log.debug("tcp: [%s]:%d", target_ipv6, port)
+    try:
+        with socket.socket(socket.AF_INET6, socket.SOCK_STREAM) as s:
+            s.settimeout(timeout)
+            start = time.time()
+            s.connect((target_ipv6, port, 0, 0))
+            rtt = (time.time() - start) * 1000.0
+            return ReachabilityResult(True, "tcp", rtt_ms=rtt, loss_pct=0,
+                                      detail=f"connected to [{target_ipv6}]:{port}")
+    except socket.timeout:
+        return ReachabilityResult(False, "tcp", detail="tcp connect timed out")
+    except OSError as e:
+        return ReachabilityResult(False, "tcp", detail=f"tcp connect failed: {e}")
+
+
+def check(target_ipv6: str, timeout: int = 5, tcp_port: int = 80,
+          prefer: str = "ping6", runner: Optional[Runner] = None) -> ReachabilityResult:
+    """Run the primary check, fall back to TCP on failure.
+
+    ``prefer`` may be ``"ping6"`` (default) or ``"tcp"``. ``runner``
+    executes commands inside the ephemeral daemon's network namespace.
+    """
+    if prefer == "tcp":
+        return tcp_probe(target_ipv6, tcp_port, timeout, runner=runner)
+
+    res = ping6(target_ipv6, timeout=timeout, runner=runner)
+    if res.reachable:
+        return res
+    # Fall back to TCP.
+    tcp_res = tcp_probe(target_ipv6, tcp_port, timeout, runner=runner)
+    if tcp_res.reachable:
+        return tcp_res
+    # Return the ping result as the canonical failure (it's the more
+    # informative one for mesh diagnostics), but note the TCP fallback.
+    res.detail = (res.detail + " | tcp_fallback: " + tcp_res.detail).strip()
+    res.method = "ping6+tcp"
+    return res
+
+
+# ---------------------------------------------------------------------------
+# ping6 output parsing
+# ---------------------------------------------------------------------------
+
+_RTT_RE = re.compile(r"rtt min/avg/max/mdev\s*=\s*[\d.]+/([\d.]+)/[\d.]+/[\d.]+\s*ms")
+_RTT_ROUNDTRIP_RE = re.compile(r"round-trip min/avg/max\s*=\s*[\d.]+/([\d.]+)/[\d.]+\s*ms")
+_LOSS_RE = re.compile(r"(\d+)% packet loss")
+
+
+def _parse_rtt(out: str) -> float | None:
+    for rx in (_RTT_RE, _RTT_ROUNDTRIP_RE):
+        m = rx.search(out)
+        if m:
+            try:
+                return float(m.group(1))
+            except ValueError:
+                pass
+    return None
+
+
+def _parse_loss(out: str) -> int | None:
+    m = _LOSS_RE.search(out)
+    if m:
+        try:
+            return int(m.group(1))
+        except ValueError:
+            pass
+    return None

mesh-scan/requirements.txt

@@ -0,0 +1,4 @@
+websocket-client>=1.6.0
+pyyaml>=6.0
+# coincurve is optional — directory.py implements bech32 (npub) encoding
+# internally so no secp256k1 dependency is required.

plans/fips-explained.md

@@ -0,0 +1,529 @@
+# How FIPS Works — A Freshman-Level Explanation
+
+FIPS (Freedom Internet Protocol Stack) is a **mesh network** — a way for
+computers to talk to each other directly, without going through a central
+server like Google or your ISP. Instead of connecting to a website through
+a data center, your computer connects to a few nearby computers, which
+connect to other computers, which connect to other computers, and so on.
+Messages hop from computer to computer until they reach their destination.
+
+Think of it like passing a note in class: you hand it to the person next
+to you, they hand it to the next person, and eventually it reaches the
+person it's addressed to. Except in FIPS, the "note" is encrypted, the
+"people" are computers running the FIPS software, and the "classroom" is
+the entire internet.
+
+This document explains how FIPS does four things:
+
+1. **Discovery** — how nodes find each other
+2. **Connection** — how nodes establish secure links
+3. **Topology** — how nodes organize themselves into a network
+4. **Routing** — how messages get from A to B through the mesh
+
+---
+
+## The Big Picture
+
+```
+                    THE INTERNET
+                    (the underlay)
+                         |
+          +--------------+--------------+
+          |              |              |
+     +--------+     +--------+     +--------+
+     | Node A |-----| Node B |-----| Node C |
+     +--------+     +--------+     +--------+
+          |              |              |
+          |              |              |
+     +--------+     +--------+     +--------+
+     | Node D |-----| Node E |     | Node F |
+     +--------+     +--------+     +--------+
+          |
+     +--------+
+     | Node G |
+     +--------+
+
+     Each line = a direct encrypted connection (a "link")
+     Each box  = a computer running fipsd (the FIPS daemon)
+```
+
+Every node in FIPS runs a program called **`fipsd`** (the FIPS daemon).
+It runs in the background, maintains connections to a few other nodes,
+and routes messages through the mesh. Each node has a unique identity
+(a cryptographic keypair) and a unique address on the mesh.
+
+---
+
+## 1. Discovery: How Nodes Find Each Other
+
+The first problem: how does a new node, starting from nothing, find
+other FIPS nodes to connect to? You can't route through the mesh yet
+because you're not in it.
+
+### The Nostr Rendezvous
+
+FIPS uses **Nostr** (a decentralized social protocol) as a bulletin
+board where nodes post "hello, I'm here" messages. Nostr itself is just
+a network of relay servers that store and forward events — think of them
+as public message boards.
+
+```
+    +----------+      "I'm Node A,          +----------+
+    |  Node A  |-----> reach me at          |  Nostr   |
+    |  (new)   |       1.2.3.4:2121"  ----->|  Relays  |
+    +----------+       (Kind 37195 event)   +----------+
+                                                    |
+                                    +---------------+
+                                    |  (stores the
+                                    |   advert for
+                                    |   ~1 hour)
+                                    |
+                    +---------------+----------+
+                    |                          |
+                    v                          v
+              +----------+              +----------+
+              |  Node B  |              |  Node C  |
+              | (reads   |              | (reads   |
+              |  adverts)|              |  adverts)|
+              +----------+              +----------+
+```
+
+Here's what happens:
+
+1. **Node A publishes an advert.** Its `fipsd` creates a Nostr event
+   (kind 37195) containing:
+   - Its identity (public key / npub)
+   - Its endpoint address (e.g., `1.2.3.4:2121` for a public IP, or
+     `nat` if behind a firewall)
+   - Its signal relays (which Nostr relays to use for hole-punching)
+   - Its STUN servers (for NAT traversal)
+   - An expiration time (~1 hour)
+
+2. **The Nostr relays store it.** Three relays by default:
+   `wss://relay.damus.io`, `wss://nos.lol`, `wss://offchain.pub`.
+   They keep the advert until it expires (~1 hour), then delete it.
+
+3. **Node A republishes every ~30 minutes.** This keeps the advert
+   fresh. If Node A goes offline, its advert expires within an hour
+   and disappears from the relays.
+
+4. **Other nodes read the adverts.** Their `fipsd` subscribes to the
+   same relays and receives every advert in real time. They cache them
+   in memory and learn "Node A exists, and here's how to reach it."
+
+### The Two Cases: Public IP vs. NAT
+
+**If Node A has a public IP** (e.g., a VPS in a data center):
+- Its advert says `endpoints: [{transport: "udp", addr: "1.2.3.4:2121"}]`
+- Other nodes can just send UDP packets directly to that address
+- Simple — no extra steps needed
+
+**If Node A is behind NAT** (e.g., a home computer behind a router):
+- Its advert says `endpoints: [{transport: "udp", addr: "nat"}]`
+- "nat" means "I can't be dialed directly — you need to help me punch
+  a hole through my firewall"
+- This requires a **NAT traversal** dance (explained below)
+
+### NAT Traversal (Hole-Punching)
+
+When Node B wants to connect to Node A, but Node A is behind NAT:
+
+```
+    Node B                    Nostr Relays                 Node A
+    (behind NAT too,          (the matchmaker)             (behind NAT)
+     or maybe not)
+        |                          |                          |
+        |  1. "Hey Node A,         |                          |
+        |     I want to connect"   |                          |
+        |----- (encrypted offer -->|                          |
+        |       via Nostr DM)      |-------> offer to A ----->|
+        |                          |                          |
+        |                          |<----- answer from A -----|
+        |<-- answer (encrypted) <--|   2. "OK, let's punch"   |
+        |                          |                          |
+        |  3. Both ask STUN:       |                    3. Both ask STUN:
+        |  "What's my public IP?"  |                    "What's my public IP?"
+        |---------> STUN server    |              -------> STUN server
+        |<--------- "You are       |              <------- "You are
+        |           5.6.7.8:9999"  |                       9.10.11.12:8888"
+        |                          |                          |
+        |  4. Node B sends UDP     |    4. Node A sends UDP   |
+        |     to 9.10.11.12:8888   |       to 5.6.7.8:9999    |
+        |-----> (hole punch!) ---->|-----> (hole punch!) ---->|
+        |                          |                          |
+        |  5. The NAT firewalls    |    5. The NAT firewalls   |
+        |     see outbound traffic |       see outbound traffic|
+        |     and allow replies    |       and allow replies   |
+        |                          |                          |
+        |  <======== UDP tunnel now open =======>             |
+        |             (encrypted with Noise IK)               |
+```
+
+The key insight: both nodes send UDP packets to each other's public
+addresses (discovered via STUN). When Node B sends a packet to Node A's
+NAT, the NAT sees it as a reply to Node A's outbound packet and lets it
+through. The Nostr relays act as the "matchmaker" that lets them
+exchange public addresses before the hole-punch.
+
+Once the UDP tunnel is open, the **Noise IK handshake** runs over it to
+authenticate both sides and establish encryption keys. After that, the
+link is a normal encrypted peer connection — Nostr is no longer needed.
+
+---
+
+## 2. Connection: How Nodes Establish Secure Links
+
+Every direct connection between two FIPS nodes is encrypted using the
+**Noise Protocol Framework** (specifically the Noise IK pattern, the
+same crypto family used by WireGuard and Signal).
+
+```
+    Node A                                          Node B
+    (initiator)                                    (responder)
+        |                                              |
+        |  msg1: A's ephemeral public key              |
+        |  (encrypted with B's static public key)      |
+        |--------------------------------------------->|
+        |                                              |
+        |                              msg2: B's ephemeral public key
+        |                              + B's static public key (encrypted)
+        |<---------------------------------------------|
+        |                                              |
+        |  Both sides now have:                        |
+        |  - Shared secret (via Diffie-Hellman)        |
+        |  - Each other's identity (public key)        |
+        |  - Symmetric encryption keys                 |
+        |                                              |
+        |  <=== Encrypted link established ===>        |
+        |  All future traffic is encrypted and         |
+        |  authenticated                               |
+```
+
+The important properties:
+- **Authentication**: both sides prove they own their private keys
+  (which match their npub / Nostr identity)
+- **Encryption**: all traffic is encrypted with modern symmetric ciphers
+- **Forward secrecy**: even if a key is compromised later, past traffic
+  stays secure
+- **Identity binding**: you know exactly who you're talking to — no
+  man-in-the-middle possible if you trust the Nostr advert's pubkey
+
+Once this handshake completes, the two nodes are **peers** — they have
+an authenticated, encrypted link they can send mesh traffic over.
+
+---
+
+## 3. Topology: How Nodes Organize Into a Network
+
+Once a node has a few peer connections, it needs to know the shape of
+the entire mesh. Not just "who am I connected to" but "who is everyone
+else, and how do I reach them?"
+
+FIPS builds a **spanning tree** — a logical structure that connects all
+nodes without loops, like a family tree or an org chart.
+
+### The Spanning Tree
+
+```
+                         [Root]
+                       Node 00001a...
+                      (smallest NodeAddr
+                       in the network)
+                      /          \
+                     /            \
+              [depth 1]          [depth 1]
+              Node B              Node C
+              /    \              |
+             /      \             |
+        [depth 2]  [depth 2]  [depth 2]
+        Node D     Node E      Node F
+           |
+        [depth 3]
+        Node G (us)
+        Node H
+
+    Each node has exactly one PARENT (closer to root)
+    A node may have multiple CHILDREN (farther from root)
+    The root is the node with the smallest NodeAddr
+    (a 16-byte hash derived from its public key)
+```
+
+**How the tree is built:**
+
+1. Every node computes its **NodeAddr** = first 16 bytes of
+   SHA-256(public key). This is a random-looking number that's unique
+   to each node.
+
+2. The node with the **smallest NodeAddr** is the **root**. This is
+   deterministic — everyone agrees on who the root is without any
+   election or voting.
+
+3. Each node picks its **parent** = the peer that's closest to the root
+   (shallowest depth). If you're not the root, your parent is someone
+   who's one step closer to the root than you.
+
+4. Nodes announce their tree position to their peers using
+   **TreeAnnounce** messages: "I am Node G, my parent is Node D, my
+   path to root is [G → D → B → Root]."
+
+5. When a node receives a TreeAnnounce, it may discover a better parent
+   (one with shallower depth) and switch. This is how the tree
+   self-organizes and heals when nodes join/leave.
+
+```
+    TreeAnnounce flow (direct peer-to-peer, NOT via Nostr):
+
+    Node G --"my parent is D, depth 3"--> Node D (its parent)
+    Node D --"my parent is B, depth 2"--> Node B (its parent)
+    Node B --"my parent is Root, depth 1"--> Root
+
+    And in the other direction (to children):
+    Node D --"I'm at depth 2, parent B"--> Node G (its child)
+    Node B --"I'm at depth 1, parent Root"--> Node D (its child)
+```
+
+TreeAnnounce messages are sent:
+- On initial connection (after handshake)
+- When a node's parent changes
+- Periodically (every ~60 seconds) as a refresh
+- Rate-limited to 1 per 500ms per peer to prevent storms
+
+### Why a Spanning Tree?
+
+The tree gives every node a **path to every other node**: go up to your
+common ancestor, then down to the destination. It also prevents routing
+loops (there's exactly one path between any two nodes in a tree).
+
+But the tree alone isn't enough for efficient routing — going all the
+way up to the root and back down is slow for distant nodes. That's where
+bloom filters come in (next section).
+
+---
+
+## 4. Routing: How Messages Get From A to B
+
+### The Problem
+
+Node G wants to send a message to Node F. G is only connected to Node D.
+How does the message get to F?
+
+```
+    G wants to send to F:
+
+          Root
+         /    \
+        B      C
+       / \     |
+      D   E    F  <-- destination
+      |
+      G  <-- source
+      H
+```
+
+The naive approach: send the message to G's parent (D), D sends to its
+parent (B), B sends to Root, Root sends to C, C sends to F. That's 5
+hops — and it forces all traffic through the root, which doesn't scale.
+
+FIPS does better using **bloom filters**.
+
+### Bloom Filters: Who Can Reach Whom?
+
+A **bloom filter** is a compact data structure that can answer "is X in
+this set?" without storing the whole set. It's probabilistic — it can
+say "definitely not" or "maybe yes" (with a small false-positive rate).
+A bloom filter for 500 nodes is about 1 KB.
+
+Every FIPS node maintains a bloom filter of **which nodes are reachable
+through it** (itself + everything in its subtree). It sends this filter
+to its direct peers via **FilterAnnounce** messages.
+
+```
+    Each node tells its peers: "here's who I can reach"
+
+    Node G's filter: {G, H}           (just itself + children)
+    Node D's filter: {D, G, H}        (itself + its subtree)
+    Node B's filter: {B, D, E, G, H}  (itself + its subtree)
+    Node C's filter: {C, F}           (itself + its subtree)
+    Root's filter:   {everyone}       (it's the root, sees all)
+
+    Filters propagate UP the tree and are shared with all peers:
+    Node D tells B: "I can reach {D, G, H}"
+    Node B tells Root: "I can reach {B, D, E, G, H}"
+    Node C tells Root: "I can reach {C, F}"
+```
+
+Now every node knows, for each of its direct peers, a compact summary of
+who's reachable through that peer. When Node G needs to send to Node F:
+
+```
+    G wants to send to F:
+
+    G checks its peers' bloom filters:
+      - Peer D's filter contains F?  No (D can reach {D,G,H})
+      - (G only has one peer, so it sends to D anyway)
+
+    G --> D: "forward this to F"
+
+    D checks its peers' bloom filters:
+      - Peer B's filter contains F?  No (B can reach {B,D,E,G,H})
+      - (D only has one peer besides G, so it sends to B)
+
+    D --> B: "forward this to F"
+
+    B checks its peers' bloom filters:
+      - Peer Root's filter contains F?  Yes! (Root can reach everyone)
+      - Peer E's filter contains F?     No
+
+    B --> Root: "forward this to F"
+
+    Root checks its peers' bloom filters:
+      - Peer B's filter contains F?  No
+      - Peer C's filter contains F?  Yes!
+
+    Root --> C: "forward this to F"
+
+    C --> F: "here's your message"
+```
+
+The bloom filters let each node make an **informed decision** about
+which peer to forward to, instead of blindly sending everything to its
+parent. This keeps traffic off the root when possible and makes routing
+efficient.
+
+### Coordinate Discovery: Finding the Exact Path
+
+Bloom filters tell you *which peer* to forward to, but not the exact
+path. For that, FIPS uses **coordinate discovery**:
+
+```
+    G wants to send to F (first time — no cached coordinates):
+
+    1. G sends a LookupRequest: "Where is F?"
+       (flooded through the tree, guided by bloom filters —
+        only forwarded to peers whose filter contains F)
+
+       G --> D --> B --> Root --> C --> F
+
+    2. F responds with a LookupResponse: "I'm here, my coordinates are
+       [F, C, Root, B, D, G]"
+       (sent back along the reverse path)
+
+       F --> C --> Root --> B --> D --> G
+
+    3. G caches F's coordinates in its coord cache.
+       Next time G sends to F, it knows the path and can route directly
+       without another lookup.
+```
+
+The coordinate cache is like a phone book — once you've looked up
+someone's number, you don't need to look it up again. Entries expire
+after 5 minutes (default), so if the network topology changes, stale
+routes get refreshed.
+
+### Putting It All Together: A Message's Journey
+
+```
+    Node G sends a web request to Node F's .fips address:
+
+    1. G's application:  "GET http://npub1...fxf.fips/index.html"
+
+    2. G's fipsd:
+       - Derives F's mesh IPv6 address from its npub (fd00::...)
+       - Checks coord cache: do I have F's coordinates?
+         - Yes: route directly using cached path
+         - No:  send LookupRequest, wait for response, cache it
+
+    3. G's fipsd wraps the HTTP request in an encrypted session
+       datagram and sends it hop-by-hop:
+
+       G --[encrypted link]--> D --[encrypted link]--> B
+       B --[encrypted link]--> Root --[encrypted link]--> C
+       C --[encrypted link]--> F
+
+       Each hop:
+       - Decrypts the outer layer
+       - Reads the destination coordinates
+       - Checks bloom filters to pick the next hop
+       - Re-encrypts and forwards
+
+    4. F's fipsd receives the datagram, decrypts the session layer,
+       and delivers the HTTP request to F's web server.
+
+    5. F's web server responds, and the response goes back the same
+       way (or a better path if one was discovered).
+```
+
+Every hop is individually encrypted (the link layer), and the
+end-to-end session is also encrypted (the session layer). So even
+intermediate nodes can't read your traffic — they can only see "this
+packet is going to Node F, forward it to peer X."
+
+---
+
+## The .fips DNS Hack
+
+FIPS gives every node a virtual IPv6 address in the `fd00::/8` range,
+derived from its public key. These addresses only exist **inside the
+mesh** — they're not real internet addresses.
+
+To make this user-friendly, `fipsd` runs a local DNS resolver that
+maps `<npub>.fips` to the node's mesh IPv6 address:
+
+```
+    You type:  curl http://npub1abc123...xyz.fips/index.html
+
+    1. Your browser asks DNS: "what's the IP for npub1abc...xyz.fips?"
+    2. Your fipsd's DNS resolver (127.0.0.1:5354) answers:
+       "It's fd5e:4c4:ce59:a5a4:829:46c7:2b33:ab21"
+    3. Your browser connects to that IPv6 address
+    4. The kernel routes it through fips0 (the TUN device)
+    5. fipsd reads it from fips0, routes it through the mesh
+    6. The message arrives at the destination node
+```
+
+This is why you can type `http://npub1....fips` in a browser and it
+just works — the DNS + TUN + mesh routing make it transparent.
+
+---
+
+## Summary: The Four Layers
+
+```
+  +-------------------------------------------------------------------+
+  |                      THE FIPS MESH                                |
+  |                                                                   |
+  |  +-------------------------------------------------------------+ |
+  |  |  Layer 4: Application                                        | |
+  |  |  HTTP, SSH, any TCP/UDP app via .fips addresses              | |
+  |  +-------------------------------------------------------------+ |
+  |  |  Layer 3: Routing                                           | |
+  |  |  Bloom filters + coordinate discovery + spanning tree       | |
+  |  |  "Which peer do I forward to?"                               | |
+  |  +-------------------------------------------------------------+ |
+  |  |  Layer 2: Links                                             | |
+  |  |  Noise IK encrypted peer-to-peer connections                | |
+  |  |  "I'm securely connected to nodes B, D, and E"              | |
+  |  +-------------------------------------------------------------+ |
+  |  |  Layer 1: Discovery                                         | |
+  |  |  Nostr Kind 37195 adverts + NAT traversal                   | |
+  |  |  "How do I find other nodes to connect to?"                  | |
+  |  +-------------------------------------------------------------+ |
+  |                                                                   |
+  +-------------------------------------------------------------------+
+                         |
+                    THE INTERNET
+                    (the underlay)
+```
+
+| Layer | What it does | How | Nostr? |
+|-------|-------------|-----|--------|
+| **Discovery** | Find other nodes | Nostr adverts + STUN hole-punch | Yes (bootstrap only) |
+| **Links** | Secure connections | Noise IK handshake over UDP/TCP/Tor | No |
+| **Routing** | Route through mesh | TreeAnnounce + FilterAnnounce + LookupRequest | No |
+| **Application** | Use the mesh | .fips DNS + TUN device + any TCP/UDP app | No |
+
+The key insight: **Nostr is only used for the initial rendezvous.**
+Once nodes are connected, everything else — topology, routing, data
+transfer — happens directly between peers over encrypted links. The
+mesh is a self-organizing overlay network that uses Nostr as its
+"phone book" and the internet as its "wires."

plans/mesh-reachability-scanner.md

@@ -0,0 +1,305 @@
+# FIPS Mesh Reachability Scanner
+
+## Goal
+
+Build a tool that answers: **"From a fresh node peered with only node X,
+can I actually reach node Y through the mesh?"** — producing a
+reachability matrix across the real public FIPS mesh.
+
+This directly diagnoses the user's problem: sometimes a node is only
+reachable by peering with it directly, and we want to know *which* seed
+nodes give good mesh reachability and *which* target nodes are
+unreachable except by direct peering — and *why* (coord cache miss,
+bloom miss, tree partition, NAT failure).
+
+## Why a standalone script, not a browser feature
+
+The browser attaches to a single long-lived FIPS daemon. The scanner
+needs **ephemeral daemons peered one-at-a-time** with different seed
+nodes, which the browser architecture can't do (see
+[`net_services.c`](../sovereign_browser/src/net_services.c) —
+`start_managed_fips` hardcodes `fips0`/UDP 4242/TCP 4243, so two
+managed daemons collide).
+
+The scanner is a standalone Python script that drives real `fipsd`
+processes via their control sockets. It reuses patterns from the
+existing [`testing/chaos/sim/control.py`](../fips/testing/chaos/sim/control.py)
+control-socket client and the browser's
+[`handle_fips_directory_json`](../sovereign_browser/src/nostr_bridge.c:3560)
+Nostr query.
+
+## Placement
+
+`~/lt/fips_playground/mesh-scan/` — the playground repo is the right
+home: it already houses FIPS experimentation tooling
+([`test_fips.sh`](../fips_playground/test_fips.sh) does single-target
+reachability via curl over `.fips`, and
+[`test_qube_pair.sh`](../fips_playground/test_qube_pair.sh) does
+two-node peering tests). The scanner generalizes those into a
+full matrix.
+
+Files (under `~/lt/fips_playground/mesh-scan/`):
+- `mesh_scan.py` — main scanner / orchestrator
+- `control.py` — control-socket client (adapted from
+  [`chaos/sim/control.py`](../fips/testing/chaos/sim/control.py),
+  standalone, no Docker dependency)
+- `directory.py` — Nostr Kind 37195 advert fetcher (ports the query
+  from
+  [`handle_fips_directory_json`](../sovereign_browser/src/nostr_bridge.c:3560)
+  to Python using `websocket-client`)
+- `fipsd.py` — ephemeral daemon lifecycle manager (config generation,
+  spawn, socket poll, teardown, TUN cleanup)
+- `reachability.py` — ping6 / curl-based reachability checks (builds
+  on the pattern in
+  [`test_fips.sh`](../fips_playground/test_fips.sh))
+- `README.md` — usage + result interpretation
+- `requirements.txt` — `websocket-client`, `coincurve`, `pyyaml`
+
+## Inputs
+
+1. **Node directory** — fetched from FIPS advert relays (Kind 37195,
+   `fips-overlay-v1`), same query as
+   [`handle_fips_directory_json`](../sovereign_browser/src/nostr_bridge.c:3560).
+   Returns each node's npub, endpoints (host:port + transport), and
+   signal relays. This is the list of "all known nodes on the mesh."
+
+2. **Seed selection** — which nodes to peer with. Options:
+   - `--seeds all` — every node in the directory (O(N) daemon lifecycles)
+   - `--seeds npub1abc,npub1def,...` — explicit list
+   - `--seeds-sample K` — random sample of K seeds (default 5)
+   - `--seeds-near-root` — pick seeds with smallest NodeAddr (tree root
+     neighborhood) — these should have the best reachability if the
+     mesh is healthy
+
+3. **Target selection** — which nodes to try to reach. Default: all
+   nodes in the directory except the current seed. `--targets` to
+   restrict.
+
+4. **Convergence budget** — `--converge-secs N` (default 45): how long
+   to wait after peering before declaring the tree stable and starting
+   reachability checks.
+
+5. **Per-target timeout** — `--ping-timeout N` (default 5s): how long
+   to wait for each reachability check.
+
+## Per-seed scan lifecycle
+
+For each seed node X:
+
+1. **Start ephemeral daemon**
+   - Create temp config dir (`/tmp/mesh-scan-<seed>-<pid>/`)
+   - Generate `fips.yaml` with:
+     - Ephemeral identity (non-persistent)
+     - Unique control socket path in the temp dir
+     - Unique TUN device name (`fips-scan-<n>`) to avoid collision
+     - Unique UDP/TCP ports (auto-allocate from ephemeral range)
+     - DNS enabled on a unique port
+     - Empty peer list
+   - Spawn `fipsd --config <path>` as a subprocess
+   - Wait for control socket to appear (poll up to 10s)
+
+2. **Peer with seed X**
+   - Send `connect` command with X's npub, endpoint, and transport
+     (from the directory entry)
+   - Wait for handshake to complete (poll `show_peers` until X appears
+     with `can_send=true`, up to 15s)
+
+3. **Wait for convergence**
+   - Poll `show_status` every 2s until:
+     - `tree_state` is stable (not `Initializing`/`Converging`), AND
+     - `peer_count >= 1`, AND
+     - `show_tree` returns a non-empty tree with a root, AND
+     - `show_identity_cache` count stops growing for 2 consecutive polls
+   - Bounded by `--converge-secs` (default 45s)
+
+4. **Record seed-side state** (for diagnostics)
+   - `show_status`: peer_count, tree_state, estimated_mesh_size
+   - `show_tree`: our depth, parent, root
+   - `show_identity_cache`: known node count, list of known npubs
+   - Per-peer bloom filter state (`show_bloom`): which targets are in
+     our seed's bloom filter
+
+5. **For each target node Y** (Y ≠ X):
+   - Resolve `<Y.npub>.fips` via the daemon's DNS resolver (or compute
+     the `fd00::` IPv6 directly from the npub)
+   - Attempt reachability check:
+     - **Primary**: `ping6 -c 3 -W <timeout> <Y.ipv6>` through the
+       daemon's TUN. Record success/fail/RTT/loss.
+     - **Fallback** (if ping6 unavailable): open a TCP connection to
+       `[<Y.ipv6>]:80` with a short timeout. Record connect success/fail.
+   - Record: target npub, success, latency, loss, and diagnostic state:
+     - Was Y in our identity cache? (`show_identity_cache`)
+     - Was Y in our seed's bloom filter? (`show_bloom`)
+     - Did we have cached coords for Y? (inferred from `show_routing`)
+     - Tree distance to Y (from `show_tree` if available)
+
+6. **Stop ephemeral daemon**
+   - Send `disconnect` for the seed peer (clean teardown)
+   - SIGTERM the process, wait up to 5s, SIGKILL if needed
+   - Clean up TUN device and temp config dir
+
+## Output
+
+### Reachability matrix
+
+A CSV / TSV table: rows = seeds, columns = targets, cells = `ok` /
+`fail` / `timeout` / RTT in ms.
+
+```
+seed            npub1abc  npub1def  npub1ghi  npub1jkl  ...
+npub1root       ok/42ms   ok/55ms   fail      ok/120ms  ...
+npub1hub        ok/38ms   ok/41ms   ok/67ms   fail      ...
+npub1leaf       fail      fail      fail      fail      ...
+```
+
+### Per-pair diagnostics JSON
+
+For each (seed, target) pair, a record with:
+
+```json
+{
+  "seed": "npub1abc",
+  "target": "npub1def",
+  "reachable": true,
+  "rtt_ms": 55,
+  "loss_pct": 0,
+  "target_in_identity_cache": true,
+  "target_in_seed_bloom": true,
+  "cached_coords": true,
+  "tree_distance": 3,
+  "seed_depth": 2,
+  "seed_parent": "npub1root",
+  "mesh_root": "npub1root",
+  "converge_secs": 12
+}
+```
+
+This lets you correlate failures with their *cause*: a `fail` where
+`target_in_identity_cache=false` is the identity-cache-gate issue from
+[`session.rs:1144`](../fips/src/node/handlers/session.rs:1144); a
+`fail` where `target_in_seed_bloom=false` is bloom propagation not
+reaching your seed; a `fail` with different `mesh_root` is a tree
+partition.
+
+### Summary report
+
+Printed to stdout at the end:
+- Total seeds scanned, total targets, total pairs
+- Overall reachability rate (X% of pairs reachable)
+- Per-seed reachability rate (which seeds are "good" / "bad" hubs)
+- Per-target reachability rate (which targets are universally
+  unreachable — likely NAT/firewall issues on that node)
+- Correlation summary: "of N failures, M had identity-cache miss, K
+  had bloom miss, J had tree partition"
+
+## How to interpret results for your problem
+
+The matrix tells you:
+
+1. **Is the mesh working at all?** If most pairs are `ok`, the mesh is
+   healthy and your direct-peering problem is specific to certain
+   nodes. If most pairs are `fail`, the mesh has systemic issues.
+
+2. **Which seeds give good reachability?** Seeds with high per-seed
+   reachability rates are well-connected hubs — peering with them
+   gives you access to most of the mesh. These are the nodes you
+   *should* peer with in production.
+
+3. **Which targets are unreachable except by direct peering?** Targets
+   that fail from *all* seeds (or all but their direct neighbors) are
+   the ones forcing you to peer directly. The diagnostics tell you
+   *why* — if it's identity-cache miss across the board, that's the
+   bug; if it's bloom miss, the target isn't propagating its existence
+   well; if it's tree partition, the target is in a different tree.
+
+4. **Is it a tree-position problem?** If seeds near the root
+   (`--seeds-near-root`) have much better reachability than seeds far
+   from the root, your problem is tree depth — peer with a
+   near-root node.
+
+## Implementation order
+
+1. `control.py` — control-socket client (port from
+   `chaos/sim/control.py`, standalone, no Docker dependency)
+2. `directory.py` — Nostr Kind 37195 fetcher (port the query from
+   [`handle_fips_directory_json`](../sovereign_browser/src/nostr_bridge.c:3560)
+   to Python using `websocket-client`)
+3. `fipsd.py` — ephemeral daemon lifecycle (config generation, spawn,
+   socket poll, teardown, TUN cleanup)
+4. `reachability.py` — ping6/curl checks (extend
+   [`test_fips.sh`](../fips_playground/test_fips.sh) pattern)
+5. `mesh_scan.py` — orchestrator: tie together directory → seeds →
+   per-seed lifecycle → reachability checks → output
+6. `README.md` — usage, output interpretation, troubleshooting
+
+## Open questions
+
+1. **TUN device cleanup** — if the scanner crashes, `fips-scan-N` TUN
+   devices may leak. Need a trap/cleanup handler and a
+   `--cleanup-stale` mode that removes any `fips-scan-*` devices.
+
+2. **Privilege** — `fipsd` needs `CAP_NET_ADMIN` for TUN. The scanner
+   must either run as root, or the `fips` binary must have the
+   capability set (same as the browser's
+   [`binary_has_net_admin`](../sovereign_browser/src/net_services.c:306)
+   check).
+
+3. **Reachability check method** — `ping6` is simplest but requires
+   ICMPv6 to pass through the FIPS TUN, which depends on the daemon's
+   ACL/firewall config. The fallback TCP check is more reliable but
+   requires the target to be running a reachable service. May need a
+   third option: a FIPS-protocol-level "echo" via the control socket
+   (if the daemon exposes one) or a Nostr-based reachability signal.
+
+4. **NAT traversal for the ephemeral node** — the scanner node itself
+   may be behind NAT, which could affect its ability to peer with
+   seeds. The FIPS Nostr discovery + STUN hole-punching should handle
+   this, but it's a variable. Running the scanner on a
+   publicly-addressable host removes this confound.
+
+5. **Rate limiting / politeness** — scanning N seeds × M targets
+   generates traffic on the real mesh. Should add a `--delay` between
+   checks and a `--max-concurrent` limit. Probably also a user-agent
+   or scanner identifier in the ephemeral node's display name so
+   others can see what's happening.
+
+## Background: why direct peering is sometimes required
+
+Mesh routing in FIPS is greedy + bloom-guided, not link-state.
+[`find_next_hop`](../fips/src/node/mod.rs:2631) tries four things in
+order:
+
+1. Local delivery (is it us?)
+2. **Direct peer** — the destination is one of our authenticated
+   peers. This branch *never* fails for a known peer.
+3. Bloom-filter-guided routing — requires **cached destination
+   coordinates** (5-min TTL in
+   [`coord_cache.rs`](../fips/src/cache/coord_cache.rs:18)).
+4. Greedy tree routing fallback — also requires cached coords, and
+   fails on local minima or different-tree.
+
+If the coord cache has no entry for the destination, `find_next_hop`
+returns `None` and the packet is dropped with a `CoordsRequired` error
+signal. The source handles this in
+[`handle_coords_required`](../fips/src/node/handlers/session.rs:1108),
+which triggers discovery — but **only if `has_cached_identity(target)`
+is true** ([`session.rs:1144`](../fips/src/node/handlers/session.rs:1144)).
+If the target's identity isn't cached, discovery is skipped entirely.
+
+Discovery itself (`maybe_initiate_lookup`,
+[`discovery.rs:487`](../fips/src/node/handlers/discovery.rs:487)) has
+two more gates: a bloom pre-check (if no peer's filter contains the
+target, skip) and tree-peer-only forwarding
+([`discovery.rs:446`](../fips/src/node/handlers/discovery.rs:446) —
+only `is_tree_peer(addr) && peer.may_reach(target)`).
+
+**Direct peering hits branch 2**, bypassing all of: coord cache, bloom
+discovery, tree routing, and per-hop transport issues. That's why it
+always works when mesh routing fails.
+
+The scanner's per-pair diagnostics map directly to these failure
+modes:
+- `target_in_identity_cache=false` → the identity-cache gate
+- `target_in_seed_bloom=false` → bloom propagation gap
+- `mesh_root` differs → tree partition
+- `cached_coords=false` → coord cache miss (cold path)