diff --git a/investigation-update.md b/investigation-update.md new file mode 100644 index 0000000..017f922 --- /dev/null +++ b/investigation-update.md @@ -0,0 +1,383 @@ +# routstrd `service install` Bug Investigation — Update + +## Executive Summary + +All 5 bugs from the initial `summary.md` were verified empirically. However, the investigation uncovered **2 additional critical bugs** that were not in the original summary: + +1. **PM2 `--interpreter bun` is broken in PM2 v6.x** — this causes *any* Bun script (even `console.log("hello")`) to exit immediately with no logs. +2. **`import.meta.main` guard in `src/daemon/index.ts` prevents `main()` from executing when PM2 wraps the script** (which PM2 does by default when using `--interpreter node`). + +The original summary identified the correct symptoms (daemon disappears from PM2) but attributed them primarily to missing cocod initialization. In reality, **the daemon cannot start under PM2 at all** due to the PM2/Bun interoperability bugs, regardless of cocod state. + +--- + +## Bug-by-Bug Verification + +### Bug 1: `Bun.resolveSync` Used Incorrectly — ✅ CONFIRMED + +**Location:** `src/cli.ts`, `service install` command handler + +**Broken code:** +```typescript +daemonPath = Bun.resolveSync("./daemon/index.js", import.meta.url); +``` + +**Evidence:** +```bash +$ cat > src/test-resolve.ts << 'EOF' +console.log("import.meta.url:", import.meta.url); +try { + const p = Bun.resolveSync("./daemon/index.js", import.meta.url); + console.log("OK:", p); +} catch (e: any) { + console.log("FAILED:", e.message); +} +EOF + +$ bun run src/test-resolve.ts +import.meta.url: file:///home/user42/.../src/test-resolve.ts +FAILED: Cannot find module './daemon/index.js' from 'file:///home/user42/.../src/test-resolve.ts' +``` + +`Bun.resolveSync(moduleID, parent)` requires `parent` to be a **directory path**. `import.meta.url` is a `file://` URL pointing to the source file, not its directory. This causes an immediate throw, so execution always falls through to the `catch` block. + +**Fix (partial):** +```typescript +const baseDir = new URL(".", import.meta.url).pathname; +daemonPath = Bun.resolveSync("./daemon/index.js", baseDir); // works +``` + +--- + +### Bug 2: Dev vs Built Path Resolution Divergence — ✅ CONFIRMED + +**Evidence:** + +**Development mode** (`bun run src/test-resolve.ts`): +``` +baseDir: /home/user42/.../src/ +resolveSync with dir path: /home/user42/.../src/daemon/index.ts +``` +Resolves to `.ts` source file because Bun resolves `.js` → `.ts` transparently in dev. + +**Built mode** (`bun run dist/test-resolve.js`): +``` +baseDir: /home/user42/.../dist/ +resolveSync with dir path: /home/user42/.../dist/daemon/index.js +``` +Resolves to the bundled `.js` artifact. + +**Fallback path behavior:** +Because Bug 1 always throws, the fallback path is always used: +```typescript +const path = require("path"); +daemonPath = path.join(path.dirname(import.meta.url).replace("file://", ""), "daemon", "index.js"); +``` + +- In **dev**: produces `.../src/daemon/index.js` → **does not exist** (source is `.ts`) +- In **built**: produces `.../dist/daemon/index.js` → **exists** if `bun run build` was run + +**User-facing impact:** +```bash +$ bun src/index.ts service install +Could not find daemon at .../src/daemon/index.js. Did you run 'bun run build'? +``` +This message is misleading. The user may want to run from source without building. + +--- + +### Bug 3: Daemon Crashes Immediately on Startup — ⚠️ PARTIALLY CONFIRMED + +**Original claim:** Daemon crashes because of missing wallet/cocod initialization. + +**Findings:** + +1. **If cocod is not installed/initialized, the daemon DOES crash** — `createWalletAdapter()` throws an unhandled error in `src/daemon/index.ts`: + ```typescript + const walletClient = createCocodClient({ cocodPath: config.cocodPath }); + const walletAdapter = await createWalletAdapter({ cocodPath: config.cocodPath, walletClient }); + ``` + This is a real bug and `service install` should check `isCocodInstalled()` before spawning PM2 (the `start` command already does this check; `service install` does not). + +2. **However, the daemon ALSO cannot start under PM2 even when cocod IS ready** due to two PM2-specific bugs (see NEW BUGS below). + +**Evidence that daemon runs fine without PM2:** +```bash +$ bun dist/daemon/index.js --port 18008 +[ProviderManager] Hydrated from store: ... +Routstr daemon listening on http://localhost:18008 +Bootstrapping providers... +Bootstrapped 22 providers +... +``` +The daemon starts successfully and stays alive when run directly with `bun`. + +--- + +### Bug 4: Bundled Build Has Native Module Issues — ❌ NOT REPRODUCED + +**Claim:** `bun run build` throws resolve errors for `@noble/hashes/crypto`. + +**Evidence:** +```bash +$ bun run build + index.js 1274.27 KB +[25ms] bundle 486 modules + index.js 1257.17 KB +[22ms] bundle 416 modules +``` +Build completed successfully. The lockfile issue may have been transient or environment-specific. + +--- + +### Bug 5: PM2 Start Command Lacks Working Directory — ⚠️ PARTIALLY CONFIRMED + +**Claim:** Missing `--cwd` causes PID file and DB to write to wrong locations. + +**Evidence:** +```typescript +// src/utils/config.ts +export const CONFIG_DIR = process.env.ROUTSTRD_DIR || `${HOME}/.routstrd`; +export const PID_FILE = process.env.ROUTSTRD_PID || `${CONFIG_DIR}/routstrd.pid`; +export const DB_PATH = `${CONFIG_DIR}/routstr.db`; +``` + +Both `PID_FILE` and `DB_PATH` are **absolute paths** under `~/.routstrd` (or `$ROUTSTRD_DIR`), so they are **not affected by CWD**. + +However: +- Missing `--cwd` can still affect **relative imports** or **runtime file resolution** inside the daemon. +- In PM2 v6, the default `exec cwd` was observed to be the project directory anyway. +- Explicit `--cwd` is still defensive and should be added. + +**Recommendation:** Add `--cwd "${process.cwd()}"` to the PM2 start command for safety, but this is not the root cause of the disappearing process. + +--- + +## 🆕 New Critical Bugs Discovered + +### New Bug A: PM2 `--interpreter bun` is Broken in PM2 v6.x + +**Environment:** PM2 v6.0.14, Bun v1.1.42, Node.js v24.15.0 + +**Symptom:** Any Bun script started with `pm2 start ... --interpreter bun` shows as "online" for ~0 seconds, then immediately disappears. No logs are written. + +**Evidence:** +```bash +$ cat > dist/test-main.js << 'EOF' +console.log("hello from bun"); +setInterval(() => console.log("alive", Date.now()), 2000); +EOF + +$ pm2 start dist/test-main.js --name test-bun --interpreter bun +[PM2] Starting ... Done. +# Shows PID, 0s uptime, online + +$ sleep 2 +$ pm2 describe test-bun +[PM2][WARN] test-bun doesn't exist + +$ cat ~/.pm2/logs/test-bun-out.log +# EMPTY — 0 bytes +``` + +**Same script with `--interpreter node` works fine and stays alive.** + +**Root cause:** PM2 v6.x has known Bun interpreter compatibility issues: +- https://github.com/Unitech/pm2/issues/5967 +- https://github.com/oven-sh/bun/issues/4949 + +**Working workaround:** +```bash +pm2 start "bun run dist/daemon/index.js" --name routstrd --interpreter none +``` + +With `--interpreter none`, PM2 runs the command through the default shell (`bash -c "..."`), and `bun` executes directly. The process stays alive and logs are captured correctly. + +--- + +### New Bug B: `import.meta.main` is False Under PM2 + +**Location:** `src/daemon/index.ts` + +**Broken code:** +```typescript +if (import.meta.main) { + main().catch((error) => { + logger.error("Failed to start Routstr daemon:", error); + process.exit(1); + }); +} +``` + +**Evidence:** +```bash +$ cat > dist/test-main.js << 'EOF' +console.log("import.meta.main:", import.meta.main); +console.log("import.meta.url:", import.meta.url); +EOF + +$ pm2 start dist/test-main.js --name test-node --interpreter node +$ pm2 logs test-node --lines 5 +0|test-node | import.meta.main: false +0|test-node | import.meta.url: file:///.../dist/test-main.js +0|test-node | argv: [ +0|test-node | '/home/user42/.local/share/nvm/v24.15.0/bin/node', +0|test-node | '/home/user42/.bun/install/global/node_modules/pm2/lib/ProcessContainerFork.js' +0|test-node | ] +``` + +PM2 wraps the target script inside `ProcessContainerFork.js`. Because the script is loaded as a module by PM2's container, `import.meta.main` evaluates to **`false`**. + +**Impact:** When PM2 starts the daemon with `--interpreter node` (or any interpreter that wraps the script), `main()` is **never called**. The process loads all modules, reaches the `if (import.meta.main)` guard, evaluates it as `false`, and exits cleanly with code 0. PM2 sees a clean exit and removes the process. + +**Why this wasn't caught before:** The summary assumed `--interpreter bun` would be used. But `--interpreter bun` is broken (New Bug A), and the fallback/default behavior (`--interpreter node`) triggers this `import.meta.main` issue. + +**Fix:** Remove the `import.meta.main` guard or add a fallback: +```typescript +// Option 1: Always run main() when loaded directly +main().catch((error) => { + logger.error("Failed to start Routstr daemon:", error); + process.exit(1); +}); + +// Option 2: Keep guard but also detect PM2 environment +if (import.meta.main || process.env.PM2_USAGE) { + main().catch(...); +} +``` + +Option 1 is simpler and safer. The guard is unnecessary because `src/daemon/index.ts` is only ever the entry point. + +--- + +## Corrected Root Cause Summary + +The `service install` command fails due to a **cascade of issues**, in order of severity: + +1. **PM2 `--interpreter bun` is broken** (New Bug A). The current command `pm2 start ... --interpreter bun` causes the process to exit silently within seconds. +2. **`import.meta.main` guard prevents execution** (New Bug B). If PM2 falls back to `node` or wraps the script, `main()` is never called. +3. **Path resolution is broken** (`Bun.resolveSync` with file URL). In dev mode, the fallback path points to a non-existent `.js` file, misleading users. +4. **No cocod readiness check** before spawning the daemon. If the user somehow gets past issues 1-3, an uninitialized wallet causes the daemon to crash. +5. **No `--cwd` passed to PM2**. Minor, but can cause unexpected behavior with relative paths. + +--- + +## Reproduction Steps (Updated) + +### Reproduce Bug 1 + 2 (Dev mode path failure): +```bash +bun src/index.ts service install +# Output: "Could not find daemon at .../src/daemon/index.js. Did you run 'bun run build'?" +``` + +### Reproduce New Bug A (PM2 `--interpreter bun` broken): +```bash +bun run build +bun dist/index.js service install +# PM2 shows "online" for 0s, then process disappears +pm2 describe routstrd +# Output: "routstrd doesn't exist" +``` + +### Reproduce New Bug B (`import.meta.main` false): +```bash +cat > dist/test-main.js << 'EOF' +if (import.meta.main) console.log("MAIN"); else console.log("NOT MAIN"); +EOF +pm2 start dist/test-main.js --name test-main --interpreter node +pm2 logs test-main +# Output: "NOT MAIN" +``` + +### Reprove daemon works fine without PM2: +```bash +bun dist/daemon/index.js --port 18008 +# Daemon starts, listens, bootstraps providers, stays alive +``` + +--- + +## Recommended Fixes (Updated) + +### Fix 1: Change PM2 start strategy (addresses New Bug A) +Replace: +```typescript +execSync(`pm2 start "${daemonPath}" --name routstrd --interpreter bun`, ...) +``` + +With: +```typescript +execSync( + `pm2 start "bun run ${daemonPath}" --name routstrd --interpreter none --cwd "${process.cwd()}"`, + { stdio: "inherit" } +); +``` + +This runs `bun` directly via shell, bypassing PM2's broken `--interpreter bun` handling. + +### Fix 2: Remove `import.meta.main` guard (addresses New Bug B) +In `src/daemon/index.ts`, replace: +```typescript +if (import.meta.main) { + main().catch((error) => { + logger.error("Failed to start Routstr daemon:", error); + process.exit(1); + }); +} +``` + +With: +```typescript +main().catch((error) => { + logger.error("Failed to start Routstr daemon:", error); + process.exit(1); +}); +``` + +### Fix 3: Correct path resolution (addresses Bug 1 + 2) +```typescript +const baseDir = new URL(".", import.meta.url).pathname; +let daemonPath: string; +try { + daemonPath = Bun.resolveSync("./daemon/index.js", baseDir); +} catch (e) { + const path = await import("path"); + daemonPath = path.join(baseDir, "daemon", "index.js"); +} + +// Differentiate dev vs built mode +if (!existsSync(daemonPath)) { + // In dev, Bun resolves .js → .ts transparently, so we can try .ts + const tsPath = daemonPath.replace(/\.js$/, ".ts"); + if (existsSync(tsPath)) { + daemonPath = tsPath; + } +} +``` + +### Fix 4: Check cocod before PM2 start (addresses Bug 3) +```typescript +const config = await loadConfig(); +if (!(await isCocodInstalled(config.cocodPath))) { + console.error("cocod is not installed or initialized. Run 'routstrd onboard' first."); + process.exit(1); +} +``` + +### Fix 5: Add server error handler (prevents silent crash on port conflict) +In `src/daemon/index.ts`: +```typescript +server.on("error", (err) => { + logger.error("Server error:", err); + process.exit(1); +}); +``` + +--- + +## Files to Modify + +| File | Fixes | +|------|-------| +| `src/cli.ts` | Fix path resolution (Bug 1/2), add cocod check (Bug 3), change PM2 command (New Bug A), add `--cwd` (Bug 5) | +| `src/daemon/index.ts` | Remove `import.meta.main` guard (New Bug B), add server error handler | diff --git a/summary.md b/summary.md new file mode 100644 index 0000000..414c594 --- /dev/null +++ b/summary.md @@ -0,0 +1,184 @@ +# routstrd `service install` Bug Investigation Summary + +## Problem Reported +The `routstrd service install` command is not working, silently failing to start or persist the daemon. + +## Investigation Findings + +When running `routstrd service install` from source (`bun src/index.ts`), the command appears to exit quickly with no output. When running from the built dist, PM2 starts but the daemon process immediately disappears. + +Multiple bugs were discovered during investigation. + +--- + +## Bug 1: `Bun.resolveSync` Used Incorrectly (CLI Broken at Runtime) + +**File:** `src/cli.ts` (in the `service install` command handler) + +**Broken code:** +```typescript +let daemonPath: string; +try { + // Try to resolve relative to this file first (works in dev and global) + daemonPath = Bun.resolveSync("./daemon/index.js", import.meta.url); +} catch (e) { + // Fallback for some bundling scenarios + const path = require("path"); + daemonPath = path.join(path.dirname(import.meta.url).replace("file://", ""), "daemon", "index.js"); +} +``` + +**Problem:** +`Bun.resolveSync(moduleID, parent)` requires `parent` to be a **directory path**, not a `file://` URL. When `import.meta.url` (which is `file:///path/to/cli.ts`) is passed as the parent, `Bun.resolveSync` throws immediately: + +``` +Cannot find module './daemon/index.js' from 'file:///home/user42/.../cli.ts' +``` + +This means the `try` block always fails and execution falls through to the fallback path. The fallback uses `require("path")` which works in Bun but the resulting path is wrong. + +**Evidence:** +```bash +$ bun -e "console.log(Bun.resolveSync('./daemon/index.js', import.meta.url))" +error: Cannot find module './daemon/index.js' from 'file:///.../[eval]' + +$ # Works correctly with a directory: +$ bun -e "console.log(Bun.resolveSync('./daemon/index.js', 'file:///home/user42/projects/routstr_main/routstrd/src'))" +/home/user42/projects/routstr_main/routstrd/src/daemon/index.js +``` + +Even when `Bun.resolveSync` is fixed, `dist/index.js` still needs to resolve the correct path because after bundling, `import.meta.url` points to the bundled file, not the original source. + +--- + +## Bug 2: Dev vs Built Path Resolution Divergence + +After `bun run build`, the `dist/` directory contains: +- `dist/index.js` — the CLI bundle +- `dist/daemon/index.js` — the daemon bundle + +When running `bun dist/index.js service install`: +- `import.meta.url` = `file:///.../dist/index.js` +- `Bun.resolveSync("./daemon/index.js", new URL(".", import.meta.url).pathname)` resolves to `dist/daemon/index.js` ✓ + +When running `bun src/index.ts service install` (development): +- `import.meta.url` = `file:///.../src/index.ts` +- `Bun.resolveSync` would try to resolve `src/daemon/index.js`, which does not exist (the source is `.ts`, not bundled) +- The daemon source is actually at `src/daemon/index.ts` + +This means the path resolution logic must differentiate between: +1. **Development mode** (`bun src/index.ts`) → the daemon should be started via `bun run src/daemon/index.ts` +2. **Production/bundled mode** (`bun dist/index.js`) → the daemon is `dist/daemon/index.js` + +--- + +## Bug 3: Daemon Crashes Immediately on Startup (Missing Wallet Initialization) + +Even after PM2 starts the daemon, it **crashes immediately** and disappears from `pm2 list`. + +**Evidence:** +```bash +$ pm2 start "dist/daemon/index.js" --name routstrd --interpreter bun +[PM2] Starting ... Done. +$ pm2 describe routstrd +[PM2][WARN] routstrd doesn't exist # <-- already crashed +``` + +The daemon (`src/daemon/index.ts`) creates a `cocod` wallet client immediately: +```typescript +const walletClient = createCocodClient({ cocodPath: config.cocodPath }); +const walletAdapter = await createWalletAdapter({ + cocodPath: config.cocodPath, + walletClient, +}); +``` + +If **cocod is not installed or initialized**, this throws an unhandled error, the process exits, and PM2 removes the crashed process. PM2 doesn't restart it because the crash happens so fast. + +The `service install` command **never checks** if cocod is installed before spawning the daemon via PM2. The `onboard`/`initDaemon` sequence does this check, but `service install` skips it entirely. + +--- + +## Bug 4: Bundled Build Has Native Module Issues (Secondary) + +`bun run build` throws resolve errors when `@noble/hashes/crypto` is missing (node_modules inconsistency). After `bun install`, the build succeeds, but this indicates the lockfile was stale/ignored. + +--- + +## Bug 5: PM2 Start Command Lacks Working Directory + +The PM2 start command uses an absolute path, but the daemon writes its PID file and opens its database relative to the current working directory. If PM2 starts it with a different CWD (which it does — PM2's internal CWD), the PID file and DB end up in the wrong place or fail to write. + +```typescript +execSync(`pm2 start "${daemonPath}" --name routstrd --interpreter bun`, ...) +``` + +Missing: `--cwd ` to ensure the daemon runs from the expected directory. + +--- + +## Root Cause Summary + +The `service install` command fails due to a **cascade of issues**: + +1. **Path resolution is broken** (`Bun.resolveSync` called with invalid parent) +2. **No cocod readiness check** before spawning the daemon under PM2 +3. **No CWD provided to PM2**, so daemon state files may write to wrong locations +4. **Daemon crashes immediately** if wallet isn't initialized, and PM2 silently removes it + +--- + +## Reproduction Steps + +1. Fresh clone / clean environment (or just ensure no PM2 process is running) +2. `bun src/index.ts service install` + - Expected: installs PM2 service, daemon starts and persists + - Actual: command may appear to work but PM2 process immediately crashes/disappears +3. `pm2 describe routstrd` → "doesn't exist" +4. `pm2 logs routstrd` → empty or abruptly ended + +--- + +## Suggested Fixes + +### Fix 1: Correct path resolution in `service install` +```typescript +const baseDir = new URL(".", import.meta.url).pathname; +let daemonPath: string; +try { + daemonPath = Bun.resolveSync("./daemon/index.js", baseDir); +} catch (e) { + // Fallback for ESM/bundler compat + daemonPath = path.join(baseDir, "daemon", "index.js"); +} +``` + +Better yet: differentiate dev vs bundled mode. In dev, the daemon should be started as `bun run src/daemon/index.ts`, not as a pre-built `.js` file. + +### Fix 2: Check cocod before asking PM2 to start +```typescript +if (!(await isCocodInstalled(config.cocodPath))) { + console.error("cocod is not installed. Run 'routstrd onboard' first."); + process.exit(1); +} +``` + +### Fix 3: Pass `--cwd` to PM2 +```typescript +execSync( + `pm2 start "${daemonPath}" --name routstrd --interpreter bun --cwd "${process.cwd()}"`, + { stdio: "inherit" } +); +``` + +### Fix 4: Provide better error visibility +The PM2 start should capture stderr or use `pm2 logs` output so users see why the daemon crashed. + +--- + +## Related Files + +- `src/cli.ts` — `service install` command handler +- `src/daemon/index.ts` — daemon entrypoint, crashes without initialized wallet +- `src/start-daemon.ts` — dev-mode daemon launcher (uses `bun run src/daemon/index.ts`) +- `src/daemon/wallet/cocod-client.ts` — `isCocodInstalled()` check