# Startup Failure Admin DM Notification

> **STATUS UPDATE (follow-up phase):** The base feature (send a failure DM) is
> implemented and verified on VM410. However, because the failure happens inside the
> systemd `Restart=on-failure` loop (`RestartSec=10`), the admin now receives a *new*
> failure DM roughly every ~25 seconds, indefinitely (observed in the live message box).
> This follow-up adds **throttling/de-duplication** so the failure DM is sent **once per
> failure type until the next successful startup**, and makes the message **more
> descriptive** (per-relay connect state + the event kinds that were needed).
> See the "Follow-up: Throttling + Descriptive Failure DM" section at the end.

# Startup Failure Admin DM Notification (original plan)

## Goal

When the agent fails to start, send a best-effort encrypted DM to the administrator
describing which startup step failed and why, so the operator learns about the failure
without having to SSH in and read `journalctl`.

This directly addresses the real-world incident on VM410 where both `didactyl.service`
and `simon.service` were stuck restart-looping at startup step 14
(`Subscribe self-skill cache: failed (no skill events found after EOSE)`) and silently
stopped answering DMs.

## Background / Code References

- Startup is a linear checklist in [`main()`](src/main.c:1012). Each step uses helpers:
  - [`startup_step_begin()`](src/main.c:303)
  - [`startup_step_ok()`](src/main.c:309)
  - [`startup_step_fail()`](src/main.c:320)
- On any fatal failure, the current pattern is:
  `startup_step_fail(...)` -> `fprintf(stderr, ...)` -> cleanup calls -> `return 1;`
  (e.g. the step-14 self-skill abort at [`src/main.c:1520`](src/main.c:1520)).
- The success path already sends an admin "started up" DM at
  [`src/main.c:1663`](src/main.c:1663) via
  [`nostr_handler_send_dm_auto()`](src/nostr_handler.c:3264).
- DM delivery requires (see [`nostr_handler_send_dm_with_role()`](src/nostr_handler.c:3133)):
  - `g_cfg` and `g_pool` set (relay pool initialized),
  - at least one **connected** relay (otherwise the kind-4 event is dropped:
    "kind 4 event not queued: no connected relays" at [`src/nostr_handler.c:3203`](src/nostr_handler.c:3203)),
  - a valid `cfg.admin.pubkey` (validated at step 4, [`src/main.c:1310`](src/main.c:1310)),
  - the private key derived (always done before the checklist).

### Delivery-window reality

A failure DM is only *deliverable* from roughly step 11 onward (relay pool created +
relays connected + admin known). The step-14 failure we actually hit is inside that
window, so it will be delivered. Earlier failures (bad nsec, missing admin, LLM config,
no relay connection) have no working channel; per decision, those are **logged only**
(best-effort, no blocking retries).

## Design Decision (confirmed)

**Best-effort only.** Attempt the failure DM whenever relays + admin are available; if
the relay pool isn't connected or admin pubkey is unknown, skip the DM and rely on
existing stderr/journal logging. No blocking retry loop, no self-healing of the
underlying failure (that can be a separate plan).

## Approach

Add a single helper that builds and sends the failure DM, then call it from every fatal
startup-abort site (the points that currently do `startup_step_fail(...) ... return 1;`).

### 1. New helper in [`src/main.c`](src/main.c)

Add a static function near the other startup helpers (after
[`startup_step_fail()`](src/main.c:320)):

```c
static void notify_admin_startup_failure(const didactyl_config_t* cfg,
                                         int step,
                                         const char* label,
                                         const char* detail);
```

Behavior:
- Guard clauses (skip + DEBUG_WARN if any precondition is missing):
  - `cfg == NULL` or `cfg->admin.pubkey[0] == '\0'` -> log "admin unknown; cannot notify".
  - `nostr_handler_connected_relay_count() <= 0` -> log "no connected relay; cannot notify".
- Build a concise human-readable message, e.g.:
  ```
  <DisplayName> FAILED to start at <YYYY-MM-DD HH:MM:SS>.
  Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.
  Version <X>. Connected relays: <n>/<m>. The agent is NOT online and will not answer DMs until fixed.
  ```
  - Reuse `nostr_handler_get_startup_display_name()` and `DIDACTYL_VERSION` exactly like
    the success DM at [`src/main.c:1645`](src/main.c:1645) for naming/version consistency.
  - Include `step`, `label`, and `detail` so each failure site is self-describing.
- Send via `nostr_handler_send_dm_auto(cfg->admin.pubkey, msg)`; if it returns non-zero,
  `DEBUG_WARN` that the failure DM could not be delivered (do not change the exit code).
- Pump the poll loop briefly after sending so the async publish can flush before process
  exit: call `nostr_handler_poll(...)` a few times (e.g. ~300--500 ms total) so the
  queued kind-4 event is actually written to the relay sockets before cleanup/`return 1`.
  (Without this, async publish may be discarded on immediate exit.)

### 2. Wire the helper into fatal abort sites

For each fatal `startup_step_fail(...)` that ends in `return 1;`, insert a call to
`notify_admin_startup_failure(&cfg, <step>, "<label>", "<detail>")` immediately after the
`startup_step_fail(...)` line and before the cleanup/`return 1`.

Priority sites (inside the deliverable window — relay pool up + admin known):
- Step 11 — discover self relay list failures
  ([`src/main.c:1420`](src/main.c:1420), [`src/main.c:1437`](src/main.c:1437)).
- Step 14 — self-skill cache failures (the incident we hit):
  - subscribe failed ([`src/main.c:1481`](src/main.c:1481)),
  - EOSE timeout ([`src/main.c:1496`](src/main.c:1496)),
  - no skill events after EOSE ([`src/main.c:1521`](src/main.c:1521)).
- Step 15 — Subscribe DMs failed ([`src/main.c:1552`](src/main.c:1552)).

Earlier-window sites (relay/admin may not be ready — helper will self-skip and just log,
but we still call it uniformly for completeness):
- Step 1/3/4/5/6/7/8/9 fatal aborts. The helper's guard clauses make these safe no-ops
  when delivery isn't possible.

Keep the existing `fprintf(stderr, ...)` and cleanup unchanged; the DM is purely additive
and must never alter the exit code or cleanup ordering.

### 3. Message size / safety

- Use a fixed stack buffer (e.g. `char msg[768];`) like the success DM at
  [`src/main.c:1644`](src/main.c:1644).
- `snprintf` with truncation is acceptable; `detail` strings are short and controlled.

## Out of Scope

- No blocking retry / wait-for-relay loop (explicitly declined).
- No auto-remediation of the step-14 self-skill cause (e.g. auto-publishing the default
  skill). Tracked separately if desired.
- No change to `deploy_lt.sh` post-deploy health checks (could be a follow-up: have the
  deploy script verify each service reaches `READY` after restart).

## Verification

1. Build the static binary (`build_static.sh`) and confirm no warnings.
2. Local reproduction of step-14 failure: run an `--nsec`-only agent for an identity that
   has connected relays but **no** published skill events, confirm:
   - startup still aborts with exit code 1 (unchanged behavior),
   - the admin receives a DM naming step 14 and the "no skill events found after EOSE"
     detail.
3. Negative case: kill relay connectivity (or use an unreachable relay) and confirm the
   helper logs "no connected relay; cannot notify" and does not crash or hang.
4. Confirm the success-path startup DM at [`src/main.c:1663`](src/main.c:1663) is
   unaffected.
5. Deploy to VM410, restart `didactyl.service` / `simon.service`, and confirm the admin
   npub receives the failure DM while the skill events are still missing; then publish a
   skill and confirm normal `READY` + startup DM resumes.

---

# Follow-up: Throttling + Descriptive Failure DM

## Problems observed in production

1. **DM flood.** With `Restart=on-failure` / `RestartSec=10`, the step-14 failure recurs
   on every restart and the agent reaches the deliverable window each time, so the admin
   gets a fresh failure DM every ~25s indefinitely (confirmed in the live message box).
2. **Insufficient detail.** The current message names the step + detail, but not which
   relays were attempted, whether each connected, nor which event kinds were required.

## Decisions (confirmed with user)

- **Throttle semantics:** send exactly once per failure type (keyed by step) until the
  agent next starts successfully. The marker is cleared on `READY`. No periodic
  reminders. A different failing step is a different "type" and notifies once.
- **Descriptive content:** include the attempted relay list with per-relay connect state,
  and the event kinds that step 14 needs (31123 / 31124 skill events; 10123 adoption list).

## Part A -- Throttle / de-duplicate via on-disk marker

### Marker file location (systemd-sandbox safe)

- The unit runs with `WorkingDirectory=/home/<user>` and (hardened variant)
  `ReadWritePaths=/home/<user>`, so only the agent home is writable.
- `cfg.tools.shell.working_directory` defaults to `"."` (see
  [`config.c`](src/config.c:1459)), which resolves to the home dir at runtime.
- Use a hidden marker in the working directory: `./.didactyl_startup_failure`.
  - For `didactyl` -> `/home/didactyl/.didactyl_startup_failure`
  - For `simon` -> `/home/simon/.didactyl_startup_failure`
  - Both inside the writable `ReadWritePaths`, so writes succeed under the sandbox.
- Resolve the directory the same way the local-file tools do: prefer
  `cfg->tools.shell.working_directory` when non-empty, else `"."`
  (mirror [`tool_local.c`](src/tools/tool_local.c:49)).

### Marker contents

Plain text, one record, easy to parse and inspect manually:

- `step=14`
- `label=Subscribe self-skill cache`
- `detail=no skill events found after EOSE`
- `last_sent_epoch=1781314692`

(Only `step` is strictly needed for the de-dup key; the rest aids debugging.)

### New helpers in [`src/main.c`](src/main.c)

Add small static helpers near
[`notify_admin_startup_failure()`](src/main.c:331):

- `startup_failure_marker_path(cfg, out, out_size)` -- builds
  `<workdir>/.didactyl_startup_failure`.
- `startup_failure_already_notified(cfg, step)` -- returns 1 if the marker exists AND its
  `step=` matches the current `step` (suppress); 0 otherwise (allow). Missing/unreadable
  marker -> 0 (allow).
- `startup_failure_marker_write(cfg, step, label, detail)` -- writes/overwrites the marker
  after an attempted send.
- `startup_failure_marker_clear(cfg)` -- `remove()` the marker; ignore ENOENT.

### Wiring

- In [`notify_admin_startup_failure()`](src/main.c:331): after the existing guard
  clauses (config/admin/relay), check `startup_failure_already_notified(cfg, step)`.
  If it returns 1, log a DEBUG_INFO suppression line and return without sending.
  - On a successful send, call `startup_failure_marker_write(cfg, step, label, detail)`.
  - If the same step keeps failing, only the first restart in the episode sends a DM;
    subsequent restarts find the matching marker and stay silent.
  - If a different step fails later, the `step=` mismatch allows exactly one new DM and
    rewrites the marker with the new step.
- On the success path, clear the marker so the next failure episode notifies again. Add
  `startup_failure_marker_clear(&cfg);` next to the READY success DM at
  [`src/main.c:1663`](src/main.c:1663).

### Edge cases

- Marker write failure (e.g. read-only FS): log DEBUG_WARN and continue. Worst case we
  fall back to current behavior (may re-send on next restart) -- acceptable, never fatal.
- Two services share the binary but use different home dirs, so markers never collide.

## Part B -- Descriptive failure message

Enhance the message built in
[`notify_admin_startup_failure()`](src/main.c:331) to append:

1. **Attempted relays + per-relay state.** Add a compact accessor to the nostr handler:
   - New `nostr_handler_relay_state_summary(void)` in
     [`nostr_handler.c`](src/nostr_handler.c) / [`nostr_handler.h`](src/nostr_handler.h)
     returning a caller-freed compact string like
     `relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connecting`.
   - Implement by reusing the existing loop over `g_cfg->relays[]` +
     `nostr_relay_pool_get_relay_status()` + `relay_status_str()` (patterns at
     [`src/nostr_handler.c:3183`](src/nostr_handler.c:3183) and
     [`src/nostr_handler.c:419`](src/nostr_handler.c:419)).
   - Preferred over embedding raw JSON from
     [`nostr_handler_relay_status_json()`](src/nostr_handler.c:3971) to keep the DM short.
2. **Needed event kinds (step-specific).** For step 14 append a fixed clause naming the
   required events: kind 31123/31124 (skills) and kind 10123 (adoption list). Implement as
   a small `startup_step_needed_events(step)` lookup in [`src/main.c`](src/main.c) that
   returns a clause for known steps (14, 11, ...) and empty for others.

### Resulting message shape (example)

- `Didactyl FAILED to start at 2026-06-13 10:38:13.`
- `Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.`
- `Version v0.2.48. Connected relays: 3/3.`
- `Relays: relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connected.`
- `Needed events: kind 31123/31124 (skills) and kind 10123 (adoption list) for this agent's pubkey.`
- `The agent is NOT online and will not answer DMs until fixed.`

Keep within the existing `failure_dm` buffer (bump to ~1536 if needed); `snprintf`
truncation remains safe.

## Implementation order

1. Add `nostr_handler_relay_state_summary()` to handler `.c`/`.h`.
2. Add marker helpers + `startup_step_needed_events()` to [`src/main.c`](src/main.c).
3. Update [`notify_admin_startup_failure()`](src/main.c:331): de-dup check, richer
   message, marker write on send.
4. Add `startup_failure_marker_clear(&cfg)` on the READY success path
   ([`src/main.c:1663`](src/main.c:1663)).
5. Build (`make`), then `build_static.sh`, deploy, verify.

## Verification (follow-up)

1. **Throttle:** reproduce step-14 failure; confirm exactly one DM on the first failing
   restart and no further DMs across subsequent restart-loop iterations (journal shows a
   suppression line).
2. **Clear-on-success:** publish a skill so startup reaches `READY`; confirm marker is
   removed and the normal startup DM is sent.
3. **Re-arm:** break startup again and confirm a single new DM is delivered.
4. **Descriptive content:** confirm the DM includes the per-relay state line and the
   needed-events clause.
5. **Sandbox write:** on VM410 confirm the marker is created at
   `/home/didactyl/.didactyl_startup_failure` (and `/home/simon/...`) without permission
   errors under the systemd sandbox.
