286 lines
14 KiB
Markdown
286 lines
14 KiB
Markdown
# Startup Failure Admin DM Notification
|
|
|
|
> **STATUS UPDATE (follow-up phase):** The base feature (send a failure DM) is
|
|
> implemented and verified on VM410. However, because the failure happens inside the
|
|
> systemd `Restart=on-failure` loop (`RestartSec=10`), the admin now receives a *new*
|
|
> failure DM roughly every ~25 seconds, indefinitely (observed in the live message box).
|
|
> This follow-up adds **throttling/de-duplication** so the failure DM is sent **once per
|
|
> failure type until the next successful startup**, and makes the message **more
|
|
> descriptive** (per-relay connect state + the event kinds that were needed).
|
|
> See the "Follow-up: Throttling + Descriptive Failure DM" section at the end.
|
|
|
|
# Startup Failure Admin DM Notification (original plan)
|
|
|
|
## Goal
|
|
|
|
When the agent fails to start, send a best-effort encrypted DM to the administrator
|
|
describing which startup step failed and why, so the operator learns about the failure
|
|
without having to SSH in and read `journalctl`.
|
|
|
|
This directly addresses the real-world incident on VM410 where both `didactyl.service`
|
|
and `simon.service` were stuck restart-looping at startup step 14
|
|
(`Subscribe self-skill cache: failed (no skill events found after EOSE)`) and silently
|
|
stopped answering DMs.
|
|
|
|
## Background / Code References
|
|
|
|
- Startup is a linear checklist in [`main()`](src/main.c:1012). Each step uses helpers:
|
|
- [`startup_step_begin()`](src/main.c:303)
|
|
- [`startup_step_ok()`](src/main.c:309)
|
|
- [`startup_step_fail()`](src/main.c:320)
|
|
- On any fatal failure, the current pattern is:
|
|
`startup_step_fail(...)` -> `fprintf(stderr, ...)` -> cleanup calls -> `return 1;`
|
|
(e.g. the step-14 self-skill abort at [`src/main.c:1520`](src/main.c:1520)).
|
|
- The success path already sends an admin "started up" DM at
|
|
[`src/main.c:1663`](src/main.c:1663) via
|
|
[`nostr_handler_send_dm_auto()`](src/nostr_handler.c:3264).
|
|
- DM delivery requires (see [`nostr_handler_send_dm_with_role()`](src/nostr_handler.c:3133)):
|
|
- `g_cfg` and `g_pool` set (relay pool initialized),
|
|
- at least one **connected** relay (otherwise the kind-4 event is dropped:
|
|
"kind 4 event not queued: no connected relays" at [`src/nostr_handler.c:3203`](src/nostr_handler.c:3203)),
|
|
- a valid `cfg.admin.pubkey` (validated at step 4, [`src/main.c:1310`](src/main.c:1310)),
|
|
- the private key derived (always done before the checklist).
|
|
|
|
### Delivery-window reality
|
|
|
|
A failure DM is only *deliverable* from roughly step 11 onward (relay pool created +
|
|
relays connected + admin known). The step-14 failure we actually hit is inside that
|
|
window, so it will be delivered. Earlier failures (bad nsec, missing admin, LLM config,
|
|
no relay connection) have no working channel; per decision, those are **logged only**
|
|
(best-effort, no blocking retries).
|
|
|
|
## Design Decision (confirmed)
|
|
|
|
**Best-effort only.** Attempt the failure DM whenever relays + admin are available; if
|
|
the relay pool isn't connected or admin pubkey is unknown, skip the DM and rely on
|
|
existing stderr/journal logging. No blocking retry loop, no self-healing of the
|
|
underlying failure (that can be a separate plan).
|
|
|
|
## Approach
|
|
|
|
Add a single helper that builds and sends the failure DM, then call it from every fatal
|
|
startup-abort site (the points that currently do `startup_step_fail(...) ... return 1;`).
|
|
|
|
### 1. New helper in [`src/main.c`](src/main.c)
|
|
|
|
Add a static function near the other startup helpers (after
|
|
[`startup_step_fail()`](src/main.c:320)):
|
|
|
|
```c
|
|
static void notify_admin_startup_failure(const didactyl_config_t* cfg,
|
|
int step,
|
|
const char* label,
|
|
const char* detail);
|
|
```
|
|
|
|
Behavior:
|
|
- Guard clauses (skip + DEBUG_WARN if any precondition is missing):
|
|
- `cfg == NULL` or `cfg->admin.pubkey[0] == '\0'` -> log "admin unknown; cannot notify".
|
|
- `nostr_handler_connected_relay_count() <= 0` -> log "no connected relay; cannot notify".
|
|
- Build a concise human-readable message, e.g.:
|
|
```
|
|
<DisplayName> FAILED to start at <YYYY-MM-DD HH:MM:SS>.
|
|
Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.
|
|
Version <X>. Connected relays: <n>/<m>. The agent is NOT online and will not answer DMs until fixed.
|
|
```
|
|
- Reuse `nostr_handler_get_startup_display_name()` and `DIDACTYL_VERSION` exactly like
|
|
the success DM at [`src/main.c:1645`](src/main.c:1645) for naming/version consistency.
|
|
- Include `step`, `label`, and `detail` so each failure site is self-describing.
|
|
- Send via `nostr_handler_send_dm_auto(cfg->admin.pubkey, msg)`; if it returns non-zero,
|
|
`DEBUG_WARN` that the failure DM could not be delivered (do not change the exit code).
|
|
- Pump the poll loop briefly after sending so the async publish can flush before process
|
|
exit: call `nostr_handler_poll(...)` a few times (e.g. ~300--500 ms total) so the
|
|
queued kind-4 event is actually written to the relay sockets before cleanup/`return 1`.
|
|
(Without this, async publish may be discarded on immediate exit.)
|
|
|
|
### 2. Wire the helper into fatal abort sites
|
|
|
|
For each fatal `startup_step_fail(...)` that ends in `return 1;`, insert a call to
|
|
`notify_admin_startup_failure(&cfg, <step>, "<label>", "<detail>")` immediately after the
|
|
`startup_step_fail(...)` line and before the cleanup/`return 1`.
|
|
|
|
Priority sites (inside the deliverable window — relay pool up + admin known):
|
|
- Step 11 — discover self relay list failures
|
|
([`src/main.c:1420`](src/main.c:1420), [`src/main.c:1437`](src/main.c:1437)).
|
|
- Step 14 — self-skill cache failures (the incident we hit):
|
|
- subscribe failed ([`src/main.c:1481`](src/main.c:1481)),
|
|
- EOSE timeout ([`src/main.c:1496`](src/main.c:1496)),
|
|
- no skill events after EOSE ([`src/main.c:1521`](src/main.c:1521)).
|
|
- Step 15 — Subscribe DMs failed ([`src/main.c:1552`](src/main.c:1552)).
|
|
|
|
Earlier-window sites (relay/admin may not be ready — helper will self-skip and just log,
|
|
but we still call it uniformly for completeness):
|
|
- Step 1/3/4/5/6/7/8/9 fatal aborts. The helper's guard clauses make these safe no-ops
|
|
when delivery isn't possible.
|
|
|
|
Keep the existing `fprintf(stderr, ...)` and cleanup unchanged; the DM is purely additive
|
|
and must never alter the exit code or cleanup ordering.
|
|
|
|
### 3. Message size / safety
|
|
|
|
- Use a fixed stack buffer (e.g. `char msg[768];`) like the success DM at
|
|
[`src/main.c:1644`](src/main.c:1644).
|
|
- `snprintf` with truncation is acceptable; `detail` strings are short and controlled.
|
|
|
|
## Out of Scope
|
|
|
|
- No blocking retry / wait-for-relay loop (explicitly declined).
|
|
- No auto-remediation of the step-14 self-skill cause (e.g. auto-publishing the default
|
|
skill). Tracked separately if desired.
|
|
- No change to `deploy_lt.sh` post-deploy health checks (could be a follow-up: have the
|
|
deploy script verify each service reaches `READY` after restart).
|
|
|
|
## Verification
|
|
|
|
1. Build the static binary (`build_static.sh`) and confirm no warnings.
|
|
2. Local reproduction of step-14 failure: run an `--nsec`-only agent for an identity that
|
|
has connected relays but **no** published skill events, confirm:
|
|
- startup still aborts with exit code 1 (unchanged behavior),
|
|
- the admin receives a DM naming step 14 and the "no skill events found after EOSE"
|
|
detail.
|
|
3. Negative case: kill relay connectivity (or use an unreachable relay) and confirm the
|
|
helper logs "no connected relay; cannot notify" and does not crash or hang.
|
|
4. Confirm the success-path startup DM at [`src/main.c:1663`](src/main.c:1663) is
|
|
unaffected.
|
|
5. Deploy to VM410, restart `didactyl.service` / `simon.service`, and confirm the admin
|
|
npub receives the failure DM while the skill events are still missing; then publish a
|
|
skill and confirm normal `READY` + startup DM resumes.
|
|
|
|
---
|
|
|
|
# Follow-up: Throttling + Descriptive Failure DM
|
|
|
|
## Problems observed in production
|
|
|
|
1. **DM flood.** With `Restart=on-failure` / `RestartSec=10`, the step-14 failure recurs
|
|
on every restart and the agent reaches the deliverable window each time, so the admin
|
|
gets a fresh failure DM every ~25s indefinitely (confirmed in the live message box).
|
|
2. **Insufficient detail.** The current message names the step + detail, but not which
|
|
relays were attempted, whether each connected, nor which event kinds were required.
|
|
|
|
## Decisions (confirmed with user)
|
|
|
|
- **Throttle semantics:** send exactly once per failure type (keyed by step) until the
|
|
agent next starts successfully. The marker is cleared on `READY`. No periodic
|
|
reminders. A different failing step is a different "type" and notifies once.
|
|
- **Descriptive content:** include the attempted relay list with per-relay connect state,
|
|
and the event kinds that step 14 needs (31123 / 31124 skill events; 10123 adoption list).
|
|
|
|
## Part A -- Throttle / de-duplicate via on-disk marker
|
|
|
|
### Marker file location (systemd-sandbox safe)
|
|
|
|
- The unit runs with `WorkingDirectory=/home/<user>` and (hardened variant)
|
|
`ReadWritePaths=/home/<user>`, so only the agent home is writable.
|
|
- `cfg.tools.shell.working_directory` defaults to `"."` (see
|
|
[`config.c`](src/config.c:1459)), which resolves to the home dir at runtime.
|
|
- Use a hidden marker in the working directory: `./.didactyl_startup_failure`.
|
|
- For `didactyl` -> `/home/didactyl/.didactyl_startup_failure`
|
|
- For `simon` -> `/home/simon/.didactyl_startup_failure`
|
|
- Both inside the writable `ReadWritePaths`, so writes succeed under the sandbox.
|
|
- Resolve the directory the same way the local-file tools do: prefer
|
|
`cfg->tools.shell.working_directory` when non-empty, else `"."`
|
|
(mirror [`tool_local.c`](src/tools/tool_local.c:49)).
|
|
|
|
### Marker contents
|
|
|
|
Plain text, one record, easy to parse and inspect manually:
|
|
|
|
- `step=14`
|
|
- `label=Subscribe self-skill cache`
|
|
- `detail=no skill events found after EOSE`
|
|
- `last_sent_epoch=1781314692`
|
|
|
|
(Only `step` is strictly needed for the de-dup key; the rest aids debugging.)
|
|
|
|
### New helpers in [`src/main.c`](src/main.c)
|
|
|
|
Add small static helpers near
|
|
[`notify_admin_startup_failure()`](src/main.c:331):
|
|
|
|
- `startup_failure_marker_path(cfg, out, out_size)` -- builds
|
|
`<workdir>/.didactyl_startup_failure`.
|
|
- `startup_failure_already_notified(cfg, step)` -- returns 1 if the marker exists AND its
|
|
`step=` matches the current `step` (suppress); 0 otherwise (allow). Missing/unreadable
|
|
marker -> 0 (allow).
|
|
- `startup_failure_marker_write(cfg, step, label, detail)` -- writes/overwrites the marker
|
|
after an attempted send.
|
|
- `startup_failure_marker_clear(cfg)` -- `remove()` the marker; ignore ENOENT.
|
|
|
|
### Wiring
|
|
|
|
- In [`notify_admin_startup_failure()`](src/main.c:331): after the existing guard
|
|
clauses (config/admin/relay), check `startup_failure_already_notified(cfg, step)`.
|
|
If it returns 1, log a DEBUG_INFO suppression line and return without sending.
|
|
- On a successful send, call `startup_failure_marker_write(cfg, step, label, detail)`.
|
|
- If the same step keeps failing, only the first restart in the episode sends a DM;
|
|
subsequent restarts find the matching marker and stay silent.
|
|
- If a different step fails later, the `step=` mismatch allows exactly one new DM and
|
|
rewrites the marker with the new step.
|
|
- On the success path, clear the marker so the next failure episode notifies again. Add
|
|
`startup_failure_marker_clear(&cfg);` next to the READY success DM at
|
|
[`src/main.c:1663`](src/main.c:1663).
|
|
|
|
### Edge cases
|
|
|
|
- Marker write failure (e.g. read-only FS): log DEBUG_WARN and continue. Worst case we
|
|
fall back to current behavior (may re-send on next restart) -- acceptable, never fatal.
|
|
- Two services share the binary but use different home dirs, so markers never collide.
|
|
|
|
## Part B -- Descriptive failure message
|
|
|
|
Enhance the message built in
|
|
[`notify_admin_startup_failure()`](src/main.c:331) to append:
|
|
|
|
1. **Attempted relays + per-relay state.** Add a compact accessor to the nostr handler:
|
|
- New `nostr_handler_relay_state_summary(void)` in
|
|
[`nostr_handler.c`](src/nostr_handler.c) / [`nostr_handler.h`](src/nostr_handler.h)
|
|
returning a caller-freed compact string like
|
|
`relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connecting`.
|
|
- Implement by reusing the existing loop over `g_cfg->relays[]` +
|
|
`nostr_relay_pool_get_relay_status()` + `relay_status_str()` (patterns at
|
|
[`src/nostr_handler.c:3183`](src/nostr_handler.c:3183) and
|
|
[`src/nostr_handler.c:419`](src/nostr_handler.c:419)).
|
|
- Preferred over embedding raw JSON from
|
|
[`nostr_handler_relay_status_json()`](src/nostr_handler.c:3971) to keep the DM short.
|
|
2. **Needed event kinds (step-specific).** For step 14 append a fixed clause naming the
|
|
required events: kind 31123/31124 (skills) and kind 10123 (adoption list). Implement as
|
|
a small `startup_step_needed_events(step)` lookup in [`src/main.c`](src/main.c) that
|
|
returns a clause for known steps (14, 11, ...) and empty for others.
|
|
|
|
### Resulting message shape (example)
|
|
|
|
- `Didactyl FAILED to start at 2026-06-13 10:38:13.`
|
|
- `Startup step [14] Subscribe self-skill cache failed: no skill events found after EOSE.`
|
|
- `Version v0.2.48. Connected relays: 3/3.`
|
|
- `Relays: relay.laantungir.net=connected; relay.damus.io=connected; relay.primal.net=connected.`
|
|
- `Needed events: kind 31123/31124 (skills) and kind 10123 (adoption list) for this agent's pubkey.`
|
|
- `The agent is NOT online and will not answer DMs until fixed.`
|
|
|
|
Keep within the existing `failure_dm` buffer (bump to ~1536 if needed); `snprintf`
|
|
truncation remains safe.
|
|
|
|
## Implementation order
|
|
|
|
1. Add `nostr_handler_relay_state_summary()` to handler `.c`/`.h`.
|
|
2. Add marker helpers + `startup_step_needed_events()` to [`src/main.c`](src/main.c).
|
|
3. Update [`notify_admin_startup_failure()`](src/main.c:331): de-dup check, richer
|
|
message, marker write on send.
|
|
4. Add `startup_failure_marker_clear(&cfg)` on the READY success path
|
|
([`src/main.c:1663`](src/main.c:1663)).
|
|
5. Build (`make`), then `build_static.sh`, deploy, verify.
|
|
|
|
## Verification (follow-up)
|
|
|
|
1. **Throttle:** reproduce step-14 failure; confirm exactly one DM on the first failing
|
|
restart and no further DMs across subsequent restart-loop iterations (journal shows a
|
|
suppression line).
|
|
2. **Clear-on-success:** publish a skill so startup reaches `READY`; confirm marker is
|
|
removed and the normal startup DM is sent.
|
|
3. **Re-arm:** break startup again and confirm a single new DM is delivered.
|
|
4. **Descriptive content:** confirm the DM includes the per-relay state line and the
|
|
needed-events clause.
|
|
5. **Sandbox write:** on VM410 confirm the marker is created at
|
|
`/home/didactyl/.didactyl_startup_failure` (and `/home/simon/...`) without permission
|
|
errors under the systemd sandbox.
|