ngit-grasp/0f73-shutdown-handler-not-triggered-by-systemd.md

Shutdown Handler Not Triggered by Systemd SIGTERM

ID: 0f73

Problem

When systemd stops the ngit-grasp service, the graceful shutdown handler in src/main.rs never runs, preventing critical state files from being saved:

• purgatory-state.json - In-memory purgatory events awaiting git data
• rejected-events-cache.json - Two-tier rejected events index

Root Cause: The application only listens for SIGINT (Ctrl+C) via signal::ctrl_c() on line 220 of src/main.rs, but systemd sends SIGTERM when stopping services. The tokio::select! never triggers the shutdown branch, so the cleanup code (lines 225-249) never executes.

Evidence:

Jan 22 12:52:16 nixos-vps1 systemd[1]: Stopping ngit-grasp GRASP relay (relay-ngit-archive)...
Jan 22 12:52:16 nixos-vps1 systemd[1]: ngit-grasp-relay-ngit-archive.service: Deactivated successfully.

No "Received shutdown signal" or "Purgatory state saved" logs appear, confirming the handler never ran.

Impact:

• Purgatory state is lost on every service restart
• Rejected events cache is lost on every service restart
• Events in purgatory must be re-synced from scratch
• Rejected events may be re-downloaded unnecessarily
• Placeholder git refs are not cleaned up

Plan

• Phase 1: Fix signal handling in src/main.rs
  • Replace signal::ctrl_c() with Unix signal handling
  • Listen for both SIGINT and SIGTERM
  • Test graceful shutdown with both signals
• Phase 2: Add systemd integration (optional enhancement)
  • Consider Type = "notify" with sd-notify protocol
  • Add proper shutdown timeout configuration
  • Document systemd service behavior
• Phase 3: Add integration test
  • Test that state files are created on shutdown
  • Verify state restoration on startup
  • Test with both SIGINT and SIGTERM

Progress

2026-01-22 [Session 13:00]

• Identified: User reported missing state files after stopping archive service
• Investigated: Checked systemd logs, confirmed SIGTERM was sent but handler didn't run
• Root cause: Application only listens for SIGINT, not SIGTERM
• Created issue to track fix

Notes

Related Code:

• src/main.rs:220 - Current signal handling (SIGINT only)
• src/main.rs:225-249 - Shutdown cleanup code that never runs
• src/purgatory/mod.rs - save_to_disk() and restore_from_disk() methods
• src/sync/rejected_index.rs - Rejected events cache persistence
• nix/module.nix:321-388 - Systemd service configuration

Solution Options:

1. Fix signal handling (recommended) - Listen for both SIGINT and SIGTERM
2. Systemd workaround - Configure KillSignal = "SIGINT" (not standard)
3. Full systemd integration - Use sd-notify protocol (best long-term)

Testing:

• Manual test: systemctl stop ngit-grasp-{instance} should create state files
• Manual test: kill -TERM <pid> should trigger graceful shutdown
• Manual test: Ctrl+C should still work in development
• Integration test: Verify state files exist after shutdown signal

References:

• Tokio signal handling: https://docs.rs/tokio/latest/tokio/signal/unix/
• Systemd signals: https://www.freedesktop.org/software/systemd/man/systemd.kill.html
• Default systemd behavior: Sends SIGTERM, waits TimeoutStopSec (default 90s), then SIGKILL