ngit-grasp/ec1f-administrator-observability-and-management-strategy.md

Administrator Observability and Management Strategy

ID: ec1f

Problem

Administrators need a cohesive strategy for monitoring their GRASP relay and taking action when problems arise. Currently we have:

1. Prometheus metrics - comprehensive time-series data (connections, git ops, events, sync)
2. Grafana dashboard - pre-built but untested and potentially out of date
3. Fragmented issues - each addressing pieces (dashboard UI, management API, storage limits, rate limiting, abuse detection) without a unified vision

Key gaps:

• No clear path from "I see a problem" to "I fix the problem"
• No strategy for configuration management (declarative vs. dynamic)
• Overlapping concerns across multiple issues without coordination
• Untested/outdated Prometheus setup and Grafana dashboard

Vision

Administrators should be able to:

1. Observe - See what's happening (metrics, dashboards, alerts)
2. Understand - Identify what needs attention (storage warnings, abusive users, large repos)
3. Act - Take corrective action (blacklist, delete, adjust quotas)
4. Enforce - Automatic protection (rate limits, storage quotas, abuse detection)

All while respecting deployment models:

• Declarative deployments (NixOS, Docker) - config from env/CLI only, dashboard read-only
• Dynamic deployments (manual) - optional GUI-based config changes that persist

Configuration Model

New Configuration Options

# Allow management dashboard to modify configuration
# Default: false (read-only dashboard)
NGIT_ALLOW_DYNAMIC_CONFIG=false

# Enforce declarative-only configuration (no local config file)
# Default: false (allow local config file if ALLOW_DYNAMIC_CONFIG=true)
NGIT_DECLARATIVE_ONLY=false

Configuration Hierarchy

1. CLI args / env vars / .env     (declarative, immutable)
         ↓
2. Local config file               (dynamic, if ALLOW_DYNAMIC_CONFIG=true)
         ↓
3. Runtime state                   (ephemeral)

Behavior Matrix

DECLARATIVE_ONLY	ALLOW_DYNAMIC_CONFIG	Behavior
true	* (ignored)	Pure declarative - no local config file, dashboard read-only
false	false	Declarative - dashboard read-only, no local config file
false	true	Dynamic - dashboard can modify config, persists to local file

NixOS/Docker deployments: Set NGIT_DECLARATIVE_ONLY=true to ensure purity

Manual deployments: Can enable NGIT_ALLOW_DYNAMIC_CONFIG=true for GUI-based management

Phased Roadmap

Phase 1: Observability (See what's happening)

Goal: Administrators can see current state and trends

Tasks:

• Test and update Prometheus setup documentation
• Test and update Grafana dashboard (verify all metrics work)
• Add missing storage metrics:
  • Total disk usage (git + database)
  • Available disk space percentage
  • Per-user storage usage (top N users)
  • Per-repository storage usage (largest repos)
• Add "attention needed" indicators:
  • Storage approaching limits (80%, 90%, 95%)
  • Flagged abusive IPs (from ConnectionTracker)
  • Large repositories (potential blacklist candidates)
  • High error rates (git operations, event rejections)

Depends on: Nothing

Related issues:

• 76fe (Repository Count Metric) - already implemented, verify it works
• 7d0b (Management Dashboard) - Phase 2 will build on this

Phase 2: Insights (Understand what needs action)

Goal: Administrators can identify problems and understand their scope

Tasks:

• Implement Management Dashboard UI (issue 7d0b)
  • Read-only view of all metrics
  • Storage breakdown visualization
  • Highlight problematic pubkeys/repos
  • Recent activity feed
  • System health indicators
• Dashboard respects configuration flags:
  • Show "read-only mode" banner when ALLOW_DYNAMIC_CONFIG=false
  • Hide action buttons in read-only mode
  • Display current config source (env, file, CLI)

Depends on: Phase 1 (metrics must exist to display)

Related issues:

• 7d0b (Management Dashboard) - PRIMARY ISSUE FOR THIS PHASE

Phase 3: Actions (Take action)

Goal: Administrators can take corrective action through authenticated API

Decision point: NIP-86 vs. simpler HTTP API?

Option A: NIP-86 Relay Management API (issue 2cdc)

• Standardized protocol, Nostr-native authentication
• More complex to implement
• Benefits entire Nostr ecosystem if we contribute patterns

Option B: Simple HTTP API

• Faster to implement
• Basic auth or API key
• Less standardization

Recommendation: Start with Option B (simple HTTP API), migrate to NIP-86 later if needed

Tasks:

• Implement authenticated management API:
  • Blacklist user (npub) from all operations
  • Blacklist repository (prevent pushes)
  • Delete repository (with confirmation)
  • Set per-user storage quota
  • Set per-repo storage quota
  • Force garbage collection on repository
  • View audit log of admin actions
• Integrate with dashboard UI (if ALLOW_DYNAMIC_CONFIG=true):
  • Action buttons for blacklist/delete/quota
  • Confirmation dialogs for destructive actions
  • Real-time feedback on action results
• Implement local config file persistence:
  • Define config file format (TOML? JSON?)
  • Load local config on startup (after env/CLI)
  • Save config changes from dashboard
  • Validate config on load

Depends on: Phase 2 (dashboard UI to trigger actions)

Related issues:

• 2cdc (NIP-86 Relay Management API) - DEFERRED in favor of simpler HTTP API first
• 7d0b (Management Dashboard) - UI integration

Phase 4: Enforcement (Automatic protection)

Goal: Relay automatically protects itself from abuse and resource exhaustion

Tasks:

• Implement storage limits and quota management (issue 8430):
  • Per-repository size limits
  • Per-user total storage limits
  • Relay-wide storage limits
  • Graceful degradation (read-only mode when approaching limits)
  • Clear error messages when quotas exceeded
• Implement defensive relay features (issue d6ee):
  • Phase 1 only: Explicit rate limits + max connections (APPROVED)
  • Defer per-IP enforcement until abuse detected in production
• Implement naughty list identification (issue 1f4f):
  • Detect invalid signatures, filter violations, DoS patterns
  • Automatic temporary bans for repeated violations
  • Integration with blacklist system from Phase 3

Depends on: Phase 3 (manual blacklist/quota system must exist for overrides)

Related issues:

• 8430 (Storage Limits and Quota Management) - PRIMARY ISSUE FOR THIS PHASE
• d6ee (Defensive Relay Features) - Phase 1 only (config-based protection)
• 1f4f (Poor Naughty List Identification) - Abuse detection

Issue Overlap Analysis

Primary Overlaps

Storage Management:

• 7d0b (Management Dashboard) - Display storage metrics
• 8430 (Storage Limits) - Enforce storage quotas
• 2cdc (NIP-86 Management API) - Set quotas via API
• Resolution: 8430 owns enforcement, 7d0b owns display, 2cdc owns API (deferred)

Abuse Detection & Response:

• d6ee (Defensive Relay Features) - Rate limiting, connection limits
• 1f4f (Poor Naughty List) - Detect malicious behavior
• 2cdc (NIP-86 Management API) - Blacklist users/repos
• Resolution: d6ee owns prevention, 1f4f owns detection, 2cdc owns manual response (deferred)

Configuration & Monitoring:

• 7d0b (Management Dashboard) - Display config and metrics
• 2cdc (NIP-86 Management API) - Modify config via API
• 76fe (Repository Count Metric) - Specific metric implementation
• Resolution: 7d0b owns display, 2cdc owns modification (deferred), 76fe is complete

Coordination Requirements

Before starting work on any overlapping issue:

1. Check this strategy issue (ec1f) for current phase
2. Verify dependencies are complete
3. Coordinate scope with related issues
4. Update this issue with progress

Implementation Sequence

Phase 1: Observability
├── Update Prometheus/Grafana setup (1-2 days)
├── Add storage metrics (2-3 days)
└── Add attention indicators (1-2 days)
    Total: ~1 week

Phase 2: Insights
├── Management Dashboard UI (3-5 days)
└── Configuration flag integration (1 day)
    Total: ~1 week
    Depends on: Phase 1

Phase 3: Actions
├── Simple HTTP management API (3-4 days)
├── Dashboard action integration (2-3 days)
└── Local config file persistence (2-3 days)
    Total: ~1.5 weeks
    Depends on: Phase 2

Phase 4: Enforcement
├── Storage limits (issue 8430) (1-2 weeks)
├── Defensive features Phase 1 (issue d6ee) (2-3 hours)
└── Naughty list detection (issue 1f4f) (1 week)
    Total: ~2-3 weeks
    Depends on: Phase 3

Total estimated time: 5-7 weeks for complete administrator experience

Success Criteria

Phase 1 Complete

• Prometheus metrics endpoint tested and documented
• Grafana dashboard loads without errors
• All metrics display current values
• Storage metrics available (disk usage, per-user, per-repo)
• Attention indicators highlight problems

Phase 2 Complete

• Management dashboard accessible via HTTP endpoint
• Dashboard displays all metrics with visualizations
• Dashboard respects ALLOW_DYNAMIC_CONFIG flag
• Mobile-friendly responsive design
• No significant performance impact on relay

Phase 3 Complete

• Authenticated management API functional
• Can blacklist users and repositories
• Can set storage quotas
• Can delete repositories with confirmation
• Audit log tracks all admin actions
• Local config file persists changes (when enabled)

Phase 4 Complete

• Storage quotas enforced automatically
• Graceful degradation when approaching limits
• Rate limiting prevents DoS attacks
• Abuse detection flags problematic users
• Clear error messages guide users

Overall Success

• Administrator can monitor relay health without SSH access
• Administrator can take corrective action through dashboard
• Declarative deployments remain pure (NixOS, Docker)
• Dynamic deployments support GUI-based management
• No manual intervention needed for common abuse scenarios

Open Questions

1. Config file format: TOML, JSON, or YAML for local config file?

   • Recommendation: TOML (matches Rust ecosystem, human-friendly)

2. Authentication for management API: Basic auth, API key, or Nostr signature?

   • Recommendation: Start with API key (simple), migrate to Nostr signatures later

3. Dashboard framework: Server-rendered HTML or SPA?

   • Recommendation: Server-rendered (htmx or Alpine.js) for simplicity

4. NIP-86 timeline: When to implement full NIP-86 compliance?

   • Recommendation: After Phase 3 proves the simple API works

5. Prometheus vs. custom metrics: Should we add non-Prometheus metrics for dashboard?

   • Recommendation: Prometheus only (single source of truth)

Related Issues

Directly related (must coordinate):

• 7d0b (Management Dashboard) - Phase 2 primary issue
• 2cdc (NIP-86 Relay Management API) - Deferred to post-Phase 3
• 8430 (Storage Limits and Quota Management) - Phase 4 primary issue
• d6ee (Defensive Relay Features) - Phase 4 (Phase 1 only)
• 1f4f (Poor Naughty List Identification) - Phase 4

Indirectly related:

• 76fe (Repository Count Metric) - Already implemented, verify in Phase 1
• cc36 (Database Backend Evaluation) - May affect storage metrics
• b905 (Deletion Request Support) - May integrate with delete repository action

Key Decisions

2026-01-14 Strategy Review

Prometheus Metrics:

• Private, API key required - Always authenticated via NGIT_METRICS_API_KEY
• Supports fleet monitoring (multiple instances reporting to central Prometheus)
• Contains sensitive admin data (per-user storage, per-repo sizes, etc.)

Public Relay Stats:

• Delivered via NIP-11 - Extend relay info document with stats section
• Cache expensive metrics rather than computing on each request
• No separate HTTP endpoint needed

User Storage Info:

• Delivered via replaceable Nostr event - User-specific storage usage/limits
• Requires NIP-42 authentication to query own data
• Preferred approach for all sensitive user-specific data via Nostr

Metrics Separation:

• Public info (NIP-11): Safe aggregate stats for relay selection
• Private metrics (Prometheus): Detailed operational data for admins
• User-specific (Nostr events): Personal storage usage for authenticated users

Next Steps

The current phased roadmap above is SUPERSEDED pending proper planning:

1. Strategy Document - docs/explanation/metrics-and-administration.md

   • Thoughtful document outlining administrator workflow
   • What metrics support each workflow step
   • How different audiences (public, users, admins) access information

2. Target Architecture - Define technical architecture based on strategy

   • Component interactions
   • Data flows
   • API designs

3. Implementation Plan - Only after strategy and architecture are complete

   • Phased delivery
   • Dependencies
   • Estimates

Progress

2026-01-14 [Session 16:30]

• Created strategy issue
• Analyzed existing issues for overlap
• Defined configuration model (DECLARATIVE_ONLY, ALLOW_DYNAMIC_CONFIG)
• Established 4-phase roadmap
• Identified decision points (NIP-86 vs. simple API)

2026-01-14 [Session ~17:00]

• Security review of public metrics exposure
• Decision: Prometheus metrics require API key authentication
• Decision: Public relay stats via NIP-11 (cached, not Prometheus)
• Decision: User storage info via replaceable Nostr events (NIP-42 auth)
• Recognized need for proper planning sequence: strategy → architecture → implementation
• Created worktree: ec1f-administrator-observability-and-management-strategy
• Wrote strategy document: docs/explanation/metrics-and-administration.md
  • Defined three audiences (public, authenticated users, administrators)
  • Documented five administrator workflows
  • Specified information architecture for each audience
  • Outlined configuration model and security considerations
• Added target architecture to strategy document:
  • System overview with component diagram
  • Stats Cache (60s refresh for NIP-11/homepage)
  • Prometheus metrics with API key auth
  • Management Service (REST API + SPA dashboard)
  • User storage info via Nostr events (application-specific kind)
  • Data flow diagrams
  • Configuration and security architecture
• Resolved open questions: event kinds (app-specific), caching (60s), dashboard (SPA), audit (log file)
• Refined architecture based on feedback:
  • Stats cache: throttled (not periodic), reuse existing homepage/NIP-11
  • Prometheus: multiple hashed API keys, generated via admin UI
  • Management API: NIP-86 (not REST), research existing implementations
  • Audit logging: standard INFO messages in app logs
  • User storage info: verify nostr-relay-builder support, NIP-44 fallback
• Next: Write implementation plan

2026-01-14 [Session ~18:00]

• Completed implementation plan in strategy document
• Broke work into 7 phases (Phase 0-7) with clear dependencies
• Identified within-phase parallelization opportunities:
  • Phase 0: 2 parallel research tracks (NIP-86, nostr-relay-builder)
  • Phase 1: 2 parallel tracks (stats cache, storage metrics)
  • Phase 2: 2 parallel tracks (NIP-11 extension, user storage events)
  • Phase 3: Single track (Prometheus auth)
  • Phase 4: Single track (NIP-86 API)
  • Phase 5: Single track (dashboard, limited internal parallelization)
  • Phase 6: Sequential tasks (quota enforcement)
  • Phase 7: 2 parallel tracks (rate limiting, abuse detection)
• Mapped all existing issues to phases:
  • 76fe: Phase 1 Track B (verification)
  • 7d0b: Phase 2 Track A + Phase 5 (9-12 days)
  • 2cdc: Phase 0 Track A + Phase 4 + Phase 5 (8-10 days)
  • 8430: Phase 4 + Phase 6 (9-11 days)
  • d6ee: Phase 7 Track A (2-3 days)
  • 1f4f: Phase 7 Track B (2-3 days)
• Effort estimates:
  • Sequential: 31-39 days (6-8 weeks)
  • With within-phase parallelization: 25-30 days (5-6 weeks)
• Added risk mitigation strategies for research and technical risks
• Defined success criteria for each phase and overall
• Each phase must commit working code with passing tests
• Code/commits must not reference phase numbers
• Ready for implementation to begin with Phase 0 research

2026-01-14 [Session ~18:30]

• Corrected implementation plan based on feedback
• Fixed incorrect parallelization claims:
  • Phase 3 and 4 cannot run in parallel (Phase 4 depends on Phase 0 Track A research)
  • Split original Phase 3 into two sequential phases (Prometheus auth, then NIP-86 API)
  • Clarified "within-phase" vs "cross-phase" parallelization
• Added explicit deliverable requirements:
  • Each phase must commit changes with all tests passing
  • No phase references in code comments or commit messages
  • Phase 0 commits research docs, all other phases commit working code
• Updated phase count from 6 to 7 phases
• Corrected dependency graph to show true sequential requirements
• More realistic effort estimates accounting for sequential dependencies

2026-01-15 [Session 09:54]

• Completed Phase 0 Track A: NIP-86 implementation research
• Used subagent to research nostr-sdk 0.43 support for NIP-86
• Key findings:
  • ✅ NIP-98 HTTP auth fully supported in nostr-sdk (can leverage for authentication)
  • ❌ NIP-86 JSON-RPC protocol NOT provided (must implement from scratch)
  • Recommended approach: Implement NIP-86 using existing NIP-98 auth
  • Estimated effort: 6-11 days for complete implementation
• Deliverable: docs/research/nip-86-implementation-options.md committed
• Next: Phase 0 Track B (nostr-relay-builder user storage events) - deferred for now
• Ready to proceed with Phase 1 implementation

2026-01-15 [Session 11:05]

• Critical architectural decision: Identified gap in design - NIP-86 is for actions, not queries
• Researched how to return sensitive data to authenticated users (admins and regular users)
• Launched 3 parallel research subagents:
  1. NIP-86 query response patterns in spec and rust-nostr
  2. Pyramid relay admin patterns (real-world implementation)
  3. Feasibility of pyramid-style approach with rust-nostr
• Key findings from research:
  • ✅ NIP-86 spec uses HTTP POST/response (NOT Nostr events for query results)
  • ✅ Pyramid uses ephemeral events to specific WebSocket connections for queries
  • ❌ nostr-relay-builder does NOT support connection-specific messaging
  • ❌ Pyramid approach requires forking nostr-relay-builder or custom WebSocket handler
• Decision: Use HTTP-based NIP-86 for all queries and actions
  • Queries: HTTP GET/POST with NIP-98 auth → JSON response
  • Actions: HTTP POST with NIP-98 auth → JSON response
  • Both admin and user queries use same pattern
  • Simpler, standard, no framework modifications needed
• Deliverables committed:
  • docs/research/nip-86-query-responses.md - NIP-86 spec analysis
  • docs/research/pyramid-admin-patterns.md - Pyramid implementation study
  • docs/research/pyramid-style-queries-design.md - Feasibility analysis for rust-nostr
• Next: Update strategy document to reflect HTTP-based approach for queries
• Phase 0 research complete, ready to finalize architecture and proceed to Phase 1

2026-01-15 [Session 11:38]

• Final Phase 0 research: Explored forking rust-nostr to support pyramid-style queries
• Comprehensive analysis of what it would take to modify nostr-relay-builder
• Key findings:
  • Implementation effort: 5-7 days (vs 1-2 days for HTTP NIP-86)
  • Ongoing maintenance: 18-29 hours/month (vs 1-2 hours/month for HTTP)
  • rust-nostr is extremely active: 305 commits in 15 days (~20/day)
  • Breaking changes in ~50% of releases
  • Pyramid uses khatru framework (Go), NOT nostr-relay-builder
  • Upstream acceptance likelihood: Low (10-20%)
• Final decision confirmed: HTTP-based NIP-86 is the correct approach
  • No fork needed
  • Standards-compliant
  • Minimal maintenance burden
  • Achieves same functional goals
• Deliverable committed:
  • docs/research/rust-nostr-fork-analysis.md - Complete fork analysis (1,316 lines)
• Phase 0 research COMPLETE ✅
• All 4 research documents committed and ready for review
• Ready to update strategy document and begin Phase 1 implementation