ngit-grasp/8430-storage-limits-and-quota-management.md

Storage Limits and Quota Management

ID: 8430

Issue Summary

We currently have no storage limits or quota enforcement, allowing users to push arbitrarily large repositories (e.g., 20GB+) and potentially fill the relay's disk. We need configurable storage limits at multiple levels and graceful handling when approaching disk capacity.

Current State

• No limits on individual repository size
• No limits on total storage per user/npub
• No limits on total relay storage
• No graceful degradation when disk approaches full
• No warnings or alerts for storage usage
• Risk of disk exhaustion causing relay crash or data corruption

Required Storage Controls

1. Repository-level limits

• Max size per individual repository
• Reject pushes that would exceed repo limit
• Clear error messages when limit hit
• Consider limits on individual objects (blob size, pack size)

2. User-level limits

• Max total storage per npub across all their repos
• Track aggregate storage usage per user
• Enforce quota on new pushes
• Consider different limits for maintainers vs contributors

3. Relay-level limits

• Max total git storage for the entire relay
• Max total database storage
• Reserved space threshold (stop accepting new data before disk 100% full)
• Emergency mode when critically low on space

4. Graceful degradation

• Warning thresholds (e.g., 80%, 90%, 95% full)
• Read-only mode when approaching limits
• Clear communication to clients about storage status
• Prevent complete disk exhaustion
• Allow emergency cleanup operations

5. Configurability

• All limits should be configurable
• Sensible defaults for different deployment sizes
• Ability to set per-user overrides (allowlist power users)
• Runtime configuration updates without restart

Investigation Tasks

Storage Tracking

• Identify where git repository data is stored
• Determine how to efficiently calculate repository sizes
• Design schema for tracking storage quotas
• Check if git provides built-in size calculation tools
• Research efficient incremental size tracking (avoid recalculating full repo size on every push)

Limit Enforcement Points

• Identify where to enforce limits in git push flow
• Determine pre-receive vs post-receive hook approach
• Find where to check quotas before accepting data
• Handle partial pushes (some refs succeed, others rejected)

Disk Space Monitoring

• Implement filesystem monitoring for available space
• Design alerting mechanism for low disk space
• Create read-only mode for emergency situations
• Determine how to communicate storage status to clients

Configuration Design

• Design configuration schema for limits
• Decide on default values for different deployment sizes
• Create override mechanism for trusted users
• Document migration path for existing deployments

Error Handling

• Design clear error messages for quota exceeded
• Provide actionable feedback (how much over limit, what to do)
• Handle edge cases (concurrent pushes, race conditions)
• Test cleanup after failed push due to quota

Research Other Implementations

• Check how GitHub/GitLab handle repository size limits
• Review how git hosting platforms enforce quotas
• Look at cgit, gitolite, gitea quota implementations
• Research Nostr relay storage limit patterns (if applicable)

Questions to Answer

1. How to efficiently track storage usage?

   • Calculate on-demand vs maintain running totals?
   • Track at object level or repository level?
   • How to handle git deduplication (shared objects)?

2. What are reasonable default limits?

   • Typical git repository sizes in practice
   • Expected usage patterns for GRASP
   • Balance between usability and resource protection

3. How to handle existing oversized repos?

   • Grandfather in existing large repos?
   • Force migration/cleanup?
   • Gradual enforcement strategy?

4. What metrics should we expose?

   • Per-user storage usage
   • Per-repo storage usage
   • Total relay storage usage
   • Available disk space

5. How to communicate limits to users?

   • NIP-11 relay info document?
   • Error messages during push?
   • Separate query endpoint for quota status?

Implementation Considerations

Git-specific challenges

• Git packs can be large but compress well
• Shared objects between repos (deduplication)
• Temporary space needed during push (may exceed final size)
• Repository size can decrease after GC/repack

Performance

• Size calculations must be fast (don't block pushes)
• Consider caching calculated sizes
• Incremental updates when possible
• Background monitoring vs synchronous checks

User experience

• Clear, actionable error messages
• Show current usage vs limit
• Suggest remediation (git gc, remove large files)
• Allow grace period for users to clean up

Findings

(To be filled in during investigation)

Implementation Plan

(To be determined after investigation)

Success Criteria

• No user can fill relay disk unintentionally
• Graceful handling of disk space exhaustion
• Clear documentation of limits and quotas
• Monitoring and alerting for storage usage
• Configurable limits for different deployment sizes