issue: update 573b and 820a - baseline analysis complete, VPS upgrade required

573b-production-timeout-diagnosis.md

@@ -502,6 +502,46 @@ Currently relay.ngit.dev runs ngit-relay (reference implementation). Before migr
   3. Use baseline for relay.ngit.dev migration decision (issue 820a)
   4. Consider fixing container stats/FD counts in future if needed
 
+### 2026-01-19 [Session 11:35] - Baseline Analysis Complete ✅
+- Completed: Comprehensive analysis of 36 hours of baseline metrics (2026-01-17 23:30 to 2026-01-19 11:35 UTC)
+- **CRITICAL FINDINGS:**
+  1. **CPU severely overloaded:** Average 6.96 load (348% utilization on 2 cores), peak 11.83 (592%)
+  2. **Memory maxed out:** 96.8% average usage, only 130MB available (3.2% free)
+  3. **Swap heavily used:** 2.0GB average (40% of 5GB), growing over time (0.75GB → 2.5GB)
+  4. **Connections NOT the problem:** Only 28-31 per relay (avg), 120-126 peak (3% of 4096 limit)
+  5. **Load does NOT correlate with connections:** 0.210 correlation (weak) - confirms git operations are bottleneck
+- **Peak Usage Patterns:**
+  - Peak hours: 16:00-20:00 UTC (load 6.5-9.9, connections 865-977)
+  - Off-peak hours: 00:00-12:00 UTC (load 6.3-7.4, connections 350-470)
+  - **Key insight:** Even off-peak shows HIGH load (6.3-7.4), indicating baseline overload independent of traffic
+- **Capacity Analysis:**
+  - Current: 2 cores (348% utilized), 3.8GB RAM (96.8% utilized)
+  - Required minimum: 4 cores, 8GB RAM (survival mode)
+  - **Recommended: 6 cores, 8GB RAM** (healthy operation + migration headroom)
+  - Optimal: 8 cores, 16GB RAM (future-proof)
+- **Migration Impact Assessment:**
+  - **Current VPS (2-core, 4GB):** 🔴 HIGH RISK - Do NOT migrate relay.ngit.dev
+  - **After upgrade to 4-core, 8GB:** 🟡 MEDIUM RISK - Proceed with caution
+  - **After upgrade to 6-core, 8GB:** 🟢 LOW RISK - Recommended safe path
+  - **After upgrade to 8-core, 16GB:** 🟢 LOW RISK - Optimal choice
+- **Data Quality Assessment:**
+  - ✅ 36 hours continuous collection, 2,160 data points (60s intervals)
+  - ✅ No gaps or missing data
+  - ✅ Critical metrics working reliably
+  - ✅ Clear patterns identified (peak hours, resource bottlenecks)
+  - ✅ **Sufficient for migration decision** - no need for additional collection
+- **Recommendations:**
+  1. **CRITICAL:** Upgrade VPS to 6-core, 8GB RAM BEFORE migrating relay.ngit.dev
+  2. Verify upgrade impact (24-48 hours monitoring)
+  3. Update migration timeline: Day 0-1 (VPS upgrade), Day 2-3 (prep), Day 4-5 (migrate), Day 5-7 (monitor)
+  4. Medium-term: Optimize git operations (global semaphore, nice processes, event-driven sync)
+  5. Long-term: Plan for growth (may need 8-core, 16GB within 6-12 months)
+- **Files Created:**
+  - `/tmp/vps-baseline-report.md` - Comprehensive 36-hour analysis report
+  - Analysis scripts: `/tmp/analyze_metrics.py`, `/tmp/hourly_analysis.py`
+- **Status:** Baseline collection COMPLETE ✅ - Proceed with VPS upgrade planning
+- **Next:** Update issue 820a with migration decision (VPS upgrade required first)
+
 ### 2026-01-16 [Phase 0 Step 1 - Initial Scripts]
 - Completed: Created three bash diagnostic scripts
 - Issue: Scripts not compatible with NixOS declarative philosophy

820a-relay-ngit-dev-migration.md

@@ -250,6 +250,49 @@ docker start ngit-relay
   3. Make migration decision based on baseline analysis
   4. Proceed with rapid preparation if baseline looks good
 
+### 2026-01-19 [Session 11:35] - Baseline Analysis Complete - VPS UPGRADE REQUIRED ⚠️
+- Completed: Comprehensive analysis of 36 hours of baseline metrics
+- **CRITICAL DECISION: Migration BLOCKED until VPS upgrade**
+- **Baseline Findings (2026-01-17 23:30 to 2026-01-19 11:35 UTC):**
+  - **CPU:** Average 6.96 load (348% utilization), peak 11.83 (592%) - 🔴 SEVERELY OVERLOADED
+  - **Memory:** 96.8% average usage, only 130MB available - 🔴 MAXED OUT
+  - **Swap:** 2.0GB average (40% of 5GB), growing 0.75GB → 2.5GB - 🟡 HEAVILY USED
+  - **Connections:** Port 8081: 28 avg/120 peak, Port 8083: 31 avg/126 peak - ✅ ONLY 3% OF CAPACITY
+  - **Correlation:** Load vs connections = 0.210 (weak) - Confirms git operations are bottleneck, not connections
+- **Peak Usage Patterns:**
+  - Peak hours: 16:00-20:00 UTC (load 6.5-9.9, connections 865-977)
+  - Off-peak hours: 00:00-12:00 UTC (load 6.3-7.4, connections 350-470)
+  - **Critical insight:** Even off-peak shows HIGH load (6.3-7.4), indicating baseline overload independent of traffic
+- **Migration Risk Assessment:**
+  - **Current VPS (2-core, 4GB):** 🔴 **HIGH RISK** - Migration will add load spike to already overloaded system
+  - **Outcome:** High probability of service degradation or outages during migration
+  - **Recommendation:** ❌ **DO NOT MIGRATE** on current VPS
+- **VPS Upgrade Requirements:**
+  - **Minimum (Survival):** 4 cores, 8GB RAM (~$20-40/month) - 🟡 MEDIUM RISK
+  - **Recommended (Healthy):** 6 cores, 8GB RAM (~$40-60/month) - 🟢 LOW RISK ✅
+  - **Optimal (Future-proof):** 8 cores, 16GB RAM (~$60-100/month) - 🟢 LOW RISK
+  - **Decision:** Upgrade to 6-core, 8GB minimum before migration
+- **Updated Migration Timeline:**
+  - **REVISED:** 5-7 days total (was 3-5 days)
+  - **Day 0-1:** VPS upgrade + verification (NEW - CRITICAL)
+  - **Day 2-3:** Rapid preparation
+  - **Day 4-5:** Migration execution
+  - **Day 5-7:** Intensive monitoring
+- **Data Quality:**
+  - ✅ 36 hours continuous collection, 2,160 data points
+  - ✅ Clear patterns identified, no anomalies
+  - ✅ **Sufficient for migration decision** - no additional collection needed
+  - ✅ Findings are conclusive: VPS upgrade required
+- **Next Steps:**
+  1. **CRITICAL:** Upgrade VPS to 6-core, 8GB RAM (BLOCKER for migration)
+  2. Verify upgrade impact (24-48 hours monitoring, expect load to drop to ~60-70%)
+  3. Proceed with accelerated migration plan (Day 2-7)
+  4. Monitor intensively during and after migration
+- **Files Created:**
+  - `/tmp/vps-baseline-report.md` - Comprehensive 36-hour analysis with migration scenarios
+- **Status:** Baseline complete ✅, Migration BLOCKED ⚠️ pending VPS upgrade
+- **Blocker:** VPS upgrade to 6-core, 8GB RAM required before migration can proceed
+
 ## Notes
 
 - **Related issue:** 573b-production-timeout-diagnosis.md (timeout investigation that led to this)