fips_playground/fips_debug.md

FIPS Debug Notes: .fips HTTP reachability incident

Summary

We investigated why this URL failed locally but worked from the remote host:

• http://npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips/index.html

Outcome:

1. Initial local failure was due to a stale fd00::/8 route pointing to an unreachable next-hop.
2. After route recovery, TCP connected but larger HTTP responses stalled.
3. Root cause for the stall was path/MTU behavior (large payload transfer blackholing/retransmission).
4. Setting local UDP transport MTU to 1280 resolved the issue; full page download succeeded.

---

What we observed

1) DNS was correct

• .fips name resolved to the expected IPv6 address.
• Name resolution was not the issue.

2) Endpoints were alive

• On remote host (ubuntu@laantungir.net), the same URL returned HTTP/1.1 200 OK with full content.
• Nginx was listening and serving properly.

3) Local routing was initially broken

• Local route had fd00::/8 via a link-local next-hop that was unreachable.
• Neighbor state for that gateway showed FAILED.
• This produced local No route to host errors.

4) After local route fix, only large responses failed

• Local curl connected to remote :80 and sent GET.
• Small responses worked:
  • direct IPv6 host request without vhost Host header got 410 small body,
  • ranged request got 206 with a small payload.
• Full index.html (10KB+) timed out with 0 bytes received locally.
• Remote socket stats indicated retransmissions and congestion-window collapse, consistent with MTU/path fragmentation loss behavior.

5) Mitigation validated

• Local config changed:

transports:
  udp:
    bind_addr: 0.0.0.0:2121
    mtu: 1280

• Restarted local fips.service.
• Retested URL: HTTP 200, full 10156 bytes downloaded.

---

Lessons learned

1. Treat .fips failures as a pipeline problem

   • Validate in this order: DNS → route → reachability → app response size behavior.

2. “Connect succeeds” does not prove path health

   • A successful TCP handshake can still hide payload delivery failures.
   • Always test both tiny and realistic/full payload responses.

3. Stale fd00::/8 routes can silently break TUN startup

   • We saw TUN init warnings when route insertion conflicted with existing state.
   • Route hygiene matters before/after service restarts.

4. MTU defaults are not always safe on every real path

   • Especially over mixed internet routes and overlays.
   • 1280 is often the practical “known-good” baseline for IPv6 paths.

5. Use control-plane telemetry early

   • fipsctl show status|peers|sessions|routing quickly distinguishes control-plane up/down from data-plane degradation.

6. Host-header/vhost behavior can mislead debugging

   • Explicit Host: header to the same IPv6 returned 200 remotely.
   • Direct IP without host header returned a different vhost response (410).
   • This is expected and should be factored into tests.

---

Recommended runbook for future incidents

1. Reproduce quickly

   • curl -v --max-time 10 http://<npub>.fips/index.html

2. Check resolution and route

   • getent ahosts <npub>.fips
   • ip -6 route get <resolved_ipv6>
   • ip -6 neigh show

3. Check FIPS health

   • sudo /usr/local/bin/fipsctl show status
   • sudo /usr/local/bin/fipsctl show peers
   • sudo /usr/local/bin/fipsctl show sessions

4. Differentiate small vs large payload behavior

   • Small: direct IP / tiny endpoint / range request
   • Large: full page/object transfer

5. If large payloads fail but small succeed, test MTU mitigation

   • Set transports.udp.mtu: 1280 locally
   • Restart fips.service
   • Retest full payload

6. Persist fix and monitor

   • Keep 1280 unless measured path supports higher values reliably.
   • Track retransmission symptoms and transfer success over time.

---

Config change made during this debug

Local /etc/fips/fips.yaml was updated to include:

transports:
  udp:
    mtu: 1280

A backup of the prior local file was created before modification.

---

Final status

The target .fips web page is reachable from this local node after lowering local UDP MTU to 1280 and restarting fips.service.