# FIPS Debug Notes: `.fips` HTTP reachability incident

## Summary

We investigated why this URL failed locally but worked from the remote host:

- `http://npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips/index.html`

Outcome:

1. Initial local failure was due to a stale `fd00::/8` route pointing to an unreachable next-hop.
2. After route recovery, TCP connected but larger HTTP responses stalled.
3. Root cause for the stall was path/MTU behavior (large payload transfer blackholing/retransmission).
4. Setting local UDP transport MTU to `1280` resolved the issue; full page download succeeded.

---

## What we observed

### 1) DNS was correct

- `.fips` name resolved to the expected IPv6 address.
- Name resolution was **not** the issue.

### 2) Endpoints were alive

- On remote host (`ubuntu@laantungir.net`), the same URL returned `HTTP/1.1 200 OK` with full content.
- Nginx was listening and serving properly.

### 3) Local routing was initially broken

- Local route had `fd00::/8` via a link-local next-hop that was unreachable.
- Neighbor state for that gateway showed `FAILED`.
- This produced local `No route to host` errors.

### 4) After local route fix, only large responses failed

- Local curl connected to remote `:80` and sent GET.
- Small responses worked:
  - direct IPv6 host request without vhost Host header got `410` small body,
  - ranged request got `206` with a small payload.
- Full `index.html` (10KB+) timed out with `0 bytes received` locally.
- Remote socket stats indicated retransmissions and congestion-window collapse, consistent with MTU/path fragmentation loss behavior.

### 5) Mitigation validated

- Local config changed:

```yaml
transports:
  udp:
    bind_addr: 0.0.0.0:2121
    mtu: 1280
```

- Restarted local `fips.service`.
- Retested URL: `HTTP 200`, full `10156` bytes downloaded.

---

## Lessons learned

1. **Treat `.fips` failures as a pipeline problem**
   - Validate in this order: DNS → route → reachability → app response size behavior.

2. **“Connect succeeds” does not prove path health**
   - A successful TCP handshake can still hide payload delivery failures.
   - Always test both tiny and realistic/full payload responses.

3. **Stale `fd00::/8` routes can silently break TUN startup**
   - We saw TUN init warnings when route insertion conflicted with existing state.
   - Route hygiene matters before/after service restarts.

4. **MTU defaults are not always safe on every real path**
   - Especially over mixed internet routes and overlays.
   - `1280` is often the practical “known-good” baseline for IPv6 paths.

5. **Use control-plane telemetry early**
   - `fipsctl show status|peers|sessions|routing` quickly distinguishes control-plane up/down from data-plane degradation.

6. **Host-header/vhost behavior can mislead debugging**
   - Explicit `Host:` header to the same IPv6 returned 200 remotely.
   - Direct IP without host header returned a different vhost response (410).
   - This is expected and should be factored into tests.

---

## Recommended runbook for future incidents

1. **Reproduce quickly**
   - `curl -v --max-time 10 http://<npub>.fips/index.html`

2. **Check resolution and route**
   - `getent ahosts <npub>.fips`
   - `ip -6 route get <resolved_ipv6>`
   - `ip -6 neigh show`

3. **Check FIPS health**
   - `sudo /usr/local/bin/fipsctl show status`
   - `sudo /usr/local/bin/fipsctl show peers`
   - `sudo /usr/local/bin/fipsctl show sessions`

4. **Differentiate small vs large payload behavior**
   - Small: direct IP / tiny endpoint / range request
   - Large: full page/object transfer

5. **If large payloads fail but small succeed, test MTU mitigation**
   - Set `transports.udp.mtu: 1280` locally
   - Restart `fips.service`
   - Retest full payload

6. **Persist fix and monitor**
   - Keep 1280 unless measured path supports higher values reliably.
   - Track retransmission symptoms and transfer success over time.

---

## Config change made during this debug

Local `/etc/fips/fips.yaml` was updated to include:

```yaml
transports:
  udp:
    mtu: 1280
```

A backup of the prior local file was created before modification.

---

## Final status

The target `.fips` web page is reachable from this local node after lowering local UDP MTU to `1280` and restarting `fips.service`.