135 lines
4.2 KiB
Markdown
135 lines
4.2 KiB
Markdown
# FIPS Debug Notes: `.fips` HTTP reachability incident
|
|
|
|
## Summary
|
|
|
|
We investigated why this URL failed locally but worked from the remote host:
|
|
|
|
- `http://npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips/index.html`
|
|
|
|
Outcome:
|
|
|
|
1. Initial local failure was due to a stale `fd00::/8` route pointing to an unreachable next-hop.
|
|
2. After route recovery, TCP connected but larger HTTP responses stalled.
|
|
3. Root cause for the stall was path/MTU behavior (large payload transfer blackholing/retransmission).
|
|
4. Setting local UDP transport MTU to `1280` resolved the issue; full page download succeeded.
|
|
|
|
---
|
|
|
|
## What we observed
|
|
|
|
### 1) DNS was correct
|
|
|
|
- `.fips` name resolved to the expected IPv6 address.
|
|
- Name resolution was **not** the issue.
|
|
|
|
### 2) Endpoints were alive
|
|
|
|
- On remote host (`ubuntu@laantungir.net`), the same URL returned `HTTP/1.1 200 OK` with full content.
|
|
- Nginx was listening and serving properly.
|
|
|
|
### 3) Local routing was initially broken
|
|
|
|
- Local route had `fd00::/8` via a link-local next-hop that was unreachable.
|
|
- Neighbor state for that gateway showed `FAILED`.
|
|
- This produced local `No route to host` errors.
|
|
|
|
### 4) After local route fix, only large responses failed
|
|
|
|
- Local curl connected to remote `:80` and sent GET.
|
|
- Small responses worked:
|
|
- direct IPv6 host request without vhost Host header got `410` small body,
|
|
- ranged request got `206` with a small payload.
|
|
- Full `index.html` (10KB+) timed out with `0 bytes received` locally.
|
|
- Remote socket stats indicated retransmissions and congestion-window collapse, consistent with MTU/path fragmentation loss behavior.
|
|
|
|
### 5) Mitigation validated
|
|
|
|
- Local config changed:
|
|
|
|
```yaml
|
|
transports:
|
|
udp:
|
|
bind_addr: 0.0.0.0:2121
|
|
mtu: 1280
|
|
```
|
|
|
|
- Restarted local `fips.service`.
|
|
- Retested URL: `HTTP 200`, full `10156` bytes downloaded.
|
|
|
|
---
|
|
|
|
## Lessons learned
|
|
|
|
1. **Treat `.fips` failures as a pipeline problem**
|
|
- Validate in this order: DNS → route → reachability → app response size behavior.
|
|
|
|
2. **“Connect succeeds” does not prove path health**
|
|
- A successful TCP handshake can still hide payload delivery failures.
|
|
- Always test both tiny and realistic/full payload responses.
|
|
|
|
3. **Stale `fd00::/8` routes can silently break TUN startup**
|
|
- We saw TUN init warnings when route insertion conflicted with existing state.
|
|
- Route hygiene matters before/after service restarts.
|
|
|
|
4. **MTU defaults are not always safe on every real path**
|
|
- Especially over mixed internet routes and overlays.
|
|
- `1280` is often the practical “known-good” baseline for IPv6 paths.
|
|
|
|
5. **Use control-plane telemetry early**
|
|
- `fipsctl show status|peers|sessions|routing` quickly distinguishes control-plane up/down from data-plane degradation.
|
|
|
|
6. **Host-header/vhost behavior can mislead debugging**
|
|
- Explicit `Host:` header to the same IPv6 returned 200 remotely.
|
|
- Direct IP without host header returned a different vhost response (410).
|
|
- This is expected and should be factored into tests.
|
|
|
|
---
|
|
|
|
## Recommended runbook for future incidents
|
|
|
|
1. **Reproduce quickly**
|
|
- `curl -v --max-time 10 http://<npub>.fips/index.html`
|
|
|
|
2. **Check resolution and route**
|
|
- `getent ahosts <npub>.fips`
|
|
- `ip -6 route get <resolved_ipv6>`
|
|
- `ip -6 neigh show`
|
|
|
|
3. **Check FIPS health**
|
|
- `sudo /usr/local/bin/fipsctl show status`
|
|
- `sudo /usr/local/bin/fipsctl show peers`
|
|
- `sudo /usr/local/bin/fipsctl show sessions`
|
|
|
|
4. **Differentiate small vs large payload behavior**
|
|
- Small: direct IP / tiny endpoint / range request
|
|
- Large: full page/object transfer
|
|
|
|
5. **If large payloads fail but small succeed, test MTU mitigation**
|
|
- Set `transports.udp.mtu: 1280` locally
|
|
- Restart `fips.service`
|
|
- Retest full payload
|
|
|
|
6. **Persist fix and monitor**
|
|
- Keep 1280 unless measured path supports higher values reliably.
|
|
- Track retransmission symptoms and transfer success over time.
|
|
|
|
---
|
|
|
|
## Config change made during this debug
|
|
|
|
Local `/etc/fips/fips.yaml` was updated to include:
|
|
|
|
```yaml
|
|
transports:
|
|
udp:
|
|
mtu: 1280
|
|
```
|
|
|
|
A backup of the prior local file was created before modification.
|
|
|
|
---
|
|
|
|
## Final status
|
|
|
|
The target `.fips` web page is reachable from this local node after lowering local UDP MTU to `1280` and restarting `fips.service`.
|