From e38d57593975223e653df1654e1b973847490cb7 Mon Sep 17 00:00:00 2001 From: Laan Tungir Date: Sun, 7 Jun 2026 11:24:02 -0400 Subject: [PATCH] gitea subpath fix --- plans/gitea-fips-subpath-fix.md | 327 ++++++++++++++++++++++++++++++++ 1 file changed, 327 insertions(+) create mode 100644 plans/gitea-fips-subpath-fix.md diff --git a/plans/gitea-fips-subpath-fix.md b/plans/gitea-fips-subpath-fix.md new file mode 100644 index 0000000..486b377 --- /dev/null +++ b/plans/gitea-fips-subpath-fix.md @@ -0,0 +1,327 @@ +# Gitea over FIPS: `/git/` Sub-Path Fix Plan + +## Problem statement + +Accessing Gitea through the FIPS mesh fails with: + +> Failed to load asset files from +> `http://npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips/assets/js/index.js?v=1.24.6`. +> Please make sure the asset files can be accessed + +The landing page on the same `.fips` host loads fine. + +## Root cause (confirmed on the server) + +The FIPS network path is **healthy** — this is **not** an MTU/PMTUD black-hole. The failure is a +Gitea sub-path reverse-proxy misconfiguration on the server (`VM410` / `laantungir.net`). + +Server facts gathered via SSH (`ubuntu@laantungir.net`): + +- One Gitea instance runs at `127.0.0.1:3000`. +- Gitea config is injected by environment variables in + `/etc/systemd/system/gitea.service` (these **override** `/var/lib/gitea/conf/app.ini`): + - `GITEA__server__DOMAIN=git.laantungir.net` + - `GITEA__server__ROOT_URL=https://git.laantungir.net/` ← root path + - `GITEA__server__HTTP_PORT=3000` +- The `.fips` host is served by `/etc/nginx/conf.d/fips-ingress.conf`, in the + `server { server_name ~^npub1.+\.fips$; ... }` block. +- That block proxies Gitea **only** under `/git/`, and it **strips** the prefix: + + ```nginx + location ^~ /git/ { + rewrite ^/git/?(.*)$ /$1 break; # <-- strips /git/ + proxy_pass http://127.0.0.1:3000; + proxy_set_header Host git.laantungir.net; + ... + } + location / { + root /var/www/html; # <-- static landing page + try_files $uri $uri/ =404; + } + ``` + +### Why assets 404 + +1. Browser opens `http://.fips/git/`. +2. nginx strips `/git/` and proxies `/` to Gitea. +3. Gitea renders HTML, but because `ROOT_URL` is **root** (`.../`), it emits asset links as + **root-relative** `/assets/js/index.js`, **not** `/git/assets/...`. +4. Browser requests `http://.fips/assets/js/index.js`. +5. That path does **not** match `location ^~ /git/`; it falls through to `location /` (static + webroot `/var/www/html`), which has no `assets` directory → **404**. +6. Gitea's JS bootstrap sees the 404 and throws "Failed to load asset files". + +```mermaid +flowchart TD + A[Browser opens npub.fips/git/] --> B[nginx strips /git/ proxies / to Gitea 3000] + B --> C[Gitea ROOT_URL is root so emits /assets/... root-relative] + C --> D[Browser requests npub.fips/assets/js/index.js] + D --> E[No /git/ match falls to location / static webroot] + E --> F[404 Not Found] + F --> G[Gitea JS error Failed to load asset files] + style B fill:#cce5ff + style E fill:#ffcccc + style F fill:#ffcccc +``` + +### Key asymmetry vs other proxied services + +The sibling `location ^~ /relay/` and `location ^~ /blossom/` blocks also strip their prefix — +and that is fine for them, because those backends do **not** generate self-referential absolute +URLs from a configured base. **Gitea does** (assets, redirects, cookies are derived from +`ROOT_URL`). Therefore Gitea's `/git/` block must be configured **differently**: the prefix must +be preserved and `ROOT_URL` must include `/git/`. + +## Roadmap context + +The operator is consolidating subdomains into apex sub-paths: + +- `relay.laantungir.net` → `laantungir.net/relay` +- `git.laantungir.net` → `laantungir.net/git` + +So the **destination** for Gitea is the `/git/` sub-path on the apex host, and equivalently +`.fips/git/` on the mesh. This makes the path-based `/git/` mount the correct, permanent +design — not a throwaway workaround. Fixing it now is a down-payment on the end state. + +## The fix (two paired changes) + +For Gitea to live correctly at `/git/` on **any** host, two things must be true together: + +1. **Gitea knows its base path**: `ROOT_URL` ends in `/git/` + → Gitea then emits assets as `/git/assets/...` (`STATIC_URL_PREFIX` follows `ROOT_URL`). +2. **nginx preserves the prefix**: `proxy_pass` passes `/git/` through (do **not** strip it), + so the prefixed asset/redirect requests reach Gitea. + +```mermaid +flowchart TD + A[Browser opens host/git/] --> B[nginx location ^~ /git/ proxy_pass keeps /git/ prefix] + B --> C[Gitea ROOT_URL ends in /git/] + C --> D[Gitea emits /git/assets/...] + D --> E[Browser requests host/git/assets/js/index.js] + E --> F[Matches /git/ location proxies to Gitea] + F --> G[200 OK assets load] + style B fill:#cce5ff + style F fill:#d4edda + style G fill:#d4edda +``` + +### Single-`ROOT_URL` consideration + +Gitea supports only **one** `ROOT_URL`. The transition has two front doors that must both serve +Gitea at the **same** sub-path `/git/`: + +- `https://laantungir.net/git/` (clearnet — the future) +- `http://.fips/git/` (FIPS — interim and ongoing) + +Because the **path component is identical** (`/git/`), one `ROOT_URL` ending in `/git/` produces +correct **root-relative** asset links (`/git/assets/...`) on **both** hosts. The only host-specific +concern is **absolute** links/redirects (login flows, some redirects) and cookie domain, which +Gitea derives from `ROOT_URL` host + forwarded headers. + +Two ways to handle the host portion during transition: + +- **Simple (recommended to start):** `ROOT_URL=https://laantungir.net/git/`. Asset loading works + on `.fips` because asset paths are root-relative. Some absolute redirects may point at + `laantungir.net`; acceptable for a transition. +- **Mesh-pure (optional polish):** add `sub_filter` on the `.fips` vhost to rewrite + `https://laantungir.net/git` → `http://.fips/git` in responses, so `.fips` sessions never + bounce to clearnet. + +--- + +## Implementation + +> All paths below are on the server `ubuntu@laantungir.net` (`VM410`). +> Make timestamped backups before editing (the server already follows this convention). + +### Step 1 — Set Gitea `ROOT_URL` to end in `/git/` + +Edit `/etc/systemd/system/gitea.service`: + +```ini +# BEFORE +Environment=GITEA__server__ROOT_URL=https://git.laantungir.net/ + +# AFTER +Environment=GITEA__server__ROOT_URL=https://laantungir.net/git/ +``` + +Notes: +- `DOMAIN`/`SSH_DOMAIN` can remain `git.laantungir.net` until the subdomain is removed; they do + not affect asset paths. (At final cutover, set `DOMAIN=laantungir.net`.) +- `STATIC_URL_PREFIX` is unset, so it defaults to follow `ROOT_URL` → `/git/assets/...`. Do **not** + set a conflicting `STATIC_URL_PREFIX`. + +Apply: + +```bash +sudo systemctl daemon-reload +sudo systemctl restart gitea +``` + +### Step 2 — Fix the `.fips` nginx vhost `/git/` block (preserve prefix) + +Edit `/etc/nginx/conf.d/fips-ingress.conf`, inside `server { server_name ~^npub1.+\.fips$; ... }`: + +```nginx +# BEFORE +location ^~ /git/ { + rewrite ^/git/?(.*)$ /$1 break; # strips prefix <-- REMOVE + proxy_pass http://127.0.0.1:3000; # no trailing /git/ + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host git.laantungir.net; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto http; + proxy_cache_bypass $http_upgrade; +} +location = /git { return 301 /git/; } +``` + +```nginx +# AFTER +location ^~ /git/ { + # Preserve the /git/ prefix toward Gitea (do NOT strip it). + proxy_pass http://127.0.0.1:3000; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host $host; # forward the .fips host + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto http; + proxy_cache_bypass $http_upgrade; + client_max_body_size 512m; # allow git pushes / large uploads +} +location = /git { return 301 /git/; } +``` + +Key changes: +- **Remove** the `rewrite ^/git/?(.*)$ /$1 break;` line so the `/git/` prefix is preserved. +- With `location ^~ /git/` + `proxy_pass http://127.0.0.1:3000;` (no URI on `proxy_pass`), nginx + forwards the **original** URI including `/git/`. (Do not add a trailing path to `proxy_pass` + here, or nginx will rewrite the matched location prefix.) +- Consider `Host $host` so Gitea/forwarded headers reflect the `.fips` host. If any Gitea host + allow-list rejects it, fall back to `Host git.laantungir.net` (assets still work either way + since they are root-relative under `/git/`). +- Add `client_max_body_size` for git over HTTP and uploads. + +### Step 3 (optional) — Keep `.fips` sessions on the mesh via `sub_filter` + +If absolute links/redirects to `https://laantungir.net/git` are undesirable for `.fips` users, +add inside the same `/git/` location: + +```nginx + proxy_set_header Accept-Encoding ""; # required for sub_filter to see bodies + sub_filter_once off; + sub_filter_types text/html application/javascript application/json; + sub_filter "https://laantungir.net/git" "http://$host/git"; +``` + +This is a transition convenience and can be removed at final cutover. + +### Step 4 — Validate and reload + +```bash +sudo nginx -t +sudo systemctl reload nginx +# gitea already restarted in Step 1 +``` + +### Step 5 — Add the clearnet `laantungir.net/git` mount (target state) + +This mirrors the `.fips` block on the apex `laantungir.net` server blocks (HTTP→HTTPS and the +`443` block) in `/etc/nginx/conf.d/default.conf`. Add to the `laantungir.net` `server` block(s): + +```nginx +location ^~ /git/ { + proxy_pass http://127.0.0.1:3000; # preserve /git/ prefix + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host $host; # laantungir.net + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_cache_bypass $http_upgrade; + client_max_body_size 512m; +} +location = /git { return 301 /git/; } +``` + +With `ROOT_URL=https://laantungir.net/git/`, this serves Gitea cleanly at the apex sub-path with +no rewriting. The existing static landing page remains at `/`. + +### Step 6 — Final subdomain removal (later, separate change) + +When ready to delete `git.laantungir.net`: +1. Confirm `laantungir.net/git/` and `.fips/git/` both fully work. +2. Set `GITEA__server__DOMAIN=laantungir.net` (and `SSH_DOMAIN` per SSH plan). +3. Remove the `git.laantungir.net` `server` blocks from nginx; reload. +4. Remove any `sub_filter` transition shims. +5. Repeat the analogous consolidation for `relay.` → `/relay` (note: relay/websocket backends can + keep prefix-stripping since they don't emit self-referential base URLs — unlike Gitea). + +--- + +## Verification + +### Server-side (on `laantungir.net`, against local nginx) + +```bash +# .fips host, /git/ landing +curl -sS -D - -o /dev/null -H "Host: npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips" \ + http://127.0.0.1/git/ | grep -Ei 'HTTP/|Set-Cookie|Location' +# expect: 200 OK and Set-Cookie: i_like_gitea (Gitea backend reached) + +# .fips host, asset under /git/ +curl -sS -o /dev/null -w '%{http_code}\n' -H "Host: npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips" \ + "http://127.0.0.1/git/assets/js/index.js?v=1.24.6" +# expect: 200 + +# clearnet apex /git/ (after Step 5) +curl -sS -o /dev/null -w '%{http_code}\n' -H "Host: laantungir.net" \ + "http://127.0.0.1/git/assets/js/index.js?v=1.24.6" +# expect: 200 (or 301->https depending on block) +``` + +### Client-side (from FIPS client VM `ai`, over the mesh) + +```bash +HOST='npub1crpldvy49ef8z34wlacwujnfudy4nd7k96aqdx5wgn6ckztz7z8q9t59ud.fips' + +# Gitea landing under /git/ +curl -6 -sS -D - -o /dev/null "http://$HOST/git/" | grep -Ei 'HTTP/|Set-Cookie' +# expect: 200 OK, Set-Cookie i_like_gitea + +# The previously-failing asset, now under /git/ +curl -6 -sS -o /dev/null -w '%{http_code} %{size_download}\n' "http://$HOST/git/assets/js/index.js?v=1.24.6" +# expect: 200 and a non-trivial size + +# Confirm root-relative asset path Gitea now emits +curl -6 -sS "http://$HOST/git/" | grep -Eo 'src="[^"]*"|href="[^"]*"' | grep '/git/assets' | head +``` + +### Browser + +- Open `http://.fips/git/` (note the trailing `/git/`). +- Clear cache / hard-reload if a stale Gitea HTML referencing root `/assets/...` was cached. +- The "Failed to load asset files" error should be gone. + +--- + +## Rollback + +- nginx: restore the timestamped backup of `fips-ingress.conf` (and `default.conf`), then + `nginx -t && systemctl reload nginx`. +- Gitea: restore the previous `ROOT_URL` line in `gitea.service`, then + `systemctl daemon-reload && systemctl restart gitea`. + +## Notes / distinguishing from the MTU failure mode + +- This failure returns a **fast HTTP 404 with a small body** for assets — the network is fine. +- An MTU/PMTUD black-hole instead presents as **small responses succeeding while large transfers + hang/stall or return `000`** (connection stalls mid-body). If that signature ever appears, + investigate TCP MSS clamping for the `fips0` MTU (1280) on `sys-fips`, not this nginx/Gitea path.