193 lines
6.2 KiB
Markdown
193 lines
6.2 KiB
Markdown
# sys-fips Setup for Qubes OS
|
|
|
|
> Goal: run a **single FIPS node** inside one ProxyVM (`sys-fips`) so any AppVM routed through it can access the FIPS mesh.
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
AppVMs → sys-fips (ProxyVM) → sys-firewall → sys-net → Internet
|
|
└─ fips0 (TUN) → FIPS mesh (encrypted)
|
|
```
|
|
|
|
This setup is intentionally single-node per VM.
|
|
If you want another independent FIPS identity/path, clone the VM (for example, `sys-fips2`).
|
|
|
|
---
|
|
|
|
## Files in this directory
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| `README.md` | This guide |
|
|
| `configs/fips.yaml` | Single-node config template |
|
|
| `scripts/00-build-fips.sh` | Run in build qube: build `fips`, `fipsctl`, `fipstop` into local `bin/` |
|
|
| `scripts/01-dom0-create-proxyvm.sh` | Run in dom0: creates `sys-fips` ProxyVM |
|
|
| `scripts/02-install-fips.sh` | Run in `sys-fips`: installs deps, binaries, and systemd units |
|
|
| `scripts/03-configure-identity.sh` | Run in `sys-fips`: configures `/etc/fips/fips.yaml`, identity mode |
|
|
| `scripts/03-generate-node-configs.sh` | Compatibility wrapper to `03-configure-identity.sh` |
|
|
| `scripts/04-start-fips.sh` | Run in `sys-fips`: starts single FIPS node |
|
|
| `scripts/04-start-fips-nodes.sh` | Compatibility wrapper to `04-start-fips.sh` |
|
|
| `scripts/05-stop-fips.sh` | Run in `sys-fips`: stops node + cleanup |
|
|
| `scripts/05-stop-fips-nodes.sh` | Compatibility wrapper to `05-stop-fips.sh` |
|
|
| `scripts/06-configure-dns.sh` | Run in `sys-fips`: dnsmasq forwards `.fips` to localhost:5354 and applies Qubes `/32` DNS compatibility patch |
|
|
| `scripts/07-route-appvms.sh` | Run in `sys-fips`: IPv6/ip6tables forwarding for routed AppVMs |
|
|
| `scripts/08-add-peer.sh` | Run in `sys-fips`: add static peer to `/etc/fips/fips.yaml` |
|
|
| `scripts/09-add-known-peers.sh` | Run in `sys-fips`: add two known bootstrap peers and restart FIPS |
|
|
| `scripts/test-connectivity.sh` | Unified test script |
|
|
| `scripts/test-local.sh` | Compatibility wrapper to `test-connectivity.sh --local` |
|
|
| `scripts/test-from-appvm.sh` | Compatibility wrapper to `test-connectivity.sh --appvm` |
|
|
| `scripts/test-full-suite.sh` | Compatibility wrapper to `test-connectivity.sh --full` |
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
### 1) Create `sys-fips` (run in dom0)
|
|
|
|
```bash
|
|
sudo bash 01-dom0-create-proxyvm.sh [template] [upstream_netvm]
|
|
```
|
|
|
|
Defaults:
|
|
- template: `debian-12`
|
|
- upstream netvm: `sys-firewall`
|
|
|
|
For VPN chaining, pass `sys-vpn` as upstream:
|
|
|
|
```bash
|
|
sudo bash 01-dom0-create-proxyvm.sh debian-12 sys-vpn
|
|
```
|
|
|
|
### 2) Build in this repo, then copy this entire folder
|
|
|
|
From this `fips_setup` repo in your build qube:
|
|
|
|
```bash
|
|
bash ./scripts/00-build-fips.sh
|
|
qvm-copy-to-vm sys-fips /home/user/anvil/fips_setup/
|
|
```
|
|
|
|
This keeps your update workflow simple: pull latest code, rebuild once in this repo, copy the folder, run install in `sys-fips`.
|
|
|
|
### 3) Install/configure/start (inside `sys-fips`)
|
|
|
|
```bash
|
|
cd ~/QubesIncoming/*/fips_setup/scripts
|
|
sudo bash 02-install-fips.sh
|
|
bash 03-configure-identity.sh
|
|
sudo bash 04-start-fips.sh
|
|
sudo bash 06-configure-dns.sh
|
|
sudo bash 07-route-appvms.sh
|
|
```
|
|
|
|
### 4) Add at least one external peer
|
|
|
|
```bash
|
|
sudo bash 08-add-peer.sh
|
|
sudo systemctl restart fips
|
|
```
|
|
|
|
Or add both known bootstrap peers automatically:
|
|
|
|
```bash
|
|
sudo bash 09-add-known-peers.sh
|
|
```
|
|
|
|
### 5) Test
|
|
|
|
```bash
|
|
sudo bash test-connectivity.sh --full
|
|
```
|
|
|
|
### 6) Point AppVMs at `sys-fips` (in dom0)
|
|
|
|
```bash
|
|
qvm-prefs my-appvm netvm sys-fips
|
|
```
|
|
|
|
---
|
|
|
|
## Clone workflow
|
|
|
|
```bash
|
|
# dom0
|
|
qvm-clone sys-fips sys-fips2
|
|
qvm-start sys-fips2
|
|
```
|
|
|
|
Inside `sys-fips2`:
|
|
|
|
```bash
|
|
sudo bash 05-stop-fips.sh
|
|
sudo rm -f /etc/fips/fips.key /etc/fips/fips.pub
|
|
bash 03-configure-identity.sh
|
|
sudo bash 04-start-fips.sh
|
|
```
|
|
|
|
Now `sys-fips` and `sys-fips2` are independent FIPS nodes.
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- One FIPS node per ProxyVM is the production target.
|
|
- The previous two-node local loopback approach was useful for local experiments but is not required for normal Qubes deployment.
|
|
- FIPS mesh encryption is end-to-end. A VPN upstream is optional and can be chained.
|
|
- `06-configure-dns.sh` removes dnsmasq `--local-service` from Debian's helper to avoid dropping AppVM DNS on Qubes `/32` links.
|
|
|
|
---
|
|
|
|
## Lessons learned from real `sys-fips` troubleshooting
|
|
|
|
These are the key fixes required to make AppVM → `sys-fips` → FIPS mesh work reliably in Qubes.
|
|
|
|
### 1) Intercept AppVM DNS in `sys-fips` with nftables
|
|
|
|
Qubes AppVM DNS normally targets the Qubes gateway addresses (for example `10.139.1.1` / `10.139.1.2`).
|
|
To make `.fips` resolution work from routed AppVMs, `sys-fips` must DNAT DNS to local dnsmasq.
|
|
|
|
Required nft hooks in `sys-fips`:
|
|
|
|
- `table ip qubes`, chain `dnat-dns`
|
|
- `table ip qubes`, chain `custom-input`
|
|
|
|
`dnat-dns` redirects UDP/TCP 53 traffic to local DNS (`10.139.1.1:53`) and `custom-input` allows DNS from AppVM interfaces (for example `vif+`).
|
|
|
|
### 2) IPv6 forwarding + route to FIPS space
|
|
|
|
`sys-fips` must forward IPv6 and route FIPS mesh space (`fd00::/8`) to `fips0`:
|
|
|
|
- enable IPv6 forwarding (`sysctl net.ipv6.conf.all.forwarding=1`)
|
|
- ensure route exists: `ip -6 route add fd00::/8 dev fips0`
|
|
|
|
Without this, AppVM traffic may resolve `.fips` names but fail to reach the resolved FIPS IPv6 destination.
|
|
|
|
### 3) AppVMs need a valid IPv6 source + route toward `sys-fips`
|
|
|
|
In testing, routed AppVM access required:
|
|
|
|
- a synthetic IPv6 on `eth0` (example: `fdaa::15/128`)
|
|
- route of FIPS space via Qubes link-local gateway:
|
|
- `ip -6 route add fd00::/8 via fe80::fcff:ffff:feff:ffff dev eth0`
|
|
|
|
This ensures traffic to resolved `.fips` AAAA records leaves the AppVM through `sys-fips`.
|
|
|
|
### 4) Persist network fixes in `/rw/config/rc.local`
|
|
|
|
For reboot safety:
|
|
|
|
- persist nftables DNS interception + IPv6 forwarding/route in `sys-fips` `/rw/config/rc.local`
|
|
- persist AppVM synthetic IPv6 + route in each AppVM `/rw/config/rc.local` that needs `.fips`
|
|
|
|
Do not rely on one-off manual commands for production.
|
|
|
|
### 5) Chromium-family browser caveat
|
|
|
|
Observed behavior:
|
|
|
|
- Firefox followed system resolver path and worked once networking was fixed.
|
|
- Chromium-family browsers (`agent-browser`, Brave) could still show `ERR_NAME_NOT_RESOLVED` even when `dig`, `getent`, `ping`, and `curl` worked.
|
|
|
|
Current operational workaround is to pin the `.fips` hostname in `/etc/hosts` for Chromium-family use-cases.
|
|
See `plans/chromium-fips-workaround.md` for tested commands and policy notes.
|