# sys-fips Setup for Qubes OS

> Goal: run a **single FIPS node** inside one ProxyVM (`sys-fips`) so any AppVM routed through it can access the FIPS mesh.

## Architecture

```text
AppVMs → sys-fips (ProxyVM) → sys-firewall → sys-net → Internet
                 └─ fips0 (TUN) → FIPS mesh (encrypted)
```

This setup is intentionally single-node per VM.
If you want another independent FIPS identity/path, clone the VM (for example, `sys-fips2`).

---

## Files in this directory

| File | Purpose |
|---|---|
| `README.md` | This guide |
| `configs/fips.yaml` | Single-node config template |
| `scripts/00-build-fips.sh` | Run in build qube: build `fips`, `fipsctl`, `fipstop` into local `bin/` |
| `scripts/01-dom0-create-proxyvm.sh` | Run in dom0: creates `sys-fips` ProxyVM |
| `scripts/02-install-fips.sh` | Run in `sys-fips`: installs deps, binaries, and systemd units |
| `scripts/03-configure-identity.sh` | Run in `sys-fips`: configures `/etc/fips/fips.yaml`, identity mode |
| `scripts/03-generate-node-configs.sh` | Compatibility wrapper to `03-configure-identity.sh` |
| `scripts/04-start-fips.sh` | Run in `sys-fips`: starts single FIPS node |
| `scripts/04-start-fips-nodes.sh` | Compatibility wrapper to `04-start-fips.sh` |
| `scripts/05-stop-fips.sh` | Run in `sys-fips`: stops node + cleanup |
| `scripts/05-stop-fips-nodes.sh` | Compatibility wrapper to `05-stop-fips.sh` |
| `scripts/06-configure-dns.sh` | Run in `sys-fips`: dnsmasq forwards `.fips` to localhost:5354 and applies Qubes `/32` DNS compatibility patch |
| `scripts/07-route-appvms.sh` | Run in `sys-fips`: IPv6/ip6tables forwarding for routed AppVMs |
| `scripts/08-add-peer.sh` | Run in `sys-fips`: add static peer to `/etc/fips/fips.yaml` |
| `scripts/09-add-known-peers.sh` | Run in `sys-fips`: add two known bootstrap peers and restart FIPS |
| `scripts/test-connectivity.sh` | Unified test script |
| `scripts/test-local.sh` | Compatibility wrapper to `test-connectivity.sh --local` |
| `scripts/test-from-appvm.sh` | Compatibility wrapper to `test-connectivity.sh --appvm` |
| `scripts/test-full-suite.sh` | Compatibility wrapper to `test-connectivity.sh --full` |

---

## Quick start

### 1) Create `sys-fips` (run in dom0)

```bash
sudo bash 01-dom0-create-proxyvm.sh [template] [upstream_netvm]
```

Defaults:
- template: `debian-12`
- upstream netvm: `sys-firewall`

For VPN chaining, pass `sys-vpn` as upstream:

```bash
sudo bash 01-dom0-create-proxyvm.sh debian-12 sys-vpn
```

### 2) Build in this repo, then copy this entire folder

From this `fips_setup` repo in your build qube:

```bash
bash ./scripts/00-build-fips.sh
qvm-copy-to-vm sys-fips /home/user/anvil/fips_setup/
```

This keeps your update workflow simple: pull latest code, rebuild once in this repo, copy the folder, run install in `sys-fips`.

### 3) Install/configure/start (inside `sys-fips`)

```bash
cd ~/QubesIncoming/*/fips_setup/scripts
sudo bash 02-install-fips.sh
bash 03-configure-identity.sh
sudo bash 04-start-fips.sh
sudo bash 06-configure-dns.sh
sudo bash 07-route-appvms.sh
```

### 4) Add at least one external peer

```bash
sudo bash 08-add-peer.sh
sudo systemctl restart fips
```

Or add both known bootstrap peers automatically:

```bash
sudo bash 09-add-known-peers.sh
```

### 5) Test

```bash
sudo bash test-connectivity.sh --full
```

### 6) Point AppVMs at `sys-fips` (in dom0)

```bash
qvm-prefs my-appvm netvm sys-fips
```

---

## Clone workflow

```bash
# dom0
qvm-clone sys-fips sys-fips2
qvm-start sys-fips2
```

Inside `sys-fips2`:

```bash
sudo bash 05-stop-fips.sh
sudo rm -f /etc/fips/fips.key /etc/fips/fips.pub
bash 03-configure-identity.sh
sudo bash 04-start-fips.sh
```

Now `sys-fips` and `sys-fips2` are independent FIPS nodes.

---

## Notes

- One FIPS node per ProxyVM is the production target.
- The previous two-node local loopback approach was useful for local experiments but is not required for normal Qubes deployment.
- FIPS mesh encryption is end-to-end. A VPN upstream is optional and can be chained.
- `06-configure-dns.sh` removes dnsmasq `--local-service` from Debian's helper to avoid dropping AppVM DNS on Qubes `/32` links.

---

## Lessons learned from real `sys-fips` troubleshooting

These are the key fixes required to make AppVM → `sys-fips` → FIPS mesh work reliably in Qubes.

### 1) Intercept AppVM DNS in `sys-fips` with nftables

Qubes AppVM DNS normally targets the Qubes gateway addresses (for example `10.139.1.1` / `10.139.1.2`).
To make `.fips` resolution work from routed AppVMs, `sys-fips` must DNAT DNS to local dnsmasq.

Required nft hooks in `sys-fips`:

- `table ip qubes`, chain `dnat-dns`
- `table ip qubes`, chain `custom-input`

`dnat-dns` redirects UDP/TCP 53 traffic to local DNS (`10.139.1.1:53`) and `custom-input` allows DNS from AppVM interfaces (for example `vif+`).

### 2) IPv6 forwarding + route to FIPS space

`sys-fips` must forward IPv6 and route FIPS mesh space (`fd00::/8`) to `fips0`:

- enable IPv6 forwarding (`sysctl net.ipv6.conf.all.forwarding=1`)
- ensure route exists: `ip -6 route add fd00::/8 dev fips0`

Without this, AppVM traffic may resolve `.fips` names but fail to reach the resolved FIPS IPv6 destination.

### 3) AppVMs need a valid IPv6 source + route toward `sys-fips`

In testing, routed AppVM access required:

- a synthetic IPv6 on `eth0` (example: `fdaa::15/128`)
- route of FIPS space via Qubes link-local gateway:
  - `ip -6 route add fd00::/8 via fe80::fcff:ffff:feff:ffff dev eth0`

This ensures traffic to resolved `.fips` AAAA records leaves the AppVM through `sys-fips`.

### 4) Persist network fixes in `/rw/config/rc.local`

For reboot safety:

- persist nftables DNS interception + IPv6 forwarding/route in `sys-fips` `/rw/config/rc.local`
- persist AppVM synthetic IPv6 + route in each AppVM `/rw/config/rc.local` that needs `.fips`

Do not rely on one-off manual commands for production.

### 5) Chromium-family browser caveat

Observed behavior:

- Firefox followed system resolver path and worked once networking was fixed.
- Chromium-family browsers (`agent-browser`, Brave) could still show `ERR_NAME_NOT_RESOLVED` even when `dig`, `getent`, `ping`, and `curl` worked.

Current operational workaround is to pin the `.fips` hostname in `/etc/hosts` for Chromium-family use-cases.
See `plans/chromium-fips-workaround.md` for tested commands and policy notes.