voice_linux/plans/gpu_passthrough_lessons_learned.md

# GPU Passthrough in Qubes OS — Summary & Lessons Learned

## What We Did

Passed an NVIDIA GTX 1080 Ti (BDF `65:00.0` / `65:00.1`) through from Qubes dom0 to a StandaloneVM called `ai`, installed NVIDIA drivers and CUDA toolkit, and rebuilt whisper.cpp with CUDA support for GPU-accelerated speech-to-text.

## Environment

| Component | Detail |
|-----------|--------|
| Host OS | Qubes OS with Xen hypervisor |
| dom0 display GPU | AMD (stays in dom0) |
| Passthrough GPU | NVIDIA GeForce GTX 1080 Ti (Pascal/GP102, 11GB VRAM, compute 6.1) |
| Second NVIDIA GPU | RTX 3050 6GB at BDF `17:00.0` (not used for this project) |
| Target VM | `ai` — originally an AppVM, converted to StandaloneVM |
| VM OS | Debian 13 (bookworm/trixie) with XFCE |
| NVIDIA driver | 550.127.05 |
| CUDA toolkit | 12.6 |

## Timeline of Steps

### Phase 0: dom0 GPU Passthrough

1. **Identified GPU BDF addresses** — `lspci` in dom0 showed two NVIDIA GPUs. We targeted the GTX 1080 Ti at `65:00.0` (GPU) and `65:00.1` (HDMI audio controller).

2. **Confirmed boot loader** — Verified Qubes was using GRUB (not systemd-boot) by checking `/etc/default/grub` existed.

3. **Hid GPU from dom0** — Added `rd.qubes.hide_pci=65:00.0,65:00.1` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`.

4. **Regenerated GRUB config** — `sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg` (EFI path, not legacy BIOS path).

5. **Rebooted dom0** — Full system reboot required for PCI hide to take effect.

6. **Verified GPU was assignable** — `xl pci-assignable-list` showed `0000:65:00.0` and `0000:65:00.1`.

7. **Attached GPU to VM** — `qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true` (and same for `65_00.1`).

### Phase 1: Driver & CUDA Installation (inside ai VM)

8. **Installed build prerequisites** — `build-essential`, `linux-headers-$(uname -r)`.

9. **Installed NVIDIA driver** — Downloaded `.run` installer, ran with `--no-opengl-files --dkms`.

10. **Installed CUDA toolkit** — Via NVIDIA's Debian repo + `cuda-toolkit-12-6`.

11. **Verified** — `nvidia-smi` showed GTX 1080 Ti, `nvcc --version` showed CUDA 12.6.

### Phase 2: Application Rebuild

12. **Rebuilt whisper.cpp with CUDA** — `WHISPER_CUDA=1 bash ./build.sh`.

13. **Verified CUDA linkage** — `ldd ./voice_linux | grep whisper|ggml` confirmed CUDA libraries linked.

14. **Tested GPU inference** — Ran with GPU mode, confirmed `nvidia-smi` showed process using GPU memory.

## Problems Encountered & Solutions

### Problem 1: AppVM vs StandaloneVM confusion

**What happened**: The `ai` VM was initially described as an AppVM (template-based). In an AppVM, anything installed outside `/home` is lost on reboot — NVIDIA drivers would vanish.

**Discovery**: User clarified it was actually a StandaloneVM, not template-based.

**Lesson**: Always verify VM type before planning driver installation. In Qubes:
- **AppVM**: Root filesystem resets to template on reboot. Drivers must go in the template or use bind-dirs.
- **StandaloneVM**: Full persistent root filesystem. Drivers persist normally.
- Check with: `qvm-prefs ai virt_mode` and `qvm-ls --fields name,klass ai`

### Problem 2: VM must be shut down before PCI attach

**What happened**: Attempted to run `qvm-pci attach` while the `ai` VM was running. The attach appeared to succeed but the device wasn't visible inside the VM.

**Discovery**: `qvm-device pci list ai` showed empty even though attach command ran.

**Lesson**: The VM must be **shut down** before attaching PCI devices with `--persistent`. The workflow is:
1. Shut down VM
2. Attach PCI device
3. Start VM
4. Verify inside VM with `lspci`

### Problem 3: "Already assigned" error on re-attach

**What happened**: After a reboot cycle, running the attach command again produced an "already assigned" message.

**Lesson**: `--persistent` means the attachment survives reboots. Don't re-run the attach command after reboot — it's already configured. Verify with `qvm-pci list ai` from dom0.

### Problem 4: lspci not found inside VM

**What happened**: Tried to verify GPU visibility inside the VM but `lspci` wasn't installed.

**Solution**: `sudo apt install pciutils` then `lspci | grep -i nvidia`.

**Lesson**: Minimal Debian VMs may not have `pciutils` installed. Include it in prerequisites.

### Problem 5: Script couldn't be copy-pasted into dom0

**What happened**: Created a comprehensive dom0 shell script for GPU passthrough, but Qubes security model prevents clipboard paste from other VMs into dom0.

**Solution**: Provided numbered step-by-step commands that could be typed manually, and also created the script as a file that could be transferred via `qvm-run` or Qubes file copy.

**Lesson**: When automating dom0 tasks in Qubes:
- Dom0 is intentionally isolated — no clipboard sharing
- Scripts must be typed manually or transferred via `qvm-copy-to-vm` (from dom0 to VM) or `qvm-run -p` pipes
- Keep dom0 scripts short and simple
- Always include a revert mechanism

### Problem 6: Reboot scope confusion

**What happened**: Unclear whether "reboot" meant just the VM or the entire Qubes system (dom0 + all VMs).

**Clarification**: GRUB changes require a **full system reboot** (dom0 reboot, which takes down all VMs). PCI attachment changes only require the target VM to be restarted.

**Lesson**: Be explicit about reboot scope:
- `sudo reboot` in dom0 = full system reboot
- `qvm-shutdown ai && qvm-start ai` = just the VM

### Problem 7: Build accidentally ran without whisper support

**What happened**: After some iteration, a build was accidentally triggered with `WITH_WHISPER=0`, producing a binary that showed "[transcription unavailable: rebuild with WITH_WHISPER=1]".

**Solution**: Rebuilt with `WITH_WHISPER=1 bash ./build.sh`.

**Lesson**: The build system defaults matter. Our `build.sh` auto-detects whisper if the vendor directory exists, but explicit `WITH_WHISPER=0` overrides that. Always verify the build output includes whisper support with `ldd ./voice_linux | grep whisper`.

### Problem 8: Dirty git worktree blocking version increment

**What happened**: The `increment_and_push.sh` script refused to run because of uncommitted changes in the working tree.

**Solution**: Committed the intended changes first, then ran the increment script.

**Lesson**: The increment script enforces a clean worktree policy. Always commit your changes before running it. If there are unrelated/untracked files, `git stash` them first.

## Key Qubes-Specific Knowledge

### PCI Passthrough Essentials

```
# Hide device from dom0 (GRUB, requires full reboot):
rd.qubes.hide_pci=BDF1,BDF2

# Attach to VM (VM must be off):
qvm-pci attach VMNAME dom0:BDF --persistent -o permissive=true

# Verify assignment:
qvm-pci list VMNAME        # from dom0
lspci | grep -i nvidia      # from inside VM
```

### NVIDIA Driver in Qubes VM

- **Always use `--no-opengl-files`** — Qubes VMs use a virtual GPU for display. Installing OpenGL files would break the display.
- **Use `--dkms`** — Ensures kernel module rebuilds on kernel updates.
- **StandaloneVM recommended** — Avoids template pollution and persistence issues.
- **`permissive=true`** — Required for NVIDIA GPUs in Qubes due to how they access PCI config space.

### What Persists Where

| VM Type | /home | /usr, /lib, /etc | Drivers |
|---------|-------|-------------------|---------|
| AppVM | ✅ Persists | ❌ Resets to template | ❌ Lost on reboot |
| StandaloneVM | ✅ Persists | ✅ Persists | ✅ Persists |
| Template | ✅ Persists | ✅ Persists | ✅ Persists (shared to AppVMs) |

### Dom0 Safety

- Dom0 has no network access by design
- Clipboard is one-way (dom0 → VM, not VM → dom0) and requires explicit Ctrl+Shift+C/V
- Always have a revert plan for GRUB changes (keep a backup of `/etc/default/grub`)
- Log what you change — our script wrote to `/var/log/gpu_passthrough.log`

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────┐
│  dom0 (Xen Hypervisor)                                   │
│                                                           │
│  AMD GPU ──── Display                                     │
│  GTX 1080 Ti ──── HIDDEN via rd.qubes.hide_pci           │
│       │                                                   │
│       │ PCI passthrough (qvm-pci attach --persistent)     │
│       ▼                                                   │
│  ┌─────────────────────────────────────────────────────┐ │
│  │  ai StandaloneVM                                     │ │
│  │                                                       │ │
│  │  NVIDIA Driver 550.x (--no-opengl-files --dkms)      │ │
│  │       │                                               │ │
│  │  CUDA Toolkit 12.6                                    │ │
│  │       │                                               │ │
│  │  whisper.cpp (GGML_CUDA=ON)                           │ │
│  │       │                                               │ │
│  │  voice_linux (GPU-accelerated transcription)          │ │
│  └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
```

## Performance Results

| Model | VRAM Used | Decode Time (10s audio) | Quality |
|-------|-----------|------------------------|---------|
| tiny.en | ~1GB | <0.5s | Fair |
| base.en | ~1GB | <1s | Good |
| small.en | ~2GB | ~1-2s | Very good |
| medium.en | ~5GB | ~2-3s | Excellent |
| large-v3-turbo | ~6GB | ~3-4s | Excellent+ |
| large-v3 | ~10GB | ~4-6s | Best |

The GTX 1080 Ti with 11GB VRAM can run all models including large-v3. For real-time dictation, medium.en provides the best accuracy-to-speed tradeoff.

## Files Created During This Process

| File | Purpose |
|------|---------|
| [`plans/gpu_enablement_plan.md`](gpu_enablement_plan.md) | Step-by-step GPU passthrough and driver installation plan |
| [`plans/architecture.md`](architecture.md) | Full system architecture including hardware details |
| [`scripts/download_models.sh`](../scripts/download_models.sh) | Downloads all compatible whisper models |

## If You Had to Do It Again

1. **Verify VM type first** — `qvm-ls --fields name,klass ai` before planning anything
2. **Shut down VM before PCI operations** — always
3. **Back up GRUB config** — `sudo cp /etc/default/grub /etc/default/grub.bak` before editing
4. **Install pciutils early** — `sudo apt install pciutils` so you can verify GPU visibility
5. **Use the .run installer, not apt packages** — NVIDIA's apt packages for Debian can conflict with Qubes' virtual GPU setup; the `.run` installer with `--no-opengl-files` is cleaner
6. **Test CPU mode first** — Build and verify voice_linux works on CPU before adding GPU complexity
7. **Keep dom0 commands minimal** — Type them manually, don't try to automate complex scripts in dom0