# GPU Enablement Plan — voice_linux on NVIDIA GTX 1080 Ti

## Summary

Enable CUDA-accelerated whisper.cpp inference for the voice_linux project by passing the GTX 1080 Ti through to the Qubes `ai` **StandaloneVM**, installing NVIDIA drivers and CUDA toolkit, and rebuilding the application with `WHISPER_CUDA=1`.

## Current State

The codebase is **already GPU-ready**:
- [`build.sh`](../build.sh) accepts `WHISPER_CUDA=1` to build whisper.cpp with CUDA
- [`src/transcribe.c:22`](../src/transcribe.c:22) sets `cparams.use_gpu` from the transcribe params
- [`src/main.c:77`](../src/main.c:77) supports `--gpu` / `--cpu` CLI flags
- [`src/transcribe.h:9`](../src/transcribe.h:9) has `use_gpu` in the params struct

What's missing is the **hardware and driver stack** underneath.

## Architecture

```mermaid
graph TD
    A[dom0 - Xen Hypervisor] -->|PCI passthrough| B[ai StandaloneVM]
    B --> C[NVIDIA Driver 550.x]
    C --> D[CUDA Toolkit 12.6]
    D --> E[whisper.cpp built with WHISPER_CUDA=ON]
    E --> F[voice_linux --gpu]
    
    style A fill:#f9f,stroke:#333
    style F fill:#9f9,stroke:#333
```

## Phase 0: GPU Passthrough in dom0

> **All commands in this phase run in a dom0 terminal.**

### Step 0.1 — Hide the GTX 1080 Ti from dom0

Edit `/etc/default/grub` in dom0 and add `rd.qubes.hide_pci=65:00.0,65:00.1` to the end of the **first** `GRUB_CMDLINE_LINUX` line (inside the quotes):

```bash
sudo nano /etc/default/grub
```

The line should look like:
```
GRUB_CMDLINE_LINUX="...existing options... rd.qubes.hide_pci=65:00.0,65:00.1"
```

The BDF addresses `65:00.0` (GPU) and `65:00.1` (HDMI audio) come from the `lspci` output documented in [`plans/architecture.md:97`](../plans/architecture.md:97).

### Step 0.2 — Regenerate GRUB and reboot

```bash
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
# If that path doesn't exist: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
```

### Step 0.3 — Verify GPU is assignable

After reboot, in dom0:

```bash
xl pci-assignable-list
```

Expected output should include:
```
0000:65:00.0
0000:65:00.1
```

If these don't appear, check IOMMU grouping with `xl pci-assignable-list -l` and verify the GRUB change took effect with `cat /proc/cmdline`.

### Step 0.4 — Attach GPU to the ai StandaloneVM

```bash
qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true
qvm-pci attach ai dom0:65_00.1 --persistent -o permissive=true
```

The `--persistent` flag ensures the GPU is attached on every boot. The `permissive=true` option is needed for NVIDIA GPUs in Qubes.

### Step 0.5 — Persistence model (already satisfied)

Your `ai` VM is already a **StandaloneVM**, so driver and CUDA installs persist normally across reboots.

No template/bind-dirs workaround is required.

## Phase 1: NVIDIA Driver Installation

> **All commands from here run inside the ai StandaloneVM.**

### Step 1.1 — Install build prerequisites

```bash
sudo apt update
sudo apt install -y build-essential linux-headers-$(uname -r)
```

### Step 1.2 — Install NVIDIA driver

For the GTX 1080 Ti (Pascal/GP102, compute capability 6.1), use the 550.x branch:

```bash
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.127.05/NVIDIA-Linux-x86_64-550.127.05.run
chmod +x NVIDIA-Linux-x86_64-550.127.05.run
sudo ./NVIDIA-Linux-x86_64-550.127.05.run --no-opengl-files --dkms
```

> **Critical**: `--no-opengl-files` prevents replacing the VM's display GL stack. We only need CUDA compute, not display rendering.

### Step 1.3 — Verify driver

```bash
nvidia-smi
```

Expected: Shows GTX 1080 Ti with 11GB VRAM and driver version 550.127.05.

## Phase 2: CUDA Toolkit Installation

### Step 2.1 — Install CUDA toolkit

```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-6
```

### Step 2.2 — Configure environment

Add to `~/.bashrc`:

```bash
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc
```

### Step 2.3 — Verify CUDA

```bash
nvcc --version
```

Expected: CUDA 12.6.

## Phase 3: Rebuild with CUDA

### Step 3.1 — Clean and rebuild

From the project directory, run:

```bash
WHISPER_CUDA=1 bash ./build.sh
```

This will:
1. Clone whisper.cpp into `vendor/whisper.cpp` (if not already present)
2. Run `cmake -DWHISPER_CUDA=ON` to configure whisper.cpp with CUDA support
3. Build whisper.cpp with CUDA kernels
4. Download the `ggml-base.en.bin` model (if not present)
5. Compile voice_linux linking against the CUDA-enabled whisper library

### Step 3.2 — Verify CUDA linkage

```bash
ldd ./voice_linux | grep -i cuda
```

Should show linkage to CUDA libraries (indirectly through libwhisper).

Also check:
```bash
ldd ./vendor/whisper.cpp/build/src/libwhisper.so | grep -i cuda
```

Should show `libcudart.so`, `libcublas.so`, etc.

## Phase 4: Run with GPU

### Step 4.1 — Launch with GPU flag

```bash
./voice_linux --gpu
```

The startup log at [`src/main.c:119`](../src/main.c:119) will print `gpu: on` confirming GPU mode. Watch for whisper.cpp log lines mentioning CUDA device initialization.

### Step 4.2 — Verify GPU utilization

In a separate terminal while transcribing:

```bash
nvidia-smi
```

Should show `voice_linux` process using GPU memory.

### Step 4.3 — Optional: Try a larger model

With 11GB VRAM on the 1080 Ti, you can comfortably run `medium.en` for much better accuracy:

```bash
cd models/
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin
```

Then update [`config.ini`](../config.ini) line 5:
```ini
model_path=./models/ggml-medium.en.bin
```

| Model | VRAM Usage | Speed (10s audio) | Accuracy |
|-------|-----------|-------------------|----------|
| base.en | ~1GB | <1s | Good |
| medium.en | ~5GB | ~2-3s | Excellent |
| large-v3 | ~10GB | ~4-6s | Best |

## Troubleshooting

| Problem | Solution |
|---------|----------|
| `xl pci-assignable-list` empty | Check GRUB cmdline with `cat /proc/cmdline`, verify `rd.qubes.hide_pci` is present |
| `nvidia-smi` not found | Driver install failed; check `dmesg` for NVIDIA errors |
| `nvcc` not found | CUDA PATH not set; run `source ~/.bashrc` or check install |
| whisper.cpp cmake fails with CUDA | Ensure `nvcc` is in PATH before running `build.sh` |
| `voice_linux` crashes on `--gpu` | Run without `--gpu` first to verify CPU mode works, then check CUDA libs with `ldd` |
| GPU passthrough causes VM crash | Try without `permissive=true` first, or check IOMMU groups in dom0 |

## No Code Changes Required

The existing codebase handles everything:
- [`build.sh:8`](../build.sh:8) — `WHISPER_CUDA` env var controls CUDA build
- [`build.sh:92-93`](../build.sh:92) — passes `-DWHISPER_CUDA=ON` to cmake
- [`src/transcribe.c:21-22`](../src/transcribe.c:21) — `whisper_context_default_params()` + `use_gpu` flag
- [`src/main.c:77-78`](../src/main.c:77) — `--gpu` / `--cpu` CLI argument parsing

The only change is the build command: `WHISPER_CUDA=1 bash ./build.sh` instead of `bash ./build.sh`.