241 lines
7.0 KiB
Markdown
241 lines
7.0 KiB
Markdown
# GPU Enablement Plan — voice_linux on NVIDIA GTX 1080 Ti
|
|
|
|
## Summary
|
|
|
|
Enable CUDA-accelerated whisper.cpp inference for the voice_linux project by passing the GTX 1080 Ti through to the Qubes `ai` **StandaloneVM**, installing NVIDIA drivers and CUDA toolkit, and rebuilding the application with `WHISPER_CUDA=1`.
|
|
|
|
## Current State
|
|
|
|
The codebase is **already GPU-ready**:
|
|
- [`build.sh`](../build.sh) accepts `WHISPER_CUDA=1` to build whisper.cpp with CUDA
|
|
- [`src/transcribe.c:22`](../src/transcribe.c:22) sets `cparams.use_gpu` from the transcribe params
|
|
- [`src/main.c:77`](../src/main.c:77) supports `--gpu` / `--cpu` CLI flags
|
|
- [`src/transcribe.h:9`](../src/transcribe.h:9) has `use_gpu` in the params struct
|
|
|
|
What's missing is the **hardware and driver stack** underneath.
|
|
|
|
## Architecture
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[dom0 - Xen Hypervisor] -->|PCI passthrough| B[ai StandaloneVM]
|
|
B --> C[NVIDIA Driver 550.x]
|
|
C --> D[CUDA Toolkit 12.6]
|
|
D --> E[whisper.cpp built with WHISPER_CUDA=ON]
|
|
E --> F[voice_linux --gpu]
|
|
|
|
style A fill:#f9f,stroke:#333
|
|
style F fill:#9f9,stroke:#333
|
|
```
|
|
|
|
## Phase 0: GPU Passthrough in dom0
|
|
|
|
> **All commands in this phase run in a dom0 terminal.**
|
|
|
|
### Step 0.1 — Hide the GTX 1080 Ti from dom0
|
|
|
|
Edit `/etc/default/grub` in dom0 and add `rd.qubes.hide_pci=65:00.0,65:00.1` to the end of the **first** `GRUB_CMDLINE_LINUX` line (inside the quotes):
|
|
|
|
```bash
|
|
sudo nano /etc/default/grub
|
|
```
|
|
|
|
The line should look like:
|
|
```
|
|
GRUB_CMDLINE_LINUX="...existing options... rd.qubes.hide_pci=65:00.0,65:00.1"
|
|
```
|
|
|
|
The BDF addresses `65:00.0` (GPU) and `65:00.1` (HDMI audio) come from the `lspci` output documented in [`plans/architecture.md:97`](../plans/architecture.md:97).
|
|
|
|
### Step 0.2 — Regenerate GRUB and reboot
|
|
|
|
```bash
|
|
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
|
|
# If that path doesn't exist: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
|
|
sudo reboot
|
|
```
|
|
|
|
### Step 0.3 — Verify GPU is assignable
|
|
|
|
After reboot, in dom0:
|
|
|
|
```bash
|
|
xl pci-assignable-list
|
|
```
|
|
|
|
Expected output should include:
|
|
```
|
|
0000:65:00.0
|
|
0000:65:00.1
|
|
```
|
|
|
|
If these don't appear, check IOMMU grouping with `xl pci-assignable-list -l` and verify the GRUB change took effect with `cat /proc/cmdline`.
|
|
|
|
### Step 0.4 — Attach GPU to the ai StandaloneVM
|
|
|
|
```bash
|
|
qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true
|
|
qvm-pci attach ai dom0:65_00.1 --persistent -o permissive=true
|
|
```
|
|
|
|
The `--persistent` flag ensures the GPU is attached on every boot. The `permissive=true` option is needed for NVIDIA GPUs in Qubes.
|
|
|
|
### Step 0.5 — Persistence model (already satisfied)
|
|
|
|
Your `ai` VM is already a **StandaloneVM**, so driver and CUDA installs persist normally across reboots.
|
|
|
|
No template/bind-dirs workaround is required.
|
|
|
|
## Phase 1: NVIDIA Driver Installation
|
|
|
|
> **All commands from here run inside the ai StandaloneVM.**
|
|
|
|
### Step 1.1 — Install build prerequisites
|
|
|
|
```bash
|
|
sudo apt update
|
|
sudo apt install -y build-essential linux-headers-$(uname -r)
|
|
```
|
|
|
|
### Step 1.2 — Install NVIDIA driver
|
|
|
|
For the GTX 1080 Ti (Pascal/GP102, compute capability 6.1), use the 550.x branch:
|
|
|
|
```bash
|
|
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.127.05/NVIDIA-Linux-x86_64-550.127.05.run
|
|
chmod +x NVIDIA-Linux-x86_64-550.127.05.run
|
|
sudo ./NVIDIA-Linux-x86_64-550.127.05.run --no-opengl-files --dkms
|
|
```
|
|
|
|
> **Critical**: `--no-opengl-files` prevents replacing the VM's display GL stack. We only need CUDA compute, not display rendering.
|
|
|
|
### Step 1.3 — Verify driver
|
|
|
|
```bash
|
|
nvidia-smi
|
|
```
|
|
|
|
Expected: Shows GTX 1080 Ti with 11GB VRAM and driver version 550.127.05.
|
|
|
|
## Phase 2: CUDA Toolkit Installation
|
|
|
|
### Step 2.1 — Install CUDA toolkit
|
|
|
|
```bash
|
|
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
|
|
sudo dpkg -i cuda-keyring_1.1-1_all.deb
|
|
sudo apt update
|
|
sudo apt install -y cuda-toolkit-12-6
|
|
```
|
|
|
|
### Step 2.2 — Configure environment
|
|
|
|
Add to `~/.bashrc`:
|
|
|
|
```bash
|
|
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
|
|
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
|
|
source ~/.bashrc
|
|
```
|
|
|
|
### Step 2.3 — Verify CUDA
|
|
|
|
```bash
|
|
nvcc --version
|
|
```
|
|
|
|
Expected: CUDA 12.6.
|
|
|
|
## Phase 3: Rebuild with CUDA
|
|
|
|
### Step 3.1 — Clean and rebuild
|
|
|
|
From the project directory, run:
|
|
|
|
```bash
|
|
WHISPER_CUDA=1 bash ./build.sh
|
|
```
|
|
|
|
This will:
|
|
1. Clone whisper.cpp into `vendor/whisper.cpp` (if not already present)
|
|
2. Run `cmake -DWHISPER_CUDA=ON` to configure whisper.cpp with CUDA support
|
|
3. Build whisper.cpp with CUDA kernels
|
|
4. Download the `ggml-base.en.bin` model (if not present)
|
|
5. Compile voice_linux linking against the CUDA-enabled whisper library
|
|
|
|
### Step 3.2 — Verify CUDA linkage
|
|
|
|
```bash
|
|
ldd ./voice_linux | grep -i cuda
|
|
```
|
|
|
|
Should show linkage to CUDA libraries (indirectly through libwhisper).
|
|
|
|
Also check:
|
|
```bash
|
|
ldd ./vendor/whisper.cpp/build/src/libwhisper.so | grep -i cuda
|
|
```
|
|
|
|
Should show `libcudart.so`, `libcublas.so`, etc.
|
|
|
|
## Phase 4: Run with GPU
|
|
|
|
### Step 4.1 — Launch with GPU flag
|
|
|
|
```bash
|
|
./voice_linux --gpu
|
|
```
|
|
|
|
The startup log at [`src/main.c:119`](../src/main.c:119) will print `gpu: on` confirming GPU mode. Watch for whisper.cpp log lines mentioning CUDA device initialization.
|
|
|
|
### Step 4.2 — Verify GPU utilization
|
|
|
|
In a separate terminal while transcribing:
|
|
|
|
```bash
|
|
nvidia-smi
|
|
```
|
|
|
|
Should show `voice_linux` process using GPU memory.
|
|
|
|
### Step 4.3 — Optional: Try a larger model
|
|
|
|
With 11GB VRAM on the 1080 Ti, you can comfortably run `medium.en` for much better accuracy:
|
|
|
|
```bash
|
|
cd models/
|
|
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin
|
|
```
|
|
|
|
Then update [`config.ini`](../config.ini) line 5:
|
|
```ini
|
|
model_path=./models/ggml-medium.en.bin
|
|
```
|
|
|
|
| Model | VRAM Usage | Speed (10s audio) | Accuracy |
|
|
|-------|-----------|-------------------|----------|
|
|
| base.en | ~1GB | <1s | Good |
|
|
| medium.en | ~5GB | ~2-3s | Excellent |
|
|
| large-v3 | ~10GB | ~4-6s | Best |
|
|
|
|
## Troubleshooting
|
|
|
|
| Problem | Solution |
|
|
|---------|----------|
|
|
| `xl pci-assignable-list` empty | Check GRUB cmdline with `cat /proc/cmdline`, verify `rd.qubes.hide_pci` is present |
|
|
| `nvidia-smi` not found | Driver install failed; check `dmesg` for NVIDIA errors |
|
|
| `nvcc` not found | CUDA PATH not set; run `source ~/.bashrc` or check install |
|
|
| whisper.cpp cmake fails with CUDA | Ensure `nvcc` is in PATH before running `build.sh` |
|
|
| `voice_linux` crashes on `--gpu` | Run without `--gpu` first to verify CPU mode works, then check CUDA libs with `ldd` |
|
|
| GPU passthrough causes VM crash | Try without `permissive=true` first, or check IOMMU groups in dom0 |
|
|
|
|
## No Code Changes Required
|
|
|
|
The existing codebase handles everything:
|
|
- [`build.sh:8`](../build.sh:8) — `WHISPER_CUDA` env var controls CUDA build
|
|
- [`build.sh:92-93`](../build.sh:92) — passes `-DWHISPER_CUDA=ON` to cmake
|
|
- [`src/transcribe.c:21-22`](../src/transcribe.c:21) — `whisper_context_default_params()` + `use_gpu` flag
|
|
- [`src/main.c:77-78`](../src/main.c:77) — `--gpu` / `--cpu` CLI argument parsing
|
|
|
|
The only change is the build command: `WHISPER_CUDA=1 bash ./build.sh` instead of `bash ./build.sh`.
|