voice_linux/plans/gpu_enablement_plan.md

GPU Enablement Plan — voice_linux on NVIDIA GTX 1080 Ti

Summary

Enable CUDA-accelerated whisper.cpp inference for the voice_linux project by passing the GTX 1080 Ti through to the Qubes ai StandaloneVM, installing NVIDIA drivers and CUDA toolkit, and rebuilding the application with WHISPER_CUDA=1.

Current State

The codebase is already GPU-ready:

• build.sh accepts WHISPER_CUDA=1 to build whisper.cpp with CUDA
• src/transcribe.c:22 sets cparams.use_gpu from the transcribe params
• src/main.c:77 supports --gpu / --cpu CLI flags
• src/transcribe.h:9 has use_gpu in the params struct

What's missing is the hardware and driver stack underneath.

Architecture

graph TD
    A[dom0 - Xen Hypervisor] -->|PCI passthrough| B[ai StandaloneVM]
    B --> C[NVIDIA Driver 550.x]
    C --> D[CUDA Toolkit 12.6]
    D --> E[whisper.cpp built with WHISPER_CUDA=ON]
    E --> F[voice_linux --gpu]
    
    style A fill:#f9f,stroke:#333
    style F fill:#9f9,stroke:#333

Phase 0: GPU Passthrough in dom0

All commands in this phase run in a dom0 terminal.

Step 0.1 — Hide the GTX 1080 Ti from dom0

Edit /etc/default/grub in dom0 and add rd.qubes.hide_pci=65:00.0,65:00.1 to the end of the first GRUB_CMDLINE_LINUX line (inside the quotes):

sudo nano /etc/default/grub

The line should look like:

GRUB_CMDLINE_LINUX="...existing options... rd.qubes.hide_pci=65:00.0,65:00.1"

The BDF addresses 65:00.0 (GPU) and 65:00.1 (HDMI audio) come from the lspci output documented in plans/architecture.md:97.

Step 0.2 — Regenerate GRUB and reboot

sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
# If that path doesn't exist: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Step 0.3 — Verify GPU is assignable

After reboot, in dom0:

xl pci-assignable-list

Expected output should include:

0000:65:00.0
0000:65:00.1

If these don't appear, check IOMMU grouping with xl pci-assignable-list -l and verify the GRUB change took effect with cat /proc/cmdline.

Step 0.4 — Attach GPU to the ai StandaloneVM

qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true
qvm-pci attach ai dom0:65_00.1 --persistent -o permissive=true

The --persistent flag ensures the GPU is attached on every boot. The permissive=true option is needed for NVIDIA GPUs in Qubes.

Step 0.5 — Persistence model (already satisfied)

Your ai VM is already a StandaloneVM, so driver and CUDA installs persist normally across reboots.

No template/bind-dirs workaround is required.

Phase 1: NVIDIA Driver Installation

All commands from here run inside the ai StandaloneVM.

Step 1.1 — Install build prerequisites

sudo apt update
sudo apt install -y build-essential linux-headers-$(uname -r)

Step 1.2 — Install NVIDIA driver

For the GTX 1080 Ti (Pascal/GP102, compute capability 6.1), use the 550.x branch:

wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.127.05/NVIDIA-Linux-x86_64-550.127.05.run
chmod +x NVIDIA-Linux-x86_64-550.127.05.run
sudo ./NVIDIA-Linux-x86_64-550.127.05.run --no-opengl-files --dkms

Critical: --no-opengl-files prevents replacing the VM's display GL stack. We only need CUDA compute, not display rendering.

Step 1.3 — Verify driver

nvidia-smi

Expected: Shows GTX 1080 Ti with 11GB VRAM and driver version 550.127.05.

Phase 2: CUDA Toolkit Installation

Step 2.1 — Install CUDA toolkit

wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-6

Step 2.2 — Configure environment

Add to ~/.bashrc:

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

Step 2.3 — Verify CUDA

nvcc --version

Expected: CUDA 12.6.

Phase 3: Rebuild with CUDA

Step 3.1 — Clean and rebuild

From the project directory, run:

WHISPER_CUDA=1 bash ./build.sh

This will:

1. Clone whisper.cpp into vendor/whisper.cpp (if not already present)
2. Run cmake -DWHISPER_CUDA=ON to configure whisper.cpp with CUDA support
3. Build whisper.cpp with CUDA kernels
4. Download the ggml-base.en.bin model (if not present)
5. Compile voice_linux linking against the CUDA-enabled whisper library

Step 3.2 — Verify CUDA linkage

ldd ./voice_linux | grep -i cuda

Should show linkage to CUDA libraries (indirectly through libwhisper).

Also check:

ldd ./vendor/whisper.cpp/build/src/libwhisper.so | grep -i cuda

Should show libcudart.so, libcublas.so, etc.

Phase 4: Run with GPU

Step 4.1 — Launch with GPU flag

./voice_linux --gpu

The startup log at src/main.c:119 will print gpu: on confirming GPU mode. Watch for whisper.cpp log lines mentioning CUDA device initialization.

Step 4.2 — Verify GPU utilization

In a separate terminal while transcribing:

nvidia-smi

Should show voice_linux process using GPU memory.

Step 4.3 — Optional: Try a larger model

With 11GB VRAM on the 1080 Ti, you can comfortably run medium.en for much better accuracy:

cd models/
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin

Then update config.ini line 5:

model_path=./models/ggml-medium.en.bin

Model	VRAM Usage	Speed (10s audio)	Accuracy
base.en	~1GB	<1s	Good
medium.en	~5GB	~2-3s	Excellent
large-v3	~10GB	~4-6s	Best

Troubleshooting

Problem	Solution
xl pci-assignable-list empty	Check GRUB cmdline with cat /proc/cmdline, verify rd.qubes.hide_pci is present
nvidia-smi not found	Driver install failed; check dmesg for NVIDIA errors
nvcc not found	CUDA PATH not set; run source ~/.bashrc or check install
whisper.cpp cmake fails with CUDA	Ensure nvcc is in PATH before running build.sh
voice_linux crashes on --gpu	Run without --gpu first to verify CPU mode works, then check CUDA libs with ldd
GPU passthrough causes VM crash	Try without permissive=true first, or check IOMMU groups in dom0

No Code Changes Required

The existing codebase handles everything:

• build.sh:8 — WHISPER_CUDA env var controls CUDA build
• build.sh:92-93 — passes -DWHISPER_CUDA=ON to cmake
• src/transcribe.c:21-22 — whisper_context_default_params() + use_gpu flag
• src/main.c:77-78 — --gpu / --cpu CLI argument parsing

The only change is the build command: WHISPER_CUDA=1 bash ./build.sh instead of bash ./build.sh.