4 Commits
12 changed files with 478 additions and 19 deletions
+1 -1
View File
@@ -27,4 +27,4 @@ vad_min_speech_ms=250
# hard cap for one speech segment
vad_max_speech_ms=15000
autotype_enabled=0
autotype_enabled=1
+253
View File
@@ -0,0 +1,253 @@
# GPU Enablement Plan — voice_linux on NVIDIA GTX 1080 Ti
## Summary
Enable CUDA-accelerated whisper.cpp inference for the voice_linux project by passing the GTX 1080 Ti through to the Qubes `ai` AppVM, installing NVIDIA drivers and CUDA toolkit, and rebuilding the application with `WHISPER_CUDA=1`.
## Current State
The codebase is **already GPU-ready**:
- [`build.sh`](../build.sh) accepts `WHISPER_CUDA=1` to build whisper.cpp with CUDA
- [`src/transcribe.c:22`](../src/transcribe.c:22) sets `cparams.use_gpu` from the transcribe params
- [`src/main.c:77`](../src/main.c:77) supports `--gpu` / `--cpu` CLI flags
- [`src/transcribe.h:9`](../src/transcribe.h:9) has `use_gpu` in the params struct
What's missing is the **hardware and driver stack** underneath.
## Architecture
```mermaid
graph TD
A[dom0 - Xen Hypervisor] -->|PCI passthrough| B[ai AppVM]
B --> C[NVIDIA Driver 550.x]
C --> D[CUDA Toolkit 12.6]
D --> E[whisper.cpp built with WHISPER_CUDA=ON]
E --> F[voice_linux --gpu]
style A fill:#f9f,stroke:#333
style F fill:#9f9,stroke:#333
```
## Phase 0: GPU Passthrough in dom0
> **All commands in this phase run in a dom0 terminal.**
### Step 0.1 — Hide the GTX 1080 Ti from dom0
Edit `/etc/default/grub` in dom0 and add `rd.qubes.hide_pci=65:00.0,65:00.1` to the end of the **first** `GRUB_CMDLINE_LINUX` line (inside the quotes):
```bash
sudo nano /etc/default/grub
```
The line should look like:
```
GRUB_CMDLINE_LINUX="...existing options... rd.qubes.hide_pci=65:00.0,65:00.1"
```
The BDF addresses `65:00.0` (GPU) and `65:00.1` (HDMI audio) come from the `lspci` output documented in [`plans/architecture.md:97`](../plans/architecture.md:97).
### Step 0.2 — Regenerate GRUB and reboot
```bash
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
# If that path doesn't exist: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
```
### Step 0.3 — Verify GPU is assignable
After reboot, in dom0:
```bash
xl pci-assignable-list
```
Expected output should include:
```
0000:65:00.0
0000:65:00.1
```
If these don't appear, check IOMMU grouping with `xl pci-assignable-list -l` and verify the GRUB change took effect with `cat /proc/cmdline`.
### Step 0.4 — Attach GPU to the ai AppVM
```bash
qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true
qvm-pci attach ai dom0:65_00.1 --persistent -o permissive=true
```
The `--persistent` flag ensures the GPU is attached on every boot. The `permissive=true` option is needed for NVIDIA GPUs in Qubes.
### Step 0.5 — Decide on driver persistence
Since `ai` is a template-based AppVM using `debian-13-xfce`, anything installed outside `/home` is lost on reboot. Three options:
| Option | Pros | Cons |
|--------|------|------|
| **A: Install in template** | Persists automatically, simple | Affects ALL AppVMs using that template |
| **B: Convert to StandaloneVM** | Full persistence, isolated | Uses more disk, no template updates |
| **C: bind-dirs** | Keeps template-based, selective persistence | More complex setup |
**Recommendation**: Option B (StandaloneVM) is simplest for a dedicated AI workload. Run in dom0:
```bash
qvm-clone --class StandaloneVM ai ai-standalone
# Then attach GPU to ai-standalone instead
qvm-pci attach ai-standalone dom0:65_00.0 --persistent -o permissive=true
qvm-pci attach ai-standalone dom0:65_00.1 --persistent -o permissive=true
```
## Phase 1: NVIDIA Driver Installation
> **All commands from here run inside the ai AppVM (or ai-standalone).**
### Step 1.1 — Install build prerequisites
```bash
sudo apt update
sudo apt install -y build-essential linux-headers-$(uname -r)
```
### Step 1.2 — Install NVIDIA driver
For the GTX 1080 Ti (Pascal/GP102, compute capability 6.1), use the 550.x branch:
```bash
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.127.05/NVIDIA-Linux-x86_64-550.127.05.run
chmod +x NVIDIA-Linux-x86_64-550.127.05.run
sudo ./NVIDIA-Linux-x86_64-550.127.05.run --no-opengl-files --dkms
```
> **Critical**: `--no-opengl-files` prevents replacing the VM's display GL stack. We only need CUDA compute, not display rendering.
### Step 1.3 — Verify driver
```bash
nvidia-smi
```
Expected: Shows GTX 1080 Ti with 11GB VRAM and driver version 550.127.05.
## Phase 2: CUDA Toolkit Installation
### Step 2.1 — Install CUDA toolkit
```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-6
```
### Step 2.2 — Configure environment
Add to `~/.bashrc`:
```bash
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc
```
### Step 2.3 — Verify CUDA
```bash
nvcc --version
```
Expected: CUDA 12.6.
## Phase 3: Rebuild with CUDA
### Step 3.1 — Clean and rebuild
From the project directory, run:
```bash
WHISPER_CUDA=1 bash ./build.sh
```
This will:
1. Clone whisper.cpp into `vendor/whisper.cpp` (if not already present)
2. Run `cmake -DWHISPER_CUDA=ON` to configure whisper.cpp with CUDA support
3. Build whisper.cpp with CUDA kernels
4. Download the `ggml-base.en.bin` model (if not present)
5. Compile voice_linux linking against the CUDA-enabled whisper library
### Step 3.2 — Verify CUDA linkage
```bash
ldd ./voice_linux | grep -i cuda
```
Should show linkage to CUDA libraries (indirectly through libwhisper).
Also check:
```bash
ldd ./vendor/whisper.cpp/build/src/libwhisper.so | grep -i cuda
```
Should show `libcudart.so`, `libcublas.so`, etc.
## Phase 4: Run with GPU
### Step 4.1 — Launch with GPU flag
```bash
./voice_linux --gpu
```
The startup log at [`src/main.c:119`](../src/main.c:119) will print `gpu: on` confirming GPU mode. Watch for whisper.cpp log lines mentioning CUDA device initialization.
### Step 4.2 — Verify GPU utilization
In a separate terminal while transcribing:
```bash
nvidia-smi
```
Should show `voice_linux` process using GPU memory.
### Step 4.3 — Optional: Try a larger model
With 11GB VRAM on the 1080 Ti, you can comfortably run `medium.en` for much better accuracy:
```bash
cd models/
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin
```
Then update [`config.ini`](../config.ini) line 5:
```ini
model_path=./models/ggml-medium.en.bin
```
| Model | VRAM Usage | Speed (10s audio) | Accuracy |
|-------|-----------|-------------------|----------|
| base.en | ~1GB | <1s | Good |
| medium.en | ~5GB | ~2-3s | Excellent |
| large-v3 | ~10GB | ~4-6s | Best |
## Troubleshooting
| Problem | Solution |
|---------|----------|
| `xl pci-assignable-list` empty | Check GRUB cmdline with `cat /proc/cmdline`, verify `rd.qubes.hide_pci` is present |
| `nvidia-smi` not found | Driver install failed; check `dmesg` for NVIDIA errors |
| `nvcc` not found | CUDA PATH not set; run `source ~/.bashrc` or check install |
| whisper.cpp cmake fails with CUDA | Ensure `nvcc` is in PATH before running `build.sh` |
| `voice_linux` crashes on `--gpu` | Run without `--gpu` first to verify CPU mode works, then check CUDA libs with `ldd` |
| GPU passthrough causes VM crash | Try without `permissive=true` first, or check IOMMU groups in dom0 |
## No Code Changes Required
The existing codebase handles everything:
- [`build.sh:8`](../build.sh:8) — `WHISPER_CUDA` env var controls CUDA build
- [`build.sh:92-93`](../build.sh:92) — passes `-DWHISPER_CUDA=ON` to cmake
- [`src/transcribe.c:21-22`](../src/transcribe.c:21) — `whisper_context_default_params()` + `use_gpu` flag
- [`src/main.c:77-78`](../src/main.c:77) — `--gpu` / `--cpu` CLI argument parsing
The only change is the build command: `WHISPER_CUDA=1 bash ./build.sh` instead of `bash ./build.sh`.
+4 -1
View File
@@ -36,7 +36,8 @@ void config_set_defaults(voice_config_t *cfg) {
cfg->vad_preroll_ms = 500;
cfg->vad_min_speech_ms = 250;
cfg->vad_max_speech_ms = 15000;
cfg->autotype_enabled = 0;
cfg->autotype_enabled = 1;
cfg->filter_bracketed_tags = 1;
}
int config_load_file(const char *path, voice_config_t *cfg) {
@@ -91,6 +92,8 @@ int config_load_file(const char *path, voice_config_t *cfg) {
cfg->vad_max_speech_ms = atoi(value);
} else if (strcmp(key, "autotype_enabled") == 0) {
cfg->autotype_enabled = atoi(value);
} else if (strcmp(key, "filter_bracketed_tags") == 0) {
cfg->filter_bracketed_tags = atoi(value);
}
}
+1
View File
@@ -20,6 +20,7 @@ typedef struct {
int vad_min_speech_ms;
int vad_max_speech_ms;
int autotype_enabled;
int filter_bracketed_tags;
} voice_config_t;
void config_set_defaults(voice_config_t *cfg);
+26 -1
View File
@@ -18,6 +18,7 @@ typedef struct {
GtkWidget *record_btn;
GtkWidget *always_on_btn;
GtkWidget *type_check;
GtkWidget *filter_tags_check;
GtkWidget *transcript_view;
GtkWidget *record_light;
GtkWidget *transcribe_light;
@@ -175,6 +176,12 @@ static void process_segment(gui_state_t *s, float *samples, size_t count) {
return;
}
if (gtk_toggle_button_get_active(GTK_TOGGLE_BUTTON(s->type_check))) {
if (typer_capture_focus() != 0) {
fprintf(stderr, "gui: failed to capture focus target for autotype\n");
}
}
s->transcribing = 1;
update_record_button(s);
set_status(s, "Transcribing... (please wait)");
@@ -194,7 +201,12 @@ static void process_segment(gui_state_t *s, float *samples, size_t count) {
append_transcript(s, text);
if (gtk_toggle_button_get_active(GTK_TOGGLE_BUTTON(s->type_check))) {
typer_type_text(text);
if (typer_type_text(text) != 0) {
fprintf(stderr, "gui: autotype injection failed\n");
set_status(s, "Autotype failed (see terminal logs)");
free(text);
return;
}
}
set_status(s, audio_msg);
@@ -413,6 +425,13 @@ static GtkWidget *add_scale_row(GtkWidget *grid,
return scale;
}
static void on_filter_tags_toggled(GtkToggleButton *btn, gpointer user_data) {
gui_state_t *s = (gui_state_t *)user_data;
int enabled = gtk_toggle_button_get_active(btn) ? 1 : 0;
s->cfg.filter_bracketed_tags = enabled;
transcribe_set_filter_bracketed_tags(enabled);
}
static void on_app_activate(GtkApplication *app, gpointer user_data) {
gui_state_t *s = (gui_state_t *)user_data;
@@ -478,6 +497,11 @@ static void on_app_activate(GtkApplication *app, gpointer user_data) {
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->type_check), s->cfg.autotype_enabled ? TRUE : FALSE);
gtk_box_pack_start(GTK_BOX(outer), s->type_check, FALSE, FALSE, 0);
s->filter_tags_check = gtk_check_button_new_with_label("Filter bracketed tags like [laughs]");
gtk_toggle_button_set_active(GTK_TOGGLE_BUTTON(s->filter_tags_check), s->cfg.filter_bracketed_tags ? TRUE : FALSE);
g_signal_connect(s->filter_tags_check, "toggled", G_CALLBACK(on_filter_tags_toggled), s);
gtk_box_pack_start(GTK_BOX(outer), s->filter_tags_check, FALSE, FALSE, 0);
GtkWidget *meter_frame = gtk_frame_new("Live Microphone Level");
gtk_box_pack_start(GTK_BOX(outer), meter_frame, FALSE, FALSE, 0);
@@ -586,6 +610,7 @@ int main(int argc, char **argv) {
snprintf(tp.language, sizeof(tp.language), "%s", st.cfg.language);
tp.use_gpu = st.use_gpu;
tp.n_threads = (int)sysconf(_SC_NPROCESSORS_ONLN);
tp.filter_bracketed_tags = st.cfg.filter_bracketed_tags;
if (audio_init(&ap) != 0) {
fprintf(stderr, "fatal: audio init failed\n");
+1
View File
@@ -87,6 +87,7 @@ int main(int argc, char **argv) {
snprintf(tp.language, sizeof(tp.language), "%s", cfg.language);
tp.use_gpu = use_gpu;
tp.n_threads = (int)sysconf(_SC_NPROCESSORS_ONLN);
tp.filter_bracketed_tags = cfg.filter_bracketed_tags;
hotkey_ctx_t hk;
+78 -1
View File
@@ -1,5 +1,6 @@
#include "transcribe.h"
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@@ -8,6 +9,71 @@
#define WITH_WHISPER 0
#endif
static int g_filter_bracketed_tags = 1;
static char *filter_bracketed_tags_copy(const char *input) {
if (!input) return NULL;
const size_t len = strlen(input);
char *no_tags = (char *)malloc(len + 1);
if (!no_tags) return NULL;
size_t w = 0;
int bracket_depth = 0;
for (size_t i = 0; i < len; ++i) {
unsigned char ch = (unsigned char)input[i];
if (ch == '[') {
bracket_depth++;
continue;
}
if (ch == ']' && bracket_depth > 0) {
bracket_depth--;
continue;
}
if (bracket_depth > 0) {
continue;
}
no_tags[w++] = (char)ch;
}
no_tags[w] = '\0';
char *out = (char *)malloc(w + 1);
if (!out) {
free(no_tags);
return NULL;
}
size_t j = 0;
int in_space = 1;
for (size_t i = 0; i < w; ++i) {
unsigned char ch = (unsigned char)no_tags[i];
if (isspace(ch)) {
if (!in_space) {
out[j++] = ' ';
in_space = 1;
}
} else {
out[j++] = (char)ch;
in_space = 0;
}
}
if (j > 0 && out[j - 1] == ' ') {
j--;
}
out[j] = '\0';
free(no_tags);
return out;
}
void transcribe_set_filter_bracketed_tags(int enabled) {
g_filter_bracketed_tags = enabled ? 1 : 0;
}
#if WITH_WHISPER
#include <whisper.h>
@@ -17,6 +83,7 @@ static transcribe_params_t g_params;
int transcribe_init(const transcribe_params_t *params) {
if (!params) return -1;
g_params = *params;
transcribe_set_filter_bracketed_tags(g_params.filter_bracketed_tags);
struct whisper_context_params cparams = whisper_context_default_params();
cparams.use_gpu = g_params.use_gpu ? true : false;
@@ -63,7 +130,17 @@ char *transcribe_buffer(const float *samples, size_t count) {
if (i + 1 < nseg) strcat(out, " ");
}
return out;
if (!g_filter_bracketed_tags) {
return out;
}
char *filtered = filter_bracketed_tags_copy(out);
if (!filtered) {
return out;
}
free(out);
return filtered;
}
void transcribe_cleanup(void) {
+2
View File
@@ -8,9 +8,11 @@ typedef struct {
char language[32];
int use_gpu;
int n_threads;
int filter_bracketed_tags;
} transcribe_params_t;
int transcribe_init(const transcribe_params_t *params);
void transcribe_set_filter_bracketed_tags(int enabled);
char *transcribe_buffer(const float *samples, size_t count);
void transcribe_cleanup(void);
+109 -13
View File
@@ -11,6 +11,16 @@
static Display *g_dpy = NULL;
static int g_delay_us = 8000;
static int g_key_hold_us = 4000;
static Window g_target_window = None;
static void sleep_us(int us) {
if (us <= 0) return;
struct timespec ts;
ts.tv_sec = us / 1000000;
ts.tv_nsec = (long)(us % 1000000) * 1000L;
nanosleep(&ts, NULL);
}
static int type_keysym(KeySym ks, int need_shift) {
KeyCode kc = XKeysymToKeycode(g_dpy, ks);
@@ -19,21 +29,21 @@ static int type_keysym(KeySym ks, int need_shift) {
KeyCode shift_kc = XKeysymToKeycode(g_dpy, XK_Shift_L);
if (need_shift && shift_kc) {
XTestFakeKeyEvent(g_dpy, shift_kc, True, CurrentTime);
if (!XTestFakeKeyEvent(g_dpy, shift_kc, True, CurrentTime)) return -1;
}
XTestFakeKeyEvent(g_dpy, kc, True, CurrentTime);
XTestFakeKeyEvent(g_dpy, kc, False, CurrentTime);
if (!XTestFakeKeyEvent(g_dpy, kc, True, CurrentTime)) return -1;
XSync(g_dpy, False);
sleep_us(g_key_hold_us);
if (!XTestFakeKeyEvent(g_dpy, kc, False, CurrentTime)) return -1;
if (need_shift && shift_kc) {
XTestFakeKeyEvent(g_dpy, shift_kc, False, CurrentTime);
if (!XTestFakeKeyEvent(g_dpy, shift_kc, False, CurrentTime)) return -1;
}
XFlush(g_dpy);
struct timespec ts;
ts.tv_sec = g_delay_us / 1000000;
ts.tv_nsec = (long)(g_delay_us % 1000000) * 1000L;
nanosleep(&ts, NULL);
XSync(g_dpy, False);
sleep_us(g_delay_us);
return 0;
}
@@ -62,32 +72,118 @@ static int type_char(char c) {
case '-': return type_keysym(XK_minus, 0);
case '_': return type_keysym(XK_minus, 1);
case '/': return type_keysym(XK_slash, 0);
case '\\': return type_keysym(XK_backslash, 0);
case '|': return type_keysym(XK_backslash, 1);
case '=': return type_keysym(XK_equal, 0);
case '+': return type_keysym(XK_equal, 1);
case '[': return type_keysym(XK_bracketleft, 0);
case '{': return type_keysym(XK_bracketleft, 1);
case ']': return type_keysym(XK_bracketright, 0);
case '}': return type_keysym(XK_bracketright, 1);
case '`': return type_keysym(XK_grave, 0);
case '~': return type_keysym(XK_grave, 1);
case '@': return type_keysym(XK_2, 1);
case '#': return type_keysym(XK_3, 1);
case '$': return type_keysym(XK_4, 1);
case '%': return type_keysym(XK_5, 1);
case '^': return type_keysym(XK_6, 1);
case '&': return type_keysym(XK_7, 1);
case '*': return type_keysym(XK_8, 1);
case '(': return type_keysym(XK_9, 1);
case ')': return type_keysym(XK_0, 1);
case '<': return type_keysym(XK_comma, 1);
case '>': return type_keysym(XK_period, 1);
default: return 0;
}
}
int typer_init(int type_delay_us) {
int ev_base = 0;
int err_base = 0;
int major = 0;
int minor = 0;
g_dpy = XOpenDisplay(NULL);
if (!g_dpy) return -1;
if (!g_dpy) {
fprintf(stderr, "typer: XOpenDisplay failed\n");
return -1;
}
if (!XTestQueryExtension(g_dpy, &ev_base, &err_base, &major, &minor)) {
fprintf(stderr, "typer: XTEST extension unavailable\n");
XCloseDisplay(g_dpy);
g_dpy = NULL;
return -1;
}
g_delay_us = type_delay_us > 0 ? type_delay_us : 8000;
g_key_hold_us = g_delay_us / 2;
if (g_key_hold_us < 4000) g_key_hold_us = 4000;
if (g_key_hold_us > 20000) g_key_hold_us = 20000;
g_target_window = None;
return 0;
}
int typer_capture_focus(void) {
if (!g_dpy) return -1;
Window focused = None;
int revert = 0;
XGetInputFocus(g_dpy, &focused, &revert);
if (focused == None || focused == PointerRoot) {
return -1;
}
g_target_window = focused;
return 0;
}
int typer_type_text(const char *text) {
if (!g_dpy || !text) return -1;
for (size_t i = 0; i < strlen(text); ++i) {
Window previous_focus = None;
int previous_revert = 0;
XGetInputFocus(g_dpy, &previous_focus, &previous_revert);
Window target = g_target_window;
if (target == None || target == PointerRoot) {
target = previous_focus;
}
if (target == None || target == PointerRoot) {
fprintf(stderr, "typer: no valid target window for autotype\n");
return -1;
}
if (previous_focus != target) {
XSetInputFocus(g_dpy, target, RevertToParent, CurrentTime);
XSync(g_dpy, False);
sleep_us(20000);
}
int failures = 0;
const size_t n = strlen(text);
for (size_t i = 0; i < n; ++i) {
if (type_char(text[i]) != 0) {
fprintf(stderr, "typer: failed at char '%c'\n", text[i]);
fprintf(stderr, "typer: failed at char 0x%02x ('%c')\n",
(unsigned char)text[i], isprint((unsigned char)text[i]) ? text[i] : '?');
failures++;
}
}
return 0;
if (previous_focus != None && previous_focus != PointerRoot && previous_focus != target) {
XSetInputFocus(g_dpy, previous_focus, RevertToParent, CurrentTime);
XSync(g_dpy, False);
}
if (failures > 0) {
fprintf(stderr, "typer: %d/%zu characters failed to inject\n", failures, n);
}
return failures == 0 ? 0 : -1;
}
void typer_cleanup(void) {
g_target_window = None;
if (g_dpy) {
XCloseDisplay(g_dpy);
g_dpy = NULL;
+1
View File
@@ -2,6 +2,7 @@
#define VOICE_TYPER_H
int typer_init(int type_delay_us);
int typer_capture_focus(void);
int typer_type_text(const char *text);
void typer_cleanup(void);
+2 -2
View File
@@ -1,9 +1,9 @@
#ifndef VOICE_VERSION_H
#define VOICE_VERSION_H
#define VOICE_LINUX_VERSION "v0.0.2"
#define VOICE_LINUX_VERSION "v0.0.4"
#define VOICE_LINUX_VERSION_MAJOR 0
#define VOICE_LINUX_VERSION_MINOR 0
#define VOICE_LINUX_VERSION_PATCH 2
#define VOICE_LINUX_VERSION_PATCH 4
#endif
BIN
View File
Binary file not shown.