voice_linux/plans/architecture.md

Voice Linux — Speech-to-Text Dictation System

Overview

A local, GPU-accelerated speech-to-text dictation tool for Qubes OS. Written in C99, using whisper.cpp with CUDA for transcription, PulseAudio for audio capture, and Xlib/XTest for global hotkey and text injection.

Press a hotkey → speak → text appears in the focused window.

System Environment

Property	Value
OS	Qubes OS (AppVM)
Session	X11 / XFCE
Audio	PipeWire 1.4.2 with PulseAudio compat
Audio source	qubes-source (virtual PipeWire device)
Python	3.13.5 (not used for main app — C99 instead)
Display GPU	AMD (dom0)
Compute GPU	NVIDIA GeForce GTX (to be passed through)

Architecture

┌─────────────────────────────────────────────────────┐
│                    voice_linux                        │
│                                                       │
│  ┌──────────┐   ┌──────────┐   ┌──────────────────┐ │
│  │  hotkey   │──▶│  audio   │──▶│   transcribe     │ │
│  │  (Xlib    │   │ (Pulse   │   │  (whisper.cpp    │ │
│  │  XGrabKey)│   │  Simple  │   │   CUDA-accel)    │ │
│  └──────────┘   │  API)    │   └────────┬─────────┘ │
│                  └──────────┘            │            │
│                                          ▼            │
│                               ┌──────────────────┐   │
│                               │     typer         │   │
│                               │  (XTest fake      │   │
│                               │   key events)     │   │
│                               └──────────────────┘   │
└─────────────────────────────────────────────────────┘

Components

Module	File	C API Used	Purpose
Hotkey	src/hotkey.c	Xlib XGrabKey	Global push-to-talk hotkey on X11
Audio	src/audio.c	PulseAudio Simple API	Record from qubes-source into PCM buffer
Transcribe	src/transcribe.c	whisper.cpp C API	Feed PCM to Whisper, get text back
Typer	src/typer.c	XTest XTestFakeKeyEvent	Type transcribed text into focused window
Config	src/config.c	stdio	Load YAML/INI config for hotkey, model, device
Main	src/main.c	—	Wire everything together

Data Flow

1. main() initializes all modules, loads config
2. hotkey_listen() blocks on X11 event loop
3. On hotkey press → audio_start_recording() begins capturing from PulseAudio
4. On hotkey release (or second press) → audio_stop_recording() returns PCM buffer
5. transcribe() feeds PCM buffer to whisper.cpp → returns char *text
6. typer_type_text(text) sends fake key events via XTest to focused window
7. Loop back to step 2

Project Structure

voice_linux/
├── Makefile                    # Build system
├── README.md                   # Setup and usage
├── config.ini                  # User config (hotkey, model, audio device)
├── plans/
│   └── architecture.md         # This document
├── src/
│   ├── main.c                  # Entry point and orchestrator
│   ├── hotkey.h / hotkey.c     # Global X11 hotkey via XGrabKey
│   ├── audio.h / audio.c       # PulseAudio Simple API recording
│   ├── transcribe.h / transcribe.c  # whisper.cpp integration
│   ├── typer.h / typer.c       # XTest text injection
│   └── config.h / config.c     # Config file parser
├── models/                     # Downloaded whisper models (gitignored)
│   └── ggml-base.en.bin
└── vendor/
    └── whisper.cpp/            # Git submodule or local clone

Phase 0: GPU Passthrough (Qubes)

Prerequisites

• AMD GPU handles dom0 display (confirmed)
• NVIDIA GTX is idle and available for passthrough (confirmed)
• IOMMU/VT-d enabled in BIOS (likely, since Qubes requires it)

Detected Hardware

17:00.0 VGA compatible controller: NVIDIA Corporation GA107 [GeForce RTX 3050 6GB] (rev a1)
17:00.1 Audio device: NVIDIA Corporation GA107 High Definition Audio Controller (rev a1)
65:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
65:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

Target GPU: GTX 1080 Ti (11GB VRAM, CUDA compute 6.1) at BDF 65:00.0 / 65:00.1

AppVM Details

• Name: ai
• Type: AppVM (template: debian-13-xfce)
• Network: sys-vpn-mullvad-app

Important

: Since ai is a template-based AppVM, NVIDIA drivers installed inside it will be lost on reboot (only /home persists). Two options:

• Option A: Install drivers in the debian-13-xfce template (affects all AppVMs using it)
• Option B: Convert ai to a StandaloneVM: qvm-clone --class StandaloneVM ai ai-standalone
• Option C: Use a bind-dirs mechanism to persist /usr/lib/nvidia and related paths

Steps (run in dom0 terminal)

0.1 — Hide the GTX 1080 Ti from dom0

Edit /etc/default/grub in dom0:

sudo nano /etc/default/grub

Find the first GRUB_CMDLINE_LINUX line:

GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-ebcd163d-dcf3-47bf-8da1-f7c13d82304b rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rhgb quiet"

Add rd.qubes.hide_pci=65:00.0,65:00.1 to the end (inside the quotes):

GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-ebcd163d-dcf3-47bf-8da1-f7c13d82304b rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rhgb quiet rd.qubes.hide_pci=65:00.0,65:00.1"

0.2 — Regenerate GRUB and reboot

# Try EFI path first (most common on modern Qubes):
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

# If that path doesn't exist, try:
# sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Reboot dom0:
sudo reboot

0.3 — Verify GPU is assignable (after reboot)

xl pci-assignable-list
# Should show:
# 0000:65:00.0
# 0000:65:00.1

0.4 — Attach GPU to the 'ai' AppVM

qvm-pci attach ai dom0:65_00.0 --persistent -o permissive=true
qvm-pci attach ai dom0:65_00.1 --persistent -o permissive=true

0.5 — Install NVIDIA drivers inside the AppVM

Start the ai VM, then inside it:

# Inside the ai AppVM:
sudo apt update
sudo apt install -y build-essential linux-headers-$(uname -r)

# For GTX 1080 Ti (Pascal/GP102), use the 550.x driver branch:
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.127.05/NVIDIA-Linux-x86_64-550.127.05.run
chmod +x NVIDIA-Linux-x86_64-550.127.05.run
sudo ./NVIDIA-Linux-x86_64-550.127.05.run --no-opengl-files --dkms

--no-opengl-files is critical in Qubes — we don't want to replace the VM's display GL, just get CUDA compute.

0.6 — Install CUDA Toolkit

# Inside the ai AppVM:
wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install -y cuda-toolkit-12-6
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

0.7 — Verify

nvidia-smi
nvcc --version

Expected output: GTX 1080 Ti with 11GB VRAM, CUDA 12.6.

Phase 1: Build and Test Core Application

1.1 — Install build dependencies

sudo apt install -y \
    build-essential \
    cmake \
    git \
    libpulse-dev \
    libx11-dev \
    libxtst-dev \
    xdotool

1.2 — Clone and build whisper.cpp with CUDA

cd vendor/
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
cmake -B build -DWHISPER_CUDA=ON
cmake --build build --config Release -j$(nproc)

1.3 — Download whisper model

cd models/
# base.en — fast, good for English dictation (~150MB)
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin

# Or medium.en for better accuracy (~1.5GB, still fast with GPU):
# wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-medium.en.bin

1.4 — Implement C99 modules

Each module is a small, focused C file with a clean header:

src/audio.h — Audio capture

• int audio_init(const char *device_name) — connect to PulseAudio
• int audio_start_recording(void) — begin capturing PCM
• int audio_stop_recording(float **samples, size_t *n_samples) — stop and return buffer
• void audio_cleanup(void) — disconnect

src/transcribe.h — Whisper integration

• int transcribe_init(const char *model_path) — load model (CUDA auto-detected)
• char *transcribe_buffer(const float *samples, size_t n_samples) — run inference
• void transcribe_cleanup(void) — free model

src/typer.h — Text injection

• int typer_init(void) — open X11 display
• int typer_type_text(const char *text) — send fake key events
• void typer_cleanup(void) — close display

src/hotkey.h — Global hotkey

• int hotkey_init(unsigned int keycode, unsigned int modifiers) — grab key
• int hotkey_wait_press(void) — block until hotkey pressed
• int hotkey_wait_release(void) — block until hotkey released
• void hotkey_cleanup(void) — ungrab and close

src/config.h — Configuration

• typedef struct { ... } config_t
• int config_load(const char *path, config_t *cfg) — parse INI file
• Default values for hotkey (Ctrl+Alt+V), model path, audio device

1.5 — Build with Makefile

CC = gcc
CFLAGS = -std=c99 -Wall -Wextra -O2
LDFLAGS = -lpulse-simple -lpulse -lX11 -lXtst -lwhisper -L./vendor/whisper.cpp/build/src
INCLUDES = -I./vendor/whisper.cpp/include

voice_linux: src/main.c src/audio.c src/transcribe.c src/typer.c src/hotkey.c src/config.c
	$(CC) $(CFLAGS) $(INCLUDES) -o $@ $^ $(LDFLAGS)

1.6 — Test end-to-end

./voice_linux --model models/ggml-base.en.bin --device qubes-source
# Press Ctrl+Alt+V, speak, press again, text appears in focused window

Phase 2: Polish

• Audio feedback: Play a short beep via PulseAudio when recording starts/stops
• Config file: config.ini with hotkey, model path, audio device, typing delay
• Systemd user service: ~/.config/systemd/user/voice-linux.service for auto-start
• Logging: Optional debug log to ~/.local/share/voice_linux/voice.log

Phase 3: Hardware

• USB microphone: Attach via Qubes device manager (qvm-usb attach) from sys-usb
• Physical button: Read HID events via /dev/input/eventN using evdev in C
• Model tuning: Try medium.en or large-v3 models with GPU — should still be fast

Key Risks and Mitigations

Risk	Mitigation
GPU passthrough fails (IOMMU grouping)	Fall back to CPU-only whisper.cpp (still works, just slower)
Older GTX lacks CUDA compute capability	whisper.cpp supports compute 5.0+; most GTX 900+ series work
PulseAudio qubes-source has no audio	Need mic attached + Qubes audio policy allowing input
XGrabKey conflicts with desktop shortcuts	Use an uncommon hotkey combo; make it configurable
whisper.cpp API changes	Pin to a specific release tag

Model Size vs Speed Reference (GPU)

Model	Size	VRAM	Speed (10s audio)	Accuracy
tiny.en	75MB	~1GB	<0.5s	Fair
base.en	150MB	~1GB	<1s	Good
small.en	500MB	~2GB	~1-2s	Very good
medium.en	1.5GB	~5GB	~2-3s	Excellent
large-v3	3GB	~10GB	~4-6s	Best

For dictation, base.en is the sweet spot to start. Upgrade to medium.en if accuracy isn't sufficient.