voice_linux/plans/always_on_plan.md

Always-On Dictation Plan (Qubes + Linux/X11)

Goal

Build an always-running speech pipeline that is safe, low-friction, and works with your mic’s physical mute button.

Design Principles

1. Mic hardware mute is king (privacy first).
2. Continuous capture, segmented transcription (VAD/endpointer).
3. Conservative text injection defaults (avoid accidental typing).
4. Dedicated StandaloneVM deployment for long-term reliability.

---

Phase A — Stabilize Current UX (1–2 sessions)

A1. Finalize GUI feedback states

• Keep current button states:
  • idle
  • recording
  • transcribing
• Add explicit status line values:
  • Mic active, waiting for speech
  • Speech detected
  • Segment ended, transcribing
  • Mic muted or near-silent

A2. Add explicit mode selector in GUI

• Modes:
  • Push-to-talk (existing)
  • Always-on preview (new)
  • Always-on auto-type (advanced)

A3. Add per-mode safety defaults

• Default to Always-on preview (no typing).
• Require explicit checkbox to enable auto-typing.

---

Phase B — Always-On Audio Engine (core)

B1. Ring buffer capture loop

Refactor audio path to run continuously:

• Thread captures audio at fixed chunk size (e.g., 20 ms frames).
• Store in ring buffer (N seconds history, e.g., 30s).
• Expose non-blocking read windows for VAD and segment extraction.

B2. Voice activity detection (VAD)

Implement lightweight VAD first:

• RMS/energy threshold + zero crossing heuristic.
• Track states:
  • silence
  • speech_started
  • speech_active
  • speech_ended
• Tunables in config:
  • vad_energy_threshold
  • vad_start_ms
  • vad_end_silence_ms

(Optionally later: replace with Silero VAD or WebRTC VAD wrapper.)

B3. Segment endpointer

When speech ends:

• Cut a segment with small pre-roll/post-roll (e.g., 250 ms / 200 ms).
• Dispatch to transcription worker queue.
• Continue capturing while worker runs.

B4. Transcription worker thread

• Single worker initially (simple, deterministic).
• Queue segment jobs FIFO.
• Return:
  • text
  • confidence surrogate (optional)
  • latency metrics

---

Phase C — Hardware Mute Integration

C1. Software mute inference (must-have)

Even without HID events:

• Detect sustained near-zero energy (e.g., >800ms).
• Mark state as muted_or_silent.
• In GUI show muted icon/state.

C2. HID/evdev mute button (nice-to-have)

If the mic exposes events:

• Open relevant /dev/input/event*.
• Parse key/switch event for mute toggle.
• Map to explicit app state hardware_muted=true/false.

Fallback remains C1 inference so behavior is robust across devices.

---

Phase D — Output Policy for Continuous Mode

D1. Preview-first pipeline

• Continuous transcript appears in GUI text area.
• Nothing typed automatically by default.

D2. Commit model

Two safe options:

1. Manual commit button: insert selected transcript into focused app.
2. Auto-commit finals: type only finalized segments (not partials).

D3. Guard rails

• Minimum confidence/length filter before typing.
• Ignore common noise artifacts ([BLANK_AUDIO], empty, punctuation-only).
• Cooldown between auto-typed segments (e.g., 300 ms).

---

Phase E — Observability and Tuning

E1. Metrics

Track and expose:

• capture RMS
• VAD state transitions
• segment durations
• transcription latency
• typed character count

E2. Debug panel in GUI

Small collapsible panel with:

• live level meter
• current VAD state
• current device name
• queue length / worker busy

E3. Config knobs

Add to config:

• mode=push_to_talk|always_on_preview|always_on_autotype
• vad_energy_threshold
• vad_end_silence_ms
• autotype_enabled
• autotype_min_chars
• autotype_cooldown_ms

---

Phase F — Qubes Deployment Model

F1. Dedicated StandaloneVM

Deploy final app in its own StandaloneVM:

• stable dependencies
• controlled attack surface
• predictable startup behavior

F2. Device routing

• Attach USB mic to this VM via sys-usb.
• Keep always-on transcription isolated from high-trust VMs.

F3. Startup behavior

• systemd --user service for auto-launch on login.
• GUI starts minimized/normal per config.

---

Implementation Order (recommended)

1. Always-on preview mode (B + D1)
2. VAD + endpoint tuning UI (E)
3. Manual commit + guarded auto-commit (D2/D3)
4. Hardware mute event support if available (C2)
5. StandaloneVM packaging + autostart (F)

---

Risks and Mitigations

• False triggers / noise typing → preview-first default, confidence/length filters.
• Long transcription latency → worker queue + clear transcribing state.
• Mic mute not exposed via HID → software mute inference fallback.
• Qubes audio routing quirks → device selector + live level meter + status diagnostics.

---

Definition of Done (MVP Always-On)

• App runs continuously with no manual start/stop needed.
• Speech automatically segmented and transcribed.
• GUI clearly indicates idle/speech/transcribing/muted.
• Preview mode stable for long sessions.
• Optional typed output only for finalized segments with safety guards.