voice_linux/plans/voice_commands_plan.md

Voice Commands Tab — Implementation Plan

Overview

Add a third GUI tab ("Commands") that lets users define trigger phrases from whisper output and map them to either text replacements or keyboard combinations. This enables meta-speech like saying "period" to insert ., or "over" to send Return.

Architecture

Data Model

typedef enum {
    VOICE_CMD_TEXT,     // replace trigger with literal text
    VOICE_CMD_KEYCOMBO  // send a key combination via XTest
} voice_cmd_type_t;

typedef struct {
    char trigger[128];          // spoken phrase, case-insensitive
    voice_cmd_type_t type;
    char text_value[256];       // for TEXT type: replacement string
    unsigned int keyval;        // for KEYCOMBO type: GDK keyval
    unsigned int modifiers;     // for KEYCOMBO type: modifier mask
    char display_key[128];      // human-readable key description
} voice_cmd_t;

typedef struct {
    voice_cmd_t *cmds;
    int count;
    int capacity;
    int enabled;                // global on/off toggle
} voice_cmd_list_t;

Processing Pipeline

transcribe_buffer() → raw text
       ↓
voice_cmd_apply(text, list) → action sequence
       ↓
For each action:
  - TEXT_CHUNK: accumulate into output string
  - KEY_EVENT: call typer_send_keycombo()
       ↓
Final text → append_transcript() + typer_type_text()

The apply function returns a sequence of interleaved actions because a single transcript may contain multiple triggers mixed with regular text. Example: "hello over goodbye over" → text:"hello " → key:Return → text:"goodbye " → key:Return.

Matching Strategy

• Case-insensitive comparison
• Whole-word boundary matching to avoid false positives
• Longest-match-first: sort triggers by length descending before scanning
• Left-to-right scan through transcript text
• Matched trigger phrases are consumed and not included in output text

New Files

src/voice_cmd.h

• voice_cmd_t and voice_cmd_list_t structs
• void voice_cmd_init(voice_cmd_list_t *list)
• void voice_cmd_free(voice_cmd_list_t *list)
• int voice_cmd_add(voice_cmd_list_t *list, const voice_cmd_t *cmd)
• int voice_cmd_remove(voice_cmd_list_t *list, int index)
• int voice_cmd_load(const char *path, voice_cmd_list_t *list)
• int voice_cmd_save(const char *path, const voice_cmd_list_t *list)

src/voice_cmd.c

• Load/save from voice_commands.ini
• voice_cmd_apply() — the core text processing function

voice_commands.ini

Persisted command rules, separate from main config.ini.

Format:

# trigger|type|value
# type: text or key
# for text: value is the replacement string (supports \n \t escapes)
# for key: value is keyval:modifiers (decimal GDK values)
period|text|.
question mark|text|?
comma|text|,
exclamation mark|text|!
new line|text|\n
new paragraph|text|\n\n
tab|text|\t
over|key|65293:0
end of command|key|65293:5

Modified Files

src/typer.h / src/typer.c

• Add: int typer_send_keycombo(unsigned int keyval, unsigned int modifiers)
• Uses XTest to send arbitrary key combinations with modifier state
• Reuses existing X11 display connection from typer_init()

src/gui_main.c

• Add third notebook tab in on_app_activate()
• Tab contains:
  • GtkTreeView showing existing commands (trigger, type, action display)
  • Delete button per row or for selected row
  • Add-new-command form:
    • GtkEntry for trigger phrase
    • GtkRadioButton pair: Text / Key Combo
    • GtkEntry for text replacement value
    • GtkButton "Capture Key" + GtkLabel showing captured key
    • GtkButton "Add Command"
  • GtkCheckButton "Enable voice commands" toggle
• Key capture workflow:
  1. User clicks Capture
  2. Button label changes to "Press key combo now..."
  3. Window key-press-event handler captures next keypress
  4. Records keyval + modifier state
  5. Formats display string like "Ctrl+Shift+Return"
  6. Disconnects capture handler, restores button label

src/gui_main.c — process_segment()

• After transcribe_buffer() returns text, call voice_cmd_apply()
• Process returned action sequence: text chunks go to transcript/autotype, key events go to typer_send_keycombo()

Makefile

• Add voice_cmd.o to build targets

build.sh

• Add src/voice_cmd.c to compilation

GUI Tab Layout

┌─ Voice Commands ──────────────────────────────────────────┐
│                                                            │
│  ☑ Enable voice commands                                   │
│                                                            │
│  ┌──────────────┬──────┬─────────────────┬───────────┐    │
│  │ Trigger      │ Type │ Action          │           │    │
│  ├──────────────┼──────┼─────────────────┼───────────┤    │
│  │ period       │ text │ .               │ [Delete]  │    │
│  │ question mark│ text │ ?               │ [Delete]  │    │
│  │ new line     │ text │ \n              │ [Delete]  │    │
│  │ over         │ key  │ Return          │ [Delete]  │    │
│  │ end of cmd   │ key  │ Ctrl+Shift+Ret  │ [Delete]  │    │
│  └──────────────┴──────┴─────────────────┴───────────┘    │
│                                                            │
│  ── Add New Command ──                                     │
│  Trigger: [____________]                                   │
│  ○ Text replacement  ● Key combination                     │
│  Text:   [____________]                                    │
│  Key:    [Press keys...] [Capture]                         │
│  [Add Command]                                             │
│                                                            │
│  Help: [hover text area]                                   │
└────────────────────────────────────────────────────────────┘

Default Commands (shipped with first install)

Trigger	Type	Action
period	text	.
question mark	text	?
exclamation mark	text	!
comma	text	,
colon	text	:
semicolon	text	;
new line	text	\n
new paragraph	text	\n\n
tab	text	\t
open paren	text	(
close paren	text	)
open bracket	text	[
close bracket	text	]
open brace	text	{
close brace	text	}

Edge Cases

1. Substring false positives: "I need a period of time" — whole-word boundary matching prevents "period" from matching inside "periodical" but this phrase would still trigger. Could add a "require end-of-phrase position" option per command, but start simple with whole-word matching.

2. Case sensitivity: Whisper may output "Period" or "period" depending on sentence position. All matching is case-insensitive.

3. Multi-word triggers: "end of command" requires multi-word scanning. Sort triggers longest-first so "end of command" matches before "end".

4. Escape sequences in text values: Support \n, \t, \ in the text replacement field. Parse on load, display escaped in UI.

5. Key capture conflicts: While capturing, consume the event so it doesn't trigger other handlers. Disconnect capture handler after first keypress.

6. Empty transcript after replacements: If the entire transcript was voice commands with no remaining text, skip autotype and transcript append.

Implementation Order

1. Create src/voice_cmd.h and src/voice_cmd.c with data structures and load/save
2. Add typer_send_keycombo() to src/typer.h / src/typer.c
3. Implement voice_cmd_apply() core matching logic
4. Build the Commands tab UI in src/gui_main.c
5. Wire key capture handler
6. Hook voice_cmd_apply() into process_segment()
7. Create default voice_commands.ini
8. Update Makefile and build.sh
9. Build and test