Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,28 @@

**English** · [简体中文](README.zh-CN.md) · [Documentation](https://github.com/Saco93/voice-input/wiki) · [中文文档](https://github.com/Saco93/voice-input/wiki/Home.zh-CN)

Voice Input is a resident dictation service with realtime transcription, a native animated HUD, full-audio final recognition, conservative LLM cleanup, and optional terminology context from the Pi or Codex session focused when dictation ends.
Voice Input is a resident dictation service with realtime transcription, a native animated HUD, full-audio final recognition, conservative LLM cleanup, and optional terminology captured from the Pi or Codex session focused when dictation starts.

## How it works

```mermaid
flowchart LR
Mic[Microphone<br/>16 kHz PCM] --> RT[Qwen Realtime ASR]
RT --> Final[Full-audio Final ASR]
Mic[Microphone<br/>16 kHz PCM] --> A3[Audio3 Streaming ASR]
Mic --> QR[Qwen Realtime ASR]
A3 --> Final[Full-audio Final ASR]
QR --> Final
Final --> LLM[LLM refinement<br/>15 s default budget]
LLM --> Out[Wayland / XWayland output]
RT -. live transcript .-> HUD[Quickshell HUD]
Agent[Pi / Codex focused at stop] -. terminology only .-> LLM
A3 -. live transcript .-> HUD[Quickshell HUD]
QR -. live transcript .-> HUD
Agent[Pi / Codex focused at start] -. terminology snapshot .-> A3
Agent -. same snapshot .-> LLM
```

1. A persistent PipeWire capture service keeps a short pre-roll buffer, so speech immediately after the hotkey is not lost. Sessions stop and finalize automatically at the configured duration limit (five minutes by default).
2. Qwen Realtime streams partial text to the HUD while Server VAD controls waveform visibility. Realtime delivery uses a bounded, nonblocking queue and fair bidirectional WebSocket processing. The worker may reconstruct the realtime session once after a pre-finish transport failure, an eight-second active-speech transcript stall, or sustained pitch-correlated local speech that receives no server event for eight seconds after text has appeared. Reconstruction replays every buffered raw PCM packet from the beginning while recording continues.
3. On toggle-off, the complete recording is optionally recognized again by the final ASR model. If the controlled reconstruction fails, its single retry is exhausted, or realtime delivery falls behind, incomplete remote text is rejected and the complete audio is recovered through the enabled final pass or local fallback.
4. The transcript is lightly cleaned by an OpenAI-compatible LLM. The window focused at toggle-off selects the refinement style: Pi and Codex receive compact Markdown that turns explicit sequences into ordered lists, unordered enumerations into bullet lists, and distinct parts into separate paragraphs; installed native messaging clients (WeChat, Feishu/Lark, Signal, and Telegram Desktop) receive conversational punctuation, preserve meaningful spoken particles, and omit a final full stop while retaining question marks, exclamation marks, and intentional ellipses; other destinations retain the lightly formal default. Refinement uses the configured timeout (15 seconds by default, capped at 30 seconds); when the budget is at least 10 seconds, contextual requests reserve five seconds for a transcript-only cleanup retry and ultimately fail open to Final ASR.
4. If Pi or Codex is focused when dictation starts and session terminology is enabled, Voice Input locally redacts and segments the latest completed assistant message once, orders unique terms from least to most frequent in that source, and retains one immutable snapshot for the operation. Audio3 Streaming receives a newline-separated view of at most 400 characters in `run-task`; a reconnect replacement receives the identical view. Refine receives up to 96 terms and 1,500 term characters from the same snapshot. High-confidence technical variants that differ only in ASCII case or separators are normalized to the snapshot spelling before and after Refine. In Adaptive mode, a healthy completed Streaming result that actually sent Session Context is not replaced by Native recognition solely because the recording exceeded 30 seconds; all degradation recovery and the explicit Always mode remain unchanged. The transcript is lightly cleaned by an OpenAI-compatible LLM. The window focused at toggle-off still selects the refinement style: Pi and Codex receive compact Markdown that turns explicit sequences into ordered lists, unordered enumerations into bullet lists, and distinct parts into separate paragraphs; installed native messaging clients (WeChat, Feishu/Lark, Signal, and Telegram Desktop) receive conversational punctuation, preserve meaningful spoken particles, and omit a final full stop while retaining question marks, exclamation marks, and intentional ellipses; other destinations retain the lightly formal default. Refinement uses the configured timeout (15 seconds by default, capped at 30 seconds); when the budget is at least 10 seconds, contextual requests reserve five seconds for a transcript-only cleanup retry and ultimately fail open to Final ASR.
5. All text is delivered through clipboard paste with automatic restoration. Native Wayland delivery marks both the transient transcript and restored content as sensitive so compatible clipboard managers do not retain or reorder them. Wayland paste shortcuts use Hyprland's `sendshortcut` dispatcher, while XWayland uses `xdotool`; Voice Input never creates a `wtype` character keymap.

No-speech sessions return to idle once realtime or final ASR confirms that no transcript exists. Audio capture, ASR, HUD rendering, persistence, and output are isolated so a slow visual or clipboard client cannot block recognition.
Expand Down Expand Up @@ -95,7 +99,7 @@ Migration is presence-aware. A configuration without `endpoint_mode` migrates to

Alibaba API keys are region-scoped. Changing the region may require replacing the encrypted Alibaba credential. Voice Input never probes another region and never migrates a key automatically. Singapore availability does not establish feature parity: each model, control combination, and language/vocabulary scenario still requires authorized live validation.

The streaming model supplies realtime text. On one recoverable transport interruption before `finish-task`, Voice Input creates a new Audio3 task, discards the old task's transcript, and replays retained PCM from the beginning at 4× realtime while recording continues. Retention is prefix-complete and limited by the configured recording duration, 300 seconds, and 10 MiB of PCM; exceeding the limit disables reconnect without evicting an audio prefix. A second interruption or a post-finish interruption uses the existing Native/local complete-audio recovery. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue.
The streaming model supplies realtime text. When explicitly enabled and dictation starts in a validated Pi or Codex session, its `run-task` also receives at most 400 characters of locally redacted, low-frequency-first Session Context terminology; no `continue-task` event is used. On one recoverable transport interruption before `finish-task`, Voice Input creates a new Audio3 task, discards the old task's transcript, and replays retained PCM from the beginning at 4× realtime while recording continues. Retention is prefix-complete and limited by the configured recording duration, 300 seconds, and 10 MiB of PCM; exceeding the limit disables reconnect without evicting an audio prefix. A second interruption or a post-finish interruption uses the existing Native/local complete-audio recovery. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue.

**Recognition preset** defaults to **Standard**, which preserves the existing `800` ms maximum sentence silence with semantic punctuation and multi-threshold mode disabled and no speech/noise threshold. **Low-latency dictation** uses `400` ms with multi-threshold mode enabled; **Long-form** uses `1300` ms with semantic punctuation enabled. Both mappings were accepted in an authorized, one-speaker evaluation and retained both clauses across a matrix with 250–2200 ms of inserted digital silence; acoustic speech boundaries remained dependent on local RMS trimming. The bounded sample does not establish a general accuracy or latency recommendation, so Standard remains the default. See [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md). **Custom** exposes all raw controls; semantic punctuation and multi-threshold mode cannot be enabled together. Its optional speech/noise threshold must be finite and between `-1` and `1`; omission preserves provider behavior because Alibaba publishes no default. Settings displays every value that a custom request can send.

Expand Down Expand Up @@ -143,7 +147,7 @@ The Wiki also covers agent context, desktop integration, privacy, and developmen

## Privacy

Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider. When agent context is explicitly enabled, Voice Input redacts and caps the latest completed Pi or Codex assistant message locally, segments it with Jieba, deduplicates the result, and sends only a bounded terminology list. The source message, window titles, process IDs, and raw desktop metadata are not included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection.
Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider. When session terminology is explicitly enabled, Voice Input captures the Pi or Codex session focused at dictation start, redacts and caps its latest completed assistant message locally, segments it with Jieba, deduplicates terms, and orders them from least to most frequent. Audio3 Streaming and Refine receive separate bounded views of that same immutable snapshot; the source message, frequencies, window titles, process IDs, and raw desktop metadata are not included in either request. The public sample disables remote refinement and session terminology. Voice Input performs no telemetry or analytics collection.

## Project status

Expand Down
Loading