Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,16 +81,24 @@ systemctl --user status voice-input.service voice-input-hud.service

## Safe support diagnostics

Use `voice-input diagnostics [--format text|json]` as the canonical output for support reports. Schema 3 contains bounded stage statuses and failure categories; aggregate streaming delivery/result timings and counts (ready, partial, nonempty partial, segment-final, audio packets and sent duration, queue delays, finish, task completion/failure, and finalization); the Audio3 native-pass mode/decision/reason; and a safe configuration summary for one session only: the active or most recently completed session. The safe summary reports whether Audio3 language hints and heartbeat are enabled, the configured maximum sentence silence, the semantic-punctuation state, and the dynamic-vocabulary entry count; it never includes vocabulary terms. A failed provider task may include an optional strictly bounded provider error identifier. That session summary remains available after completion and is reset when the next recording starts. It contains no audio, credentials, endpoints, model names, provider messages, window/application data, prompt context, tooltips, session history, or normal recognized/refined transcript text.
Use `voice-input diagnostics [--format text|json]` as the canonical output for support reports. Schema 4 contains bounded stage statuses and failure categories; aggregate streaming delivery/result timings and counts (ready, partial, nonempty partial, segment-final, audio packets and sent duration, queue delays, finish, task completion/failure, and finalization); bounded Audio3 timestamp aggregates (timestamp-bearing results, accepted timed units, truncated units, results with rejected timestamp metadata, and the latest event-reported valid numeric audio-relative end in milliseconds); the Audio3 native-pass mode/decision/reason; and a safe configuration summary for one session only: the active or most recently completed session. Each normal result contributes at most one rejected-timestamp-metadata count when its timestamp block contains any invalid scalar, relationship, words shape, or processed unit; truncation alone contributes zero. The latest valid end is overwritten by a later event that supplies one, with no cross-event monotonicity assumption. Timestamp diagnostics never include sentence IDs, timed-unit text or punctuation, transcripts, or provider messages. The safe summary reports the Audio3 endpoint mode and region, only whether a Workspace ID is configured, whether language hints and heartbeat are enabled, the effective maximum sentence silence and semantic-punctuation state after preset resolution, and the dynamic-vocabulary entry count. It never includes the Workspace ID, endpoint/host, model, key, or vocabulary terms. A failed provider task may include an optional strictly bounded provider error identifier. That session summary remains available after completion and is reset when the next recording starts. It contains no audio, credentials, endpoints, model names, provider messages, window/application data, prompt context, tooltips, session history, or normal recognized/refined transcript text.

Do not paste `voice-input status` output into reports, with or without `--extended`. Status output is intended for local UI integration and can include the current or most recent transcript and tooltip text.

## Experimental Qwen-Audio-3

Qwen-Audio-3 is available as an explicit experimental provider. It is disabled by default and is not offered by the stable setup wizard. The explicit experimental gate and setup-wizard omission remain intentional while the beta is prepared. To try it, open Settings, choose **Qwen-Audio-3 (experimental)**, acknowledge the experimental-provider warning, and save. The existing encrypted Alibaba credential is shared with this provider.

**Endpoint mode** defaults to **Regional**, with **Beijing** as the default region; **Singapore** is also selectable. In Regional mode, an empty **Workspace ID** uses Alibaba's reviewed legacy host for the selected region, while a nonempty value routes both Streaming and Native through that workspace's reviewed regional hostname. Workspace ID is treated as an opaque provider value. Because it occupies one hostname label, Voice Input enforces only the transport constraint of 1–63 ASCII letters, digits, or hyphens with no leading or trailing hyphen; this is not Alibaba business-ID validation. Voice Input does not add a workspace header, query field, or request-body field. **Custom** mode uses the configured raw Streaming and Native URLs exactly and ignores the dormant Workspace ID.

Migration is presence-aware. A configuration without `endpoint_mode` migrates to Regional Beijing only when both old URLs exactly equal the canonical Beijing pair, or to Regional Singapore only when both exactly equal the canonical Singapore pair. Either migration preserves an explicitly configured Workspace ID exactly; an absent or empty value remains empty. Mixed pairs, current workspace hosts, loopback endpoints, proxies, and any custom path, port, or query migrate to Custom with both URL strings preserved byte-for-byte; Voice Input never infers a Workspace ID from a host. An explicit endpoint mode takes precedence, and an explicit region takes precedence with Beijing as its missing-field default. Dormant raw URLs remain preserved.

Alibaba API keys are region- and workspace-scoped. Changing either setting may require replacing the single encrypted Alibaba credential. Voice Input never probes another region/workspace and never migrates a key automatically. This phase intentionally does not add separate credentials per region. Singapore availability does not establish feature parity: each model, control combination, language/vocabulary scenario, and workspace route still requires authorized live validation.

The streaming model supplies realtime text. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue.

**Recognition preset** defaults to **Standard**, which preserves the existing `800` ms maximum sentence silence with semantic punctuation and multi-threshold mode disabled and no speech/noise threshold. **Low-latency dictation** uses `400` ms with multi-threshold mode enabled; **Long-form** uses `1300` ms with semantic punctuation enabled. Both mappings were accepted in an authorized, one-speaker evaluation and retained both clauses across a matrix with 250–2200 ms of inserted digital silence; acoustic speech boundaries remained dependent on local RMS trimming. The bounded sample does not establish a general accuracy or latency recommendation, so Standard remains the default. See [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md). **Custom** exposes all raw controls; semantic punctuation and multi-threshold mode cannot be enabled together. Its optional speech/noise threshold must be finite and between `-1` and `1`; omission preserves provider behavior because Alibaba publishes no default. Settings displays every value that a custom request can send.

Optional **Dynamic vocabulary** entries are global to Audio3 and are sent to both streaming and native requests only when configured. Settings displays every remotely sent entry as one JSON object per line, for example `{"term":"Voice Input","weight":5}`. Terms use weights `1`–`5` or `50`; the local validator enforces the provider's term, duplicate, count, and weight limits. Dynamic terms are deliberately absent from routine support diagnostics.

**Native final pass** has three modes. **Streaming only** (the default) never sends the complete recording. **Adaptive** runs native recognition when realtime delivery is overloaded, a backend/event worker is interrupted, streaming is empty/failed/degraded, the server does not send an explicit `Finished` completion, or captured audio lasts at least 30 seconds. It skips only a usable, non-overloaded, explicitly finished stream shorter than 30 seconds. **Always** is the explicit maximum-accuracy choice and runs native recognition for every non-cancelled, nonempty recording. Existing configurations whose legacy boolean was `true` migrate to **Always**; `false` migrates to **Streaming only**.
Expand All @@ -104,7 +112,7 @@ voice-input asr stream-test --file sample.wav # WebSocket streaming
voice-input asr test --file sample.wav # native full-audio request
```

Both commands require Qwen-Audio-3 to be selected and explicitly enabled in the active configuration. Remote tests send the supplied audio to the configured Alibaba endpoint and may incur API charges. Treat the provider, model names, endpoint compatibility, and transcript behavior as subject to change while this option remains experimental.
Both commands require Qwen-Audio-3 to be selected and explicitly enabled in the active configuration. Remote tests send the supplied audio to the resolved Regional route or exact Custom endpoint and may incur API charges. Treat the provider, model names, endpoint compatibility, and transcript behavior as subject to change while this option remains experimental.

## Highlights

Expand Down Expand Up @@ -135,7 +143,7 @@ The Wiki also covers agent context, desktop integration, privacy, and developmen

## Privacy

Remote Qwen modes send audio to the configured Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider; only when agent context is explicitly enabled does it also send a capped and redacted session excerpt. Window titles, process IDs, and raw desktop metadata are never included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection.
Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider; only when agent context is explicitly enabled does it also send a capped and redacted session excerpt. Window titles, process IDs, and raw desktop metadata are never included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection.

## Project status

Expand Down
Loading