diff --git a/README.md b/README.md index 17376ed..39fc9ca 100644 --- a/README.md +++ b/README.md @@ -81,7 +81,7 @@ systemctl --user status voice-input.service voice-input-hud.service ## Safe support diagnostics -Use `voice-input diagnostics [--format text|json]` as the canonical output for support reports. Schema 3 contains bounded stage statuses and failure categories; aggregate streaming delivery/result timings and counts (ready, partial, nonempty partial, segment-final, audio packets and sent duration, queue delays, finish, task completion/failure, and finalization); the Audio3 native-pass mode/decision/reason; and a safe configuration summary for one session only: the active or most recently completed session. The safe summary reports whether Audio3 language hints and heartbeat are enabled, the configured maximum sentence silence, the semantic-punctuation state, and the dynamic-vocabulary entry count; it never includes vocabulary terms. A failed provider task may include an optional strictly bounded provider error identifier. That session summary remains available after completion and is reset when the next recording starts. It contains no audio, credentials, endpoints, model names, provider messages, window/application data, prompt context, tooltips, session history, or normal recognized/refined transcript text. +Use `voice-input diagnostics [--format text|json]` as the canonical output for support reports. Schema 4 contains bounded stage statuses and failure categories; aggregate streaming delivery/result timings and counts (ready, partial, nonempty partial, segment-final, audio packets and sent duration, queue delays, finish, task completion/failure, and finalization); bounded Audio3 timestamp aggregates (timestamp-bearing results, accepted timed units, truncated units, results with rejected timestamp metadata, and the latest event-reported valid numeric audio-relative end in milliseconds); the Audio3 native-pass mode/decision/reason; and a safe configuration summary for one session only: the active or most recently completed session. Each normal result contributes at most one rejected-timestamp-metadata count when its timestamp block contains any invalid scalar, relationship, words shape, or processed unit; truncation alone contributes zero. The latest valid end is overwritten by a later event that supplies one, with no cross-event monotonicity assumption. Timestamp diagnostics never include sentence IDs, timed-unit text or punctuation, transcripts, or provider messages. The safe summary reports the Audio3 endpoint mode and region, only whether a Workspace ID is configured, whether language hints and heartbeat are enabled, the effective maximum sentence silence and semantic-punctuation state after preset resolution, and the dynamic-vocabulary entry count. It never includes the Workspace ID, endpoint/host, model, key, or vocabulary terms. A failed provider task may include an optional strictly bounded provider error identifier. That session summary remains available after completion and is reset when the next recording starts. It contains no audio, credentials, endpoints, model names, provider messages, window/application data, prompt context, tooltips, session history, or normal recognized/refined transcript text. Do not paste `voice-input status` output into reports, with or without `--extended`. Status output is intended for local UI integration and can include the current or most recent transcript and tooltip text. @@ -89,8 +89,16 @@ Do not paste `voice-input status` output into reports, with or without `--extend Qwen-Audio-3 is available as an explicit experimental provider. It is disabled by default and is not offered by the stable setup wizard. The explicit experimental gate and setup-wizard omission remain intentional while the beta is prepared. To try it, open Settings, choose **Qwen-Audio-3 (experimental)**, acknowledge the experimental-provider warning, and save. The existing encrypted Alibaba credential is shared with this provider. +**Endpoint mode** defaults to **Regional**, with **Beijing** as the default region; **Singapore** is also selectable. In Regional mode, an empty **Workspace ID** uses Alibaba's reviewed legacy host for the selected region, while a nonempty value routes both Streaming and Native through that workspace's reviewed regional hostname. Workspace ID is treated as an opaque provider value. Because it occupies one hostname label, Voice Input enforces only the transport constraint of 1–63 ASCII letters, digits, or hyphens with no leading or trailing hyphen; this is not Alibaba business-ID validation. Voice Input does not add a workspace header, query field, or request-body field. **Custom** mode uses the configured raw Streaming and Native URLs exactly and ignores the dormant Workspace ID. + +Migration is presence-aware. A configuration without `endpoint_mode` migrates to Regional Beijing only when both old URLs exactly equal the canonical Beijing pair, or to Regional Singapore only when both exactly equal the canonical Singapore pair. Either migration preserves an explicitly configured Workspace ID exactly; an absent or empty value remains empty. Mixed pairs, current workspace hosts, loopback endpoints, proxies, and any custom path, port, or query migrate to Custom with both URL strings preserved byte-for-byte; Voice Input never infers a Workspace ID from a host. An explicit endpoint mode takes precedence, and an explicit region takes precedence with Beijing as its missing-field default. Dormant raw URLs remain preserved. + +Alibaba API keys are region- and workspace-scoped. Changing either setting may require replacing the single encrypted Alibaba credential. Voice Input never probes another region/workspace and never migrates a key automatically. This phase intentionally does not add separate credentials per region. Singapore availability does not establish feature parity: each model, control combination, language/vocabulary scenario, and workspace route still requires authorized live validation. + The streaming model supplies realtime text. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue. +**Recognition preset** defaults to **Standard**, which preserves the existing `800` ms maximum sentence silence with semantic punctuation and multi-threshold mode disabled and no speech/noise threshold. **Low-latency dictation** uses `400` ms with multi-threshold mode enabled; **Long-form** uses `1300` ms with semantic punctuation enabled. Both mappings were accepted in an authorized, one-speaker evaluation and retained both clauses across a matrix with 250–2200 ms of inserted digital silence; acoustic speech boundaries remained dependent on local RMS trimming. The bounded sample does not establish a general accuracy or latency recommendation, so Standard remains the default. See [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md). **Custom** exposes all raw controls; semantic punctuation and multi-threshold mode cannot be enabled together. Its optional speech/noise threshold must be finite and between `-1` and `1`; omission preserves provider behavior because Alibaba publishes no default. Settings displays every value that a custom request can send. + Optional **Dynamic vocabulary** entries are global to Audio3 and are sent to both streaming and native requests only when configured. Settings displays every remotely sent entry as one JSON object per line, for example `{"term":"Voice Input","weight":5}`. Terms use weights `1`–`5` or `50`; the local validator enforces the provider's term, duplicate, count, and weight limits. Dynamic terms are deliberately absent from routine support diagnostics. **Native final pass** has three modes. **Streaming only** (the default) never sends the complete recording. **Adaptive** runs native recognition when realtime delivery is overloaded, a backend/event worker is interrupted, streaming is empty/failed/degraded, the server does not send an explicit `Finished` completion, or captured audio lasts at least 30 seconds. It skips only a usable, non-overloaded, explicitly finished stream shorter than 30 seconds. **Always** is the explicit maximum-accuracy choice and runs native recognition for every non-cancelled, nonempty recording. Existing configurations whose legacy boolean was `true` migrate to **Always**; `false` migrates to **Streaming only**. @@ -104,7 +112,7 @@ voice-input asr stream-test --file sample.wav # WebSocket streaming voice-input asr test --file sample.wav # native full-audio request ``` -Both commands require Qwen-Audio-3 to be selected and explicitly enabled in the active configuration. Remote tests send the supplied audio to the configured Alibaba endpoint and may incur API charges. Treat the provider, model names, endpoint compatibility, and transcript behavior as subject to change while this option remains experimental. +Both commands require Qwen-Audio-3 to be selected and explicitly enabled in the active configuration. Remote tests send the supplied audio to the resolved Regional route or exact Custom endpoint and may incur API charges. Treat the provider, model names, endpoint compatibility, and transcript behavior as subject to change while this option remains experimental. ## Highlights @@ -135,7 +143,7 @@ The Wiki also covers agent context, desktop integration, privacy, and developmen ## Privacy -Remote Qwen modes send audio to the configured Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider; only when agent context is explicitly enabled does it also send a capped and redacted session excerpt. Window titles, process IDs, and raw desktop metadata are never included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection. +Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider; only when agent context is explicitly enabled does it also send a capped and redacted session excerpt. Window titles, process IDs, and raw desktop metadata are never included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection. ## Project status diff --git a/README.zh-CN.md b/README.zh-CN.md index 06c1c3f..e22fa58 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -81,7 +81,7 @@ systemctl --user status voice-input.service voice-input-hud.service ## 安全的支持诊断信息 -提交支持报告时，请使用 `voice-input diagnostics [--format text|json]` 作为标准诊断输出。Schema 3 只提供当前 session 或最近完成的 session，并使用固定字段记录各阶段状态和失败类别、流式投递与结果的汇总耗时和计数（ready、partial、非空 partial、segment-final、音频包及已发送音频时长、队列延迟、finish、任务完成或失败以及 finalization）、Audio3 原生最终处理的模式/决定/原因，以及安全的配置摘要。安全摘要说明 Audio3 语言提示和 heartbeat 是否启用、配置的最大句末静音时长、语义标点状态和动态词汇表的词条数量；它不会包含词条内容。提供商任务失败时，诊断信息可能包含一个可选且长度和字符范围受到严格限制的提供商错误标识符。完成后的摘要会保留到下一次录音开始，并在新录音开始时重置。输出不包含音频、凭据、端点、模型名称、提供商消息、窗口或应用信息、prompt context、tooltip、session 历史记录，也不包含正常的识别或整理后 transcript 文本。 +提交支持报告时，请使用 `voice-input diagnostics [--format text|json]` 作为标准诊断输出。Schema 4 只提供当前 session 或最近完成的 session，并使用固定字段记录各阶段状态和失败类别、流式投递与结果的汇总耗时和计数（ready、partial、非空 partial、segment-final、音频包及已发送音频时长、队列延迟、finish、任务完成或失败以及 finalization）、有明确数量上限的 Audio3 时间戳汇总数据（包含时间戳的结果数、接受的计时单元数、因超过上限而截断的单元数、时间戳元数据被拒绝的结果数，以及最近一个事件报告的有效音频相对结束毫秒数）、Audio3 原生最终处理的模式/决定/原因，以及安全的配置摘要。如果一个普通结果的时间戳块包含任何无效标量、关系、`words` 结构或已处理单元，该结果最多使“时间戳元数据被拒绝的结果数”增加 1；仅发生截断时不会增加该计数。如果后续事件提供有效结束时间，诊断信息会直接覆盖此前的值，不假设不同事件之间单调递增。时间戳诊断信息不会包含句子 ID、计时单元文本或标点、transcript 或提供商消息。安全摘要说明 Audio3 的端点模式和区域、是否配置了工作空间 ID、语言提示和 heartbeat 是否启用、解析预设后生效的最大句末静音时长和语义标点状态，以及动态词汇表的词条数量。它不会包含工作空间 ID、端点或主机、模型、key 或词条内容。提供商任务失败时，诊断信息可能包含一个可选且长度和字符范围受到严格限制的提供商错误标识符。完成后的摘要会保留到下一次录音开始，并在新录音开始时重置。输出不包含音频、凭据、端点、模型名称、提供商消息、窗口或应用信息、prompt context、tooltip、session 历史记录，也不包含正常的识别或整理后 transcript 文本。 请勿把 `voice-input status` 的输出粘贴到报告中，无论是否使用 `--extended`。状态输出用于本地 UI 集成，可能包含当前或最近一次 transcript 和 tooltip 文本。 @@ -89,8 +89,16 @@ systemctl --user status voice-input.service voice-input-hud.service Qwen-Audio-3 目前作为需要明确启用的实验性提供商使用。该选项默认关闭，稳定安装向导也不会提供它。在 beta 准备期间，明确的实验功能开关以及安装向导不提供该选项都是有意保留的设计。如需试用，请打开 Settings，选择 **Qwen-Audio-3（实验性）**，确认实验功能警告后保存。该提供商与现有 Alibaba 实时识别共用同一份加密凭据。 +**端点模式**默认使用**区域路由**，默认区域为**北京**，也可以选择**新加坡**。在区域路由模式下，**工作空间 ID** 留空时会使用所选区域经过审核的 Alibaba 旧版主机；填写该值后，流式请求和原生请求都会通过该工作空间对应且经过审核的区域主机名发送。程序把工作空间 ID 视为不透明的提供商值。因为该值会占用一个主机名 label，Voice Input 只验证传输约束：必须包含 1–63 个 ASCII 字母、数字或连字符，且首尾不能是连字符。这项检查不代表 Alibaba 的业务 ID 规则。Voice Input 不会添加工作空间 header、query 字段或请求正文（request body）字段。**自定义**模式会原样使用已配置的流式 URL 和原生 URL，并忽略处于非活动状态的工作空间 ID。 + +迁移过程会区分字段是否存在。未包含 `endpoint_mode` 的配置只有在两个旧 URL 与北京标准组合完全相同时，才会迁移到北京区域路由；只有在两个 URL 与新加坡标准组合完全相同时，才会迁移到新加坡区域路由。两种迁移都会原样保留明确配置的工作空间 ID；该字段缺失或为空时仍保持为空。混合组合、当前工作空间主机、回环端点、代理，以及带有自定义路径、端口或 query 的配置都会迁移到自定义模式，并逐字节保留两个 URL 字符串；Voice Input 绝不会根据主机推断工作空间 ID。明确配置的端点模式具有优先级；明确配置的区域也具有优先级，缺少区域字段时默认使用北京。程序还会保留处于非活动状态的原始 URL。 + +Alibaba API key 受区域和工作空间范围约束。更改其中任何一项后，用户可能需要替换同一份加密 Alibaba 凭据。Voice Input 绝不会探测其他区域或工作空间，也不会自动迁移 key。本阶段有意不增加按区域分别保存的凭据。支持选择新加坡区域并不表示已经实现完整功能一致性；每个模型、控制项组合、语言或词汇表场景以及工作空间路由仍需完成经过授权的在线验证。 + 流式模型负责提供实时文本。**语言提示**和**流式 heartbeat** 是两个相互独立的选用设置，默认均为关闭。启用语言提示后，程序会把现有语言选项发送给 Audio3：英语使用 `en`；简体中文和繁体中文使用 `zh,en`；日语使用 `ja,en`；韩语使用 `ko,en`。中文、日语和韩语的额外英语提示用于保留英语混合识别；关闭该开关会保留服务商的自动检测行为。启用流式 heartbeat 后，只要程序继续发送格式正确的音频帧，它就能使长时间静音的按键说话 session 保持连接。 +**识别预设**默认使用**标准**。该预设保留现有行为：最大句末静音时长为 `800` 毫秒，语义标点和多阈值模式均关闭，并且不发送语音/噪声阈值。**低延迟听写**使用 `400` 毫秒并启用多阈值模式；**长篇语音**使用 `1300` 毫秒并启用语义标点。经过授权的单说话人评估确认服务端接受这两个映射；在插入了 250–2200 毫秒数字静音的测试矩阵中，两者都保留了静音前后的内容。声学语音边界仍取决于本地 RMS 裁剪。有限样本无法形成通用的准确率或延迟建议，因此标准预设仍为默认值。详见 [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md)。**自定义**会显示全部原始控制项；语义标点与多阈值模式不能同时启用。可选的语音/噪声阈值必须是 `-1` 到 `1` 之间的有限数值；Alibaba 未公布默认值，因此省略该字段可以保留服务商行为。Settings 会显示自定义请求可能发送的每一个值。 + 用户可以配置可选的全局**动态词汇表**。程序只在配置不为空时，才会把词条发送给 Audio3 的流式请求和原生请求。Settings 会用每行一个 JSON 对象的方式显示所有将发送到远程服务的词条，例如 `{"term":"Voice Input","weight":5}`。权重可以是 `1`–`5` 或 `50`；本地验证会检查服务商规定的词条、重复项、数量和权重限制。常规支持诊断信息不会包含动态词条。 **原生最终处理**提供三种模式。默认的**仅流式识别**不会发送完整录音。**自适应**模式会在实时音频传输过载、后端或事件 worker 中断、流式识别为空/失败/降级、服务端未发送明确的 `Finished` 完成事件，或者录音达到 30 秒时运行原生识别。只有可用、未过载、明确完成且短于 30 秒的流式识别才会跳过原生识别。**始终运行**是明确请求最高准确度的选项，会对每段未取消且非空的录音运行原生识别。旧配置中的 boolean 为 `true` 时会迁移到**始终运行**，为 `false` 时会迁移到**仅流式识别**。 @@ -104,7 +112,7 @@ voice-input asr stream-test --file sample.wav # WebSocket 流式识别 voice-input asr test --file sample.wav # 原生完整音频识别 ``` -两个命令都要求当前配置已经选择并明确启用 Qwen-Audio-3。远程测试会把指定音频发送到配置的 Alibaba endpoint，并且可能产生 API 费用。在该选项仍处于实验阶段时，提供商行为、模型名称、端点兼容性和识别结果都可能发生变化。 +两个命令都要求当前配置已经选择并明确启用 Qwen-Audio-3。远程测试会把指定音频发送到解析后的区域路由或完全按原值使用的自定义 Alibaba 端点，并且可能产生 API 费用。在该选项仍处于实验阶段时，提供商行为、模型名称、端点兼容性和识别结果都可能发生变化。 ## 主要能力 @@ -135,7 +143,7 @@ Wiki 还包含 Agent context、桌面集成、安全隐私和开发说明。 ## 隐私 -远程 Qwen 模式会把音频发送到配置的 Alibaba endpoint。LLM refinement 会把 transcript 和粗粒度的目标风格（coding agent 结构化 Markdown、`instant-messaging` 或默认风格）通过 system prompt 发送到配置的 provider；只有在用户明确启用 Agent context 时，才会额外发送经过截断与脱敏的会话片段。LLM 请求不会包含窗口标题、进程 ID 或原始桌面元数据。公开示例配置默认关闭远程 refinement 和 Agent context。Voice Input 不收集遥测或分析数据。 +远程 Qwen 模式会把音频发送到所选的区域路由或完全按原值使用的自定义 Alibaba 端点。LLM refinement 会把 transcript 和粗粒度的目标风格（coding agent 结构化 Markdown、`instant-messaging` 或默认风格）通过 system prompt 发送到配置的 provider；只有在用户明确启用 Agent context 时，才会额外发送经过截断与脱敏的会话片段。LLM 请求不会包含窗口标题、进程 ID 或原始桌面元数据。公开示例配置默认关闭远程 refinement 和 Agent context。Voice Input 不收集遥测或分析数据。 ## 项目状态 diff --git a/assets/config.toml b/assets/config.toml index 64b9c93..58f241e 100644 --- a/assets/config.toml +++ b/assets/config.toml @@ -36,12 +36,17 @@ final_pass_enable_itn = false [asr.alibaba_audio3] experimental_enabled = false +endpoint_mode = "regional" +region = "beijing" +workspace_id = "" endpoint = "wss://dashscope.aliyuncs.com/api-ws/v1/inference" model = "qwen-audio-3.0-asr-flash-streaming" language_hints_enabled = false heartbeat_enabled = false +recognition_preset = "standard" max_sentence_silence_ms = 800 semantic_punctuation_enabled = false +multi_threshold_mode_enabled = false vocabulary = [] native_endpoint = "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation" native_model = "qwen-audio-3.0-asr-flash" diff --git a/assets/quickshell-settings/I18n.qml b/assets/quickshell-settings/I18n.qml index ff464bf..8dfdd0a 100644 --- a/assets/quickshell-settings/I18n.qml +++ b/assets/quickshell-settings/I18n.qml @@ -106,6 +106,7 @@ QtObject { "Realtime recognition behavior.": "设置实时识别行为。", "Replace Alibaba API key": "替换 Alibaba API key", "Enter a new credential": "输入新凭据", + "API keys are region- and workspace-scoped; Voice Input never probes another route or migrates a key automatically.": "API key 受区域和工作空间范围约束；语音输入绝不会探测其他路由，也不会自动迁移 key。", "Realtime model": "实时模型", "Turn mode": "分段模式", "Server VAD": "服务端 VAD", @@ -113,10 +114,26 @@ QtObject { "Experimental provider; behavior and API compatibility may change.": "此提供商仍处于实验阶段，其行为和 API 兼容性可能发生变化。", "I understand and enable experimental Qwen-Audio-3": "我了解相关风险并启用实验性 Qwen-Audio-3", "Selecting the provider does not enable this acknowledgement.": "仅选择此提供商不会自动确认并启用实验功能。", + "Endpoint mode": "端点模式", + "Regional": "区域路由", + "Regional routing uses reviewed Alibaba hosts. Custom keeps the raw streaming and native endpoints in Advanced settings.": "区域路由使用经过审核的 Alibaba 主机。自定义模式会在高级设置中保留原始流式端点和原生端点。", + "Region": "区域", + "Beijing": "北京", + "Singapore": "新加坡", + "Select the region that owns the configured Alibaba API key. Singapore controls still require scenario-specific live validation.": "请选择当前 Alibaba API key 所属的区域。新加坡区域的各项控制仍需按具体场景完成在线验证。", + "Workspace ID": "工作空间 ID", + "Optional": "可选", + "Optional. Empty uses the regional legacy route. A nonempty value must satisfy the DNS-label transport constraint (1–63 ASCII letters, digits, or hyphens; no leading or trailing hyphen). This is not provider business-ID validation. API keys are region- and workspace-scoped; Voice Input never probes or migrates them.": "可选。留空时使用对应区域的旧版路由。非空值必须满足 DNS label 传输约束：包含 1–63 个 ASCII 字母、数字或连字符，且首尾不能是连字符。这不是对提供商业务 ID 规则的验证。API key 受区域和工作空间范围约束；语音输入绝不会探测或迁移 key。", "Enable language hints": "启用语言提示", "Opt in to sending the selected language as an Audio3 recognition hint.": "选择将所选语言作为 Audio3 识别提示发送。", "Enable streaming heartbeat": "启用流式 heartbeat", "Opt in to keeping long silent push-to-talk sessions alive while audio frames continue.": "选择在音频帧持续发送时，使长时间静音的按键说话 session 保持连接。", + "Recognition preset": "识别预设", + "Standard": "标准", + "Low-latency dictation": "低延迟听写", + "Long-form": "长篇语音", + "Custom": "自定义", + "Standard preserves existing behavior. Low-latency dictation and long-form are evaluation candidates pending live pause and noise validation. Custom exposes every raw recognition control.": "标准预设会保留现有行为。低延迟听写和长篇语音是评估候选值，仍需完成暂停和噪声的在线验证。自定义预设会显示所有原始识别控制项。", "Dynamic vocabulary": "动态词汇表", "Optional global Audio3 terms. Enter one JSON object per line with term and weight; weights are 1–5 or 50. Every listed term is sent remotely.": "可选的全局 Audio3 词条。每行输入一个包含 term 和 weight 的 JSON 对象；权重可以是 1–5 或 50。列出的每个词条都会发送给远程服务。", "{\"term\":\"Voice Input\",\"weight\":5}": "{\"term\":\"语音输入\",\"weight\":5}", @@ -161,6 +178,13 @@ QtObject { "Milliseconds from 200 to 6000. Lower values finalize speech segments sooner.": "范围为 200 到 6000 毫秒。较小的值会更快完成语音分段。", "Enable semantic punctuation": "启用语义标点", "Allow the streaming model to use semantic punctuation when finalizing speech segments.": "允许流式模型在完成语音分段时使用语义标点。", + "Enable multi-threshold mode": "启用多阈值模式", + "Use the documented adaptive VAD threshold mode. It cannot be combined with semantic punctuation.": "使用文档说明的自适应 VAD 阈值模式。此模式不能与语义标点同时启用。", + "Override speech/noise threshold": "覆盖语音/噪声阈值", + "Send an optional threshold from -1 to 1. Lower values classify more noise as speech; Alibaba publishes no default.": "发送一个 -1 到 1 的可选阈值。数值越低，越多噪声会被归类为语音；Alibaba 未公布默认值。", + "Speech/noise threshold": "语音/噪声阈值", + "Finite value from -1 to 1. This value is sent remotely only while the override is enabled.": "请输入 -1 到 1 的有限数值。仅在启用覆盖时，程序才会将该值发送给远程服务。", + "Enter a finite number.": "请输入有限数值。", "Qwen-Audio-3 native final pass": "Qwen-Audio-3 原生最终处理", "Native endpoint": "原生端点", "Native model": "原生模型", @@ -297,6 +321,7 @@ QtObject { "must not contain embedded credentials": "不得包含嵌入式凭据", "must include a host": "必须包含主机名", "must use transport encryption unless the host is loopback": "除回环主机外，必须使用加密传输", + "must be a valid DNS label for hostname transport": "必须是可用于主机名传输的有效 DNS label", "must be true when the experimental provider is selected": "选择实验性提供商时必须明确启用此选项", "must not exceed maximum recording duration": "不得超过最长录音时长", "must be finite and between 0 and 1": "必须是 0 到 1 之间的有限数值", @@ -350,6 +375,10 @@ QtObject { if (match) return "已通过 " + match[1] + " 配置"; + match = source.match(/^(.*)\. Blank keeps it unchanged\. API keys are region- and workspace-scoped; Voice Input never probes another route or migrates a key automatically\.$/); + if (match) + return root.tr(match[1]) + "。留空将保持不变。API key 受区域和工作空间范围约束；语音输入绝不会探测其他路由，也不会自动迁移 key。"; + match = source.match(/^(.*)\. Blank keeps it unchanged\.$/); if (match) return root.tr(match[1]) + "。留空将保持不变。"; diff --git a/assets/quickshell-settings/SettingsController.qml b/assets/quickshell-settings/SettingsController.qml index 8f7c043..0eccf6f 100644 --- a/assets/quickshell-settings/SettingsController.qml +++ b/assets/quickshell-settings/SettingsController.qml @@ -88,12 +88,18 @@ QtObject { }, "alibaba_audio3": { "experimental_enabled": false, + "endpoint_mode": "regional", + "region": "beijing", + "workspace_id": "", "endpoint": "wss://dashscope.aliyuncs.com/api-ws/v1/inference", "model": "qwen-audio-3.0-asr-flash-streaming", "language_hints_enabled": false, "heartbeat_enabled": false, + "recognition_preset": "standard", "max_sentence_silence_ms": 800, "semantic_punctuation_enabled": false, + "multi_threshold_mode_enabled": false, + "speech_noise_threshold": null, "vocabulary": [], "native_endpoint": "https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation", "native_model": "qwen-audio-3.0-asr-flash", @@ -299,7 +305,7 @@ QtObject { } function fieldNeedsAdvanced(path) { - const audio3Advanced = path.indexOf("asr.alibaba_audio3.") === 0 && path !== "asr.alibaba_audio3.experimental_enabled" && path !== "asr.alibaba_audio3.language_hints_enabled" && path !== "asr.alibaba_audio3.heartbeat_enabled" && path !== "asr.alibaba_audio3.vocabulary" && path !== "asr.alibaba_audio3.native_final_pass_mode"; + const audio3Advanced = path.indexOf("asr.alibaba_audio3.") === 0 && path !== "asr.alibaba_audio3.experimental_enabled" && path !== "asr.alibaba_audio3.endpoint_mode" && path !== "asr.alibaba_audio3.region" && path !== "asr.alibaba_audio3.workspace_id" && path !== "asr.alibaba_audio3.language_hints_enabled" && path !== "asr.alibaba_audio3.heartbeat_enabled" && path !== "asr.alibaba_audio3.recognition_preset" && path !== "asr.alibaba_audio3.vocabulary" && path !== "asr.alibaba_audio3.native_final_pass_mode"; return path === "audio.sample_rate" || path === "audio.partial_interval_ms" || path === "audio.pre_roll_ms" || path === "asr.connect_timeout_ms" || path === "asr.finalize_timeout_ms" || path.indexOf("asr.alibaba.endpoint") === 0 || path.indexOf("asr.alibaba.vad_") === 0 || path.indexOf("asr.alibaba.silence_") === 0 || path.indexOf("asr.alibaba.final_pass_") === 0 || audio3Advanced || path === "llm.api_base_url" || path === "llm.timeout_ms" || path === "llm.provider_sort" || path === "llm.agent_context_max_chars" || path === "output.type_delay_ms" || path === "output.pre_type_delay_ms" || path === "output.paste_keys" || path === "output.prefer_paste_for_xwayland" || path === "output.xwayland_paste_keys" || path === "hud.margin_bottom" || path === "hud.height" || path === "hud.offset_x" || path === "hud.offset_y" || path === "hud.nudge_step"; } @@ -438,6 +444,17 @@ QtObject { for (let i = 0; i < unsignedIntegers.length; ++i) valid = asNumber(config, unsignedIntegers[i][0], true, unsignedIntegers[i][1]) && valid for (let i = 0; i < signedIntegers.length; ++i) valid = asNumber(config, signedIntegers[i], true, null) && valid valid = asNumber(config, "asr.alibaba.vad_threshold", false, null) && valid; + const threshold = config.asr.alibaba_audio3.speech_noise_threshold; + if (threshold !== null && threshold !== undefined) { + if (String(threshold).trim().length === 0) { + const next = clone(fieldErrors); + next["asr.alibaba_audio3.speech_noise_threshold"] = "Enter a finite number."; + fieldErrors = next; + valid = false; + } else { + valid = asNumber(config, "asr.alibaba_audio3.speech_noise_threshold", false, null) && valid; + } + } const vocabulary = parseAudio3Vocabulary(); if (vocabulary === null) valid = false; diff --git a/assets/quickshell-settings/SpeechPage.qml b/assets/quickshell-settings/SpeechPage.qml index 60e6987..a649f71 100644 --- a/assets/quickshell-settings/SpeechPage.qml +++ b/assets/quickshell-settings/SpeechPage.qml @@ -153,7 +153,7 @@ SettingsPage { theme: root.theme label: "Replace Alibaba API key" value: root.controller.alibabaCredential - help: root.controller.credentialLabel("alibaba-api-key") + ". Blank keeps it unchanged." + help: root.controller.credentialLabel("alibaba-api-key") + ". Blank keeps it unchanged. API keys are region- and workspace-scoped; Voice Input never probes another route or migrates a key automatically." password: true placeholderText: "Enter a new credential" error: root.controller.errorFor("credentials.alibaba-api-key") @@ -217,6 +217,49 @@ SettingsPage { } } + SettingCombo { + theme: root.theme + label: "Endpoint mode" + value: root.controller.value("asr.alibaba_audio3.endpoint_mode", "regional") + labels: ["Regional", "Custom"] + values: ["regional", "custom"] + help: "Regional routing uses reviewed Alibaba hosts. Custom keeps the raw streaming and native endpoints in Advanced settings." + error: root.controller.errorFor("asr.alibaba_audio3.endpoint_mode") + enabled: !root.controller.busy + onSelected: (value) => { + return root.controller.setValue("asr.alibaba_audio3.endpoint_mode", value); + } + } + + SettingCombo { + visible: root.controller.value("asr.alibaba_audio3.endpoint_mode", "regional") === "regional" || root.controller.errorFor("asr.alibaba_audio3.region").length > 0 + theme: root.theme + label: "Region" + value: root.controller.value("asr.alibaba_audio3.region", "beijing") + labels: ["Beijing", "Singapore"] + values: ["beijing", "singapore"] + help: "Select the region that owns the configured Alibaba API key. Singapore controls still require scenario-specific live validation." + error: root.controller.errorFor("asr.alibaba_audio3.region") + enabled: !root.controller.busy + onSelected: (value) => { + return root.controller.setValue("asr.alibaba_audio3.region", value); + } + } + + SettingTextField { + visible: root.controller.value("asr.alibaba_audio3.endpoint_mode", "regional") === "regional" || root.controller.errorFor("asr.alibaba_audio3.workspace_id").length > 0 + theme: root.theme + label: "Workspace ID" + value: root.controller.value("asr.alibaba_audio3.workspace_id", "") + placeholderText: "Optional" + help: "Optional. Empty uses the regional legacy route. A nonempty value must satisfy the DNS-label transport constraint (1–63 ASCII letters, digits, or hyphens; no leading or trailing hyphen). This is not provider business-ID validation. API keys are region- and workspace-scoped; Voice Input never probes or migrates them." + error: root.controller.errorFor("asr.alibaba_audio3.workspace_id") + enabled: !root.controller.busy + onEdited: (value) => { + return root.controller.setValue("asr.alibaba_audio3.workspace_id", value); + } + } + SettingSwitch { theme: root.theme label: "Enable language hints" @@ -241,6 +284,20 @@ SettingsPage { } } + SettingCombo { + theme: root.theme + label: "Recognition preset" + value: root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") + labels: ["Standard", "Low-latency dictation", "Long-form", "Custom"] + values: ["standard", "low-latency-dictation", "long-form", "custom"] + help: "Standard preserves existing behavior. Low-latency dictation and long-form are evaluation candidates pending live pause and noise validation. Custom exposes every raw recognition control." + error: root.controller.errorFor("asr.alibaba_audio3.recognition_preset") + enabled: !root.controller.busy + onSelected: (value) => { + return root.controller.setValue("asr.alibaba_audio3.recognition_preset", value); + } + } + SettingTextArea { theme: root.theme label: "Dynamic vocabulary" @@ -464,12 +521,13 @@ SettingsPage { SectionCard { id: audio3StreamingCard - visible: root.controller.value("asr.provider", "local-cli") === "alibaba-qwen-audio3" || root.controller.hasErrorPrefix("asr.alibaba_audio3.endpoint") || root.controller.hasErrorPrefix("asr.alibaba_audio3.model") + visible: root.controller.value("asr.provider", "local-cli") === "alibaba-qwen-audio3" || root.controller.hasErrorPrefix("asr.alibaba_audio3.endpoint") || root.controller.hasErrorPrefix("asr.alibaba_audio3.model") || root.controller.hasErrorPrefix("asr.alibaba_audio3.max_sentence_silence_ms") || root.controller.hasErrorPrefix("asr.alibaba_audio3.semantic_punctuation_enabled") || root.controller.hasErrorPrefix("asr.alibaba_audio3.multi_threshold_mode_enabled") || root.controller.hasErrorPrefix("asr.alibaba_audio3.speech_noise_threshold") theme: root.theme title: "Qwen-Audio-3 streaming" showDivider: audio3NativeCard.visible SettingTextField { + visible: root.controller.value("asr.alibaba_audio3.endpoint_mode", "regional") === "custom" || root.controller.errorFor("asr.alibaba_audio3.endpoint").length > 0 theme: root.theme label: "Streaming endpoint" value: root.controller.value("asr.alibaba_audio3.endpoint", "") @@ -492,6 +550,7 @@ SettingsPage { } SettingTextField { + visible: root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") === "custom" || root.controller.errorFor("asr.alibaba_audio3.max_sentence_silence_ms").length > 0 theme: root.theme label: "Maximum sentence silence" value: root.controller.value("asr.alibaba_audio3.max_sentence_silence_ms", 800) @@ -504,6 +563,7 @@ SettingsPage { } SettingSwitch { + visible: root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") === "custom" || root.controller.errorFor("asr.alibaba_audio3.semantic_punctuation_enabled").length > 0 || root.controller.errorFor("asr.alibaba_audio3.multi_threshold_mode_enabled").length > 0 theme: root.theme label: "Enable semantic punctuation" checked: root.controller.value("asr.alibaba_audio3.semantic_punctuation_enabled", false) @@ -515,6 +575,51 @@ SettingsPage { } } + SettingSwitch { + visible: root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") === "custom" || root.controller.errorFor("asr.alibaba_audio3.multi_threshold_mode_enabled").length > 0 + theme: root.theme + label: "Enable multi-threshold mode" + checked: root.controller.value("asr.alibaba_audio3.multi_threshold_mode_enabled", false) + help: "Use the documented adaptive VAD threshold mode. It cannot be combined with semantic punctuation." + error: root.controller.errorFor("asr.alibaba_audio3.multi_threshold_mode_enabled") + enabled: !root.controller.busy + onToggled: (checked) => { + return root.controller.setValue("asr.alibaba_audio3.multi_threshold_mode_enabled", checked); + } + } + + SettingSwitch { + visible: root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") === "custom" || root.controller.errorFor("asr.alibaba_audio3.speech_noise_threshold").length > 0 + theme: root.theme + label: "Override speech/noise threshold" + checked: root.controller.value("asr.alibaba_audio3.speech_noise_threshold", null) !== null + help: "Send an optional threshold from -1 to 1. Lower values classify more noise as speech; Alibaba publishes no default." + error: root.controller.errorFor("asr.alibaba_audio3.speech_noise_threshold") + enabled: !root.controller.busy + onToggled: (checked) => { + if (checked && root.controller.value("asr.alibaba_audio3.speech_noise_threshold", null) === null) + return root.controller.setValue("asr.alibaba_audio3.speech_noise_threshold", 0); + + if (!checked) + return root.controller.setValue("asr.alibaba_audio3.speech_noise_threshold", null); + + return true; + } + } + + SettingTextField { + visible: (root.controller.value("asr.alibaba_audio3.recognition_preset", "standard") === "custom" && root.controller.value("asr.alibaba_audio3.speech_noise_threshold", null) !== null) || root.controller.errorFor("asr.alibaba_audio3.speech_noise_threshold").length > 0 + theme: root.theme + label: "Speech/noise threshold" + value: root.controller.value("asr.alibaba_audio3.speech_noise_threshold", "") + help: "Finite value from -1 to 1. This value is sent remotely only while the override is enabled." + error: root.controller.errorFor("asr.alibaba_audio3.speech_noise_threshold") + enabled: !root.controller.busy && root.controller.value("asr.alibaba_audio3.speech_noise_threshold", null) !== null + onEdited: (value) => { + return root.controller.setValue("asr.alibaba_audio3.speech_noise_threshold", value); + } + } + } SectionCard { @@ -526,6 +631,7 @@ SettingsPage { showDivider: false SettingTextField { + visible: root.controller.value("asr.alibaba_audio3.endpoint_mode", "regional") === "custom" || root.controller.errorFor("asr.alibaba_audio3.native_endpoint").length > 0 theme: root.theme label: "Native endpoint" value: root.controller.value("asr.alibaba_audio3.native_endpoint", "") diff --git a/docs/qwen-audio3-milestone2-api-decisions.md b/docs/qwen-audio3-milestone2-api-decisions.md new file mode 100644 index 0000000..43b91ab --- /dev/null +++ b/docs/qwen-audio3-milestone2-api-decisions.md @@ -0,0 +1,184 @@ +# Qwen-Audio-3 Milestone 2 API Decisions + +Research date: 2026-08-05 + +This document records provider-facing decisions for Milestone 2 before runtime implementation. It uses current official Alibaba Cloud Model Studio documentation as the source of truth. No provider request was made during this research. + +## Scope + +Milestone 2 is limited to: + +1. Audio3 streaming VAD and sentence-boundary controls with named presets; +2. bounded timestamp parsing and timestamp-aware assembly foundations; +3. Beijing/Singapore regional and workspace endpoint routing. + +It does not add reconnect/replay, `continue-task` context, application/window collection, Filetrans, diarization, or new credential-discovery behavior. + +## VAD and sentence-boundary controls + +All confirmed controls belong under `payload.parameters` in the initial `run-task` event. + +| Field | Type | Official default | Valid range | Decision | +| --- | --- | --- | --- | --- | +| `semantic_punctuation_enabled` | Boolean | `false` | Boolean | Implement. When enabled, semantic segmentation replaces VAD segmentation. | +| `max_sentence_silence` | Integer milliseconds | `1300` | 200–6000 | Already implemented. Voice Input's existing `800` default remains the standard preset for backward compatibility. | +| `multi_threshold_mode_enabled` | Boolean | `false` | Boolean | Implement. It is effective only when semantic punctuation is disabled. | +| `speech_noise_threshold` | Float | Not documented | -1.0–1.0 | Implement as optional custom tuning. Omission preserves provider behavior because no official default is published. | +| `heartbeat` | Boolean | `false` | Boolean | Already implemented. It remains independent of segmentation controls. | + +There are no documented Audio3 fields named `speech_threshold` or `noise_threshold`. Voice Input will not send or expose those inferred names. + +Official interaction rules: + +- `semantic_punctuation_enabled = true` disables VAD segmentation. In this mode, `max_sentence_silence` does not cause `sentence_end`, although an excessively low value may still reduce recognition quality. +- `multi_threshold_mode_enabled` takes effect only when semantic punctuation is disabled. +- Values of `speech_noise_threshold` near `-1` classify more noise as speech; values near `1` classify more speech as noise. Alibaba publishes no default. +- Official documentation does not state whether unknown, wrongly typed, or unsupported fields are rejected or ignored. Deterministic client validation will reject invalid local values; live tests must confirm accepted combinations. + +### Preset decisions + +The configuration retains raw custom values and resolves a named preset: + +- `standard`: preserve the existing effective request (`800` ms, semantic punctuation disabled, multi-threshold disabled, speech/noise threshold omitted). +- `low-latency-dictation`: use `400` ms, semantic punctuation disabled, multi-threshold enabled, and speech/noise threshold omitted. +- `long-form`: use `1300` ms, semantic punctuation enabled, multi-threshold disabled, and speech/noise threshold omitted. +- `custom`: send the user's validated raw controls. `speech_noise_threshold` remains optional. + +`standard` remains the default. Authorized bounded pause/noise evaluation accepted both named mappings and retained all tested pause clauses, so they remain explicit options. The one-speaker sample does not establish a general accuracy or latency recommendation, and neither mapping becomes the default based on API documentation or this bounded evaluation alone. + +Any configured raw `speech_noise_threshold` must be finite so it can round-trip through Settings JSON. The provider's `-1.0`–`1.0` range is enforced only when that threshold is effective for the active Audio3 custom preset; finite dormant out-of-range values remain preserved under named presets and inactive providers. + +## Timestamp schema and safe use + +Audio3 streaming returns timestamps without a request opt-in. + +Confirmed `result-generated` fields under `payload.output.sentence`: + +| Field | Shape and meaning | +| --- | --- | +| `sentence_id` | Integer sequence identifier. Normal results begin at 1 and increment; heartbeat results use 0. | +| `begin_time` | Integer sentence start in milliseconds. | +| `end_time` | Integer sentence end in milliseconds; it may be `null` for an intermediate result. | +| `sentence_end` | `false` for intermediate output and `true` for a final sentence. | +| `words[]` | Timestamped word/segment entries containing integer-millisecond `begin_time` and `end_time`, plus text and punctuation. | + +The Chinese documentation describes `words[]` as character-level in one place, while its examples contain multi-character entries and the English documentation calls it word-level. Voice Input will call these entries timed units and will not promise per-character timestamps. It will not retain a second copy of their text or punctuation. + +A heartbeat result has `heartbeat = true` and `sentence_id = 0`; it must be discarded before transcript or timestamp processing. + +Official documentation does not guarantee: + +- that every partial and final revision for one sentence keeps the same `sentence_id`; +- that partial text, times, or timed units cannot be revised; +- that a final event cannot repeat; +- correction, retraction, reconnect, replay, or idempotency semantics. + +Milestone 2 item 6 now implements the initial safe subset locally: + +- a borrowed typed parser for normal `result-generated` events, with heartbeat filtering before transcript or timestamp event construction; +- positive-integer `sentence_id`, integer `begin_time`, and explicit integer-or-null partial `end_time` validation; final `end_time` must be an integer, and missing required bounds reject all timed units while preserving text; +- partial results with explicit null `end_time` accept only nonoverlapping units whose starts are at or after the sentence start, with no upper bound; results with valid integer sentence bounds require every accepted unit to remain inside those bounds; +- a limit of 512 processed timed units per result, with every excess array entry counted as truncated through saturating counters; +- a text-free numeric telemetry delta for every normal result and best-effort diagnostics persistence; +- diagnostics schema 4 counters for timestamp-bearing results, accepted timed units, truncated timed units, and results with rejected timestamp metadata, plus the latest event-reported valid numeric audio-relative end time; schema-3 snapshots remain readable; +- unchanged transcript event/output assembly for partial, segment-final and authoritative `task-finished` text. + +The rejection counter's unit is one normal result whose timestamp block contains any invalid scalar, relationship, `words` shape, or processed unit. A malformed result increments it exactly once regardless of the number of defects; absent metadata, valid metadata, and truncation alone increment it zero times. The latest valid end is overwritten by each later event that supplies one, because partial revisions are not assumed to be monotonic across events. + +The parser borrows transcript text only long enough to apply the existing 16 KiB transcript bound and copy it into the unchanged event type. It retains at most 512 begin/end candidates. Unknown timed-unit fields, including text and punctuation, are consumed with Serde's `IgnoredAny` without copying or retaining their values; entries beyond the candidate limit are also ignored after counting. Complete WebSocket messages and frames are capped at 1 MiB. Messages beyond that transport cap remain protocol errors, while timestamp-array overflow within an accepted message is semantically truncated and never drops otherwise valid transcript text. `sentence_id` is validated and then discarded. No provider event-shape live observation was performed as part of this local implementation. + +Duplicate-final suppression and correction replacement remain blocked unless Alibaba publishes a stable revision-identity contract or an implementation is proven correct even when IDs and ranges are reused or revised. A finite live capture can validate parser compatibility and reveal counterexamples, but it cannot establish an undocumented identity guarantee. Reconnect and replay remain Milestone 3 and will not be implemented here. + +## Regional and workspace endpoints + +Both Audio3 model variants are officially available in Beijing and Singapore with unchanged model IDs. + +### Canonical endpoint matrix + +| API | Region | Legacy endpoint | Current workspace endpoint template | +| --- | --- | --- | --- | +| Streaming WebSocket | Beijing | `wss://dashscope.aliyuncs.com/api-ws/v1/inference` | `wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference` | +| Streaming WebSocket | Singapore | `wss://dashscope-intl.aliyuncs.com/api-ws/v1/inference` | `wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/inference` | +| Native HTTP | Beijing | `https://dashscope.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | `https://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | +| Native HTTP | Singapore | `https://dashscope-intl.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | `https://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api/v1/services/aigc/multimodal-generation/generation` | + +Alibaba recommends workspace-specific hosts and states that legacy hosts remain usable. Current workspace routing places Workspace ID in the hostname's leftmost label. It is not a query parameter, path component, or request-body field. + +The legacy WebSocket documentation also describes an optional `X-DashScope-WorkSpace` header. The Native HTTP reference does not describe that header and instead documents workspace-specific hosts. Voice Input will use one consistent hostname-derived mechanism for both APIs rather than inventing Native header behavior. + +### Configuration and migration decisions + +- Add `regional` and `custom` endpoint modes. +- Add explicit `beijing` and `singapore` regions. +- In regional mode, an empty Workspace ID uses the official legacy endpoint; a configured Workspace ID derives the reviewed workspace hostname. +- Existing exact Beijing legacy endpoint pairs migrate to regional Beijing with an empty Workspace ID. +- Existing exact Singapore legacy endpoint pairs migrate to regional Singapore with an empty Workspace ID. +- Any noncanonical, mixed, proxied, loopback, query-bearing, or otherwise overridden pair migrates to custom and remains byte-for-byte preserved. +- Regional hosts are selected from constants. Region input is never concatenated into a host. +- Workspace ID has no published provider grammar or length. Voice Input treats it as opaque and applies only the RFC-compatible DNS-label constraints required to place it safely in a hostname. These constraints are transport validation, not a claim about Alibaba's business identifier format. + +API keys are region-specific and workspace-scoped. A key cannot be assumed to work across regions or workspaces. Voice Input will retain one encrypted Alibaba credential, warn that changing region/workspace may require replacing it, and never probe another region automatically. + +### Deterministic implementation state + +Milestone 2 item 7 is implemented and validated locally without network access: + +- kebab-case `regional`/`custom` endpoint modes and `beijing`/`singapore` regions use Regional Beijing for new configurations; +- presence-aware migration recognizes only the two exact canonical legacy pairs and preserves any explicitly configured Workspace ID exactly (absent or empty stays empty); all mixed, workspace-host, loopback, proxy, custom path/port/query, and otherwise changed pairs remain Custom byte-for-byte, without inferring a Workspace ID; +- one pure typed resolver selects reviewed constants for both Streaming and Native, uses legacy hosts for empty workspaces, and constructs the reviewed regional workspace host only after value-free DNS-label transport validation; +- Custom mode ignores its dormant Workspace ID, while Regional mode ignores and preserves dormant custom URLs; neither mode sends a workspace header, query field, or request-body field; +- production WebSocket and Native requests use the resolved target, while authorization, request bodies, response sanitization, redirect behavior, models, and controls remain unchanged; +- normal Settings displays mode, region, optional Workspace ID, and the credential-scope warning without displaying a derived URL; raw URLs are shown only for Custom routing or a routed validation error; +- schema 4 diagnostics add only endpoint mode, region, and a Workspace-ID-configured boolean. They never include the Workspace ID, endpoint/host, model, or key, and schema-3/schema-4 compatibility defaults remain readable. + +Authorized live canaries subsequently succeeded for the Beijing Regional empty-workspace Streaming and Native routes. Singapore and workspace-specific routes remain untested because no matching scoped credential is configured. The implementation and Beijing canaries do not establish Singapore feature parity or workspace-route success. + +Official pages do not provide a complete per-region matrix for language hints, heartbeat, vocabulary, sentence controls, thresholds, and timestamp behavior. The shared API reference documents these controls without a regional exclusion. Each model, field combination, and scenario used in Singapore still requires its own authorized live validation; the result will not be presented as complete feature parity. + +## Implementation and live-test gates + +### Implement deterministically + +- recognition preset/config migration and an effective-control resolver; +- exact request-envelope fields for confirmed VAD controls; +- bounded timestamp parsing and aggregate-only diagnostics; +- regional/custom endpoint migration and pure endpoint resolution; +- hostname-based workspace routing; +- safe settings controls and privacy regression tests. + +### Block pending evidence + +- separate speech/noise threshold fields; +- guaranteed character-level timing; +- identity-dependent timestamp duplicate suppression or correction replacement without an official contract or an algorithm that remains correct under revisions; +- inferred Workspace ID business grammar; +- automatic credential-region probing; +- claims of complete regional feature parity. + +### Authorized live-validation status + +Completed within the bounded scope documented in [`qwen-audio3-milestone2-evaluation.md`](qwen-audio3-milestone2-evaluation.md): + +- Low-latency and Long-form field combinations and observable segmentation effects across a private pause/noise corpus; +- aggregate-only parser compatibility observation for live partial/final timestamp metadata, without retaining sentence IDs or inferring a revision identity contract; +- Beijing Regional empty-workspace Streaming and Native canaries. + +Still requires a matching scoped credential and separate authorization: + +- Singapore model/control scenarios, including dynamic vocabulary; +- workspace-specific Streaming and Native routing; +- any broader regional feature-parity claim. + +## Official sources + +- [Realtime recognition client events](https://help.aliyun.com/zh/model-studio/fun-asr-client-events) +- [Realtime recognition server events](https://help.aliyun.com/zh/model-studio/fun-asr-server-events) +- [Realtime WebSocket API](https://help.aliyun.com/zh/model-studio/fun-asr-realtime-websocket-api) +- [Realtime recognition guide](https://help.aliyun.com/zh/model-studio/real-time-speech-recognition-user-guide) +- [Native Flash API](https://help.aliyun.com/zh/model-studio/non-real-time-speech-recognition-for-fun-asr-flash) +- [Native recognition guide](https://help.aliyun.com/zh/model-studio/non-realtime-speech-recognition-user-guide) +- [ASR model specifications](https://help.aliyun.com/zh/model-studio/asr-model/) +- [Model Studio regional Base URLs](https://help.aliyun.com/zh/model-studio/base-url) +- [Obtain Workspace ID](https://help.aliyun.com/zh/model-studio/obtain-the-app-id-and-workspace-id) +- [API key acquisition](https://help.aliyun.com/zh/model-studio/get-api-key) +- [Workspace permission management](https://help.aliyun.com/zh/model-studio/permission-management-overview) diff --git a/docs/qwen-audio3-milestone2-evaluation.md b/docs/qwen-audio3-milestone2-evaluation.md new file mode 100644 index 0000000..285f236 --- /dev/null +++ b/docs/qwen-audio3-milestone2-evaluation.md @@ -0,0 +1,87 @@ +# Qwen-Audio-3 Milestone 2 Evaluation + +Evaluation date: 2026-08-05 + +Candidate commit: `8c11b9e` + +This report contains aggregate results only. It does not include audio, reference text, normal or refined transcripts, provider messages, credentials, endpoints, Workspace IDs, model names, window/application data, or prompt context. Source recordings, generated WAV files, per-request transcripts, errors, detailed per-file construction metadata, and raw per-session diagnostics remain in a private local directory excluded from Git. + +## Scope + +The authorized evaluation covered: + +- Standard, Low-latency dictation, and Long-form Streaming presets; +- exact inserted digital pauses, natural multi-sentence speech, stable background noise, one impulse-noise sample, silence, and noise-only input; +- one real microphone session per preset; +- one live cancellation canary; +- bounded timestamp parser compatibility; +- Beijing Regional routing with an empty Workspace ID for Streaming and Native. + +Singapore and workspace-specific routing were not attempted because no matching region/workspace credential was configured. Their live status remains unknown. + +## Fixed private corpus + +Twelve private source recordings produced 17 fixed WAV files. Four independently recorded clause fragments were trimmed with a local RMS threshold and combined into nine pause samples. The inserted digital silence matrix was: + +- Chinese: 250, 450, 800, 1,100, 1,600, and 2,200 ms; +- English: 450, 800, and 1,600 ms. + +Inserted digital silence duration is exact and reproducible. The acoustic boundary between the last speech sound and first subsequent speech sound still depends on threshold-based trimming, so these files support controlled relative comparison rather than instrument-grade VAD threshold calibration. The corpus also included two short utterances, two natural multi-sentence utterances, speech with stable background noise, speech with one impulse noise, silence, and noise-only input. + +The 17 WAV files totaled 96.97 seconds. Identical files were replayed once under each preset, totaling 290.90 input seconds and 51 requests. All 51 requests succeeded without timeout or retry. + +### Aggregate results + +Latency includes local command startup and rapid prerecorded-audio submission. It is not realtime endpoint latency. + +| Preset | Successful | Speech nonempty | Silence/noise false positive | Normalized CER | Latency p50 | Latency p95 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Standard | 17/17 | 15/15 | 0/2 | 2.31% | 1,141 ms | 1,911 ms | +| Low-latency dictation | 17/17 | 15/15 | 0/2 | 1.28% | 1,189 ms | 2,411 ms | +| Long-form | 17/17 | 15/15 | 0/2 | 3.08% | 1,104 ms | 1,933 ms | + +All nine pause samples retained both clause portions under all three presets. Each preset produced an exact normalized reference match on 5/9 pause files. The other results contained ordinary recognition differences of 1–5 normalized character edits; no complete clause was dropped or duplicated. Both natural multi-sentence samples and the impulse-noise speech sample matched exactly under all presets. The stable-background-noise speech sample had the same small recognition difference under all presets. Pure silence and noise-only input remained empty under all presets. + +One replay per condition cannot separate preset effects from provider variability. The lower Low-latency CER in this corpus is therefore an observation, not a general accuracy claim. Rapid replay also cannot establish that one preset has lower live endpoint latency. + +## Real microphone sessions + +One controlled toggle-mode microphone session was completed under each preset. Each session contained two spoken clauses separated by a natural approximately 1–2 second pause. Diagnostics are measured from recording start; first-nonempty latency therefore includes user timing before and during the first clause. + +| Preset | Audio sent | Ready | First nonempty partial | Segment finals | Finalize after stop | Timestamp-bearing results | Accepted timed units | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| Standard | 8,948 ms | 136 ms | 1,847 ms | 2 | 652 ms | 27 | 38 | +| Low-latency dictation | 8,564 ms | 130 ms | 1,665 ms | 3 | 328 ms | 27 | 37 | +| Long-form | 8,564 ms | 127 ms | 2,188 ms | 2 | 349 ms | 19 | 31 | + +All three sessions completed through Streaming and skipped the adaptive Native final pass as healthy short streams. Across the three sessions, the bounded parser accepted 73 timestamp-bearing results and 106 timed units. It rejected zero timestamp-bearing results and truncated zero units. These observations establish compatibility with the event shapes received during this evaluation. They do not establish a stable revision identity contract, monotonic sentence IDs, or character-level alignment. + +Each preset has only one realtime sample, and the spoken timing was human-controlled. The apparent finalize difference is not sufficient for a stable median, p95, or recommendation. + +## Cancellation + +A live Long-form session was cancelled with F8 while toggle-mode recording was active. No final text was delivered. The daemon journal recorded the `cancel` control and an immediate transition to idle, with no output operation before the next session. The single-session diagnostics slot was subsequently replaced by the user's next normal recording, so no cancelled-session diagnostic snapshot is claimed. Deterministic cancellation tests continue to provide exhaustive state-machine coverage, including suppression of Native final pass and output. + +## Regional route canaries + +With Regional Beijing selected and Workspace ID empty, one private WAV completed successfully through each API: + +| Route | Result | Command elapsed | +| --- | --- | ---: | +| Beijing Streaming legacy host | Success | 1,364 ms | +| Beijing Native legacy host | Success | 984 ms | + +The existing encrypted credential was not probed against Singapore or a workspace host. No automatic fallback or cross-region attempt was made. + +## Approximate request cost + +Using the Beijing rates already recorded for Milestone 1—¥0.00033 per Streaming input second and ¥0.00022 per Native input second—the instrumented corpus, protocol/route canaries, and three completed realtime sessions account for approximately ¥0.113. The short cancellation input adds only a small uninstrumented amount. This estimate excludes free quota and is not billing evidence. + +## Decisions + +1. **Keep Standard as the default.** It preserves the pre-milestone request and completed every evaluated scenario without a silence/noise false positive. +2. **Keep Low-latency dictation as an explicit option.** Its field combination was accepted and it preserved all pause clauses. Its lower corpus CER and shorter one-sample finalize time are bounded observations, not general recommendations. +3. **Keep Long-form as an explicit option.** Its field combination was accepted, semantic punctuation produced valid results, and it preserved all pause clauses. This corpus does not show a general accuracy advantage. +4. **Keep Custom controls and validation unchanged.** The provider documentation still exposes one optional combined speech/noise threshold; separate threshold fields remain unsupported. +5. **Retain bounded timestamp parsing without identity-based transcript changes.** Live events were parser-compatible, but finite captures cannot prove stable revision identity semantics. +6. **Mark Beijing empty-workspace routing live-validated for the exercised Streaming and Native calls.** Singapore and workspace-specific routes remain pending a credential scoped to those routes; no feature-parity claim is made. diff --git a/docs/qwen-audio3-optimization-checklist.md b/docs/qwen-audio3-optimization-checklist.md index 3bb7098..13995d1 100644 --- a/docs/qwen-audio3-optimization-checklist.md +++ b/docs/qwen-audio3-optimization-checklist.md @@ -72,25 +72,31 @@ Complete items 1–4, then stop and evaluate their combined effect before starti ## Milestone 2 — Advanced recognition controls +Provider-facing decisions, confirmed fields, endpoint constants, ambiguities, and stop conditions are recorded in `docs/qwen-audio3-milestone2-api-decisions.md`. + ### 5. VAD and sentence-boundary settings -- [~] Foundations only: validated `max_sentence_silence` and semantic-punctuation fields are included in Milestone 1 closeout at user direction. Multi-threshold mode and speech/noise thresholds remain unimplemented and must be added only where officially supported. -- [ ] Define presets for low-latency dictation and longer-form speech instead of exposing unexplained raw values by default. -- [ ] Complete pause/noise evaluation across endpoint latency, long pauses, background noise, cancellation, and final transcript assembly. +- [x] API verification, deterministic runtime implementation, and bounded live evaluation complete. Named preset resolution, presence-aware migration, documented `multi_threshold_mode_enabled`, optional `speech_noise_threshold`, universal finite-threshold validation for Settings JSON serialization, active-custom effective range/combination validation, request omission rules, Settings controls, diagnostics, privacy coverage, and bilingual documentation are implemented. Separate speech/noise threshold fields do not exist in the Audio3 documentation and remain blocked. +- [x] Define `standard`, `low-latency-dictation`, `long-form`, and `custom` presets instead of exposing unexplained raw values by default. Standard preserves the existing request and remains the default. The `400` ms Low-latency and `1300` ms semantic Long-form mappings are retained as explicit options after bounded evaluation, without a general accuracy or latency recommendation. +- [x] Complete a private fixed-input pause/noise evaluation across rapid-replay transcript assembly, exact inserted 250–2200 ms digital pauses, natural multi-sentence speech, background and impulse noise, silence/noise-only false positives, one realtime session per preset, and one live cancellation canary. Limitations and aggregate results are recorded in `docs/qwen-audio3-milestone2-evaluation.md`. ### 6. Parse and use timestamps -- [ ] Parse sentence and word/character timestamps without changing transcript output. -- [ ] Validate monotonicity and bound stored in-memory metadata. -- [ ] Use timestamps for duplicate suppression, partial/final assembly, and future reconnect recovery. -- [ ] Keep timestamp diagnostics aggregate-only and transcript-free. +- [x] Bounded parsing and aggregate diagnostics are complete and live parser compatibility was observed. Three controlled sessions supplied 73 timestamp-bearing results and 106 accepted timed units with zero rejected or truncated units. Identity-dependent assembly remains blocked. +- [x] Parse optional sentence and timed word/segment ranges without changing transcript output; the provider does not guarantee one timestamp per Unicode character. Missing or rejected timestamp metadata does not alter otherwise valid text. +- [x] Validate positive integer sentence IDs, required sentence bounds, unit ranges and event-local unit ordering without assuming monotonic revisions. Partial null ends use a sentence-start lower bound with no upper bound; integer sentence bounds contain all accepted units. Process at most 512 timed units per result and count every excess entry as truncated with saturating aggregate counters. +- [x] Parse normal results through a borrowed typed visitor that retains at most 512 numeric begin/end candidates and skips timed-unit text, punctuation, unknown fields, and excess entries with `IgnoredAny`. The existing transcript remains bounded to 16 KiB. WebSocket complete messages and frames use an explicit 1 MiB transport cap; larger messages remain protocol errors, while accepted timestamp overflow never drops text. +- [!] Keep transcript assembly unchanged initially. Identity-dependent duplicate suppression and correction replacement require an official stable revision contract or an algorithm proven correct when IDs/ranges are revised; finite live captures cannot establish that guarantee. Reconnect/replay remains Milestone 3. +- [x] Keep timestamp diagnostics aggregate-only and transcript-free. Schema 4 records timestamp-bearing result, accepted-unit and truncated-unit counts, plus one rejected-result count for each normal result containing any timestamp metadata defect. Truncation alone is not rejection. The latest valid numeric end is overwritten when a later event supplies one; no cross-event monotonicity is assumed. +- [x] Observe authorized live Audio3 partial/final timestamp metadata for parser compatibility and counterexample discovery only. Aggregate diagnostics showed no parser rejection in the bounded sample; no stable revision identity contract is inferred from finite captures, and sentence IDs/text were not retained. ### 7. Workspace regional endpoints -- [ ] Add explicit Beijing/Singapore region selection and derive official workspace endpoints safely. -- [ ] Validate workspace identifiers and regional credential expectations without logging endpoint or credential values. -- [ ] Preserve custom endpoints for testing and compatibility. -- [ ] Document migration from legacy DashScope endpoints. +- [~] Deterministic implementation, migration, Settings, diagnostics, privacy coverage, and offline validation are complete. Authorized Beijing Regional empty-workspace Streaming and Native canaries succeeded. Item 7 remains in progress because Singapore and workspace-specific routes require a matching scoped credential; no Singapore feature-parity or workspace live-success claim is made. +- [x] Add explicit Beijing/Singapore region selection and derive official Streaming and Native workspace endpoints from fixed reviewed constants. +- [x] Validate the optional Workspace ID only as a DNS-label hostname transport constraint and document regional/workspace credential expectations without logging or diagnosing endpoint, host, Workspace ID, model, or credential values. +- [x] Preserve custom Streaming and Native endpoints byte-for-byte for testing and compatibility; Custom ignores its dormant Workspace ID and sends no workspace header/query/body field. +- [x] Document presence-aware migration from exact canonical legacy DashScope endpoint pairs while preserving an explicitly configured Workspace ID exactly; an absent or empty value remains empty, and mixed, workspace-host, loopback, proxy, and custom path/port/query pairs remain Custom without Workspace ID inference. ## Milestone 3 — Context, resilience, and separate workflows @@ -120,6 +126,11 @@ Complete items 1–4, then stop and evaluate their combined effect before starti | 2026-08-02 | Milestone 1 live evaluation | Completed with documented limits | Real streaming/native APIs accepted the new controls. Seven controlled, transcript-free diagnostic samples covered short Chinese/English/mixed speech, configured technical terms, silence, repeated noise, long speech, and silence beyond 60 seconds. Healthy short streams skipped native; empty and long cases invoked it; no fallback or ASR failure occurred. One of two noise attempts produced a false positive. Aggregate timings, invocation rate, approximate Beijing pricing, decisions, and limitations are in `docs/qwen-audio3-milestone1-evaluation.md`. | | 2026-08-05 | Milestone 1 closeout | Completed locally; no new live evaluation | Added deterministic production-used Audio3 lifecycle coverage for 65 simulated heartbeat-enabled seconds and cancellation, corrected terminal/committed-only streaming telemetry, made malformed persisted provider error codes degrade to unavailable, and advanced diagnostics to schema v3. Included validated `max_sentence_silence` and semantic-punctuation configuration foundations at user direction; Milestone 2 item 5 remains in progress because presets, multi-threshold/noise controls, and full pause/noise evaluation are still outstanding. | | 2026-08-05 | Fixed-corpus vocabulary A/B | Completed | Retained a private local Git-excluded corpus with 30 clips and replayed identical audio under no vocabulary, weight 5, and weight 50 in Streaming and Native. All 180 requests succeeded. Weight 50 produced 10/10 recall for each tested term in both modes; weight 5 showed no recall uplift. Streaming had no negative-control insertion, while Native weight 50 inserted a configured term for one deliberately sound-alike control. Only aggregate results are committed. | +| 2026-08-05 | Milestone 2 Phase 0 API verification | Completed; implementation not started | Confirmed the Audio3 VAD field schema, timestamp paths and units, Beijing/Singapore endpoint matrix, hostname-based workspace routing, and region/workspace credential scope from official documentation. Separate speech/noise thresholds and guaranteed character timestamps are unsupported. Identity-dependent timestamp deduplication remains blocked without an official revision contract or a revision-safe algorithm; finite captures are observational only. Workspace ID has no published business grammar, so only hostname transport safety can be validated. Decisions and live-test gates are in `docs/qwen-audio3-milestone2-api-decisions.md`. | +| 2026-08-05 | Milestone 2 item 5 deterministic implementation | Completed | Added presence-aware preset migration, pure effective-control resolution, universal finite-threshold validation for Settings JSON serialization, active-custom range/combination validation, exact streaming request omission behavior, custom-only advanced Settings controls, effective diagnostics summaries, deterministic migration/request/settings/privacy tests, sample configuration, and bilingual documentation. Standard remains unchanged and is the default. | +| 2026-08-05 | Milestone 2 item 6 bounded timestamp implementation | Parsing, aggregate diagnostics, and bounded live compatibility observation completed; identity use blocked | Added borrowed typed Audio3 sentence/timed-unit parsing with a 512-candidate per-result bound, a 1 MiB WebSocket transport cap, required-bound and event-local range validation, per-malformed-result rejection counting, event-latest overwrite semantics, saturating aggregate-only telemetry, best-effort daemon persistence, and diagnostics schema 4 with schema-3 compatibility. Transcript event and authoritative task-finished assembly remain unchanged. Timed-unit text, punctuation and sentence IDs are discarded before telemetry. Three controlled live sessions accepted 73 timestamp-bearing results and 106 timed units without rejection or truncation; finite captures do not establish an identity contract. | +| 2026-08-05 | Milestone 2 bounded live evaluation | Items 5 and 6 completed within documented limits; Beijing portion of item 7 validated | Replayed 17 fixed private WAV files under three presets for 51/51 successful Streaming requests; all pause clauses were retained, silence/noise-only stayed empty, and one realtime session per preset plus one cancellation canary completed. Aggregate results, one-sample latency limits, parser observations, cost estimate, and decisions are in `docs/qwen-audio3-milestone2-evaluation.md`. | +| 2026-08-05 | Milestone 2 item 7 deterministic regional/workspace routing | Implemented; Beijing empty-workspace live canaries passed; other scoped routes pending | Added presence-aware exact-pair migration, typed Regional/Custom and Beijing/Singapore configuration, fixed-constant Streaming/Native endpoint resolution, hostname-only workspace routing with value-free DNS-label transport errors, Custom byte preservation, active-provider isolation, normal Settings controls and credential warnings, schema-4-safe routing summaries, bilingual documentation, and offline resolver/request/privacy tests. Authorized Beijing Regional empty-workspace Streaming and Native calls succeeded. Singapore and workspace-specific calls were not attempted without a matching credential. | ## Official references diff --git a/src/app.rs b/src/app.rs index 66d8195..02e2663 100644 --- a/src/app.rs +++ b/src/app.rs @@ -505,6 +505,9 @@ mod tests { config.asr.alibaba_audio3.model = "SENTINEL_MODEL".into(); config.asr.alibaba_audio3.native_model = "SENTINEL_NATIVE_MODEL".into(); config.asr.alibaba_audio3.endpoint = "https://SENTINEL_ENDPOINT.example".into(); + config.asr.alibaba_audio3.native_endpoint = + "https://SENTINEL_NATIVE_ENDPOINT.example".into(); + config.asr.alibaba_audio3.workspace_id = "SENTINEL_WORKSPACE".into(); config.asr.alibaba_audio3.api_key = "SENTINEL_CREDENTIAL".into(); let mut snapshot = Snapshot::idle(&config); @@ -525,6 +528,8 @@ mod tests { "SENTINEL_MODEL", "SENTINEL_NATIVE_MODEL", "SENTINEL_ENDPOINT", + "SENTINEL_NATIVE_ENDPOINT", + "SENTINEL_WORKSPACE", "SENTINEL_CREDENTIAL", "SENTINEL_TOOLTIP", "SENTINEL_TRANSCRIPT", @@ -547,7 +552,7 @@ mod tests { let config = Config::default(); let unavailable = SupportPayload::new(&config, None); let text = format_diagnostics(&unavailable, OutputFormat::Text).unwrap(); - assert!(text.starts_with("Voice Input diagnostics (schema 3)\n")); + assert!(text.starts_with("Voice Input diagnostics (schema 4)\n")); assert!(text.contains("Runtime: unavailable\n")); assert!(text.contains("Session: none\n")); @@ -580,7 +585,7 @@ mod tests { let json = format_diagnostics(&payload, OutputFormat::Json).unwrap(); let parsed: serde_json::Value = serde_json::from_str(&json).unwrap(); - assert_eq!(parsed["schema_version"], 3); + assert_eq!(parsed["schema_version"], 4); assert_eq!(parsed["session"]["session_id"], 7); assert_eq!(parsed["session"]["asr_outcome"], "completed"); assert!(parsed["session"].get("outcome").is_none()); diff --git a/src/backend.rs b/src/backend.rs index 2152d0a..5113510 100644 --- a/src/backend.rs +++ b/src/backend.rs @@ -53,6 +53,15 @@ impl AsrControl { } } +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct TimestampDiagnosticsDelta { + pub timestamp_bearing_result_count: u64, + pub accepted_timed_unit_count: u64, + pub result_with_rejected_timestamp_metadata_count: u64, + pub truncated_timed_unit_count: u64, + pub latest_valid_audio_end_ms: Option, +} + #[derive(Debug, Clone)] pub enum AsrEvent { Ready, @@ -67,6 +76,9 @@ pub enum AsrEvent { max_queue_delay_ms: u64, last_queue_delay_ms: u64, }, + TimestampDiagnostics { + delta: TimestampDiagnosticsDelta, + }, Partial { committed: String, unstable: String, diff --git a/src/backend/qwen_audio3/native.rs b/src/backend/qwen_audio3/native.rs index c81bdf7..82cffe9 100644 --- a/src/backend/qwen_audio3/native.rs +++ b/src/backend/qwen_audio3/native.rs @@ -20,6 +20,9 @@ pub(crate) fn transcribe_full_audio(config: &Config, wav_path: &Path) -> Result< bail!("Qwen-Audio-3 native ASR requires a configured credential"); } + let endpoints = audio3 + .resolve_endpoints() + .context("failed to resolve Qwen-Audio-3 routing")?; let wav_bytes = read_bounded_wav(wav_path)?; let body = request_body( &audio3.native_model, @@ -29,7 +32,7 @@ pub(crate) fn transcribe_full_audio(config: &Config, wav_path: &Path) -> Result< &audio3.vocabulary, )?; let response = http_client::post_json_sanitized( - audio3.native_endpoint.as_str(), + endpoints.native(), audio3.api_key.trim(), config.asr.connect_timeout_ms, audio3.native_timeout_ms, @@ -149,7 +152,7 @@ mod tests { MAX_RAW_AUDIO_BYTES, enforce_raw_audio_limit, parse_response, request_body, transcribe_full_audio, }; - use crate::config::{Audio3VocabularyTerm, Config, Language}; + use crate::config::{Audio3EndpointMode, Audio3VocabularyTerm, Config, Language}; #[test] fn request_body_matches_official_native_shape_and_data_uri() { @@ -295,6 +298,23 @@ mod tests { } } + #[test] + fn native_custom_endpoint_construction_errors_are_value_free() { + const ENDPOINT_SENTINEL: &str = "private native endpoint construction sentinel"; + let temp = tempfile::tempdir().unwrap(); + let wav_path = temp.path().join("input.wav"); + std::fs::write(&wav_path, b"wav").unwrap(); + let mut config = Config::default(); + config.asr.alibaba_audio3.api_key = "test-key".into(); + config.asr.alibaba_audio3.endpoint_mode = Audio3EndpointMode::Custom; + config.asr.alibaba_audio3.native_endpoint = ENDPOINT_SENTINEL.into(); + + let error = transcribe_full_audio(&config, &wav_path) + .expect_err("malformed custom native target must fail generically"); + assert_eq!(error.to_string(), "native HTTP request failed"); + assert!(!format!("{error:#}").contains(ENDPOINT_SENTINEL)); + } + #[test] fn loopback_timeout_is_bounded_and_does_not_expose_request_details() { const QUERY_SENTINEL: &str = "private-endpoint-query-sentinel"; @@ -316,6 +336,7 @@ mod tests { std::fs::write(&wav_path, AUDIO_SENTINEL).unwrap(); let mut config = Config::default(); config.asr.alibaba_audio3.api_key = CREDENTIAL_SENTINEL.into(); + config.asr.alibaba_audio3.endpoint_mode = Audio3EndpointMode::Custom; config.asr.alibaba_audio3.native_endpoint = endpoint.clone(); config.asr.alibaba_audio3.native_timeout_ms = 300; @@ -333,7 +354,10 @@ mod tests { #[test] fn loopback_request_uses_bearer_auth_and_exact_body() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); - let endpoint = format!("http://{}/native", listener.local_addr().unwrap()); + let endpoint = format!( + "http://{}/native?route=exact%2Fvalue", + listener.local_addr().unwrap() + ); let (request_tx, request_rx) = mpsc::channel(); let server = thread::spawn(move || { let (mut stream, _) = listener.accept().unwrap(); @@ -361,6 +385,8 @@ mod tests { term: "Voice Input".into(), weight: 5, }]; + config.asr.alibaba_audio3.endpoint_mode = Audio3EndpointMode::Custom; + config.asr.alibaba_audio3.workspace_id = "dormant-workspace".into(); config.asr.alibaba_audio3.native_endpoint = endpoint; config.asr.alibaba_audio3.native_model = "test-native-model".into(); @@ -372,12 +398,18 @@ mod tests { server.join().unwrap(); let (headers, body) = request.split_once("\r\n\r\n").unwrap(); - assert!(headers.starts_with("POST /native HTTP/1.1\r\n")); + assert!(headers.starts_with("POST /native?route=exact%2Fvalue HTTP/1.1\r\n")); assert!( headers .lines() .any(|line| line.eq_ignore_ascii_case("authorization: Bearer test-bearer-token")) ); + assert!( + !headers + .to_ascii_lowercase() + .contains("x-dashscope-workspace") + ); + assert!(!headers.contains("dormant-workspace")); let body: Value = serde_json::from_str(body).unwrap(); assert_eq!(body["parameters"]["language_hints"], json!(["ko", "en"])); assert_eq!(body["parameters"]["vocabulary"], json!({"Voice Input": 5})); diff --git a/src/backend/qwen_audio3/streaming.rs b/src/backend/qwen_audio3/streaming.rs index 0cb4083..291bada 100644 --- a/src/backend/qwen_audio3/streaming.rs +++ b/src/backend/qwen_audio3/streaming.rs @@ -1,4 +1,6 @@ use std::{ + borrow::Cow, + fmt, net::{TcpStream, ToSocketAddrs}, sync::{ Arc, @@ -10,6 +12,10 @@ use std::{ }; use anyhow::{Context, Result, anyhow, bail}; +use serde::{ + Deserialize, Deserializer, + de::{IgnoredAny, MapAccess, SeqAccess, Visitor}, +}; use serde_json::{Value, json}; use tungstenite::{ Message, WebSocket, @@ -21,8 +27,14 @@ use tungstenite::{ }; use crate::{ - backend::{ASR_CONTROL_QUEUE_CAPACITY, AsrControl, AsrEvent, AsrSessionHandle, AudioSpec}, - config::{Audio3VocabularyTerm, Config, Language}, + backend::{ + ASR_CONTROL_QUEUE_CAPACITY, AsrControl, AsrEvent, AsrSessionHandle, AudioSpec, + TimestampDiagnosticsDelta, + }, + config::{ + AlibabaAudio3Config, Audio3VocabularyTerm, Config, EffectiveAudio3RecognitionControls, + Language, + }, diagnostics::{FailureKind, ProviderErrorCode}, }; @@ -75,8 +87,12 @@ impl DeadlineClock for ProductionClock { } const MAX_CONTROLS_PER_TICK: usize = 8; -const MAX_SERVER_MESSAGE_BYTES: usize = 64 * 1024; +// Complete messages beyond this transport cap are protocol errors. Within the +// cap, timestamp overflow is handled semantically and never drops transcript +// text, so oversized timing arrays can be received and counted as truncated. +const MAX_SERVER_MESSAGE_BYTES: usize = 1024 * 1024; const MAX_TRANSCRIPT_BYTES: usize = 16 * 1024; +const MAX_TIMED_UNITS_PER_RESULT: usize = 512; static TASK_SEQUENCE: AtomicU64 = AtomicU64::new(1); pub(super) fn spawn_session(config: &Config, spec: AudioSpec) -> Result { @@ -118,7 +134,7 @@ fn run_session( // Preserve the existing startup budget: connection establishment consumes // time from the same deadline as the task-started wait. let connect_deadline = clock.deadline_after(connect_timeout); - let mut socket = open_socket(&audio3.endpoint, &audio3.api_key, connect_timeout)?; + let mut socket = open_socket(audio3, &audio3.api_key, connect_timeout)?; configure_socket(socket.get_mut())?; run_established_socket( &mut socket, @@ -160,6 +176,7 @@ fn run_established_socket( event_tx, } = session; let audio3 = &config.asr.alibaba_audio3; + let recognition = audio3.effective_recognition_controls(); send_json( socket, run_task_envelope( @@ -170,8 +187,7 @@ fn run_established_socket( language: config.asr.language, language_hints_enabled: audio3.language_hints_enabled, heartbeat_enabled: audio3.heartbeat_enabled, - max_sentence_silence_ms: audio3.max_sentence_silence_ms, - semantic_punctuation_enabled: audio3.semantic_punctuation_enabled, + recognition, vocabulary: &audio3.vocabulary, }, ), @@ -258,7 +274,11 @@ fn run_established_socket( ServerEvent::ResultGenerated { text, sentence_final, + timestamp_summary, } => { + let _ = event_tx.send(AsrEvent::TimestampDiagnostics { + delta: timestamp_summary.into(), + }); if sentence_final { let text = assembler.apply_segment_final(text); if !text.is_empty() { @@ -377,8 +397,12 @@ fn report_task_failure( bail!("Qwen-Audio-3 streaming ASR failed ({})", kind.as_str()) } -fn open_socket(endpoint: &str, api_key: &str, timeout: Duration) -> Result { - let request = websocket_request(endpoint, api_key)?; +fn open_socket( + audio3: &AlibabaAudio3Config, + api_key: &str, + timeout: Duration, +) -> Result { + let request = websocket_request_for_config(audio3, api_key)?; let websocket_config = WebSocketConfig { max_message_size: Some(MAX_SERVER_MESSAGE_BYTES), max_frame_size: Some(MAX_SERVER_MESSAGE_BYTES), @@ -388,14 +412,24 @@ fn open_socket(endpoint: &str, api_key: &str, timeout: Duration) -> Result