Add resilient Audio 3 streaming and local terminology

Cargo.toml

@@ -14,6 +14,7 @@ anyhow = "1.0.102"
 base64 = "0.22.1"
 dirs = "5.0.1"
 libc = "0.2.177"
+jieba-rs = "0.10.3"
 reqwest = { version = "=0.12.9", default-features = false, features = ["blocking", "json", "rustls-tls-webpki-roots"] }
 rustls = { version = "0.23.37", features = ["ring"] }
 serde = { version = "1.0.214", features = ["derive"] }

README.md

@@ -95,7 +95,7 @@ Migration is presence-aware. A configuration without `endpoint_mode` migrates to
 
 Alibaba API keys are region-scoped. Changing the region may require replacing the encrypted Alibaba credential. Voice Input never probes another region and never migrates a key automatically. Singapore availability does not establish feature parity: each model, control combination, and language/vocabulary scenario still requires authorized live validation.
 
-The streaming model supplies realtime text. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue.
+The streaming model supplies realtime text. On one recoverable transport interruption before `finish-task`, Voice Input creates a new Audio3 task, discards the old task's transcript, and replays retained PCM from the beginning at 4× realtime while recording continues. Retention is prefix-complete and limited by the configured recording duration, 300 seconds, and 10 MiB of PCM; exceeding the limit disables reconnect without evicting an audio prefix. A second interruption or a post-finish interruption uses the existing Native/local complete-audio recovery. **Language hints** and **streaming heartbeat** are independent opt-in settings and are disabled by default. Enabling language hints sends the existing language selection to Audio3: English uses `en`; Simplified and Traditional Chinese use `zh,en`; Japanese uses `ja,en`; and Korean uses `ko,en`. The extra English hint retains mixed-English recognition for Chinese, Japanese, and Korean; leaving the switch disabled preserves the provider's automatic detection. Enabling streaming heartbeat keeps long silent push-to-talk sessions alive while correctly formatted audio frames continue.
 
 **Recognition preset** defaults to **Standard**, which preserves the existing `800` ms maximum sentence silence with semantic punctuation and multi-threshold mode disabled and no speech/noise threshold. **Low-latency dictation** uses `400` ms with multi-threshold mode enabled; **Long-form** uses `1300` ms with semantic punctuation enabled. Both mappings were accepted in an authorized, one-speaker evaluation and retained both clauses across a matrix with 250–2200 ms of inserted digital silence; acoustic speech boundaries remained dependent on local RMS trimming. The bounded sample does not establish a general accuracy or latency recommendation, so Standard remains the default. See [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md). **Custom** exposes all raw controls; semantic punctuation and multi-threshold mode cannot be enabled together. Its optional speech/noise threshold must be finite and between `-1` and `1`; omission preserves provider behavior because Alibaba publishes no default. Settings displays every value that a custom request can send.
 
@@ -143,7 +143,7 @@ The Wiki also covers agent context, desktop integration, privacy, and developmen
 
 ## Privacy
 
-Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider; only when agent context is explicitly enabled does it also send a capped and redacted session excerpt. Window titles, process IDs, and raw desktop metadata are never included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection.
+Remote Qwen modes send audio to the selected Regional route or exact Custom Alibaba endpoint. LLM refinement sends the transcript and a coarse destination style through its system prompt (structured coding-agent Markdown, `instant-messaging`, or the default style) to the configured provider. When agent context is explicitly enabled, Voice Input redacts and caps the latest completed Pi or Codex assistant message locally, segments it with Jieba, deduplicates the result, and sends only a bounded terminology list. The source message, window titles, process IDs, and raw desktop metadata are not included in the LLM request. The public sample disables remote refinement and agent context. Voice Input performs no telemetry or analytics collection.
 
 ## Project status
 

README.zh-CN.md

@@ -95,7 +95,7 @@ Qwen-Audio-3 目前作为需要明确启用的实验性提供商使用。该选
 
 Alibaba API key 受区域范围约束。更改区域后，用户可能需要替换加密的 Alibaba 凭据。Voice Input 绝不会探测其他区域，也不会自动迁移 key。支持选择新加坡区域并不表示已经实现完整功能一致性；每个模型、控制项组合以及语言或词汇表场景仍需完成经过授权的在线验证。
 
-流式模型负责提供实时文本。**语言提示**和**流式 heartbeat** 是两个相互独立的选用设置，默认均为关闭。启用语言提示后，程序会把现有语言选项发送给 Audio3：英语使用 `en`；简体中文和繁体中文使用 `zh,en`；日语使用 `ja,en`；韩语使用 `ko,en`。中文、日语和韩语的额外英语提示用于保留英语混合识别；关闭该开关会保留服务商的自动检测行为。启用流式 heartbeat 后，只要程序继续发送格式正确的音频帧，它就能使长时间静音的按键说话 session 保持连接。
+流式模型负责提供实时文本。如果在发送 `finish-task` 前发生一次可恢复的传输中断，Voice Input 会创建新的 Audio3 task，使旧 task 的 transcript 失效，并且以 4 倍实时速度从头重放保留的 PCM，同时继续录音。保留的 PCM 必须包含完整前缀，其上限取配置的最大录音时长、300 秒和 10 MiB PCM 三者中的最小值；超过上限会停用重连，同时不会丢弃前缀后继续重放。第二次中断或发送 `finish-task` 后的中断会使用现有的 Native 或本地完整音频恢复。**语言提示**和**流式 heartbeat** 是两个相互独立的选用设置，默认均为关闭。启用语言提示后，程序会把现有语言选项发送给 Audio3：英语使用 `en`；简体中文和繁体中文使用 `zh,en`；日语使用 `ja,en`；韩语使用 `ko,en`。中文、日语和韩语的额外英语提示用于保留英语混合识别；关闭该开关会保留服务商的自动检测行为。启用流式 heartbeat 后，只要程序继续发送格式正确的音频帧，它就能使长时间静音的按键说话 session 保持连接。
 
 **识别预设**默认使用**标准**。该预设保留现有行为：最大句末静音时长为 `800` 毫秒，语义标点和多阈值模式均关闭，并且不发送语音/噪声阈值。**低延迟听写**使用 `400` 毫秒并启用多阈值模式；**长篇语音**使用 `1300` 毫秒并启用语义标点。经过授权的单说话人评估确认服务端接受这两个映射；在插入了 250–2200 毫秒数字静音的测试矩阵中，两者都保留了静音前后的内容。声学语音边界仍取决于本地 RMS 裁剪。有限样本无法形成通用的准确率或延迟建议，因此标准预设仍为默认值。详见 [`docs/qwen-audio3-milestone2-evaluation.md`](docs/qwen-audio3-milestone2-evaluation.md)。**自定义**会显示全部原始控制项；语义标点与多阈值模式不能同时启用。可选的语音/噪声阈值必须是 `-1` 到 `1` 之间的有限数值；Alibaba 未公布默认值，因此省略该字段可以保留服务商行为。Settings 会显示自定义请求可能发送的每一个值。
 
@@ -143,7 +143,7 @@ Wiki 还包含 Agent context、桌面集成、安全隐私和开发说明。
 
 ## 隐私
 
-远程 Qwen 模式会把音频发送到所选的区域路由或完全按原值使用的自定义 Alibaba 端点。LLM refinement 会把 transcript 和粗粒度的目标风格（coding agent 结构化 Markdown、`instant-messaging` 或默认风格）通过 system prompt 发送到配置的 provider；只有在用户明确启用 Agent context 时，才会额外发送经过截断与脱敏的会话片段。LLM 请求不会包含窗口标题、进程 ID 或原始桌面元数据。公开示例配置默认关闭远程 refinement 和 Agent context。Voice Input 不收集遥测或分析数据。
+远程 Qwen 模式会把音频发送到所选的区域路由或完全按原值使用的自定义 Alibaba 端点。LLM refinement 会把 transcript 和粗粒度的目标风格（coding agent 结构化 Markdown、`instant-messaging` 或默认风格）通过 system prompt 发送到配置的 provider。只有在用户明确启用 Agent context 时，Voice Input 才会在本地对最近一条已完成的 Pi 或 Codex assistant message 进行脱敏和截断，使用 Jieba 分词并去重，然后只发送数量受限的术语列表。LLM 请求不会包含 Agent source message、窗口标题、进程 ID 或原始桌面元数据。公开示例配置默认关闭远程 refinement 和 Agent context。Voice Input 不收集遥测或分析数据。
 
 ## 项目状态
 

assets/config.toml

@@ -72,6 +72,7 @@ model = ""
 timeout_ms = 15000
 provider_sort = ""
 agent_context_enabled = false
+# Local redacted source budget before segmentation; only bounded terminology is sent.
 agent_context_max_chars = 6000
 
 [hud]

assets/quickshell-settings/I18n.qml

@@ -195,7 +195,7 @@ QtObject {
         "Replace OpenRouter API key": "替换 OpenRouter API key",
         "Context": "上下文",
         "Use agent context": "使用 Agent 上下文",
-        "Send a redacted excerpt from the Pi or Codex session focused when dictation ends.": "发送 dictation 结束时聚焦的 Pi 或 Codex 会话中经过脱敏的片段。",
+        "Locally segment a redacted Pi or Codex excerpt and send only bounded, deduplicated terminology.": "在本地对经过脱敏的 Pi 或 Codex 片段进行分词，并且只发送数量受限且已去重的术语。",
         "Test refinement": "测试文本优化",
         "Test the current LLM draft and credential without saving it.": "无需保存即可测试当前 LLM 设置草稿和凭据。",
         "Testing…": "正在测试…",
@@ -207,7 +207,7 @@ QtObject {
         "Optional OpenRouter provider ordering expression.": "可选的 OpenRouter 提供商顺序表达式。",
         "Agent context": "Agent 上下文",
         "Context limit": "上下文限制",
-        "Maximum characters sent from a redacted agent-session excerpt.": "经过脱敏的 Agent 会话片段最多发送的字符数。",
+        "Maximum redacted agent-session characters (500–12000).": "本地分词前最多读取 500–12000 个经过脱敏的 Agent 会话字符。",
         "Clipboard delivery and input-method coordination.": "控制剪贴板粘贴和输入法协同。",
         "Delivery": "输出方式",
         "Mode": "模式",

assets/quickshell-settings/RefinementPage.qml

@@ -72,7 +72,7 @@ SettingsPage {
                 theme: root.theme
                 label: "Use agent context"
                 checked: root.controller.value("llm.agent_context_enabled", false)
-                help: "Send a redacted excerpt from the Pi or Codex session focused when dictation ends."
+                help: "Locally segment a redacted Pi or Codex excerpt and send only bounded, deduplicated terminology."
                 enabled: !root.controller.busy
                 onToggled: (checked) => {
                     return root.controller.setValue("llm.agent_context_enabled", checked);
@@ -154,7 +154,7 @@ SettingsPage {
                     theme: root.theme
                     label: "Context limit"
                     value: root.controller.value("llm.agent_context_max_chars", 6000)
-                    help: "Maximum characters sent from a redacted agent-session excerpt."
+                    help: "Maximum redacted agent-session characters (500–12000)."
                     error: root.controller.errorFor("llm.agent_context_max_chars")
                     enabled: !root.controller.busy
                     onEdited: (value) => {

docs/qwen-audio3-optimization-checklist.md

@@ -99,22 +99,25 @@ Provider-facing decisions, confirmed fields, endpoint constants, ambiguities, an
 - [x] Document presence-aware migration from exact canonical DashScope endpoint pairs; mixed pairs, noncanonical hosts, loopback endpoints, proxies, and custom path/port/query pairs remain Custom without host inference.
 - [x] Keep diagnostics schema 4 while reporting only endpoint mode and region; endpoint values and route identifiers remain private.
 
-## Milestone 3 — Context, resilience, and separate workflows
+## Milestone 3 — Streaming resilience and local refinement context
 
-### 8. Context enhancement and streaming reconnect
+Alibaba Session Context and Filetrans are deferred and are not part of this milestone.
 
-- [ ] Design an explicit-consent context model; never ingest clipboard, window, transcript history, or agent context automatically.
-- [ ] Bound and visibly disclose all context sent through initial requests or `continue-task`.
-- [ ] Design reconnect/replay with retained audio, timestamp-based deduplication, retry limits, and cancellation safety.
-- [ ] Add deterministic disconnect, replay, duplicate, timeout, and privacy tests before live testing.
+### 8. Audio3 streaming reconnect
 
-### 9. Filetrans file-transcription workflow
+- [x] Retain a prefix-complete, bounded in-memory PCM sequence while the first task is active; never persist retained PCM or include it in logs or diagnostics.
+- [x] On one recoverable pre-finish transport interruption, create a new Audio3 task, invalidate every transcript result from the old task, and replay retained PCM from the beginning at bounded speed while capture continues.
+- [x] Use only the replacement task's authoritative transcript. Do not merge transcripts across tasks or depend on sentence IDs or timestamps for deduplication.
+- [x] Preserve cancellation, finish-during-replay, retry, retention, and deadline safety, then use the existing full-audio Native/local recovery after terminal failure.
+- [~] Add deterministic disconnect, exact-replay, stale-transcript reset, second-failure, timeout, cancellation, retention-bound, and privacy tests before live testing. Deterministic tests are complete; an authorized live disconnect canary remains pending.
 
-- [ ] Keep Filetrans separate from push-to-talk voice input.
-- [ ] Define asynchronous task creation, polling/callback behavior, cancellation, file limits, retention, and cleanup.
-- [ ] Add optional diarization and speaker-count controls only for supported inputs.
-- [ ] Ensure local files and transcripts are never uploaded without an explicit command and confirmation.
-- [ ] Add a dedicated CLI/UI workflow, documentation, and privacy-safe diagnostics.
+### 9. Local terminology extraction for Refine
+
+- [x] Treat the latest completed Pi/Codex assistant message as the terminology source; do not derive correction vocabulary from the potentially incorrect ASR transcript.
+- [x] Redact the source locally before tokenization, then perform bounded local segmentation, stable deduplication, and filtering without sending the original unredacted message anywhere.
+- [~] Compare the previous bounded excerpt, a terminology-only payload, and a bounded excerpt plus terminology payload for payload size, extraction latency, terminology correction, and false replacement. Deterministic synthetic size/latency measurements are complete; the authorized private-corpus accuracy A/B remains pending.
+- [x] Keep context opt-in and untrusted. The Refine prompt must use extracted terms only when the transcript has a clear spoken match and must never follow instructions found in context.
+- [x] Record only aggregate evaluation results; never commit private session text, transcripts, paths, extracted terms, or provider responses. See `docs/refine-local-terminology-experiment.md`.
 
 ## Work log
 
@@ -132,6 +135,8 @@ Provider-facing decisions, confirmed fields, endpoint constants, ambiguities, an
 | 2026-08-05 | Milestone 2 item 6 bounded timestamp implementation | Parsing, aggregate diagnostics, and bounded live compatibility observation completed; identity use blocked | Added borrowed typed Audio3 sentence/timed-unit parsing with a 512-candidate per-result bound, a 1 MiB WebSocket transport cap, required-bound and event-local range validation, per-malformed-result rejection counting, event-latest overwrite semantics, saturating aggregate-only telemetry, best-effort daemon persistence, and diagnostics schema 4 with schema-3 compatibility. Transcript event and authoritative task-finished assembly remain unchanged. Timed-unit text, punctuation and sentence IDs are discarded before telemetry. Three controlled live sessions accepted 73 timestamp-bearing results and 106 timed units without rejection or truncation; finite captures do not establish an identity contract. |
 | 2026-08-05 | Milestone 2 bounded live evaluation | Items 5 and 6 completed within documented limits; Beijing portion of item 7 validated | Replayed 17 fixed private WAV files under three presets for 51/51 successful Streaming requests; all pause clauses were retained, silence/noise-only stayed empty, and one realtime session per preset plus one cancellation canary completed. Aggregate results, one-sample latency limits, parser observations, cost estimate, and decisions are in `docs/qwen-audio3-milestone2-evaluation.md`. |
 | 2026-08-05 | Milestone 2 item 7 deterministic regional routing | Implemented; Beijing live canaries passed; Singapore pending | Added presence-aware exact-pair migration, typed Regional/Custom and Beijing/Singapore configuration, fixed-constant Streaming/Native endpoint resolution, Custom byte preservation, active-provider isolation, normal Settings controls and region-scoped credential warnings, schema-4-safe routing summaries, bilingual documentation, and offline resolver/request/privacy tests. Authorized Beijing Regional Streaming and Native calls succeeded. Singapore was not attempted without a matching region credential. |
+| 2026-08-12 | Milestone 3 Audio3 streaming reconnect | Deterministic implementation completed; live disconnect canary pending | Added one bounded pre-finish replacement task, prefix-complete PCM retention, 4× replay while capture continues, authoritative transcript reset, finish/cancellation/retry safety, aggregate schema-4 diagnostics, and deterministic exact-replay/second-failure/protocol/task-failure/retention tests. Task transcripts are never merged and timestamps are not used for deduplication. |
+| 2026-08-12 | Refine local terminology extraction | Experimental implementation and synthetic measurements completed; private-corpus A/B pending | Replaced the remotely sent agent excerpt with locally redacted, Jieba-segmented, stably deduplicated terminology capped at 96 terms and 1,500 characters. The source remains the latest completed Pi/Codex assistant message, never the ASR transcript. Aggregate size, latency, and binary-size measurements are in `docs/refine-local-terminology-experiment.md`. |
 
 ## Official references