Skip to content

feat(dictation): mouse middle/side buttons with Hold refcount (split from #724)#727

Open
HKLHaoBin wants to merge 18 commits into
Open-Less:betafrom
HKLHaoBin:feat/718-mouse-dictation
Open

feat(dictation): mouse middle/side buttons with Hold refcount (split from #724)#727
HKLHaoBin wants to merge 18 commits into
Open-Less:betafrom
HKLHaoBin:feat/718-mouse-dictation

Conversation

@HKLHaoBin

@HKLHaoBin HKLHaoBin commented Jun 21, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

User description

摘要

Fixes #718（部分：鼠标触发）。

由已关闭的 #724 拆分而来（第 2/3 部分）：独立 prefs 开关；Win/macOS 全局鼠标 hook；Hold 模式 HoldSourceTracker 多源 refcount。Linux 输入走 PR3 evdev。可与 #726 并行审查/合并。

关联：#718、#724。

修复 / 新增 / 改进

  • 鼠标中键 / 侧键听写开关与 i18n
  • mouse_dictation + hold_source_tracker
  • 设置变更时 refresh_mouse_dictation

兼容

  • 不包含：side_aware_combo、侧向录制 UI、Linux evdev
  • Linux 鼠标/侧向组合需 PR3

测试计划

本地实机验证（回应 @jiangmuran CHANGES_REQUESTED）：

  • Win/macOS：开启中键/侧键 Toggle → Hold 与 Toggle 听写均可触发
  • Hold refcount：键盘 Hold + 鼠标 Hold 同时按下，分别释放 → 仅最后一个源释放后才结束
  • Android：cargo metadata --filter-platform aarch64-linux-android（stub 编译）
  • npm run build

请维护者：在真机验证鼠标触发；Linux 需 PR3 合并后复测。

PR Type

Enhancement, Tests

Description

  • Add mouse middle/side button dictation triggers with independent prefs

  • Implement HoldSourceTracker for multi-source hold release refcount

  • Integrate with Win/macOS platform hooks and coordinator

  • Add i18n and UI toggles for mouse dictation

  • Include mobile stubs and tests

Diagram Walkthrough

flowchart LR
  Input["Input Events (Keyboard/Mouse)"]
  Platform["Platform Hooks\nWindows/macOS low-level"]
  MouseDict["mouse_dictation\n(Middle/Side Button Press)"]
  HoldTracker["hold_source_tracker\n(Refcount Multiple Sources)"]
  Coordinator["Coordinator\n(Start/Stop Dictation)"]
  Input --> Platform
  Platform --> MouseDict
  MouseDict --> HoldTracker
  HoldTracker --> Coordinator
Loading

File Walkthrough
Relevant files
Tests
3 files
mod.rs
Add mouse dictation refresh mock                                                 
+5/-0     
mock-data.ts
Add mouse dictation fields to mock settings                           
+2/-0     
stylePrefs.test.ts
Add mouse dictation fields to test prefs                                 
+2/-0     
Enhancement
11 files
settings.rs
Add refresh_mouse_dictation to settings persistence           
+16/-0   
coordinator.rs
Integrate mouse_dictation and hold_sources into coordinator
 +181/-5 
dictation.rs
Refactor pressed/released edge to accept TriggerSource     
+44/-16 
hotkey_loops.rs
Add mouse dictation supervisor, bridge, hold source release
 +194/-6 
hold_source_tracker.rs
Implement HoldSourceTracker with refcount logic                   
+125/-0 
hotkey.rs
Dispatch mouse events on macOS event tap                                 
+12/-3   
lib.rs
Register mouse_dictation module and start listener             
+7/-0     
mouse_dictation.rs
Provide mobile stub for mouse dictation                                   
+38/-0   
mouse_dictation.rs
Implement global mouse dictation monitor (Win/macOS)         
+356/-0 
types.ts
Add mouse dictation fields to UserPreferences                       
+6/-2     
RecordingInputSection.tsx
Add mouse dictation toggle UI                                                       
+23/-0   
Configuration changes
1 files
types.rs
Add mouse_middle/side_button_dictation prefs                         
+16/-0   
Documentation
5 files
en.ts
Add English i18n for mouse dictation toggles                         
+4/-0     
ja.ts
Add Japanese i18n for mouse dictation toggles                       
+4/-0     
ko.ts
Add Korean i18n for mouse dictation toggles                           
+4/-0     
zh-CN.ts
Add Simplified Chinese i18n for mouse dictation toggles   
+4/-0     
zh-TW.ts
Add Traditional Chinese i18n for mouse dictation toggles 
+4/-0     

This was referenced Jun 21, 2026
Open
Closed
@github-actions

github-actions Bot commented Jun 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

(Review updated until commit 53450fc)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

718 - Partially compliant

Compliant requirements:

  • Mouse middle button dictation toggle added

Non-compliant requirements:

724 - Partially compliant

Compliant requirements:

  • Mouse middle/side button dictation triggers with independent prefs
  • HoldSourceTracker implementation
  • Windows low-level mouse hook and macOS CGEvent dispatch
  • UI toggles and i18n (en, zh-CN, zh-TW, ja, ko)
  • Mobile stubs for mouse_dictation
  • Unit tests for hold source tracking and coordinator hold scenarios

Non-compliant requirements:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Bug: Windows hook thread orphaned on monitor drop

When MouseDictationMonitor is dropped (e.g., user disables mouse dictation), the Windows low-level hook thread spawned by ensure_hook_thread is never signaled to exit. The thread continues running its message loop indefinitely, causing a resource leak. If the user later re-enables mouse dictation, HOOK_THREAD_STARTED prevents a new thread from being spawned, leaving the old (stale) hook active. No new hooks are installed on subsequent toggles, and mouse events stop being processed. The Drop impl (lines 64–75) only releases held sources and clears the global state but does not uninstall the hook or terminate the thread.

 
impl Drop for MouseDictationMonitor {
    fn drop(&mut self) {
        if let Some(slot) = ACTIVE_MOUSE.get() {
            if let Ok(mut guard) = slot.write() {
                if let Some(state) = guard.as_mut() {
                    release_held_sources(state, true, true);
                }
                *guard = None;
            }
        }
    }
}

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 71787ba

@HKLHaoBin

HKLHaoBin commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

稍等，我处理一下ci问题。

@Felix201209 Felix201209 mentioned this pull request Jun 21, 2026
Merged
6 tasks
Felix201209
Felix201209 requested changes Jun 21, 2026
View reviewed changes

@Felix201209 Felix201209 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

方向对、拆得也清楚，但有一个会影响使用的 HIGH bug 必须先修，所以暂时 request changes 🙏

🔴 HIGH：按住鼠标键时禁用鼠标听写 → 听写卡死直到重启

mouse_dictation.rs + coordinator/hotkey_loops.rs（update_mouse_dictation_binding_now）

复现（Hold 模式）：

  1. 按住鼠标中键 → Pressed → hold_sources.press(MouseMiddle) → count=1 → 听写开始；
  2. 此时在设置里把"中键听写"关掉；
  3. update_mouse_dictation_binding_now drop 掉 monitor（ACTIVE_MOUSE=None），但从不重置 hold_sources；
  4. 物理松开按键时 handle_button 已经收不到事件了 → handle_released_edge 永不触发 → active_count 永远停在 1；
  5. 听写会话一直跑到 app 重启。

修法（二选一）：update_mouse_dictation_binding_now 在禁用分支返回前调用 inner.hold_sources.reset()；或 update_config 清 middle_held/side_held 之前，对当前仍持握的源补发合成 HotkeyEvent::Released。

🟠 与 #726 必然冲突（已批准，建议本 PR rebase 去重）

本 PR 在 types.ts 给 HotkeyTrigger 加了 leftCommand/leftShift/rightShift，并在 5 个语言文件加了对应文案——#726 在同样位置加了同样的行。#726 已经 approve、排在前面合并，所以本 PR 需要 rebase 到最新 beta 并删掉这几处重复新增，否则二合一时会冲突。

其它（不阻断，建议一并处理）

  • 并发 press 的 TOCTOU（coordinator/dictation.rs）：press() 后又单独读 active_count()，键鼠几乎同时按下时两边都读到 2、都提前 return，begin_session 漏调。建议直接用 fetch_add 的返回旧值判断"是否首次按下"。
  • Linux 上鼠标听写 UI 可设但无效：mouse_dictation.rs 没有 Linux 平台实现，但 RecordingInputSection.tsx 在 Linux 仍显示开关。建议 toggle 加 os !== 'linux' 或 start() 在 Linux 返回错误（Linux 的鼠标触发本就归 #728 的 evdev）。
  • dead i18n：mouseMiddleDesc/mouseSideDesc 五语言都翻译了，但 SettingRow 没传 description，要么接上要么删掉。

测试缺口

hold_source_tracker / coordinator 的 happy path 测得不错，但缺：①上面"持握中禁用→卡死"的回归测试；②并发同时按下的 TOCTOU。建议至少给 HIGH 那条补个测试。

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 60672bf

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit df02187

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 645b35b

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 2135949

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 1bf48fe

@HKLHaoBin

HKLHaoBin commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

@Felix201209 已按 review 处理，请 re-review 🙏

HIGH（持握中禁用鼠标听写）

  • mouse_dictation 层：Drop / update_config 禁用路径补发 Released（21359494）
  • coordinator 层：新增 sync_release_mouse_hold_sources，在 update_mouse_dictation_binding_now 与 supervisor 禁用分支、take() 前同步释放 mouse hold 源并在最后一个源释放时 end_session（1bf48fec）

TOCTOU + debounce

  • HoldSourceTracker::press() 返回递增前的 count；handle_pressed_edge 用 prev_count == 0 判断是否首个源
  • debounce 丢弃时 rollback hold_sources / hotkey_trigger_held

其它

CI 请在 GitHub 上确认。

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit a841adc

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit e866c41

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit f718636

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 02a3b6f

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit c85b192

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 8027eca

@HKLHaoBin

HKLHaoBin commented Jun 21, 2026

Copy link
Copy Markdown
Contributor Author

@Felix201209 感谢 review 🙏 针对 review #4539504044 里的各点，最新 head 8027ecae 已全部处理，CI 绿。

🔴 HIGH：持握中禁用鼠标听写 → 卡死

采用你建议的「补发合成 Released」方案，并加了 coordinator 侧兜底：

  • mouse_dictation.rs：Drop / update_config 禁用路径在清 middle_held/side_held 前补发 HotkeyEvent::Released（21359494）
  • hotkey_loops.rs：新增 sync_release_mouse_hold_sources，在 update_mouse_dictation_binding_now、supervisor 禁用分支、以及 take() 前同步释放仍持握的 mouse 源；最后一个源释放时走 end_session（1bf48fec）

🟠 与 #726 重复

types.ts / 五语言 i18n 里 PR1 的 side-modifier 字段与文案已去掉，本 PR 只保留 mouse 相关（71787bad、60672bf7）。

其它建议

点 处理
TOCTOU press() 返回递增前 count；handle_pressed_edge 用 prev_count == 0 判首个源；debounce 丢弃时 rollback hold_sources / hotkey_trigger_held
Linux UI showMouseDictation = showDesktopHotkey && os !== 'linux'，等 #728 evdev
dead i18n SettingRow 已接 mouseMiddleDesc / mouseSideDesc

测试

  • hold_mode_mouse_disable_while_holding_ends_session — HIGH 回归
  • hold_mode_concurrent_press_starts_once — TOCTOU
  • hold_mode_sync_release_mouse_only_keeps_keyboard_hold — 只释 mouse 时 keyboard hold 不受影响（a841adce）
  • hold_mode_hotkey_rebind_while_holding_clears_sources_and_ends_session — 持握中改快捷键/模式也会清源（e866c414）

CI 侧 nested-runtime 问题也已修完（f7186364 → 8027eca），Windows / macOS / Linux / Android check 均 pass。

方便的话请再扫一眼，谢谢！

Felix201209
Felix201209 previously approved these changes Jun 21, 2026
View reviewed changes

@Felix201209 Felix201209 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

响应得又快又到位 👍 重新逐条核对后，上一轮提的问题全部解决，改判 approve ✅

  • 🔴 HIGH 卡死 bug — 已解决：monitor drop（mouse_dictation.rs 的 Drop → release_held_sources）、disable（sync_release_mouse_hold_sources 先清零再 .take()）、rebind mid-hold（update_hotkey_binding 首步调 clear_active_hold_sources_on_hotkey_rebind）三条路径都会补发 Released / 重置 refcount，且有状态机 + 幂等保护，不会再卡。
  • 🟠 与 #726 的冲突 — 已解决：types.ts 和五个语言文件里重复的 leftCommand/leftShift/rightShift 已移除，本 PR 现在只加 mouseMiddle*/mouseSide*，#726 先合后不会冲突。
  • 并发 press TOCTOU — 已解决：改用 swap(true) 闩 + fetch_add 返回旧值（prev_count==0 才是首次按下），不再有单独 active_count() 读的竞态。
  • 测试 — 已补齐：hold_mode_*（含 disable-mid-hold、rebind-mid-hold、键鼠重叠）四个集成测试 + update_config/Drop 两个单测，正好覆盖之前的缺口。CI 四平台全绿。

一个 LOW nit，留作后续即可、不拦合并：coordinator/hotkey_loops.rs 的 mouse_dictation_supervisor_loop 里那对互补 #[cfg(not(linux))] / #[cfg(linux)] 块体一样、等价于无 cfg，是重构残留，可直接去掉 cfg。

合并顺序提醒：本 PR 依赖 #726 的类型定义，请先合 #726 再合本 PR。另：beta 的 ruleset 只有 @jiangmuran 能 bypass，最终 merge 仍需他来点。

appergb
appergb reviewed Jun 25, 2026
View reviewed changes

@appergb appergb left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review note: this PR was previously approved and CI is green, but it is currently marked CONFLICTING against beta after related hotkey changes landed. It needs a rebase/conflict resolution before it can be merged.

HKLHaoBin and others added 16 commits June 26, 2026 01:01
@HKLHaoBin @cursoragent
feat(dictation): mouse middle/side buttons with Hold refcount
170a1a5 
Split from Open-Less#724: Win/macOS global mouse hooks, prefs toggles, HoldSourceTracker for multi-source hold release. Linux evdev path deferred to PR3.

Fixes Open-Less#718 (partial: mouse triggers).

Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin @cursoragent
fix(dictation): keep PR2 types mouse-only without side trigger variants
77bfb36 
Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin @cursoragent
fix(dictation): tighten PR2 scope for mouse-only split
5acbcc0 
- Fix hold-mode tests to pass TriggerSource::KeyboardDictation

- Hide mouse toggles on Linux until PR3 evdev

- Remove PR1 side-modifier type/i18n/mock remnants

- Complete mouse pref wire/deserialize/default in types.rs

Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin
Trigger CI
fc9301a
@HKLHaoBin @cursoragent
docs(types): drop PR1 side-modifier wording from ShortcutBinding comm…
2956aa2 
…ents

Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin @cursoragent
fix(dictation): emit mouse release when monitor drops or disables hel…
b68b565 
…d source

On update_config disable or Drop, swap held flags and send Released events so HoldSourceTracker does not retain stale mouse sources after prefs refresh.

Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin
feat(dictation): add mouse hold mode tests and improve mouse source h…
33de4cb 
…andling
@HKLHaoBin @cursoragent
test(dictation): sync_release_mouse preserves keyboard hold source
0924e7c 
Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin @cursoragent
fix(dictation): clear hold sources when hotkey binding changes mid-hold
eb0a463 
Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin
fix(ci): import HotkeyMode in hotkey_loops.rs
407ab26
@HKLHaoBin
fix(ci): avoid nested runtime in tests by extracting async release_mo…
813911f 
…use_hold_sources
@HKLHaoBin
fix(ci): extract async clear_active_hold_sources_on_hotkey_rebind_asy…
9b22bbc 
…nc to avoid nested runtime in tests
@HKLHaoBin
fix(ci): use futures::executor::block_on instead of async_runtime::bl…
faad142 
…ock_on to avoid nested runtime
@HKLHaoBin
fix(ci): use tokio::spawn + mpsc channel instead of block_on when in …
a3e0201 
…tokio runtime
@HKLHaoBin
fix(ci): use std::thread::spawn + handle.block_on to avoid blocking t…
f5232ae 
…okio worker thread
@HKLHaoBin @cursoragent
fix(rebase): restore lib modules and side-aware HotkeyTrigger literals
5dc5b50 
Co-authored-by: Cursor <cursoragent@cursor.com>
@HKLHaoBin HKLHaoBin force-pushed the feat/718-mouse-dictation branch from 8027eca to 5dc5b50 Compare June 25, 2026 17:09
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 5dc5b50

@HKLHaoBin @cursoragent
fix(rebase): restore side-aware HotkeyTrigger variants in Rust types
98f90d1 
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 98f90d1

@HKLHaoBin @cursoragent
fix(tests): implement refresh_mouse_dictation on MockWriter
53450fc 
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Persistent review updated to latest commit 53450fc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@appergb appergb appergb left review comments
@Felix201209 Felix201209 Felix201209 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

Review effort 3/5

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[feature] 支持左/右修饰键快捷键与鼠标中键唤起语音识别

3 participants

@HKLHaoBin @Felix201209 @appergb