[WIP] new: add local whisper.cpp provider + optional Wayland paste workaround - #1
Open
njeudy wants to merge 6 commits into
Open
[WIP] new: add local whisper.cpp provider + optional Wayland paste workaround#1njeudy wants to merge 6 commits into
njeudy wants to merge 6 commits into
Conversation
- ``whisper-rs`` 0.16 with ``cuda`` and ``log_backend`` features for GPU-accelerated whisper.cpp inference. - ``hound`` for decoding 16 kHz mono WAV input files in pure Rust. - ``tempfile`` promoted from dev-dependencies to runtime dependencies: the streaming transcribe path buffers chunks to a temp file before decoding (no streaming OGG/Opus decoder is available yet).
- ``Provider::WhisperLocal`` variant alongside Mistral and OpenAI,
with ``Display``, ``FromStr`` and serde aliases for
``whisper-local`` / ``whisperlocal``.
- ``ProvidersConfig.whisper_local: Option<WhisperLocalConfig>`` so
configs that don't use the local backend stay untouched.
- ``WhisperLocalConfig`` fields: ``model_path`` (required GGML file),
``language`` (default ``auto``), ``use_gpu`` (default ``true``),
``initial_prompt`` (whisper.cpp equivalent of the cloud
``context_bias``), ``threads`` (default 4).
- ``TALK_RS_PROVIDERS_WHISPER_LOCAL_{MODEL_PATH,LANGUAGE,INITIAL_PROMPT}``
environment overrides mirroring the existing Mistral/OpenAI pattern.
- Provider rename to ``snake_case`` serde casing so YAML keys are
``whisper_local`` (existing ``mistral``/``openai`` keys are
unchanged because they have no underscore).
…pport
``WhisperLocalBatchTranscriber`` implements ``BatchTranscriber`` using
the ``whisper-rs`` bindings to whisper.cpp. Inference runs on the
GPU when the binary is built with the ``cuda`` feature (or any other
``_gpu`` backend), or on the CPU otherwise.
Key design points:
- The ``WhisperContext`` is loaded lazily inside a ``OnceLock`` on the
first ``transcribe_file`` call. This keeps the daemon spawn fast:
the multi-second model-load cost is paid only when the recording
has already been captured, not before audio capture starts.
- Inference is dispatched to ``tokio::task::spawn_blocking`` so the
reactor isn't stalled by the synchronous whisper.cpp call.
- Audio decoding is pure Rust using existing dependencies:
- OGG/Opus via the ``ogg`` + ``opus`` crates (the format produced
by ``talk-rs record``).
- WAV via ``hound`` (16 kHz mono, 16-bit int or 32-bit float).
- Other formats return a clear error pointing at ``ffmpeg``.
- Streaming transcription buffers OGG chunks to a temp file then
delegates to ``transcribe_file`` — true streaming decode is not
available yet for whisper.cpp.
- Diarization is not supported by whisper.cpp; the factory rejects
``--diarize`` with a helpful error.
- Realtime is not supported either; the factory returns a clear
error explaining how to fall back to batch mode.
- ``initial_prompt`` is wired to whisper's ``set_initial_prompt`` so
users can bias the decoder toward names and technical terms
(the local equivalent of the cloud ``context_bias`` field).
The transcription dispatcher (``create_batch_transcriber``,
``create_realtime_transcriber``, ``is_model_error``,
``enrich_model_error``) is extended to route ``Provider::WhisperLocal``
to the new backend.
The model resolver and the retry-candidates builder gain a ``WhisperLocal`` branch: - ``resolve_model`` returns the GGML file stem (e.g. ``ggml-large-v3``) as the effective "model name" when ``Provider::WhisperLocal`` is selected — the local backend has no API-served model catalog, so the file name is the closest semantic equivalent. - ``add_known_models_with_streaming`` and ``add_known_realtime_models`` return empty slices for ``WhisperLocal``: we can't enumerate other local GGML files on disk, so there is no retry catalog to offer. - The realtime ``provider_specific`` metadata builder returns ``None`` for ``WhisperLocal``. This branch is unreachable in practice because the realtime factory rejects ``WhisperLocal`` upstream, but the match must stay exhaustive.
Under a Wayland compositor the X11 ``XTEST`` fake keystroke used by the standard paste path is blocked, leaving the transcription out of reach of the focused application. Worse: talk-rs restores the previous clipboard after the failed paste, so even a manual ``Ctrl+Shift+V`` finds nothing useful. Add an opt-in mode ``paste.no_paste: true`` that skips both the fake keystroke and the clipboard restoration. The transcription is written to the X11 clipboard and, when ``$WAYLAND_DISPLAY`` is set, also piped into ``wl-copy`` (best-effort) so Wayland-native apps have access to it. Combined with a clipboard manager (GPaste, Pano, Copyous, …) this yields a "transcribe → Ctrl+V" workflow that works reliably on GNOME Wayland. - ``PasteConfig.no_paste`` defaults to ``false`` so existing X11 installs are unaffected. - ``paste_text_to_target`` takes a new ``no_paste: bool`` argument and short-circuits before any XTest call when the flag is set. - Both call sites (dictate batch path, picker selection path) read the value from ``config.paste.no_paste`` and propagate it. - ``write_to_wayland_clipboard`` is a small ``wl-copy`` wrapper that fails silently with a single warn-level log when the binary is missing — no hard runtime dependency on ``wl-clipboard``.
Cover three topics that the README was missing: - ``libssl-dev`` / ``openssl-devel`` is required for the reqwest TLS stack to compile. Document it next to the other build deps. - A full "Local whisper.cpp" subsection in *Prerequisites*: optional ``nvidia-cuda-toolkit`` for the GPU build, link to the GGML model catalog, and a ready-to-run ``curl`` line for ``ggml-large-v3.bin``. - A "Wayland tips" subsection explaining why the default XTEST paste path breaks on Wayland, how ``paste.no_paste`` solves it, the optional ``wl-clipboard`` package, and pointers to popular clipboard managers (GPaste, Pano, Copyous). The configuration tables now list ``providers.whisper_local.*`` and ``paste.no_paste``; ``config.example.yaml`` gains commented sample sections for both with inline documentation.
Author
|
@vaab I use ia for this, so you can accept or not this PR, modify it, i use it locally for now :) |
Owner
|
Many thanks for the interest. Alas, I'm currently totally underwater, and I don't have neither a local Whisper (I would recommend local voxtral BTW), nor a good graphic card, and even less so a Wayland install. I also committed a lot in between, and even had a deep look at your PR to integrate still what I could, and I'm afraid this will be too hard to just integrate as such, which is a pity. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey Valentin,
WIP draft — I've been running this locally for a few days and figured I'd share the diff in case it's useful, or in case you'd like to leave feedback on the approach.
What's in here
1. Local Whisper provider
Adds a third transcription backend that runs whisper.cpp via the
whisper-rscrate, optionally CUDA-accelerated. Useful when you'd rather not send audio to a cloud provider. The user drops a GGML model file on disk, pointsproviders.whisper_local.model_pathat it, and that's it.On my RTX 4070,
large-v3loads in ~700 ms (file in page cache) and transcribes faster than realtime. Cloud providers are untouched — same config, same factory dispatch.Realtime + diarization aren't supported by the local backend; the factory returns clear errors when those flags are passed.
2. Optional Wayland paste workaround
Under GNOME Wayland, the X11
XTESTfakeCtrl+Shift+Vkeystroke used bypaste_text_to_targetgets blocked by the compositor for Wayland-native apps. Worse: the clipboard restore at the end of the function then wipes the transcription, so even a manualCtrl+Shift+Vfinds nothing.This PR adds an opt-in
paste.no_paste: trueconfig flag that short-circuits both the XTEST call and the clipboard restoration. The transcription is left in the X11 clipboard and piped intowl-copy(best-effort) so the Wayland-native clipboard has it too. Combined with a clipboard manager (GPaste, Pano, Copyous…) this gives a reliable ""transcribe → Ctrl+V"" workflow on Wayland.The flag is off by default — existing X11 setups are unaffected.
Caveats / things you might want to push back on
whisper_localmodule and the new config types. Happy to revisit any pattern you'd write differently (error mapping, theOnceLock<WhisperContext>lazy init, the audio decoding viaogg/opus/hound, thespawn_blockingwrapping, etc.). Feedback very welcome — feel free to be blunt.no_pastecommit is needed for everyone on Wayland — I had a confusing test earlier where I was actually runningtalk-rs dictate(no--toggle) in a terminal, which printed everything to stdout. After fixing that I still observed the auto-paste failing on Wayland-native apps, so I think the flag is real, but feel free to drop the commit if you'd rather keeppaste_text_to_targetsimple.libssl-devwas missing from the Debian build deps in the README — I added it. The build fails without it (transitive dep ofreqwest/tokio-tungsteniteviaopenssl-sys).whisper_localProvider variant participates in everymatch provider { … }site (3 indictate/models.rs, 1 indictate/realtime.rs); I went through the exhaustive cases rather than adding a catch-all wildcard.Commits
Verification on my side
cargo fmt,cargo clippy --all-targetsclean (0 warnings)cargo test --releasepasses (only the pre-existingxclipintegration test fails on my Wayland box; not introduced by this PR)cargo build --releaseproduces a 129 MB binary (statically linked whisper.cpp + CUDA)talk-rs transcribe synthetic.wav --provider whisper_localvalidated; daemon--toggleworkflow validated under GNOME Wayland withpaste.no_paste: trueCheers — and thanks for talk-rs, it's become my daily dictation setup. ❤️