feat: add OpenAI-compatible speech recognition provider#261
Conversation
Upstream split STT routing into HybridTranscriptionService; re-home the OpenAI provider there and revert CactusTranscriptionService to local-only.
|
@crc-32 could weigh in - not sure how much this complicates things. We already get a lot of bug reports...don't want to get even more from people who footgun themselves some how with this |
|
I'm not against the idea of this! Haven't reviewed it yet, but I'm mindful of the slowly building complexity of the transcription code with more and more services, this might need to wait for some restructuring to ensure more transcription services don't add more code debt. Also (speaking to anyone reading this PR) note that this would be the kind of advanced feature where we would be providing minimal support, if it doesn't work the solution is to try it out with the official options we're able to maintain then report a bug if it's still not working, or otherwise PR a fix. This is only because it's such a wildcard since we don't know what the exact backend you'd be using is and we already built out our own options to be supportable by our team. |
Move the WisprFlow+Kirinki and OpenAI remote logic out of HybridTranscriptionService into RemoteTranscriber strategies selected by cloudProvider. The router now owns only local/remote mode routing; adding a cloud backend is a new RemoteTranscriber plus its registration, with no change to the routing code.
|
@crc-32 Fair point. The way I had it, every new backend added more branches to the router. I've reworked it so each cloud backend (WisprFlow+Kirinki, and OpenAI) is its own RemoteTranscriber that handles its own availability + timeout/fallback. Does that match what you had in mind? If you've already got a restructure in progress for that file, just say so and I'll rebase onto yours, no point in you untangling two versions. On support: happy to mark it advanced/community-supported, add a "Test endpoint" button in settings so a bad URL/key/model shows up before someone hits record.... @ericmigi I get the support concern. The idea is to make it clearly advanced/community-supported and hard to break silently. There'd be a "Test endpoint" button in settings that actually calls the endpoint, so a wrong URL/key/model fails there instead of when through a recording. Bug reports should also note which provider was used, so if one comes in from a custom endpoint you can point them at the official options. Basically a broken self-hosted setup should break where the user can see it, not quietly on your end. Also, when the custom endpoint is slected, it could show a warning or a text mentioning that this is not an officially supported feature, or whatever. Anyway, I think that this feature adds a lot of value and freedom for users. Personally, I don't want to use WisprFlow, and the local Parakeet model is not good for my particular use case. However, I have my own server with Whisper large v3, and I'd love to use it. |
|
I would recommend using the webhook for now - you can route all your recordings there easily. |
|
@ericmigi I think the webhook's one-way only, right? It'd get my audio to my server, but the transcription wouldn't come back into the app. What I'm after is using my own endpoint as the actual STT backend, so replying to messages and taking notes from the watch runs through my own Whisper. That's the part the webhook doesn't cover, unless I'm missing something. |
|
just have your server text you the response? or make a watch app? |
|
From what I understand, the webhook won't cover this. It fires after the app has already transcribed the recording with its own backend, then just Posts the result out. The response never gets read, so a self-hosted whisper's output has nowhere to go. It's also only wired into the recordings queue, not the watch voice path at all. Dictation goes voiceSessionManager -> STTRouter -> cactusTranscriptionService A watchapp can't do it, because watchapps never get the raw mic audio. The dictation API sends text the companion already transcribed with whatever backend it's configured for. Audio only flows watch -> companion, so the companion's transcriptionService is the only spot a custom backend can go FWIW I'd sketched out basically this same feature for myself recently and landed on the same shape (separate backend class) so +1 from me |
|
Try this: press ring button -> talk into ring -> recording sent to webhook -> your server -> process recording -> open watchapp on watch, receive data from your server. I think this gets as close as you can right now. We're not planning to expand to alternative STT services at this time as per Alice's comment earlier. |
|
In my case, I do not own an Index ring. I am using the PT2; and I am referring to the transcription backend used when replying to messages, dictating for a notes app, etc. on the watch itself. This feature just gives more freedom to users, so they can choose the backend they prefer the most; being it local, WisprFlow, or a provider of their choice (even self-hosted) with OpenAI compatible APIs. |
|
@pluja I initially started working on this (referenced in a separate issue), but you got much further ahead, so let me know if you need help testing this or polishing. I think this is an important feature if freedom of choice is the end goal, and it could even be hidden behind the "Show Debug options" feature, or a similar flag, as many things are. |
|
Hi, I've pulled and tested this PR in it's current iteration, it works with whisper.cpp using Looking forward to future development/refinement on this! |
|
I put up an Android build with the feature here: https://github.com/pluja/mobileapp-custom-asr/releases I'll try to keep the fork updated with it. Note that the social logins (GitHub, Google, etc.) and OTA updates don't work in this build since I don't have the Firebase/Memfault keys. Everything else (pairing, STT, etc) works fine. |
What
This adds a third speech-recognition backend: any OpenAI-compatible
/audio/transcriptionsendpoint. You enter an endpoint, API key, and model under Settings → Speech Recognition, and the app sends cloud transcription there.Why
The only cloud option today is Wispr Flow. Many users want to choose the provider themselves: OpenAI, a self-hosted Whisper server, or any compatible service they trust. Pairing this with the existing on-device mode also lets the app transcribe without a Core account.
Related issue: coredevices/PebbleOS#1512
Changes
STTConfiggainscloudProvider(CloudSTTProvider) andopenAi(OpenAiSTTConfig: endpoint, apiKey, model). Both have defaults, so existing saved settings load unchanged.OpenAiTranscriptionServicesends the clip as one request: buffer the audio, encode 16 kHz mono WAV, post multipart, read{ "text": ... }. It mirrorsKirinkiTranscriptionService.CactusTranscriptionServiceroutes every cloud path (Cloud Only, Cloud+Local fallback, Local+Cloud fallback) andisAvailable()through the selected provider.STTConfigfrom scratch and wiped the provider, OpenAI config, and spoken language.Behavior notes
/audio/transcriptions(a full path works too). Android blocks plainhttp://endpoints by default, which affects a LAN Whisper server.Testing
:composeApp:assembleDebugandassembleReleasebuild clean.