Skip to content

feat: add OpenAI-compatible speech recognition provider#261

Open
pluja wants to merge 4 commits into
coredevices:masterfrom
pluja:feat/openai-compatible-stt
Open

feat: add OpenAI-compatible speech recognition provider#261
pluja wants to merge 4 commits into
coredevices:masterfrom
pluja:feat/openai-compatible-stt

Conversation

@pluja

@pluja pluja commented Jun 20, 2026

Copy link
Copy Markdown

What

This adds a third speech-recognition backend: any OpenAI-compatible /audio/transcriptions endpoint. You enter an endpoint, API key, and model under Settings → Speech Recognition, and the app sends cloud transcription there.

Why

The only cloud option today is Wispr Flow. Many users want to choose the provider themselves: OpenAI, a self-hosted Whisper server, or any compatible service they trust. Pairing this with the existing on-device mode also lets the app transcribe without a Core account.

Related issue: coredevices/PebbleOS#1512

Changes

  • STTConfig gains cloudProvider (CloudSTTProvider) and openAi (OpenAiSTTConfig: endpoint, apiKey, model). Both have defaults, so existing saved settings load unchanged.
  • OpenAiTranscriptionService sends the clip as one request: buffer the audio, encode 16 kHz mono WAV, post multipart, read { "text": ... }. It mirrors KirinkiTranscriptionService.
  • CactusTranscriptionService routes every cloud path (Cloud Only, Cloud+Local fallback, Local+Cloud fallback) and isAvailable() through the selected provider.
  • Settings gains a selectable "Cloud Recognition Provider". Wispr Flow keeps its logo row. Pick OpenAI to open a dialog for the endpoint, API key (masked), and model.
  • If you're signed out and have configured an OpenAI backend, picking a cloud mode no longer triggers the sign-in prompt; the app uses that backend. This also fixes the model-download dialog, which rebuilt STTConfig from scratch and wiped the provider, OpenAI config, and spoken language.

Behavior notes

  • Pick a cloud mode while signed out and the app sets the provider to OpenAI-Compatible. That choice sticks after you sign in; switch it back from the provider dropdown. Wispr Flow still needs a Core account.
  • The app treats your endpoint as a base URL and appends /audio/transcriptions (a full path works too). Android blocks plain http:// endpoints by default, which affects a LAN Whisper server.

Testing

  • :composeApp:assembleDebug and assembleRelease build clean.
  • I ran it on-device against an OpenAI-compatible endpoint and confirmed watch dictation transcribes through it.

Claude Code wrote the code in this PR. I planned the feature, reviewed the changes, and tested it on-device before opening the PR.

@CLAassistant

CLAassistant commented Jun 20, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Upstream split STT routing into HybridTranscriptionService; re-home the
OpenAI provider there and revert CactusTranscriptionService to local-only.
@ericmigi

ericmigi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@crc-32 could weigh in - not sure how much this complicates things. We already get a lot of bug reports...don't want to get even more from people who footgun themselves some how with this

@crc-32

crc-32 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

I'm not against the idea of this! Haven't reviewed it yet, but I'm mindful of the slowly building complexity of the transcription code with more and more services, this might need to wait for some restructuring to ensure more transcription services don't add more code debt.

Also (speaking to anyone reading this PR) note that this would be the kind of advanced feature where we would be providing minimal support, if it doesn't work the solution is to try it out with the official options we're able to maintain then report a bug if it's still not working, or otherwise PR a fix. This is only because it's such a wildcard since we don't know what the exact backend you'd be using is and we already built out our own options to be supportable by our team.

Move the WisprFlow+Kirinki and OpenAI remote logic out of
HybridTranscriptionService into RemoteTranscriber strategies selected by
cloudProvider. The router now owns only local/remote mode routing; adding a
cloud backend is a new RemoteTranscriber plus its registration, with no change
to the routing code.
@pluja

pluja commented Jul 4, 2026

Copy link
Copy Markdown
Author

@crc-32 Fair point. The way I had it, every new backend added more branches to the router. I've reworked it so each cloud backend (WisprFlow+Kirinki, and OpenAI) is its own RemoteTranscriber that handles its own availability + timeout/fallback. HybridTranscriptionService now just does the local-vs-remote mode routing and grabs a backend by cloudProvider. That pulled ~140 lines out of the router, and adding another service is a new class plus one DI line now, with nothing touching the routing.

Does that match what you had in mind? If you've already got a restructure in progress for that file, just say so and I'll rebase onto yours, no point in you untangling two versions.

On support: happy to mark it advanced/community-supported, add a "Test endpoint" button in settings so a bad URL/key/model shows up before someone hits record....

@ericmigi I get the support concern. The idea is to make it clearly advanced/community-supported and hard to break silently. There'd be a "Test endpoint" button in settings that actually calls the endpoint, so a wrong URL/key/model fails there instead of when through a recording. Bug reports should also note which provider was used, so if one comes in from a custom endpoint you can point them at the official options. Basically a broken self-hosted setup should break where the user can see it, not quietly on your end. Also, when the custom endpoint is slected, it could show a warning or a text mentioning that this is not an officially supported feature, or whatever.

Anyway, I think that this feature adds a lot of value and freedom for users. Personally, I don't want to use WisprFlow, and the local Parakeet model is not good for my particular use case. However, I have my own server with Whisper large v3, and I'd love to use it.

@ericmigi

ericmigi commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

I would recommend using the webhook for now - you can route all your recordings there easily.

@pluja

pluja commented Jul 4, 2026

Copy link
Copy Markdown
Author

@ericmigi I think the webhook's one-way only, right? It'd get my audio to my server, but the transcription wouldn't come back into the app.

What I'm after is using my own endpoint as the actual STT backend, so replying to messages and taking notes from the watch runs through my own Whisper. That's the part the webhook doesn't cover, unless I'm missing something.

@ericmigi

ericmigi commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

just have your server text you the response? or make a watch app?

@castawait

Copy link
Copy Markdown

From what I understand, the webhook won't cover this. It fires after the app has already transcribed the recording with its own backend, then just Posts the result out. The response never gets read, so a self-hosted whisper's output has nowhere to go. It's also only wired into the recordings queue, not the watch voice path at all. Dictation goes voiceSessionManager -> STTRouter -> cactusTranscriptionService

A watchapp can't do it, because watchapps never get the raw mic audio. The dictation API sends text the companion already transcribed with whatever backend it's configured for. Audio only flows watch -> companion, so the companion's transcriptionService is the only spot a custom backend can go

FWIW I'd sketched out basically this same feature for myself recently and landed on the same shape (separate backend class) so +1 from me

@ericmigi

ericmigi commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Try this: press ring button -> talk into ring -> recording sent to webhook -> your server -> process recording -> open watchapp on watch, receive data from your server.

I think this gets as close as you can right now. We're not planning to expand to alternative STT services at this time as per Alice's comment earlier.

@pluja

pluja commented Jul 5, 2026

Copy link
Copy Markdown
Author

In my case, I do not own an Index ring. I am using the PT2; and I am referring to the transcription backend used when replying to messages, dictating for a notes app, etc. on the watch itself.

This feature just gives more freedom to users, so they can choose the backend they prefer the most; being it local, WisprFlow, or a provider of their choice (even self-hosted) with OpenAI compatible APIs.

@bbutkovic

Copy link
Copy Markdown

@pluja I initially started working on this (referenced in a separate issue), but you got much further ahead, so let me know if you need help testing this or polishing.

I think this is an important feature if freedom of choice is the end goal, and it could even be hidden behind the "Show Debug options" feature, or a similar flag, as many things are.

@breakingspell

Copy link
Copy Markdown

Hi, I've pulled and tested this PR in it's current iteration, it works with whisper.cpp using whisper-server --inference-path "/v1/audio/transcriptions". I do agree the custom endpoint field should be marked as an expert/debug feature.

Looking forward to future development/refinement on this!

@pluja

pluja commented Jul 7, 2026

Copy link
Copy Markdown
Author

I put up an Android build with the feature here: https://github.com/pluja/mobileapp-custom-asr/releases

I'll try to keep the fork updated with it. Note that the social logins (GitHub, Google, etc.) and OTA updates don't work in this build since I don't have the Firebase/Memfault keys. Everything else (pairing, STT, etc) works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants