voice: add on-device audio recording with playback and SDK API#1641
voice: add on-device audio recording with playback and SDK API#1641julpel8 wants to merge 36 commits into
Conversation
Add the ability to record microphone audio to flash on the watch, independent of a phone connection, and play it back through the speaker. Audio is Speex-encoded (reusing the dictation codec) into a PFS file, so a recording can later be replayed to the phone for deferred transcription. - services/voice: a voice_recording service captures mic frames into a pre-allocated temp file (no flash erase on the capture path) and compacts it to an exact-size file on stop. Adds a Speex decoder and a speaker playback path that streams decoded PCM. Recording and live dictation are mutually exclusive as they share the single microphone. - applib/voice: audio_recording_* SDK API (start/stop/cancel/is_active) with syscalls, exported symbols and an SDK revision bump. - settings: a "Record audio (beta)" menu to record, play and delete memos directly from the watch. The record/play trigger is a temporary test affordance (VOICE_REC_TEST_TRIGGER) until an app-facing trigger lands. Refs coredevices#719 Co-Authored-By: Julien Pelletier <pelletier.julien@pm.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: gpt5.5-codex <noreply@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Leaving the "Record audio (beta)" page mid-recording never stopped the capture: the service stayed in RecState_Recording with the microphone running. On re-entry the fresh page had active_id == 0, so tapping "Stop recording" called voice_recording_stop(0), which failed the id == s_active_id check and did nothing, leaving the UI stuck. Add voice_recording_stop_active() to stop and finalize whichever recording is active, and call it from the page's unload callback. This saves the in-progress memo on exit and also recovers an already-stuck state. Co-Authored-By: Julien Pelletier <pelletier.julien@pm.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: gpt5.5-codex <noreply@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…lbacks from writing to or stopping replacement audio Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…, playback control, and resource management Co-Authored-By: gpt5.5-codex <noreply@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Co-Authored-By: gpt5.5-codex <noreply@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…ild sources Co-Authored-By: gpt5.5-codex <noreply@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…yscalls Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add VoiceEventTypeSessionProgress and a progress field so the voice service can report upload progress (0-100) to the UI while streaming a recording for transcription. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add voice_start_dictation_from_recording(), which streams a stored recording's Speex frames to the phone over the existing dictation transport instead of the live microphone. The stored frames already match the audio endpoint wire format, so they are sent as-is without transcoding. A paced feed timer keeps the send rate near real-time so the audio endpoint send buffer never overflows and drops frames (which would corrupt the transcription), reports upload progress through the new VoiceEventTypeSessionProgress event, and, once the whole payload has been sent, stops the transfer and awaits the result. Results come back through the same path as live dictation. Only mono recordings can be transcribed. Session bring-up common to the microphone and recording sources is factored into prv_start_session(). Adds voice_recording_storage_get_metadata() and a syscall. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
voice_window_create_for_recording() transcribes a stored recording: it skips the microphone capture phase, jumps straight to the upload state, and drives the progress bar from the real upload-progress events reported by the voice service (rather than the estimated animation used for live dictation). Confirmation and error dialogs are disabled and the transcription is delivered through the existing dictation event path. Also allow the StateStartWaitForReady -> StateWaitForResponse transition, since a recording session can become ready before the fly-in animation completes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add a one-shot audio_recording_transcribe() that sends a stored recording to the phone for transcription, shows a progress screen while it uploads, and delivers the resulting text (or a failure status) to a callback. It reuses the recording-source voice window and the dictation event path. Registered under the existing AudioRecording SDK group (revision 106); no SDK revision bump. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…E_SIZE constant Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
voice_recording_storage_total_bytes() re-opened and header-read every stored recording on each call, and voice_recording_start() calls it on the capture path just before grabbing the microphone. Cache the sum instead: compute it once at init, add each finalized file's exact size, and invalidate on delete (recomputed lazily) / zero it on delete-all. All mutators run under the voice_recording lock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Reject app-originated recording operations when the recording UUID does not match the current app. Keep privileged system callers able to manage all recordings. Co-authored-by: Codex <codex@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
An interrupted finalize (reset or power loss during stop) leaves a vrec_ file whose header was never written. Such a file fails the header validation, so it is excluded from listing and the storage quota, yet it still occupies flash with no way to see or reclaim it short of deleting all recordings. Validate the header of every recording file at init and remove the invalid ones, alongside the existing temp-file cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
pfs_remove() panics when the file is held open, and the transcription stream keeps a recording's payload descriptor open for the whole (real-time) streaming duration. Deleting that recording while it streams would reboot the watch. Track the id of the recording being streamed in the voice service and expose it via voice_transcribing_recording_id(). Deleting a single recording now refuses if it is being transcribed, and delete-all skips it (invalidating the cached total so it is recomputed lazily). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
test_voice_recording_playback.c was added without a wscript_build entry, so it never built or ran. Register it; all three test cases pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
… app Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
|
cool! keep in touch with @jplexer as they will help merge this if it's good enough |
… error handling Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…recordings Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
PEBBLE_DICTATION_EVENT is broadcast to every subscriber, so concurrent consumers (a recording transcription plus a dictation session, or two transcriptions) all received each other's results: the first event delivered the wrong status to every listener and tore down in-flight transcriptions. Tag the event with a per-window id and make the dictation session and audio transcription subscribers ignore results that are not from their own window. PebbleEvent grows by one byte, as invited by the size assert comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Playback started by an app kept running after the app exited or crashed, and sys_audio_recording_stop_playback had no ownership check, so any app could silence a playback it did not own (e.g. one started from the Settings UI). Track whether the active playback was app-initiated, stop it in the process cleanup path alongside capture, and only let elevated callers stop an app-owned playback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add persistent recording quality, capture gain, and playback gain controls to the Voice Memos settings. Apply digital gain with saturation to recording and playback samples. Co-authored-by: GPT-5 Codex <codex@openai.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
When the recording feed reaches end of payload it called audio_endpoint_stop_transfer() from the new_timer callback, which sends a BT message with a 4 s blocking timeout while holding the voice lock. Right after streaming a whole payload the send buffer is likely congested, so this could stall every new_timer client in the system. Schedule the stop on the system task instead, guarded by the session generation so it no-ops if the session ended in the meantime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The startup scan reset the next id in file order, so seeing vrec_65535 before vrec_1 primed the allocator onto an id that still had a stored file, and voice_recording_start() never checked for collisions: after the uint16 id space wrapped, finalize could silently overwrite an existing memo. Prime the allocator from the highest stored id and probe for a genuinely free id before creating the temp file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Recordings owned by an uninstalled app were orphaned forever: invisible to every other app (ownership-filtered listing) while still counting against the shared storage budget, eventually making every recording attempt fail with StorageFull. Remove the app's recordings alongside its persist file on uninstall (not on upgrade), skipping a file held open by an active transcription stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The size-only static assert let the applib and service structs drift apart silently: reordering or retyping fields keeps sizes equal while the list syscall cast garbles every field. Check offset and size of each field instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The length-prefixed frame parsing (read length byte, validate against remaining payload and max frame size, read payload) was duplicated between the playback feed and the transcription feed, so a container format change fixed in one path would silently corrupt the other. Extract voice_recording_storage_read_frame() and use it in both; adapt the playback unit test to stub the shared reader (and the playback gain accessor it now needs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
|
would be cool to see an easy to use companion pkjs side that receives/decodes/streams audio from watch |
|
are you using a debug or release build? debug builds enable UART RX which is very high power consumption |
Yes, I don't set CONFIG_RELEASE. Thank you ! |
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
|
Would really love to see this merged! There are so many apps I want to make with this feature... |
|
we're tracking it! just haven't had time yet sorry |
|
This is fantastic! offline on-device recording is going to be a really welcome addition. However, I think there's a missing piece: Two reasons from my own use case (I'm building a Roam Research capture watchapp for PT2) that specifically need the recording, not just a transcript:
Others in #719 have asked for the same (saving audio + transcript), so exposing an API to get the audio out would help a range of apps, not just mine. One possible shape (drafted with AI assistance; the technical claims were checked against the PR source): a read API like Happy to help in any way I can: whether testing or by contributing a stab at a follow-up PR (DCO + AI-disclosure understood). Thank you! |
|
Following up on my comment above, I just noticed @ericmigi's earlier note about "an easy to use companion pkjs side that receives/decodes/streams audio from watch." That's exactly the shape that would work for me, and it's a better fit than the read API I mentioned above. (Below is from Claude, so please take with grain of salt again)
It also preserves this PR's offline model: keep the recording on-watch until the export is acknowledged, then the app can delete it. (A byte-read API like Happy to help spec, test, or contribute a follow-up PR. |
|
Thanks @baibhavbista! Getting the raw audio out makes a lot of sense. I’ve actually already forked Brain Dump by @adrienthiery and got it listing the on-device recordings via audio_recording_list(). |
|
@baibhavbista After reviewing it, I realized I hadn’t pushed the changes that make it easier to list, edit and listen to the audio. |

Summary
Adds on-device audio recording to flash, playback through the watch speaker, and deferred transcription through the phone.
Recordings use the existing Speex codec and can be created without a phone connection. Once Bluetooth is available, an app can send a stored recording through the existing dictation
transport and receive the transcription result.
This addresses the offline-recording use case described in #719.
Refs #719
What's included
On-device microphone recording
Speaker playback
Deferred transcription
SDK API
Settings test interface
user-facing Voice Memos interface is an open question.
Open questions
Recording lifecycle: Should recordings remain stored until the app
deletes them, be deleted automatically after transcription, or be managed
by an OS retention policy? What should happen when the owning app is
uninstalled or the recording storage quota is reached?
Audio quality: Should recording quality remain configurable through a
system menu, use a fixed default, or be exposed to apps through the SDK?
Should recording and playback gain remain configurable as well?
Storage information: Should the watch display the storage used and
available for recordings? If so, should this represent the dedicated
recording quota or the actual free flash space? Should this information also
be exposed through the SDK?
Settings UI: Should the current settings page remain available as a
user-facing Voice Memos interface, or should it be removed once SDK
integration is validated?
Validation
obelix@pvt(Pebble Time 2).Screenshots
Notes for reviewers
This PR covers recording, playback, and deferred transcription as one end-to-end feature. It is relatively large and can be split into separate reviewable changes if preferred.
The capture stop path still shares the microphone tail-drop behavior described in #1423: residual PDM data and the final partial Speex frame are not flushed.