Skip to content

voice: add on-device audio recording with playback and SDK API#1641

Open
julpel8 wants to merge 36 commits into
coredevices:mainfrom
julpel8:audio/recording
Open

voice: add on-device audio recording with playback and SDK API#1641
julpel8 wants to merge 36 commits into
coredevices:mainfrom
julpel8:audio/recording

Conversation

@julpel8

@julpel8 julpel8 commented Jul 1, 2026

Copy link
Copy Markdown

Summary

Adds on-device audio recording to flash, playback through the watch speaker, and deferred transcription through the phone.

Recordings use the existing Speex codec and can be created without a phone connection. Once Bluetooth is available, an app can send a stored recording through the existing dictation
transport and receive the transcription result.

This addresses the offline-recording use case described in #719.

Refs #719

What's included

  • On-device microphone recording

    • Stores Speex-encoded audio in PFS.
    • Uses pre-allocated temporary files to avoid flash erases during capture.
    • Compacts recordings to their final size when stopped.
    • Supports storage quotas, enumeration, deletion, validation, and cleanup.
    • Enforces recording ownership for third-party apps.
    • Keeps recording and live dictation mutually exclusive.
  • Speaker playback

    • Decodes stored Speex frames to 16 kHz PCM.
    • Streams audio through the existing speaker service.
    • Isolates playback sessions from stale callbacks.
  • Deferred transcription

    • Streams stored Speex frames to the phone without transcoding.
    • Paces transfers to prevent send-buffer overflow and dropped frames.
    • Reports upload progress to the UI.
    • Prevents deletion of a recording while it is being transcribed.
    • Delivers results through the existing dictation event path.
  • SDK API

    • Adds APIs to start, stop, cancel, query, enumerate, play, delete, and transcribe recordings.
    • Includes the required syscalls and exported SDK symbols.
    • Bumps the SDK revision.
    • Returns transcription results asynchronously through a callback.
  • Settings test interface

    • Adds a “Record audio (beta)” page for recording, playback, transcription, and deletion directly on the watch.
    • This currently serves as a test interface. Whether it should remain as a
      user-facing Voice Memos interface is an open question.

Open questions

  • Recording lifecycle: Should recordings remain stored until the app
    deletes them, be deleted automatically after transcription, or be managed
    by an OS retention policy? What should happen when the owning app is
    uninstalled or the recording storage quota is reached?

  • Audio quality: Should recording quality remain configurable through a
    system menu, use a fixed default, or be exposed to apps through the SDK?
    Should recording and playback gain remain configurable as well?

  • Storage information: Should the watch display the storage used and
    available for recordings? If so, should this represent the dedicated
    recording quota or the actual free flash space? Should this information also
    be exposed through the SDK?

  • Settings UI: Should the current settings page remain available as a
    user-facing Voice Memos interface, or should it be removed once SDK
    integration is validated?

Validation

  • Built for obelix@pvt (Pebble Time 2).
  • Tested on real Pebble Time 2 hardware:
    • record and stop;
    • display the saved recording with its duration and size;
    • play audio through the speaker;
    • transcribe a stored recording;
    • delete recordings.
  • Registered and ran the voice-recording playback unit tests successfully.

Screenshots

image image image image

Notes for reviewers

This PR covers recording, playback, and deferred transcription as one end-to-end feature. It is relatively large and can be split into separate reviewable changes if preferred.

The capture stop path still shares the microphone tail-drop behavior described in #1423: residual PDM data and the final partial Speex frame are not flushed.

julpel8 and others added 3 commits July 1, 2026 12:06
Add the ability to record microphone audio to flash on the watch,
independent of a phone connection, and play it back through the
speaker. Audio is Speex-encoded (reusing the dictation codec) into a
PFS file, so a recording can later be replayed to the phone for
deferred transcription.

- services/voice: a voice_recording service captures mic frames into a
  pre-allocated temp file (no flash erase on the capture path) and
  compacts it to an exact-size file on stop. Adds a Speex decoder and a
  speaker playback path that streams decoded PCM. Recording and live
  dictation are mutually exclusive as they share the single microphone.
- applib/voice: audio_recording_* SDK API (start/stop/cancel/is_active)
  with syscalls, exported symbols and an SDK revision bump.
- settings: a "Record audio (beta)" menu to record, play and delete
  memos directly from the watch. The record/play trigger is a temporary
  test affordance (VOICE_REC_TEST_TRIGGER) until an app-facing trigger
  lands.

Refs coredevices#719

Co-Authored-By: Julien Pelletier <pelletier.julien@pm.me>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: gpt5.5-codex <noreply@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Leaving the "Record audio (beta)" page mid-recording never stopped the
capture: the service stayed in RecState_Recording with the microphone
running. On re-entry the fresh page had active_id == 0, so tapping
"Stop recording" called voice_recording_stop(0), which failed the
id == s_active_id check and did nothing, leaving the UI stuck.

Add voice_recording_stop_active() to stop and finalize whichever
recording is active, and call it from the page's unload callback. This
saves the in-progress memo on exit and also recovers an already-stuck
state.

Co-Authored-By: Julien Pelletier <pelletier.julien@pm.me>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: gpt5.5-codex <noreply@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…lbacks from writing to or stopping replacement audio

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
@julpel8
julpel8 force-pushed the audio/recording branch from 96d36a0 to ab10187 Compare July 1, 2026 11:54
julpel8 and others added 11 commits July 1, 2026 14:08
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…, playback control, and resource management

Co-Authored-By: gpt5.5-codex <noreply@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Co-Authored-By: gpt5.5-codex <noreply@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…ild sources

Co-Authored-By: gpt5.5-codex <noreply@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…yscalls

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add VoiceEventTypeSessionProgress and a progress field so the voice
service can report upload progress (0-100) to the UI while streaming a
recording for transcription.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add voice_start_dictation_from_recording(), which streams a stored
recording's Speex frames to the phone over the existing dictation
transport instead of the live microphone. The stored frames already
match the audio endpoint wire format, so they are sent as-is without
transcoding.

A paced feed timer keeps the send rate near real-time so the audio
endpoint send buffer never overflows and drops frames (which would
corrupt the transcription), reports upload progress through the new
VoiceEventTypeSessionProgress event, and, once the whole payload has
been sent, stops the transfer and awaits the result. Results come back
through the same path as live dictation.

Only mono recordings can be transcribed. Session bring-up common to the
microphone and recording sources is factored into prv_start_session().
Adds voice_recording_storage_get_metadata() and a syscall.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
voice_window_create_for_recording() transcribes a stored recording: it
skips the microphone capture phase, jumps straight to the upload state,
and drives the progress bar from the real upload-progress events
reported by the voice service (rather than the estimated animation used
for live dictation). Confirmation and error dialogs are disabled and
the transcription is delivered through the existing dictation event
path.

Also allow the StateStartWaitForReady -> StateWaitForResponse
transition, since a recording session can become ready before the
fly-in animation completes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add a one-shot audio_recording_transcribe() that sends a stored
recording to the phone for transcription, shows a progress screen while
it uploads, and delivers the resulting text (or a failure status) to a
callback. It reuses the recording-source voice window and the dictation
event path.

Registered under the existing AudioRecording SDK group (revision 106);
no SDK revision bump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
@julpel8

julpel8 commented Jul 2, 2026

Copy link
Copy Markdown
Author

@jplexer @gmarull My PR is heavy, should I split it in 3 PR (record, playing and transcribe) ?

julpel8 and others added 10 commits July 2, 2026 14:59
…E_SIZE constant

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
voice_recording_storage_total_bytes() re-opened and header-read every
stored recording on each call, and voice_recording_start() calls it on
the capture path just before grabbing the microphone. Cache the sum
instead: compute it once at init, add each finalized file's exact size,
and invalidate on delete (recomputed lazily) / zero it on delete-all.
All mutators run under the voice_recording lock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Reject app-originated recording operations when the recording UUID does not match the current app. Keep privileged system callers able to manage all recordings.

Co-authored-by: Codex <codex@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
An interrupted finalize (reset or power loss during stop) leaves a
vrec_ file whose header was never written. Such a file fails the
header validation, so it is excluded from listing and the storage
quota, yet it still occupies flash with no way to see or reclaim it
short of deleting all recordings.

Validate the header of every recording file at init and remove the
invalid ones, alongside the existing temp-file cleanup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
pfs_remove() panics when the file is held open, and the transcription
stream keeps a recording's payload descriptor open for the whole
(real-time) streaming duration. Deleting that recording while it
streams would reboot the watch.

Track the id of the recording being streamed in the voice service and
expose it via voice_transcribing_recording_id(). Deleting a single
recording now refuses if it is being transcribed, and delete-all skips
it (invalidating the cached total so it is recomputed lazily).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
test_voice_recording_playback.c was added without a wscript_build
entry, so it never built or ran. Register it; all three test cases
pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
… app

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
@ericmigi

ericmigi commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

cool! keep in touch with @jplexer as they will help merge this if it's good enough

julpel8 and others added 3 commits July 3, 2026 08:47
… error handling

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
…recordings

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
PEBBLE_DICTATION_EVENT is broadcast to every subscriber, so concurrent
consumers (a recording transcription plus a dictation session, or two
transcriptions) all received each other's results: the first event
delivered the wrong status to every listener and tore down in-flight
transcriptions.

Tag the event with a per-window id and make the dictation session and
audio transcription subscribers ignore results that are not from their
own window. PebbleEvent grows by one byte, as invited by the size
assert comment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
julpel8 and others added 8 commits July 3, 2026 10:52
Playback started by an app kept running after the app exited or
crashed, and sys_audio_recording_stop_playback had no ownership check,
so any app could silence a playback it did not own (e.g. one started
from the Settings UI).

Track whether the active playback was app-initiated, stop it in the
process cleanup path alongside capture, and only let elevated callers
stop an app-owned playback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Add persistent recording quality, capture gain, and playback gain controls to the Voice Memos settings. Apply digital gain with saturation to recording and playback samples.

Co-authored-by: GPT-5 Codex <codex@openai.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
When the recording feed reaches end of payload it called
audio_endpoint_stop_transfer() from the new_timer callback, which sends
a BT message with a 4 s blocking timeout while holding the voice lock.
Right after streaming a whole payload the send buffer is likely
congested, so this could stall every new_timer client in the system.

Schedule the stop on the system task instead, guarded by the session
generation so it no-ops if the session ended in the meantime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The startup scan reset the next id in file order, so seeing vrec_65535
before vrec_1 primed the allocator onto an id that still had a stored
file, and voice_recording_start() never checked for collisions: after
the uint16 id space wrapped, finalize could silently overwrite an
existing memo.

Prime the allocator from the highest stored id and probe for a genuinely
free id before creating the temp file.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
Recordings owned by an uninstalled app were orphaned forever: invisible
to every other app (ownership-filtered listing) while still counting
against the shared storage budget, eventually making every recording
attempt fail with StorageFull.

Remove the app's recordings alongside its persist file on uninstall
(not on upgrade), skipping a file held open by an active transcription
stream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The size-only static assert let the applib and service structs drift
apart silently: reordering or retyping fields keeps sizes equal while
the list syscall cast garbles every field. Check offset and size of
each field instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
The length-prefixed frame parsing (read length byte, validate against
remaining payload and max frame size, read payload) was duplicated
between the playback feed and the transcription feed, so a container
format change fixed in one path would silently corrupt the other.

Extract voice_recording_storage_read_frame() and use it in both; adapt
the playback unit test to stub the shared reader (and the playback gain
accessor it now needs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
@julpel8
julpel8 marked this pull request as ready for review July 3, 2026 12:08
@julpel8
julpel8 requested review from gmarull and jplexer as code owners July 3, 2026 12:08
@ericmigi

ericmigi commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

would be cool to see an easy to use companion pkjs side that receives/decodes/streams audio from watch

@julpel8

julpel8 commented Jul 4, 2026

Copy link
Copy Markdown
Author

Is this heavy battery consumption related to audio feature ?

image

@ericmigi

ericmigi commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

are you using a debug or release build? debug builds enable UART RX which is very high power consumption

@julpel8

julpel8 commented Jul 4, 2026

Copy link
Copy Markdown
Author

are you using a debug or release build? debug builds enable UART RX which is very high power consumption

Yes, I don't set CONFIG_RELEASE. Thank you !

Signed-off-by: Julien Pelletier <pelletier.julien@pm.me>
@eiis1000

Copy link
Copy Markdown

Would really love to see this merged! There are so many apps I want to make with this feature...

Copy link
Copy Markdown
Collaborator

we're tracking it! just haven't had time yet sorry

@baibhavbista

baibhavbista commented Jul 21, 2026

Copy link
Copy Markdown

This is fantastic! offline on-device recording is going to be a really welcome addition.

However, I think there's a missing piece: audio_recording_transcribe() already transports the stored recording to the phone, but only exposes Pebble's transcript — there's currently no supported way for the initiating app (or its PKJS companion) to receive the recording itself. And a class of apps needs the audio, not just text.

Two reasons from my own use case (I'm building a Roam Research capture watchapp for PT2) that specifically need the recording, not just a transcript:

  • My own transcription — the built-in model struggles with my accent, so I need to re-transcribe the audio on my own backend.
  • Storing the recording — I want to save the recording in Roam alongside a timestamped transcript, so it's replayable and navigable later. In our equivalent apple watch app, this recording has proved to be a godsend in noisy situations where the AI misspells a lot of stuff.

Others in #719 have asked for the same (saving audio + transcript), so exposing an API to get the audio out would help a range of apps, not just mine.

One possible shape (drafted with AI assistance; the technical claims were checked against the PR source): a read API like audio_recording_read(id, offset, buf, len) (plus the codec params) that lets an app pull the stored Speex frames and forward them to a server. That said, see my follow-up below for a refinement I'd actually prefer, aligned with @ericmigi's earlier comment.

Happy to help in any way I can: whether testing or by contributing a stab at a follow-up PR (DCO + AI-disclosure understood). Thank you!

cc @ericmigi @jplexer to hear your thoughts on my request

@baibhavbista

baibhavbista commented Jul 21, 2026

Copy link
Copy Markdown

Following up on my comment above, I just noticed @ericmigi's earlier note about "an easy to use companion pkjs side that receives/decodes/streams audio from watch." That's exactly the shape that would work for me, and it's a better fit than the read API I mentioned above.

(Below is from Claude, so please take with grain of salt again)


audio_recording_transcribe() already streams the stored recording to the phone — if that same transport could hand the audio to the initiating app's PKJS companion (decoded, or as frames + codec metadata) instead of only the built-in transcriber, that would be all I need. PKJS could own the HTTP upload to my backend and return the result to the watch over AppMessage. An app-scoped export/stream that preserves the recording ID (AudioRecordingInfo.id) as a correlation ID, plus completion/error reporting, would be enough.

It also preserves this PR's offline model: keep the recording on-watch until the export is acknowledged, then the app can delete it.

(A byte-read API like audio_recording_read() would also work, but it'd make every app re-chunk the audio through AppMessage even though this PR already has a native transport — so I'd favor the companion stream if that matches where you're headed.)


Happy to help spec, test, or contribute a follow-up PR.

@julpel8

julpel8 commented Jul 22, 2026

Copy link
Copy Markdown
Author

Thanks @baibhavbista!

Getting the raw audio out makes a lot of sense.

I’ve actually already forked Brain Dump by @adrienthiery and got it listing the on-device recordings via audio_recording_list().
Adrien was interested in this feature too. I’ll add the PKJS code to ship the audio out to a backend in my fork branch soon.

@julpel8

julpel8 commented Jul 22, 2026

Copy link
Copy Markdown
Author

@baibhavbista After reviewing it, I realized I hadn’t pushed the changes that make it easier to list, edit and listen to the audio.
PS: I vibe write a doc in docs/reference/voice_recordings.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants