fix(tts): decode the raw PCM Gemini returns

[Joilence]

Why

Two things stopped the Gemini voice working.

The default model, gemini-2.5-flash, is not a speech model: asked for audio it returns 400: This model only supports text output. That is the documented call.

Past that, the audio is raw PCM with no header. gemini.ts hands it to convertToWav, which runs ffmpeg -i pipe:0 with no format flag, so ffmpeg has nothing to recognise and exits with Invalid data found when processing input.

What

Default to gemini-3.1-flash-tts-preview. The native-audio models speak only the Live API socket, and every Gemini TTS model is preview.

convertToWav takes an optional input format and declares it with -f; other engines are unchanged.

parseRawAudioMime reads format and rate off the media type and refuses one it cannot read, since a wrong rate is silently wrong rather than an error. Little-endian contradicts RFC 2586, but it is what Google sends.

Test

21 tests, 16 failing on main. One decodes real audio through the ffmpeg CI installs.

Live calls confirm both models return audio, both payloads fail on main, and big-endian decodes to clipped noise where little-endian gives speech. 3.1 spells the media type audio/l16; rate=24000; channels=1, 2.5 audio/L16;codec=pcm;rate=24000; a test covers both.

Reproducing both failures

Steps 1 and 2 need only ffmpeg. Steps 3 and 4 also need GEMINI_API_KEY, curl and jq; step 3 costs nothing because it errors before generating anything.

#!/usr/bin/env bash
set -uo pipefail

TTS_MODEL=gemini-3.1-flash-tts-preview

# One second of headerless mono s16le at 24kHz: the shape Gemini's TTS models
# return, minus the RIFF header they do not send.
pcm() { ffmpeg -f lavfi -i sine=frequency=440:duration=1:sample_rate=24000 -f s16le - 2>/dev/null; }

# main's conversion: no format flag, so there is nothing for ffmpeg to sniff.
on_main() { ffmpeg -i pipe:0 -f wav -acodec pcm_f32le -ac 1 -ar 24000 pipe:1 2>&1 >/dev/null; }
fixed()   { ffmpeg -f s16le -ar 24000 -ac 1 -i pipe:0 -f wav -acodec pcm_f32le -ac 1 -ar 24000 pipe:1 2>/dev/null; }

say() { echo "   $*"; }

echo "1. main's conversion, on synthesized L16"
pcm | on_main | grep -i -m1 'invalid data' | sed 's/^/   /'

echo "2. same bytes, declaring the format"
say "$(pcm | fixed | wc -c | tr -d ' ') bytes of WAV"

if [ -z "${GEMINI_API_KEY:-}" ]; then
  echo "3-4. skipped, GEMINI_API_KEY not set"
  exit 0
fi

echo "3. main's default model, asked for audio"
curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"hello"}]}],"generationConfig":{"responseModalities":["AUDIO"]}}' \
  | jq -r '.error | "   \(.code) \(.message)"'

echo "4. real audio from $TTS_MODEL, through main's conversion"
resp=$(curl -s -X POST "https://generativelanguage.googleapis.com/v1beta/models/$TTS_MODEL:generateContent" \
  -H "x-goog-api-key: $GEMINI_API_KEY" -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"Say: testing one two three."}]}],
       "generationConfig":{"responseModalities":["AUDIO"],
       "speechConfig":{"voiceConfig":{"prebuiltVoiceConfig":{"voiceName":"Kore"}}}}}')
part=$(jq -r '.candidates[0].content.parts[0].inlineData' <<<"$resp")
say "media type: $(jq -r .mimeType <<<"$part")"
jq -r .data <<<"$part" | base64 -d > /tmp/gemini31.pcm
say "$(wc -c < /tmp/gemini31.pcm | tr -d ' ') raw bytes returned"
on_main < /tmp/gemini31.pcm | grep -i -m1 'invalid data' | sed 's/^/   /'
say "with the flag: $(fixed < /tmp/gemini31.pcm | wc -c | tr -d ' ') bytes of WAV"
rm -f /tmp/gemini31.pcm

Output here:

1. main's conversion, on synthesized L16
   Error opening input: Invalid data found when processing input
2. same bytes, declaring the format
   96080 bytes of WAV
3. main's default model, asked for audio
   400 This model only supports text output.
4. real audio from gemini-3.1-flash-tts-preview, through main's conversion
   media type: audio/l16; rate=24000; channels=1
   186240 raw bytes returned
   Error opening input: Invalid data found when processing input
   with the flag: 372560 bytes of WAV

Step 1 is the decode bug with argo taken out of the picture: that is the argv convertToWav builds on main today, and ffmpeg has nothing to sniff. Step 4 is the same failure on bytes the newest TTS model actually returned, so the decode fix is needed whichever model you point the engine at.

The failing tests need no key either, since the transport is mocked:

git checkout main
git checkout <this-branch> -- tests/tts/
npx vitest run tests/tts        # 16 failed | 84 passed (100)
git checkout HEAD -- tests/tts/ # restore

[shreyaskarnik]

Thanks @Joilence — this is a thorough piece of work on a feature that was, as you found, broken in two independent places at once.

The part I want to call out is the decision to throw when an L16 media type carries no readable rate, and the reasoning in that comment. Guessing 24000 for a 16000 stream doesn't fail — it returns a shorter clip at 1.5x pitch with exit code 0, and since Argo derives scene durations from clip length, every wait in the recording quietly shortens. That's the worst class of bug in this codebase and the comment names it exactly. Refusing is right.

Verified rather than read:

• Argument order is correct. -f s16le -ar -ac land before -i, so they configure the demuxer rather than the output. The apparent -ar/-ac duplication on both sides of -i is right too — input description vs. output normalisation — and you have a test pinning it.
• The roundtrip test is real. Reverting ...inputArgs produces four failures, one of them a genuine Command failed: ffmpeg -i pipe:0 ... — so it's exercising ffmpeg, not just asserting on mocked argv. Checking that byte-swapped data reads as near-full-scale noise is a nice touch: it proves the s16le vs s16be distinction actually matters rather than just asserting the string.
• Other engines are untouched. The third parameter is optional, so the ElevenLabs and Sarvam call sites are unaffected; 774 tests pass on the branch.
• Noting the little-endian deviation from RFC 2586 in the comment, rather than silently matching what Google sends, is the kind of thing that saves the next person an afternoon.

One follow-up, not a change request. gemini-3.1-flash-tts-preview is a preview SKU with no fallback, and the model test necessarily pins a literal string — it catches a regression back to a non-TTS model, but by construction it can't detect Google retiring the SKU, since the transport is mocked. When it rotates, the failure surfaces as a bare SDK 404 that reads like a bad API key. Worth an error hint that names the requirement ("this must be a TTS model") if it ever bites.

Also minor: docs/specs/2026-03-14-extensible-tts-design.md:92 still documents default: 'gemini-2.5-flash'.

Merging. Thanks again.