Map audio/mpeg to the OpenAI mp3 input audio format - #6710
Open
chabinhwang wants to merge 1 commit into
Open
Conversation
The audio input format was derived by looking for "mp3" in the media MIME type, so "audio/mpeg", the media type registered for MP3 and the one Spring's MimeType detection returns, was silently sent to OpenAI as wav. Resolve the format in a dedicated method that also accepts "audio/mpeg" and warns when an unsupported audio type is forwarded. Signed-off-by: chabinhwang <7chabin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The OpenAI audio input format was chosen with
mimeType.contains("mp3"), falling back to WAV for anything else.audio/mpegis the media type registered for MP3 (RFC 3003) and does not contain the substringmp3, so MP3 audio declared that way was silently sent to OpenAI labeled aswav.This moves the decision into
toInputAudioFormat(String), which acceptsaudio/mp3andaudio/mpeg(case-insensitively) for MP3 and warns when an audio type that is neither MP3 nor WAV is forwarded as WAV.Why
audio/mpegmattersMedia.Formatexposes no audio constants, so callers build the MIME type themselves, andaudio/mpegis what standard detection produces for an.mp3file.Spring AI's own speech documentation uses
audio/mpegfor MP3 output, so feeding generated speech back in as chat input hits exactly this path.The failure is silent at the Spring AI layer — the request is built and sent with the wrong format declaration.
The reference documentation previously listed only
audio/mp3andaudio/wav; it now listsaudio/mpegas well.Testing
Two parameterized tests added to
OpenAiChatModelTests, both going throughcreateRequestso they assert on theChatCompletionCreateParamsactually sent:userMessageMp3MediaMapsToMp3InputAudioFormat—audio/mp3,audio/mpeg,audio/MPEGuserMessageWavMediaMapsToWavInputAudioFormat—audio/wav,audio/x-wav,audio/vnd.waveBoth also assert the base64 payload is unchanged.
Verified locally:
./mvnw -Dmaven.build.cache.enabled=false -pl models/spring-ai-openai clean package—Tests run: 176, Failures: 0, Errors: 0, checkstyle andspring-javaformatpassaudio/mpeghandling and re-ran the new tests: theaudio/mpegandaudio/MPEGcases fail withexpected: mp3 but was: wav, while theaudio/mp3and WAV cases pass either wayIntegration tests requiring an OpenAI API key were not executed.
Scope
Only the input audio format resolution and the corresponding documentation line changed. The output audio path and the transcription/speech models are untouched.