Add ordered content parts to UserMessage - #6729
Open
dmitriierokhin wants to merge 7 commits into
Open
Conversation
A UserMessage pairs a single text with a collection of media and defines no ordering between them, so providers can only serialize it as text followed by all media. Prompts that need media positioned relative to text are not representable: a per-page document prompt where each page's image must directly follow that page's own text loses the association the model relies on. Introduce ContentPart, a sealed TextPart/MediaPart pair, and let a UserMessage carry an ordered list of them. The existing text and media accessors become faithful projections of the parts and vice versa, so a message built either way reads correctly through either accessor, providers that cannot express ordering keep working unchanged, and token counting still sees a single source of truth. Builder.text and Builder.media discard any previously set parts eagerly. That flattens ordered content rather than leaving a stale projection behind, so components which rewrite user text keep working; Builder.appendText is added as the structure-preserving alternative for them to migrate to. hasInterleavedContent() reports whether flattening would lose information, letting providers with a flat text-plus-media shortcut keep taking it for content that is flat-equivalent. getMedia() now returns an unmodifiable view, since mutating it would desynchronize it from the content parts. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Gemini's Content carries an ordered Part list, which maps one-to-one onto a user message's content parts, so an interleaved prompt survives verbatim. The media mapping is extracted into mediaToPart so a single MediaPart can reuse it. For a message built in the flat form the parts are the text followed by the media, so this produces the same Part list as before. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Anthropic's content is an ordered ContentBlockParam list, so a user message's content parts map onto it directly. The block-building branch is now also entered when flattening the content would lose information, otherwise a message carrying several text parts would fall into the flat fast path and be collapsed into one block. Messages whose content is flat-equivalent still take that fast path, so their requests are unchanged. Cache control moves from the first text block to the last one: a cache prefix has to cover the whole content, and with several text blocks the first no longer marks its end. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Bedrock's Converse content is an ordered ContentBlock list, so a user message's content parts map onto it directly. Walking the parts also stops an empty text block being sent for a message whose content is media only, which the previous unconditional ContentBlock.fromText produced and which Bedrock rejects. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Mistral's content is an ordered ContentChunk list, so a user message's content parts map onto it directly. The parts are handled before the assertion that the message text is non-null, since a message whose content starts with media has no leading text to assert on. mapToImageUrlChunks is removed, its only caller being the stream-concat this replaces. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
OpenAI's content is an ordered ChatCompletionContentPart array, so a user message's content parts map onto it directly. The media mapping is extracted verbatim from the inline lambda into toContentPart, returning null for media it cannot represent so the caller skips it exactly as the lambda did. The content-parts array is now also built when flattening the content would lose information; a message with no media whose content is flat-equivalent still takes the plain string path, and system messages are untouched. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Ollama's message format is a text plus a separate image list and cannot express an ordering between them, so ordered content parts are flattened to text-then-images by the projections. Record that at the conversion site; no behaviour changes. Signed-off-by: Dmitrii Erokhin <dmitrii.erokhin@webbfontaine.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
UserMessageholds a singletextplus aList<Media>with no ordering relation betweenthem, so every provider converter can only ever serialize a user turn as all text, then all
media.
Some multimodal prompts depend on a finer ordering. The case that led me here is a per-page
document prompt: each page contributes a marker, its OCR text, and its page image, and the
image must directly follow that page's own text so the model associates the two. Flattened to
text-then-media, the page↔image association is gone. There is currently no way to express this
through the core message API, even for providers whose wire format is itself an ordered list of
parts.
Solution
Add
ContentPart(spring-ai-commons,org.springframework.ai.content) — a sealedTextPart|MediaPart— and let aUserMessagecarry an orderedList<ContentPart>:The design point that keeps this additive:
textandmediaremain faithful projections ofthe parts, and vice versa. They are never a second source of truth.
getContentParts()as the derived[text, media…]sequence, so providers need only one code path.
getText()(text parts joined with\n) andgetMedia()(media parts in order).
with no change at all, and token counting still sees one source of truth. I deliberately did
not add anything to
MediaContent, soJTokkitTokenCountEstimatorcannot double-count.AbstractMessage's non-null-text invariant is untouched: a media-only message projects to"".hasInterleavedContent()reports whether flattening would lose information (i.e. anything otherthan one leading text followed by media). Providers with a flat text-plus-media shortcut keep
taking it while that is
false, so existing requests are byte-identical; only genuinely orderedcontent takes the parts path. This is what stops the change rewriting the wire format for every
existing Anthropic and OpenAI user message.
Provider adoption
spring-ai-google-genaiList<Part>mediaToPartextracted for reusespring-ai-anthropicList<ContentBlockParam>spring-ai-openaiChatCompletionContentPart[]toContentPart(Media)extracted verbatim from the inline lambda, then parts walkspring-ai-bedrock-converseList<ContentBlock>spring-ai-mistral-aiContentChunkspring-ai-ollamacontent+imagesspring-ai-deepseekTwo pre-existing issues were fixed incidentally, because walking the parts replaced the code
that contained them:
ContentBlock.fromText(userMessage.getText())unconditionally, so amedia-only message produced an empty text block, which Bedrock rejects.
media threw.
Backwards compatibility
Every existing
UserMessageconstructor,Buildermethod,getText(),getMedia()contents,toString(),equals()andhashCode()keep their exact signatures and semantics. For amessage built the flat way
getText()returns the identical string, including empty andwhitespace-only values, and
getMedia()returns the identical contents in the identical order.copy()/mutate()round-trip faithfully. No provider requires a change. Nothing isdeprecated.
Three behaviours do change, all intentional and covered by tests:
getMedia()now returns an unmodifiable view. Mutating the returned list was never asupported contract, and it would desynchronize the list from the content parts. One test in
spring-ai-mistral-aidid this (userMessage.getMedia().add(...)) and is updated to use thebuilder.
Builder.text(...)/Builder.media(...)discard previously set content parts. This isenforced eagerly in each setter rather than validated in
build(), deliberately: componentsthat rewrite user text (
ChatModelCallAdvisor,StructuredOutputValidationAdvisor,Prompt.augmentUserMessage) all domutate().text(...), and throwing there would break thewhole advisor chain the moment one interleaved message reached it. Flattening loses ordering
but keeps all content, which is exactly today's behaviour.
Builder.appendText(String)isadded as the structure-preserving alternative for those callers to migrate to; I left the
migration itself out of this PR.
getText()is"". Componentsthat treat user text as a query would see an empty query for such a message.
Known limitation
UserMessagestill does not overrideequals/hashCode, so neither media nor part orderingparticipates in equality — unchanged from today, and pinned by a test with a comment. Including
parts in equality now would give identity semantics, because
Mediahas noequals/hashCode;that would make messages round-tripped through the Redis and Neo4j chat-memory repositories stop
comparing equal to their originals. The follow-up is value-based
equalsonMediafirst.Deliberately out of scope
ChatClient's fluent surface.DefaultChatClientUtilsonly materializes aUserMessagewhenStringUtils.hasText(processedUserText), so a blocks-only spec would be silently dropped, andDefaultPromptUserSpecholds media as a flat list. Threading parts throughPromptUserSpec→DefaultChatClientRequestSpec→DefaultChatClientUtilsis a separatedesign question (including how
param()rendering should apply per part), andChatClientRequestSpec.messages(Message...)already accepts a pre-builtUserMessagein themeantime. Happy to follow up if you'd like it in the same change.
Tests
ContentPartTests(new) — construction, null rejection, blank text allowed, record equality(and that
MediaPartequality is necessarily identity-based givenMediahas noequals).UserMessageTests— all pre-existing tests pass unmodified, which is the main regressionsignal. Added: projection in both directions, the
\njoin separator, no media marker leakinginto the text projection, builder channel exclusivity in both orders,
copy()/mutate()fidelity including that
mutate().text(...)leaves no stale part carrying the old text,appendTextin both modes,hasInterleavedContent()truth table, immutability of bothreturned lists, and equality semantics pinned as-is.
CreateGeminiRequestTests— interleaved parts asserted index-by-index on the resultingGeminiRequest(oneContent, N parts, alternatingtext()/fileData().fileUri()), partsalongside a
SystemMessage, and a case pinning the legacy flat path unchanged../mvnw clean packagepasses locally.Notes
Happy to split this into a core-only PR plus per-provider follow-ups if that is easier to
review, or to reshape the API — the naming (
ContentPart/TextPart/MediaPart) avoidscollisions with the Anthropic and Bedrock SDKs' own
ContentBlock/TextBlocktypes, but I haveno attachment to it.
This was developed with the help of an AI coding agent; I have reviewed every line and am
accountable for it, per CONTRIBUTING.md. The per-commit
Co-authored-bytrailers record that.