Skip to content

Support image input for Grok - #58

Closed
hassaans wants to merge 2 commits into
raine:mainfrom
hassaans:grok-image-input
Closed

Support image input for Grok#58
hassaans wants to merge 2 commits into
raine:mainfrom
hassaans:grok-image-input

Conversation

@hassaans

@hassaans hassaans commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Grok's translator only modelled text content, so any Anthropic image block was rejected with unsupported content block: image — even though grok-4.5 is multimodal. This makes vision unusable when driving Grok from Claude Code, and it is more than a missing feature: a single pasted screenshot bricks the whole conversation, because the image stays in history and every subsequent request re-sends it (so every following turn 400s until the image is removed).

Change

Add an input_image content part and translate Anthropic image blocks in user messages to it, matching the Responses-style shape the Grok provider already uses:

  • base64 sources become a data:<media_type>;base64,<data> URL.
  • url sources pass the URL through.
  • Unknown image sources and unexpected block keys still fail loudly (consistent with the existing strict validation).

count_tokens gains a rough per-image estimate so its match stays exhaustive.

Verification

  • Verified end to end against grok-4.5: a solid magenta test image is described correctly ("magenta") through the translated request — so the input_image shape is accepted by xAI and vision actually works, not just compiles.
  • 3 new tests (base64 → input_image, url → input_image, unknown source / stray key rejected), matching the existing test conventions.
  • cargo test --locked: all passing. cargo fmt --check clean. No new clippy warnings.

hassaans and others added 2 commits July 19, 2026 20:16
Grok's translator only modelled text content, so any Anthropic `image`
block was rejected with "unsupported content block: image" — even though
grok-4.5 is multimodal. This makes vision unusable when driving Grok from
Claude Code (a pasted screenshot bricks the whole conversation, since the
image stays in history and every subsequent request re-sends it).

Add an `input_image` content part and translate Anthropic image blocks in
user messages to it, matching the Responses-style shape the Grok provider
already uses:

- base64 sources become a `data:<media_type>;base64,<data>` URL.
- url sources pass the URL through.
- Unknown image sources / unexpected block keys still fail loudly.

Verified end to end against grok-4.5: a solid magenta test image is
described correctly ("magenta") through the translated request.
count_tokens gains a rough per-image estimate so it stays exhaustive.
Image support needs to cover screenshots returned by tools as well as images in
ordinary user messages. Otherwise an image-producing MCP or browser tool leaves
the conversation in a request shape the Grok translator cannot accept.

Represent function outputs as either text or Responses-style content arrays so
text and image children retain their ordering. Validate each Anthropic image
source variant, reject formats outside Grok's documented JPEG and PNG support,
and accept explicit null cache controls. Extend token estimation and regression
coverage to include images inside tool results.
@raine

raine commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Thanks for putting this together. I verified that the PR fixes the original failure for ordinary user images and image-bearing tool results, including end-to-end vision against Grok.

I am closing this because #69 has been merged and supersedes this implementation. It includes the same image support plus graceful omission defaults, opt-in vision modes, upstream image validation and size gates, request-wide limits, traffic redaction, and broader regression coverage. Those safeguards also address a verified edge case here where a valid 1x1 PNG passes local translation but Grok rejects it, producing a 502.

The branch also conflicts with current main in both changed files, so there is no remaining code to carry forward from this PR.

@raine raine closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants