Support v1 inline raw multimodal images

[eligotts]

Summary

Intermediate multimodal storage mode — the inline bundle. Keeps the load-bearing architecture of the raw multimodal redesign — the image processor runs nowhere near the env worker: renderers compute token layout via pure geometry math, PrimeRlServingTokens materializes pixels in front of vLLM (with the byte-bounded single-flight materialize cache), and the trainer re-materializes from the same source via family adapters — but images travel inline as base64 data URLs through messages, traces, transport, and refs. No shared image directory, no offload.

Companion inline PRs:

• Renderers: Inline raw image refs for multimodal rendering renderers#110
• Verifiers: Inline raw multimodal images in the v1 train client verifiers#2120

Design:

• Serving: raw mmraw: refs embed the inline image source; materialization decodes and sha256-verifies the inline payload. Expected descriptor failures—including hash, fingerprint, grid, and placeholder-length mismatches—return invalid_mm_image_ref 400s. The byte-bounded single-flight cache keys on a digest of the complete ref plus its modality, outer hash, placeholder length, processor model, and trust setting, so a distinct or stale descriptor cannot alias a validated entry. Cache size is controlled by PRIME_RL_MM_MATERIALIZE_CACHE_GB.
• Trainer: RawImageMaterializer decodes inline bytes from the descriptor. MMImageRef drops the uri field — the image lives once, in the descriptor.
• Removed entirely (nothing to configure, nothing to go missing): [multimodal].offload_dir / MultimodalConfig, utils/run_assets.py, VF_RENDERER_IMAGE_OFFLOAD_DIR launcher + SLURM plumbing, and missing_mm_image_policy + the adapters' synthesize_placeholder machinery — an inline image cannot disappear, so the zero-loss placeholder path has nothing to guard.
• Kept: SFT renderers default to multimodal_output='processed' with a hard guard against 'raw' (storage-independent).
• Rollout records/traces keep the inline base64 (deliberate; noted in the monitor-run skill — expect payloads and results.jsonl to grow with turn count).
• The local RL launcher preserves the inherited process environment when spawning the orchestrator, including the virtualenv PATH and credentials.

Relationship to the offload bundle

This branch is cut from feat/v1-raw-mm-offload (#2836) and targets main independently. The offload bundle (renderers #89 / verifiers #1746 / prime-rl #2836) is the follow-on that layers content-addressed file:// storage on top: ingress offload walker, offload-dir plumbing, missing-image policy, and disk-backed refs.

Validation

• Original branch validation: tests/unit minus GPU-kernel model tests — 482 passed (398 non-train + 84 train non-models); uv lock --check clean; ruff check + format clean.
• Materialization hardening: uv run pytest -q tests/unit/inference/test_serving_tokens.py — 21 passed; touched-file ruff check and format check clean.

Update: hardening + restored main's multimodal packing (7d85b170, fb6ef306)

• Review hardening (7d85b170): subprocess env inheritance restored in rl_local's orchestrator launch; the materialize-cache key now digests the full ref ((sha256(raw_ref), modality, mm_hash, placeholder_len, model, trust_remote_code)) so a distinct descriptor can never alias an already-validated entry; adapter validation errors map to invalid_mm_image_ref 400s.
• Packing parity with main (fb6ef306): the merge had left raw-ref multimodal samples never packing, regressing main's feature. Same-family raw-ref samples now pack with each other and with text from the same run/LoRA; _materialize_bin merges refs with placeholder offsets rebased to the packed stream. Mirrors the offload branch's fix; tests restored to main's semantics.

Validation: tests/unit minus GPU-kernel model tests: 484 passed.

Update: unwrapped the ref payload (603118b / e40dded5)

raw_mm_ref no longer base64-wraps its payload — it serializes compact JSON directly. The ref travels as a string inside a JSON request body, so the only encoding cost is escaping the payload's own quotes (~200 bytes); the base64 wrapper instead inflated the whole payload by a third, and in inline mode the payload contains the image.

Measured on a 75 KiB JPEG: 130.3 KiB → 97.9 KiB per image slot on the wire (the 97.7 KiB data URL plus ~200 bytes of metadata/escapes) — a 25% cut, applied to every image slot of every request. Refs now parse with a single partition(":") since the payload contains colons; the base64-alphabet guard retired with the wrapper.

Update: merged main — vLLM 0.26 + agent metrics (0722181f3)

Merged prime-rl main (~23 commits), catching this branch up to the offload bundle's baseline: the vLLM 0.26 move (serving_tokens imports from scale_out.token_in_token_out, mm_input takes a MultiModalKwargsItems wrapper, online_renderer rename) resolved identically to #2836 with the inline raw_image_data decode path re-applied on top, plus main's agent metrics rename (#3165). Submodule pins advance to the companion PR merges (renderers #110 27fe3c6, verifiers #2120 10da86af0); uv.lock relocked with submodules at their merged pins. Unit suite: 411 passed.

Update: trust the train-time checkpoint contract at materialize (8e9c8c75a)

Render and materialize are bound to the same model.name, so materialize stops re-litigating layout policy (companion to renderers ce7078a):

• processor_fingerprint and the fingerprint comparisons are deleted from both materialize paths (vLLM front + trainer), along with RawMMItem.layout_fingerprint and the adapter-protocol method.
• The output-level grid and placeholder-length asserts stay: they compare actual processor output against the ref payload with values already in hand (zero extraction machinery), and they convert checkpoint skew into a clean per-request invalid_mm_image_ref error / trainer exception instead of an engine crash or silent training corruption.
• Adapters read the few knobs they still need (merge_size, patch_size) straight off the live processor; the renderers-side extractors are config-JSON-only now.

Materialize is now: read image → verify content hash → image_processor(images=…) → grid/length assert → pack. Unit suite: 411 passed.

Update: pruned RawMMItem to what materialize reads (56087658b)

Companion to renderers 20b3f2d: RawMMItem keeps family (adapter routing), the image source, and the adapter-owned payload; the unread modality/raw_ref/vllm_modality mirror fields and the now-orphaned _optional_str helper are gone. Unit suite: 411 passed.

---

Note

High Risk
Large cross-cutting change to multimodal data paths (inference serving, orchestrator transport, trainer batching/forward) with strict validation assumptions; regressions would surface as training failures or silent image/token misalignment rather than isolated UI bugs.

Overview
This PR switches v1 multimodal RL to inline base64 image descriptors end to end instead of shipping preprocessed pixel tensors through transport. Orchestrator samples carry lightweight mm_refs (descriptor + hash + placeholder span); inference and trainer each decode and verify bytes locally through family adapters (qwen_vl, kimi_k25).

Inference materializes every raw ref in PrimeRlServingTokens before vLLM sees the request, with a byte-bounded LRU cache (PRIME_RL_MM_MATERIALIZE_CACHE_GB, default 2 GiB), single-flight dedup, and invalid_mm_image_ref 400s on hash/grid/placeholder mismatches. Trainer uses RawImageMaterializer at batch load; packing groups same-family raw-ref samples and rebases image offsets in packed streams. Truncation cuts only at whole-image boundaries; orchestrator validates placeholder spans against mm_token_type_ids before samples ship.

Config/docs: SFT rejects multimodal_output='raw' and defaults to 'processed'; RL launcher preserves inherited subprocess env for orchestrator/trainer/inference. Docs and monitor-run skill cover inline payloads, cache metrics, and larger rollout artifacts.

^{Reviewed by Cursor Bugbot for commit 8e9c8c7. Bugbot is set up for automated code reviews on this repo. Configure here.}

Update: current-main integration (6353659ae)

Merged prime-rl main at f50ec5c4c. Resolved the current player.harness / player.runtime config shape, regenerated the two stale lockfile metadata lines, and pinned the companion PR heads exactly: renderers #110 at 50ce3c3e, verifiers #2120 at 1e35f768b.

Validation after the merge: uv lock --check, touched-file Ruff, and format checks passed; 48 targeted multimodal serving, batch, raw-ref packing, and schema tests passed. One unrelated trainer progress test could not import in the local CPU-only environment because flash_attn is not installed; GitHub CPU/GPU checks cover the full environment.

Update: fingerprints derived from renderer layout specs (d453dfa82)

Companion to renderers 9208b40: the renderer-side layout dataclasses are now the single canonical knob list per family, so the adapters stop re-listing the fields. processor_fingerprint is one line per family — qwen_layout_from(image_processor).fingerprint() / kimi_layout_from(...).fingerprint() — and the kimi materialize path reads patch shapes through the same extractor. Deletes the hand-duplicated _processor_value / _processor_layout helper stacks that previously had to be kept in sync with the renderers by hand. Fingerprint values are unchanged. Unit suite: 405 passed.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e40dded. Configure here.}

[cursor]

Unknown family yields 500

Medium Severity

get_multimodal_adapter raises NotImplementedError for an unknown family, but _materialize_raw_image_ref_sync only maps TypeError/ValueError to _MMImageRefError. An unrecognized descriptor family therefore bypasses the invalid_mm_image_ref 400 path and surfaces as an unhandled 500 from serving.

Additional Locations (1)

• src/prime_rl/multimodal/registry.py#L12-L17

 

^{Reviewed by Cursor Bugbot for commit e40dded. Configure here.}