Skip to content

Train from inline multimodal rollouts - #3320

Draft
eligotts wants to merge 5 commits into
mainfrom
feat/v1-inline-mm-vllm
Draft

Train from inline multimodal rollouts#3320
eligotts wants to merge 5 commits into
mainfrom
feat/v1-inline-mm-vllm

Conversation

@eligotts

@eligotts eligotts commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • accept raw multimodal content_parts on the pinned vLLM token endpoint and return its effective expanded prompt_token_ids
  • build on main's renderer-free orchestrator path and trace-stamped mm_token_type_id_map, carrying only inline image URLs and expanded placeholder ranges through transports.batch
  • preserve raw image refs through truncation, packing, and workload splitting so distributed runs avoid unnecessary dummy multimodal batches
  • materialize one trainer microbatch at a time through a small model-family adapter registry with explicit forward and context-parallel policies
  • support Qwen-VL and Kimi K2.5 while leaving the existing multimodal SFT path unchanged
  • pin Renderers PR length rewards #133 and Verifiers PR fix(ring_attn): make FA3 causal/window_size kwargs version-agnostic #2417 at 36f8636f3

Stack

Validation

  • uv run pytest -q tests/unit/inference/test_serving_tokens.py tests/unit/orchestrator/test_batch.py tests/unit/orchestrator/test_qwen3_vl_e2e.py tests/unit/train/test_multimodal.py tests/unit/train/test_model_forward.py (44 passed)
  • Ruff check and format passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant