Skip to content

feat(video): support typed input references - #13706

Draft
furionw wants to merge 1 commit into
qiwa/video-reference-download-limitsfrom
qiwa/video-typed-input-references
Draft

feat(video): support typed input references#13706
furionw wants to merge 1 commit into
qiwa/video-reference-download-limitsfrom
qiwa/video-typed-input-references

Conversation

@furionw

@furionw furionw commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Based on #13705.

Why

The video API exposes only one untyped image reference, while vLLM-Omni pipelines can condition generation on image, video, and audio inputs. Dynamo needs a typed contract that preserves per-modality order, materializes remote media within request-level limits, and adapts singleton values to existing pipeline ABIs. Backends without this capability must reject the field instead of silently producing unconditioned video.

What Change

  • Add typed image, video, and audio references to Rust and Python protocols.
  • Materialize validated references with count, byte, and cleanup boundaries.
  • Scalarize generic singleton inputs and reject unsupported backend usage.
classDiagram
    NvCreateVideoRequest --> VideoReferenceMaterializer : supplies typed references
    VideoReferenceMaterializer --> OmniHandler : returns pipeline-ready paths
    OmniHandler --> OmniTextPrompt : attaches multimodal data

    note for VideoReferenceMaterializer "Bounds, stages, groups, and cleans request media"
    note for OmniHandler "Adapts references to the selected pipeline ABI"
Loading

Test Plan

  • Run Rust/Python video protocol and materializer tests.
  • Run SGLang and TensorRT-LLM rejection tests.

@github-actions github-actions Bot added the feat label Aug 24, 2026
@github-actions github-actions Bot added backend::vllm Relates to the vllm backend backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal labels Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend backend::trtllm Relates to the trtllm backend backend::vllm Relates to the vllm backend feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant