Skip to content

feat(vllm-omni): preserve generated video audio - #13707

Draft
furionw wants to merge 1 commit into
qiwa/video-typed-input-referencesfrom
qiwa/omni-video-audio-output
Draft

feat(vllm-omni): preserve generated video audio#13707
furionw wants to merge 1 commit into
qiwa/video-typed-input-referencesfrom
qiwa/omni-video-audio-output

Conversation

@furionw

@furionw furionw commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Based on #13706.

Why

Joint audio-video diffusion models return video frames as the primary output and audio plus media rates as multimodal metadata. Dynamo currently reads only the frames and re-encodes a silent VP9 MP4, losing a model-generated modality. The formatter should preserve every generated video, mux synchronized audio when present, and leave the existing royalty-free silent-video path unchanged.

What Change

  • Normalize batched tensor and array video layouts without dropping outputs.
  • Mux generated audio into H.264/AAC MP4 responses with rate metadata.
  • Add an opt-in codec overlay with a real encode-and-reopen smoke test.
classDiagram
    DiffusionFormatter ..> MediaMuxer : muxes joint outputs
    DiffusionFormatter ..> VideoData : records actual media metadata
    DiffusionFormatter --> MediaStorage : uploads completed MP4 bytes

    note for DiffusionFormatter "Normalizes, pairs, encodes, and formats diffusion outputs"
Loading

Test Plan

  • Run formatter tests for silent, joint, tensor, and batched outputs.
  • Build the codec overlay and validate both MP4 streams.

@github-actions github-actions Bot added the feat label Aug 24, 2026
@github-actions github-actions Bot added backend::vllm Relates to the vllm backend frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal labels Aug 24, 2026
@furionw
furionw force-pushed the qiwa/omni-video-audio-output branch from 6aff7de to 0e3911f Compare August 24, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant