A Reinforcement Learning Framework for Unified Multimodal Models

U(you)·ni(need)·RL for unified multimodal intelligence

News 🚀

• [2026-06] DRPO released — "Rethinking the Divergence Regularization in LLM RL" (arXiv).
• [2026-06] Flow-DPPO released — "FlowDPPO: Divergence Proximal Policy Optimization for Flow Matching Models" (arXiv).
• [2026-06] CPPO released — "Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning" (arXiv).

About 💡

UniRL applies one RL post-training loop — generate samples, score them, compute advantages, update the policy, and sync weights back to rollout workers — across multimodal model families.

UniRL is a layered, composable system. Each entrypoint (train_diffusion, train_ar, train_pe, train_unified_model) loads a Hydra example config covering model, algorithm, rollout, reward, placement, and sync, then creates the matching domain trainer (DiffusionTrainer, ARTrainer, PETrainer, UnifiedModelTrainer). The trainer coordinates the RL loop across pluggable rollout engines, algorithms, model bundles, reward services, and the shared distributed runtime: Ray DevicePool, FSDP, Transfer Queue (TQ), and LoRA/full-weight sync. See unirl/README.md for the runtime loop, deployment modes, and module map.

Agentic entrypoints extend the AR path with multi-turn tool or environment interaction. They preserve each turn as a Sample lineage and support barrier, colocated partial-rollout, and disaggregated asynchronous trainers.

Team-Proposed Algorithms 🌟

🌟 These algorithms are proposed by our team — the highlight of UniRL. Each algorithm's folder holds a step-by-step tutorial and a runnable example recipe. We highly recommend trying them in our framework!

Algorithm	Paper	Tutorial	Notes
Flow-DPPO	"Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models"	FlowDPPO/	Diffusion/flow RL with an exact divergence-based trust-region mask.
DRPO	"Rethinking the Divergence Regularization in LLM RL"	DRPO/	Token-level LLM RL with a smooth advantage-weighted quadratic regularizer.
CPPO	"Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning"	CPPO/	Token-level LLM RL with a position-weighted, cumulative-prefix-budget Binary-TV mask.

UniRL also wires in standard reference algorithms — (LLM's)GRPO, DiffusionNFT, DanceGRPO, and MixGRPO — in unirl/algorithms/.

Model Support 🎨

Model and algorithm support are two independent dimensions that compose within a domain: any diffusion algorithm (see above) runs on a diffusion model, AR algorithms on AR models — so UniRL covers many more model × algorithm combinations than the shipped example recipes alone. The table below is the model dimension; all listed models are supported (✅).

Model	Category	Modality	Status
Stable Diffusion 3 / 3.5	Image diffusion	Text → Image	✅
Qwen-Image	Image diffusion	Text → Image	✅
FLUX.2-Klein	Image diffusion	Text → Image / Text + Image → Image	✅
Z-Image	Image diffusion	Text → Image	✅
WAN 2.1	Video diffusion	Text / Image → Video	✅
WAN 2.2	Video diffusion	Text / Image → Video	✅
HunyuanVideo 1.0 / 1.5	Video diffusion	Text → Video	✅
LTX-Video-2	Video diffusion	Text → Video	✅
LTX-Video-2.3	Video diffusion	Text → Audio + Video	✅
Qwen-VL	Vision-language AR	Text + Image → Text	✅
Qwen3	LLM AR	Text → Text	✅
Prompt-Enhancer	LLM + diffusion	Text → Text → Image	✅
HunyuanImage3	Unified AR + diffusion	Text → Image	✅
Bagel	Unified AR + diffusion	Text / Text + Image → Image	✅

Each model maps to a domain entrypoint (train_diffusion, train_ar, train_pe, train_unified_model); see Getting Started below to run any of them.

Training Modes 🧩

UniRL unifies four training modes, one Hydra example bucket and entrypoint each. Examples are self-contained YAML files selected with --config-name=<domain>/<example>:

Domain	Trains	Entrypoint	Example
diffusion/	Image / video diffusion models	train_diffusion	diffusion/sd3/sd3_sglang_rollout_colocate
ar/	Autoregressive models — vision-language (VLM) + text-only (LLM)	train_ar	ar/qwen_vl_grpo_geo3k_mc_4x8, ar/qwen3_drpo_4b_base_dapo_sglang
pe/	Prompt-enhancer (AR rewriter + diffusion reward)	train_pe	pe/pe_sglang_full_pickscore
unified_model/	Unified AR + diffusion models	train_unified_model	unified_model/hi3_vllmomni

See examples/README.md for the full launch guide, naming schema, and how to add a recipe.

Agentic Workflows 🤖

The agentic rollout engine repeatedly performs model generation followed by a tool or environment step, returning a trajectory of Sample objects. Entrypoints are named for their reward source and execution topology, not for a benchmark — each serves any recipe with a matching contract:

Reward source	Entrypoints	Example recipes
Environment's per-trajectory return	train_agentic_env, train_agentic_env_partial, train_agentic_env_async	examples/alfworld/
Graded terminal answer	train_agentic, train_agentic_partial, train_agentic_async	examples/deep_research/

The base entrypoint waits for a complete rollout batch. The partial variant keeps training and rollout colocated, while the async variant places training and rollout on separate GPU slabs and overlaps their producer/consumer loops. Both use an explicit tail policy: stateless tool trajectories may be carried when their Sample contains everything needed to resume, while stateful environment episodes and tool sessions must currently be dropped. Cross-worker stateful resume is deferred until its resource ownership and teardown contract is implemented.

See the agent-loop guide for the environment, tool, trajectory, and partial-resume contracts.

Getting Started ⚡

Install dependencies first — see INSTALL.md.

# compose-check, then launch a single-node example
python -m unirl.train_diffusion --config-name=diffusion/sd3/sd3_trainside --cfg job --resolve
bash examples/run_experiment_single_node.sh diffusion/sd3/sd3_trainside

Full launch guide — multi-node, every entrypoint, mooncake.

Roadmap 🗺️

We are actively expanding model and algorithm coverage. Near-term directions:

• Broaden algorithm coverage for the newer model families — FLUX.2-Klein, HunyuanVideo 1.0 / 1.5, and Bagel.
• Extend the team-proposed algorithms (Flow-DPPO, DRPO) to more model families.
• Broaden reward backends and rollout-engine coverage across domains.

Want a model or algorithm prioritized? Open an issue to discuss.

Contributing 🤝

Contributions and questions are welcome. Before opening a pull request, read the repository conventions in AGENTS.md, run the pre-PR checks for the files you touched, and fill in the pull request template. For questions, bug reports, and feature requests, open an issue.

Acknowledgement 🙏

UniRL builds on ideas and infrastructure from the open-source RL and inference ecosystem. We especially thank vLLM, SGLang, slime, and verl.

Citation 📚

If you find UniRL helpful, please cite:

@misc{unirl_github,
  title        = {{UniRL: A Reinforcement Learning Framework for Unified Multimodal Models}},
  author       = {Haonan Wang and Linyu Wu and Qian Qiu and Lewei Jin and Bowen Ping and Jianghai Chen and Yiheng Du and Guangxin He and Yu Shi and Yongguang Lin and Zhuoxin Zhou and Zhanchao Zhou and Keming Wu and Rizhen Hu and Xuefei Ning and Lvfang Tao and Feiyu Hu and Xiangyan Liu and Siqi Kou and Jiarui Yao and Xiangxin Zhou and Liefeng Bo and Wenxi Zhu and Tianyu Pang},
  year         = {2026},
  howpublished = {\url{https://github.com/Tencent-Hunyuan/UniRL}},
  urldate      = {2026-06-05}
}

If you use DRPO, please also cite:

@article{yao2026rethinking,
  title={Rethinking the Divergence Regularization in LLM RL},
  author={Yao, Jiarui and Zhou, Xiangxin and Qi, Penghui and Lee, Wee Sun and Bo, Liefeng and Pang, Tianyu},
  journal={arXiv preprint arXiv:2606.09821},
  year={2026}
}

If you use Flow-DPPO, please also cite:

@article{ping2026flow,
  title={Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models},
  author={Ping, Bowen and Zhou, Xiangxin and Qi, Penghui and Luo, Minnan and Bo, Liefeng and Pang, Tianyu},
  journal={arXiv preprint arXiv:2606.11025},
  year={2026}
}