Why is DMD needed if CPS gives tractable reverse KL

[ssshamiii]

RTDMD uses CPS sampling, where each denoising transition can be written as an isotropic Gaussian:

$$p_\theta(x_{t-1}\mid x_t)=\mathcal N(\mu_\theta,\rho_t^2 I)$$

So the reverse KL to a frozen reference/teacher policy seems available in closed form:

$$ D_{\mathrm{KL}}(p_\theta || p_{\mathrm{ref}}) = \frac{|\mu_\theta-\mu_{\mathrm{ref}}|^2}{2\rho_t^2} $$

assuming the same CPS noise schedule, summed over denoising steps.

If the final goal is reverse-KL-regularized policy optimization, what is the necessity of the extra DMD + fake-score stage?

Is DMD mainly a warm start / variance reduction trick, or is there a theoretical reason direct CPS-based RL/KL optimization is insufficient?

Looking forward to your clarification.

[Harahan]

Of course, you can directly employ CPS-based RL for a flow model. But in this work, we try to do both few-step distillation and RL in the same pipeline.

[ssshamiii]

Thanks for the reply!

I get that you want to combine distillation and RL in one pipeline. But I'm wondering: is DMD strictly necessary for few-step RL?

Take Flow-GRPO for example, it just reduces sampling steps during RL (from 30 to 10) directly (no distillation or fake-score stage) and works great.

So what exactly does the DMD stage bring to the table that pure few-step RL can't? Did you happen to compare your method against a pure few-step RL baseline? I'm just trying to better understand the specific advantages the DMD stage adds on top of direct RL. Thanks again for your time!