[Remyx Recommendation] Bring consistency distillation to text-to-audio pipelines (ConsistencyTTA)

[github-actions]

Recommended paper: ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation
Confidence: moderate
Research interest: (pin-arxiv)

License & code availability

🟢 Permissive license — safe to adopt.

• Code: https://github.com/Bai-YT/ConsistencyTTA
• License: MIT (class: permissive, compat: 1.00, source: arxiv_html_retry)

---

Optional subtitle: Single-step, CFG-aware latent consistency for AudioLDM2/Tango + optional CLAP closed-loop fine-tuning.

Why this paper is interesting for the team

ConsistencyTTA cuts text-to-audio (TTA) diffusion inference to a single
non-autoregressive denoiser query (~400x fewer queries) by adapting latent
consistency models to the audio latent space and folding classifier-free
guidance (CFG) directly into the consistency training, then optionally
fine-tuning closed-loop against an audio-space text metric (CLAP score). For
Diffusers — which already ships AudioLDM2 / MusicLDM / Tango-style pipelines
and an image-only consistency-distillation example — this is a natural
bridge from slow N-step TTA sampling to one-step TTA generation, reusing the
library's existing latent-consistency machinery on the audio side.

What blocks a clean implementation

• No TTA consistency-distillation trainer exists. examples/consistency_distillation/
  is present but image-only (Latent Consistency Models for SDXL/SD via
  train_lcm_distill_*.py). There is no audio/TTA variant to port
  ConsistencyTTA's CFG-aware latent consistency objective into, and no audio
  latent + UNet training path is wired for it.
• CLAP is used only as a conditioning encoder, not as an eval metric.
  Inside audioldm2 / musicldm / audioldm pipelines, CLAP encodes the
  text prompt for generation. ConsistencyTTA's closed-loop fine-tuning needs
  CLAP as an audio↔text alignment score used as a training/reward signal —
  no such scorer or TTA eval harness exists in the repo (and scripts/ is
  entirely checkpoint-conversion utilities, so there is no eval hook to wire
  one into).
• The core contribution is a training procedure, not a drop-in module.
  Delivering the result means a distillation trainer plus an audio-metric
  reward loop, not a scorer or filter that slots into an existing scripts/
  call site. No specific call site was named by the selection pass, and the
  natural home (examples/, not scripts/) is a new training example rather
  than an edit to existing code.

What we'd need to know / decide first

1. Scope: trainer or eval-only? Is adding a TTA consistency-distillation
   trainer in scope (mirroring examples/consistency_distillation/ but
   targeting AudioLDM2/Tango, with CFG folded into the consistency loss), or
   should a first step be a standalone CLAP-based TTA quality scorer that the
   existing audio pipelines can opt into?
2. Which closed-loop metric? CLAP (transformers.ClapModel is available,
   but Diffusers does not currently depend on it for eval) versus a lighter
   audio-space proxy — and whether closed-loop fine-tuning is part of the
   first PR or a follow-up.
3. Target anchor pipeline. AudioLDM2 (VAE latent + UNet) is the closest
   existing surface for latent consistency. Do we commit to it as the anchor,
   or keep the distillation path TTA-generic (AudioLDM2 / Tango / MusicLDM)?

---

Opened by the Remyx Recommendation orchestrator — the coding agent elected Issue-mode rather than scaffolding a PR for this paper.

Reopen this Issue if you want Outrider to revisit this paper later. While it stays closed, the orchestrator will not re-recommend the same paper.