[SongBloom]: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement

We propose SongBloom, a novel framework for full-length song generation that leverages an interleaved paradigm of autoregressive sketching and diffusion-based refinement. SongBloom employs an autoregressive diffusion model that combines the high fidelity of diffusion models with the scalability of language models. Specifically, it gradually extends a musical sketch from short to long and refines the details from coarse to fine-grained. The interleaved generation paradigm effectively integrates prior semantic and acoustic context to guide the generation process. Experimental results demonstrate that SongBloom outperforms existing methods across both subjective and objective metrics and achieves performance comparable to the state-of-the-art commercial music generation platforms.

Demo page: https://cypress-yang.github.io/SongBloom_demo

ArXiv: https://arxiv.org/abs/2506.07634

Prepare Environments

conda create -n SongBloom python==3.8.12
conda activate SongBloom

# yum install libsndfile
# pip install torch==2.2.0 torchaudio==2.2.0 --index-url https://download.pytorch.org/whl/cu118 # For different CUDA version
pip install -r requirements.txt

Data Preparation

A .jsonl file, where each line is a json object:

{
	"idx": "The index of each sample", 
	"lyrics": "The lyrics to be generated",
	"prompt_wav": "The path of the style prompt audio",
}

One example can be refered to as: example/test.jsonl

The prompt wav should be a 10-second, 48kHz audio clip.

The details about lyric format can be found in docs/lyric_format.md.

Inference

source set_env.sh

python3 infer.py --input-jsonl example/test.jsonl

TODO List