This repository contains the implementation of Regression MLPs for V2A-Mapper Strategies, which is part of the "V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models" project. Please note that this repository is unofficial. Paper, Project page
Our lightweight method only requires the training of a V2A-Mapper to bridge the domain gap between the vision representative FM CLIP and the audio generative FM AudioLDM. The V2A-Mapper is supervised by the audio representative FM CLAP to learn the translation from visual space to auditory space. Leveraging the generalization and knowledge transfer ability of foundation models, the V2A-Mapper is trained with the same modestly sized dataset but the overall system can achieve much better performance.
Visualize the domain gap between CLIP image space and CLAP audio space
BEFORE Training
AFTER Training
The web APP currently only support Text-to-Audio generation. For full functionality please refer to the Commandline Usage
- Start the web application (powered by Gradio)
python app.py - A link will be printed out. Click the link to open the browser and play.
Part of the code is borrowed from the following repos. We would like to thank the authors of these repos for their contribution. [AudioLDM](https://github.com/LAION-AI/CLAP](https://github.com/haoheliu/AudioLDM)