Are the models on the demo site and in the repository the same?

[Wolfman1219]

I segmented several audios using the sam-audio-large model. I got ideal results from https://aidemos.meta.com/segment-anything/editor/segment-audio/ , but I didn't get good results in a local environment with the same prompt and audio.

[lematt1991]

The demo model uses 8 candidate re-ranking, and for text based separation, uses span prediction. You should be able to replicate with:

with torch.inference_mode():
    result = model.separate(batch, predict_spans=True, reranking_candidates=8)

The README has been updated to clarify this.

[Wolfman1219]

Thank you for quick answer. I don't have enough memory for even small model and while trying with reranking_candidates=4. I got this error:

File /mnt/data07/llm/deploys/sam-audio/sam_audio/model/model.py:322, in SAMAudio.separate(self, batch, noise, ode_opt, reranking_candidates, predict_spans)
317 elif reranking_candidates > 1 and self.text_ranker is not None:
318 input_audio = [
319 audio[:, :size].expand(reranking_candidates, -1)
320 for audio, size in zip(batch.audios, sizes, strict=False)
321 ]
--> 322 scores = self.text_ranker(
323 extracted_audio=target_wavs,
324 input_audio=input_audio,
325 descriptions=batch.descriptions,
326 sample_rate=self.audio_codec.sample_rate,
...
--> 467 idx = np.random.integers(0, overflow + 1)
468 audio_data = audio_data[idx: idx + max_len]
470 else: # padding if too short

AttributeError: module 'numpy.random' has no attribute 'integers'

I think this is bug with version mismatching.

[lematt1991]

Thanks for reporting! This is indeed an issue with CLAP. I've created a PR to fix: LAION-AI/CLAP#185. In the meantime, you can pip install git+https://github.com/lematt1991/CLAP.git. #40 provides a temporary workaround until the PR is merged.

[Wolfman1219]

Thanks! Adding reranking_candidates worked well for the large model. However, it’s still failing on the small and base versions. Given that my audio quality is very poor, this result isn't entirely unexpected.

[aliiice00ai]

The demo model uses 8 candidate re-ranking, and for text based separation, uses span prediction. You should be able to replicate with:

with torch.inference_mode():
result = model.separate(batch, predict_spans=True, reranking_candidates=8)
The README has been updated to clarify this.

Hi, thx for your advice, but my local result still doesn't match the online demo after I set predict_spans=True, reranking_candidates=8:

Should any other parameters be aligned?

[Wolfman1219]

Did you use large model?

[aliiice00ai]

Did you use large model?

Yes

[chevalierNoir]

@aliiice00ai Hi, there does exist some randomness and model is sensitive to duration especially for speech separation. Can you try the following: (1). disabling the span prediction and set the reranking_candidates=8 (2). try use the checkpoint: sam-audio-large-tv? If that still doesn't work, please share the audio example and prompt if that's okay?

[peterpans01]

@chevalierNoir I also feel like the duration is a factor here. I tried with different durations and results are hits-or-miss.

[chevalierNoir]

Many of the training data are of ~10s audios. So I recommend to try 10s as the default duration.

[Wolfman1219]

Am I need to reopen the issue?

[chevalierNoir]

Am I need to reopen the issue?

We can leave it as is. Currently just seem to be randomness.