[enhancement] multi-talker Word Error Rate (mtWER) as a metric?

[namgyaaal]

Hello,

Is there any interest in adding mtWER as a metric to this library?

This metric is found in some papers working on multi-speaker evaluation [1, 2] and seems like it would be a good fit to have for this library.

Thanks!

[1] Joint ASR and Speaker Role Tagging with Serialized Output Training
[2] The CHiME-8 MMCSG Challenge: Multi-modal conversations in smart glasses

[thequilo]

Hey @namgyaaal!

Do you know if an open implementation of the mtWER exists? The CHiME-8 paper is not particularly clear about how the speaker attribution errors are determined. Quote:

These error counts are obtained by jointly searching for the best alignment of the reference words of SELF and OTHER with the words in the hypothesis. Note that some words can be aligned to a wrong word of a wrong speaker, thus being both substituted and mis-attributed.

It is unclear how the joint search works. My guess is it is similar to the word-level MIMO WER (or what is done in ascilte) and from the assignment (which words matched across all speakers) the number of confusions is estimated. If my guess is right, it is totally possible to implement in meeteval.

[namgyaaal]

Hey @namgyaaal!

Do you know if an open implementation of the mtWER exists? The CHiME-8 paper is not particularly clear about how the speaker attribution errors are determined. Quote:

These error counts are obtained by jointly searching for the best alignment of the reference words of SELF and OTHER with the words in the hypothesis. Note that some words can be aligned to a wrong word of a wrong speaker, thus being both substituted and mis-attributed.

It is unclear how the joint search works. My guess is it is similar to the word-level MIMO WER (or what is done in ascilte) and from the assignment (which words matched across all speakers) the number of confusions is estimated. If my guess is right, it is totally possible to implement in meeteval.

Thanks for getting back!!

Did some digging and found out that the implementation was here: https://github.com/facebookresearch/MMCSG/tree/main/tools/multitalker_wer

I don't know the DP algorithm for WER to give a good answer but looking at compute and get_result here https://github.com/facebookresearch/MMCSG/blob/6683f9aea58c77dbe9f12eedf0c8e09163102222/tools/multitalker_wer/MultiTalkerEditDistance.cpp#L36 it looks like it keeps scores for all speakers then and estimates from them to get the confusion, which seems to be what you're saying?