for test on MVSA

[radiateall777]

When reproducing, in 5 experiments on the MVSA dataset, the best result was 69 when ε = 5.0, while the result in the paper was 73.85 ± 1.42，are there some other settings for inference? @QingyangZhang

[QingyangZhang]

Hi, we recently re-run the experiment by training one model on MVSA and evaluating 5 seeds on the test set, achieving an accuracy of 73.06±1.77 under Gaussian noise. We suspect that the result of 69 is due to the evaluation script's default use of salt-pepper noise. You can add the 'noise type' parameter to the script, as in: python eval_qmf.py ..... --noise $noise --noise_type Gaussian. Besides, please kindly notice that MVSA is relatively small and there is no official data split. Thus the performance on this dataset is usually highly unstable. Please let me informed whether you still have trouble in reproducing.

[nollexu]

I also reproduced the MVSA dataset, and the average accuracy was indeed only around 66% when ε = 5.0.

[QingyangZhang]

I also reproduced the MVSA dataset, and the average accuracy was indeed only around 66% when ε = 5.0.

Sorry for the late response. After carefully reviewing the code and re-run the results, I found two possible reason for this issue:

• The default hyparameters in the original training script is not optimal. For example, the number of image embedding should be 3 not 1. You need to follow the demand line we provided in the original readme.

• The text modality's noise is directly implemented in get_item() function. This causes the training code to not differentiate between training and testing set———when args.noise is not 0, it adds text noise to both training and test data.

Would you mind try the latest code by

python train_qmf.py --noise_level 5.0 --noise_type Gaussian

The results of QMF under Gaussian noise with epsilon=5 is as follow, similar to that in the paper:

Feel free to contact if the problem remains.