[Reproduce] Cannot Reproduce the results.

[bbhbbxwy]

Hello, could you provide the pre-trained weights for the second stage? Following the steps you provided, the results I reproduced are MAE: 9.04 and RMSE: 95.87, which are quite different from the test results you provided. Thank you!

[jerpelhan]

Hi!
Are these the results you obtain when running the training script, or when running evaluate.py? In evaluate.py, additional upsampling and SAM mask postprocessing are applied.

[bbhbbxwy]

Hi! Are these the results you obtain when running the training script, or when running evaluate.py? In evaluate.py, additional upsampling and SAM mask postprocessing are applied.

Hello! These are the results I obtained when running evaluate.py. The validation results are quite similar to yours, but the test results, with MAE 9.04 and RMSE 95.87, are significantly different from what you provided.

[xian-stack]

The results i got:
val:
MAE: 10.09
RMSE: 42.10
'AP': 33.48, 'AP50': 62.73

test:
MAE: 8.72
RMSE: 62.04
'AP': 42.80
'AP50': 75.22

Close to reported. Could you please help with 0shot finetuning?

[jerpelhan]

Hi!

The performance variance on the test set—both in this method and in other state-of-the-art approaches—is typically quite high in FSCD. This is largely due to a single image (1123.jpg). That said, I haven't observed the variance to be as extreme as what @bbhbbxwy reported. It might be worth trying to adjust the threshold to better match the validation and this test split?

@xian-stack For fine-tuning, if you're doing zero-shot, you should freeze the entire pre-trained network. Then, in counter.py, add:

self.exemplars = nn.Parameter(torch.randn(1, emb_dim))

and use it as it is used in counter_infer.py, and thanks for confirming results.

[xian-stack]

Thank you for your response!

After reruning training I achieved:
Val set MAE: 9.29 RMSE: 36.30
Test set MAE: 8.89 RMSE: 62.36

Val set MAE: 9.27 RMSE: 41.42
Test set MAE: 7.74 RMSE: 54.51

Val set MAE: 9.40 RMSE: 43.50
Test set MAE: 8.91 RMSE: 66.93

and AP AP50 always very close to reported in paper. There is noise in training, but most of result are close (second even better).

[cripsgreen]

Hello, I would like to ask: when running this code evaluate_bboxes.py, it cannot find the files geco_val.json and geco_test.json. Do you know how to locate these two files? Thank you very much.
@xian-stack @bbhbbxwy