Model Pretraining does not improve

[olivierdullaert]

Hello,

When running pretrain.py, the model does not seem to improve significantly after c. 20 epochs. If that is correct, then why must de model be pretrained for 150 epochs? Am I missing something? Also pretraining currently takes c. 4 days. do you have any advice on how to speed up this process as I noted that one forward pass takes c. 4 sec with a batch size of 4 (on 2 Nvidia v-110 gpu's)? Thanks in advance!

[jerpelhan]

Yes, as you’ve observed, training on FSC can be quite unstable, mostly due to noisy/bad annotations. This is one of the reasons we opted for a longer pretraining schedule (150 epochs), to give the model more opportunity to learn. Even though the performance metrics (MAE and loss) may appear to plateau early, we’ve found that continued training can still yield small but meaningful improvements.

That said, we're aware of the challenges around training speed and stability. We've since developed a much faster and more general model with improved performance, which is currently under review. This new model is also trained on a significantly larger dataset to support better generalization to real-world scenarios. We look forward to sharing more details about it soon!

[olivierdullaert]

Hi Thanks for the response! In that case, as performance does improve despite MAE not decreasing significantly would you recommend using the Last.pth model weights or the best.pth weights? As for me, when I train, the best.pth weights are consistently saved somewhere around epoch 30 as this is when the val MAE is lowest.

Also, great to hear that you are already working on an improved model. I am looking forward to it. It there any chance you could share a rough indication of when the model will become available?

[jerpelhan]

Yes, in our case we recommend using the last.pth weights.

We're hoping to make the new model available within the next couple of months, mostly depending on the outcome of the review process.

[cripsgreen]

To reproduce your results, should I mainly use pre-training, i.e., run pretrain.py to generate a model like GeCo.pth, or should I directly use the already trained GeCo.pth and run train.py to produce a new model? @jerpelhan