Model Fitting Runs Not Perfectly Reproducible

[SimWdm]

As pointed out by by @rdrighetto in #8 (comment), model fitting on the same data with the same random seed yields fitting and validation curves that are very similar but not identical as expected.

I found that the random rotations of the subtomos during model fitting are not seeded which is a likely explanation. I will implement proper seeding in a new branch and see if this fixes the issue.

[rdrighetto]

Hi again,

As stated in my last reply in #8, I was running comparisons between the current stable release based on MRC subtomograms and the torch_subtomos branch.
I ran the test 4 times in total, 2x with MRC and 2x with torch subtomos, all else being equal. Of the three runs that went to completion, all results look similar and have similar training and validation curves, but are not identical, including both runs of the torch_subtomos branch. This might be explained by the random rotations not being seeded, as you found out, @SimWdm.

Please let me know if I can help with more tests or if you need more details!

[SimWdm]

Hi Ricardo, just a quick heads up: Unfortunately I have not yet had the time to look into this as I am currently pushing towards a paper deadline. Please be assured that I have not forgotten about this issue!

[rdrighetto]

Hi Simon, no worries, take your time and good luck the paper!

…

-- Ricardo Diogo Righetto Em qua., 18 de set. de 2024 às 11:32, Simon Wiedemann < ***@***.***> escreveu:

Hi Ricardo, just a quick heads up: Unfortunately I have not yet had the time to look into this as I am currently pushing towards a paper deadline. Please be assured that I have not forgotten about this issue! — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2W676CSY7ZCPWZY7ZCRVLZXFCDLAVCNFSM6AAAAABNZYGHZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJXHE3DOMRQGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[asarnow]

GPU execution is usually not reproducible even just because summation order may change. If you run Relion or cryoSPARC refinements with all parameters fixed, including the same random seed, the particle angles and even resolution will be different as well. Could be the same here.

[rdrighetto]

Thanks for this insight @asarnow! Unfortunately, in the case of DDW this non-reproducibility is not negligible, as it seems to lead to crashes. I need to get back investigating this, will keep you guys posted.