About stage4 fine-tuning

[haloya-0328]

Thanks for your work.
Can it run on 8 A100(80GB) GPUs?

[LanXingXuan]

Thank you for your interest in our work!
Our approach allows model training to be conducted on 8 A100(80GB) GPUs or fewer. To mitigate the instability that may arise from a reduced number of GPUs, we recommend increasing the gradient accumulation steps during training.