gradient norm plateau

[QuQingY]

Problem description

there is a plateau for gradient norm, why does this plateau appear?

How to reproduce

1. open comfyResearch: https://online.comfy-research.com/
2. Use Lottery mode and Type in 'pineapple' in the input field
3. draw the random graph and run(there is some parameter you need to change especially the formula in dataset)

[QuQingY]

Watch the comparison between train-test-loss-gap and gradient norm, the gradient norm plateau period seems to overlap with model choosing overfitting or generalization:

overfit:

generalize: