Death of many neurons

[KindXiaoming]

Problem description

ReLU nonlinear count (the number of active neurons) suddenly drop around step 200.

How to reproduce

1. open comfyResearch: https://online.comfy-research.com/
2. Use Lottery mode and Type in 'Liu Lab' in the input field
3. draw the random graph and run(there is some parameter you need to change)

[KindXiaoming]

Many things are happening simultaneously around step 200:

1. death of relu neurons
2. slight plateau for loss, weight L2 and maximum Hessian eigenvalue
3. spike for gradient norm