With a memorization dataset, it's hard to find a good learning rate for Muon, either too small (progress is slow), or too large (large oscillations). By contrast, a SOAP or Adam optimizer (leave for readers to try) can nicely work for memorization dataset.
Muon (lr = 0.01)
Muon (lr = 0.1)
How to reproduce
- open comfyResearch: https://online.comfy-research.com/
- Use Lottery mode and Type in '142859' in the input field
- draw the random graph and run
With a memorization dataset, it's hard to find a good learning rate for Muon, either too small (progress is slow), or too large (large oscillations). By contrast, a SOAP or Adam optimizer (leave for readers to try) can nicely work for memorization dataset.
Muon (lr = 0.01)
Muon (lr = 0.1)
How to reproduce