Expose num_ns_steps - #283
Conversation
It can be useful to obtain the `num_ns_steps` in certain cases, e.g., to balance compute around existing instances' `num_ns_steps`. Could also refactor this to allow people to set `num_ns_steps` post-init by either (a) creating a new `scaled_orthogonalize_fn` when setting `num_ns_steps` or (b) simply passing `self` into the `scaled_orthogonalize_fn` closure, so that the instance's current `num_ns_steps` are always accessed. Signed-off-by: janEbert <janpabloe@nvidia.com>
Greptile SummaryExposes Muon’s construction-time
Confidence Score: 5/5The PR appears safe to merge with no actionable defects identified. The new property is initialized from the same validated constructor argument used by the orthogonalization closure, and current subclass and serialization paths preserve that value consistently. Important Files Changed
Reviews (1): Last reviewed commit: "Expose `num_ns_steps`" | Re-trigger Greptile |
It is very dangerous to modify Having said that, we designed for relatively simple customization, like QKV split. If more complex customization is needed, we should revisit the entire design of OrthogonalizedOptimizer. |
It can be useful to obtain the
num_ns_stepsin certain cases, e.g., to balance compute around existing instances'num_ns_steps.Could also refactor this to allow people to set
num_ns_stepspost-init by either (a) creating a newscaled_orthogonalize_fnwhen settingnum_ns_stepsor (b) simply passingselfinto thescaled_orthogonalize_fnclosure, so that the instance's currentnum_ns_stepsare always accessed.