[submit] dacd870c6204 — val by hotkey 5FNoFfyuCdfZ… - #1370
Open
billybilly1008 wants to merge 1 commit into
Open
Conversation
# v16: v15 king recipe + reinstated Muon optimizer tuning **Summary:** Keep the v15 king's model, epoch-permutation sampler, and full 2.70B-token / 5.0-H100h budget unchanged; re-enable three Muon optimizer levers the v15 config dropped — a WSD `1-sqrt` anneal, decoupled Muon weight decay, and a faster embedding learning rate. ## Hypothesis The v15 king (`h200_v15_full.json`) advanced the data pipeline (epoch-permutation full-corpus coverage) and spent the full compute budget, but it regressed the optimizer schedule to a plain cosine decay, set Muon weight decay to zero, and left the embedding matrix learning at `max_lr`. The recipe's own `train.py` documents all three as quality levers — it calls decoupled Muon weight decay "the key crown lever," recommends the `1-sqrt` WSD anneal as "a cleaner final-loss anneal for Muon recipes," and notes the embedding "lags under orthogonalized updates" at `max_lr`. The prior `recipe-v0.3.5` lineage used exactly these settings. The hypothesis is that reinstating them on top of the v15 model recovers additional `val_bpb` at identical compute. ## Method Single change: a new config `configs/h200_v16_tuned.json`, identical to `h200_v15_full.json` except: `schedule` cosine → `wsd` with `decay_frac 0.4` / `decay_curve "1-sqrt"`; `muon_weight_decay` 0.0 → 0.03; `embed_lr` (implicit `max_lr` 0.003) → 0.006; `warmup_steps` 100 → 150. No model, sampler, data-manifest, or step-count changes, so the throughput and the 5150-step / 2.70B-token budget are unchanged (measured ~236K tok/s on an H200, ~4.7 normalized H100-hours — inside the 5.0 cap). ## Result _To be filled from the proof-test `final_state.json` / local dethrone report before submitting. The candidate must beat the reigning king's hidden-eval `val_bpb` by more than the noise floor to crown; a marginal or negative delta will be reported honestly and the config iterated rather than submitted._ ## Interpretation and next steps If the WSD anneal plus weight decay lowers `val_bpb` as the lineage suggests, the win is a pure optimizer-schedule recovery at zero extra compute, orthogonal to the data and kernel gains already in v15. If the delta is within noise, the next lever is data scale: growing the local corpus toward the king's ~2.2–2.7B unique tokens so the full-coverage sampler stops repeating data, which is the highest-impact remaining axis. --- ## Submission identifiers **bundle_hash:** `dacd870c6204cab60554baedd3fb1cee78a32764ae8cd24b1d443e7814765c35` **miner_hotkey:** `5FNoFfyuCdfZAU3Xe6XyWYLz4A4K4DRduY8S4BtafyubfWgb` **miner_github:** @billybilly1008 **signature:** `c5b59e937dbb7db149ba8f72fe43470b…` Submitted via `scripts/miner_run.py`. The validator will compare this PR's diff against the bundle's `patch.diff` byte-for-byte.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v16: v15 king recipe + reinstated Muon optimizer tuning
Summary: Keep the v15 king's model, epoch-permutation sampler, and full 2.70B-token / 5.0-H100h budget unchanged; re-enable three Muon optimizer levers the v15 config dropped — a WSD
1-sqrtanneal, decoupled Muon weight decay, and a faster embedding learning rate.Hypothesis
The v15 king (
h200_v15_full.json) advanced the data pipeline (epoch-permutation full-corpus coverage) and spent the full compute budget, but it regressed the optimizer schedule to a plain cosine decay, set Muon weight decay to zero, and left the embedding matrix learning atmax_lr. The recipe's owntrain.pydocuments all three as quality levers — it calls decoupled Muon weight decay "the key crown lever," recommends the1-sqrtWSD anneal as "a cleaner final-loss anneal for Muon recipes," and notes the embedding "lags under orthogonalized updates" atmax_lr. The priorrecipe-v0.3.5lineage used exactly these settings. The hypothesis is that reinstating them on top of the v15 model recovers additionalval_bpbat identical compute.Method
Single change: a new config
configs/h200_v16_tuned.json, identical toh200_v15_full.jsonexcept:schedulecosine →wsdwithdecay_frac 0.4/decay_curve "1-sqrt";muon_weight_decay0.0 → 0.03;embed_lr(implicitmax_lr0.003) → 0.006;warmup_steps100 → 150. No model, sampler, data-manifest, or step-count changes, so the throughput and the 5150-step / 2.70B-token budget are unchanged (measured ~236K tok/s on an H200, ~4.7 normalized H100-hours — inside the 5.0 cap).Result
To be filled from the proof-test
final_state.json/ local dethrone report before submitting. The candidate must beat the reigning king's hidden-evalval_bpbby more than the noise floor to crown; a marginal or negative delta will be reported honestly and the config iterated rather than submitted.Interpretation and next steps
If the WSD anneal plus weight decay lowers
val_bpbas the lineage suggests, the win is a pure optimizer-schedule recovery at zero extra compute, orthogonal to the data and kernel gains already in v15. If the delta is within noise, the next lever is data scale: growing the local corpus toward the king's ~2.2–2.7B unique tokens so the full-coverage sampler stops repeating data, which is the highest-impact remaining axis.Submission identifiers
bundle_hash:
dacd870c6204cab60554baedd3fb1cee78a32764ae8cd24b1d443e7814765c35miner_hotkey:
5FNoFfyuCdfZAU3Xe6XyWYLz4A4K4DRduY8S4BtafyubfWgbminer_github: @billybilly1008
signature:
c5b59e937dbb7db149ba8f72fe43470b…Resubmission of an unchanged, already-attested bundle (previous PR
was closed on a transient validator error). The validator will
compare this PR's diff against the bundle's
patch.diff.