Skip to content

Add K3 three-stage RL path: proxy checkpoint tooling, RL data prep, and fused-layout checkpoint cleanup - #20

Draft
ISEEKYAN wants to merge 18 commits into
mainfrom
feat/k3-three-stage-rl-clean
Draft

Add K3 three-stage RL path: proxy checkpoint tooling, RL data prep, and fused-layout checkpoint cleanup#20
ISEEKYAN wants to merge 18 commits into
mainfrom
feat/k3-three-stage-rl-clean

Conversation

@ISEEKYAN

@ISEEKYAN ISEEKYAN commented Aug 1, 2026

Copy link
Copy Markdown
Owner

What

The implementation side of the K3 three-stage RL path, cut fresh from main with no experiment scaffolding.

  • lite/checkpoint.py: the fused attention input projection is no longer concatenated with self-computed offsets, and per-projection semantics are no longer derived by matching substrings of the parameter name. The layout is declared and the offsets are derived from that declaration.
  • lite/fused_layout.py: K3 now owns the minimal row-axis checkpoint layout declarations it needs, removing the dependency on the unmerged MLite fused-weights module.
  • lite/protocol.py: K3 owns decoder-layer selection for R3 router replay; this folds the former Fix K3 R3 router discovery with current MLite #21 into this PR.
  • lite/model.py, config.py: the declaration surface (per-projection name, head count, head dim, order) and the config plumbing behind it.
  • primitive/router.py: explicit FP32 gating contract for the shared router.
  • tools/build_proxy_checkpoint.py, tools/prepare_gsm8k_rl_from_sft.py: reproducible construction of the proxy checkpoint and of the RL dataset from the SFT split.

Why

Weight synchronisation into the inference engine was producing a deterministic, silent corruption of one segment of the fused seven-way projection: the same element index and the same bad-element count reproduced byte-for-byte across independent runs, while the source-side tensor was entirely finite. Hand-computed fusion offsets and name-substring parsing were the mechanism; declaring the layout removes the class of bug rather than one instance of it.

Keeping the K3-specific layout declaration in this package also removes the import closure that previously required an unmerged MLite PR.

Validation

  • K3 checkpoint layout contracts: 5 passed.
  • K3 R3 replay-root contract: 1 passed.
  • Ruff format check, Ruff check, compileall, and git diff --check passed.
  • A full local run against the available MLite checkout produced 61 passed, 1 skipped, and 6 failures in pre-existing QAT/MXFP4 paths because that checkout lacks megatron.lite.primitive.quantization.qat and differs in its MXFP4 export contract. These paths were not changed by this update.
  • GPU validation of the end-to-end path is not included here and is tracked separately.

Note

Opened as a draft: the GPU re-verification of the synchronisation path has not run yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant